remove json column v4 format writers

[clintropolis]

Description

Removes the ability to write JSON columns in the older version '4' format, leaving only the latest current version, '5', as the only option. (Internally it isn't really v5, rather v0 of the 'auto' format, but it is branded as v5 for the sake of continuity).

I have left the structures to control which version is written added in #14955, in the event we need to introduce any newer versions in the future, we just only allow v5 right now.

Release note

JSON columns can no longer be written in the older V4 format, the current latest version, V5, is the only version which can be written to new segments. This means both the formatVersion of a JSON column spec on JSON based ingestion specs, e.g.

          {
            "type": "json",
            "name": "someColumn",
            "formatVersion": 5
          }

and druid.indexing.formats.nestedColumnFormatVersion in runtime.properties only allow 5.

All older version of JSON columns can still be read and queried just fine, this change just prevents writing new segments in the older format.

[cryptoe]

I was hoping there would be a place where we fail the query/service if nestedColVersion is set to 4. Could not find such a guardrail. Is that not required ?

[clintropolis]

A failure would happen in DefaultColumnFormatConfig which throws an exception if the format is not 5.

I guess I could think about switching this over to use the PropertiesValidator introduced in #17634, though this config itself isn't really deprecated, I did intend that thing to be used for validation of runtime.properties.

The more I think about it, I'm not actually sure failing is that useful, since v5 isn't really much of a behavior change, the risk is that with v5 segments you cannot roll back to Druid versions older than 26.0.0, which was covered in prior release notes and doesn't seem to be worth exploding over. The main difference of v5 format is that there are nested array fields which means you can use json_value function to get arrays of primitives out more efficiently and there are optimizations for the cases where there are no actual objects in a json column (e.g. it writes a scalar column instead if there is no nesting), so there isn't really much of a behavioral risk to using the new format.

I think we could also consider just log.warn about having a bad config value that is no longer supported and just ignoring it, though I suppose older druid versions also fail if you didn't put the value of 4 or 5 into the config, so 🤷