Allow setting PK Chunking for "sf data query --bulk"

[joachim-roesecke-bpt]

Unless I missed something, the command "sf data query" does not allow switching on primary key chunking when data is queried via the Bulk API. The idea would be to have a new flag like "--pk-chunking" to switch this on in case the Bulk API is used.

[mshanemc]

it currently doesn't. That header Sforce-Enable-PKChunking can be boolean or a few other things and can set multiple values.

Which ones are you interested in? Just "on/off" or do you see a need for the other options?

https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/async_api_headers_enable_pk_chunking.htm

[joachim-roesecke-bpt]

Hi. I believe the most important options would be "TRUE", "FALSE" (default) and "chunkSize". I personally do not have a need for "parent" or "startRow" right now...

[AllanOricil]

I think these changes I did would also be needed in case you use bulk api v2 instead of v1: #1995

Edited: I remembered why I could not use pk chunking. It was because Bulk Api v2 does not support it. This is a feature of V1 only.

I spent some hours to find this out because the documentation "title header" kind deceived me into thinking that pk chunking was a feature of bulk Api V2.

Also, I remember that this command is currently using Bulk API V2 https://github.com/salesforcecli/plugin-data/blob/d21a32c334dc5e92e4298241f7856ee8c0aae4dd/src/commands/data/query.ts#L135C3-L165. And this means it won't be possible to use pk chunking unless you go back to use bulk api v1.

There is also this issue in jsforce's repo asking maintainers to enable pk chunking:
jsforce/jsforce#605

and a PR that "enables it" (I don't know if it really does), but it was never merged:
jsforce/jsforce#780

[AllanOricil]

I propose turning the --bulk flag into an optional options flag

sf data query ... # does not use bulk
sf data query ... --bulk v1 # uses v1
sf data query ... --bulk v2 # uses v2
sf data query ... --bulk # uses SF_DEFAULT_BULK_API_VERSION env variable

SF_DEFAULT_BULK_API_VERSION can be: undefined, v1 or v2

If SF_DEFAULT_BULK_API_VERSION = undefined and the --bulk flag is provided, an error as follows is thrown "you must either set SF_DEFAULT_BULK_API_VERSION or pass a value to bulk api version"

If --bulk flag value resolves to v1, passing the --pk-chuncking flag is allowed, otherwise it throws an exception as such "pk chuncking is supported in bulk api v1 only. Read more at DOC LINK". The same logic could be done for all other bulk features that are dependant on the api version.

This solution could solve this issue as well #2667

[cristiand391]

While working with the Bulk V2 API I saw this field begin included in the response:
https://github.com/jsforce/jsforce/blob/3a88038617494cd2c877f152363218f17939c400/src/api/bulk2.ts#L47

I talked with the API team at TDX and they said pk chunking is enabled by default in V2 (starting v60), if the object supports it (most do) we get that field set to true or false if not.

Since it's always enabled we discussed making the CLI print a warning if the object being queried didn't support it, any thoughts?