Cannot create S3 volume on ceph, having "The provided 'x-amz-content-sha256' header does not match what was computed." error

[zentavr]

I'd created the PVC like:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: zabbix-db-dump
  annotations:
    helm.sh/resource-policy: keep
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 750Gi
  storageClassName: csi-s3

The CSI driver is the latest 0.41.1 and installed with the values:

---
storageClass:
  # Specifies whether the storage class should be created
  create: true
  singleBucket: "cti-k8s-csi"

secret:
  # Specifies whether the secret should be created
  create: true
  # Name of the secret
  name: "csi-s3-secret"
  # S3 Access Key
  accessKey: "*******redacted*******"
  # S3 Secret Key
  secretKey: "*******redacted*******"
  # Endpoint
  endpoint: "http://rgw-slow-dev01.ti.local:80"

There are other volumes from the old deployments which are fine. What could be wrong here?
Events:
Reason: ExternalProvisioning (source: persistentvolume-controller):

Waiting for a volume to be created either by the external provisioner 'ru.yandex.s3.csi' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

Reason: ProvisioningFailed (source: ru.yandex.s3.csi_csi-s3-provisioner-0_4d383a87-31f4-42c5-b9c9-58d2c6c41063)

failed to provision volume with StorageClass "csi-s3": rpc error: code = Unknown desc = failed to create prefix pvc-a555057d-22e8-489d-bc7c-86e678f2d83f: The provided 'x-amz-content-sha256' header does not match what was computed.

[zentavr]

S3 browser says:

s3cmd        --acl-private        --access_key=*******redacted*******        --secret_key=*******redacted*******        --host=127.0.0.1        --host-bucket=127.0.0.1        --ssl        --no-check-certificate ls s3://cti-k8s-csi
                       DIR   s3://cti-k8s-csi/pvc-cbd4812e-f26f-4d62-b591-65bb41f42a00/
                       DIR   s3://cti-k8s-csi/pvc-cf7ff348-77c6-4f9e-a3e1-54f00d645d7f/
                       DIR   s3://cti-k8s-csi/pvc-ec4d9b57-c291-447d-8e92-e7825b9ac77c/

...so it shows pvc-folders for the old PVCs

[zentavr]

We upgraded Ceph cluster from 17.2.7 to 18.2.4 and the issue is still there

[zentavr]

I was able to capture HTTP requests, they are here https://gist.github.com/zentavr/facecab1db8376cb9fc011e1e955e68d

[zentavr]

Probably related to: https://tracker.ceph.com/issues/63153
I'd noticed in my sniffed data that when there is a header like:

X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD

...the whole request:

PUT /cti-k8s-csi/pvc-8c2c2930-681a-4ad1-9f35-64b4ecdba2e0/ HTTP/1.1
Host: rgw-slow-dev01.ti.local
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5
Transfer-Encoding: chunked
Authorization: AWS4-HMAC-SHA256 Credential=1Z8NJSBMVHQK32YFVK04/20240821/default/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,Signature=dc4974e5608a1bc6b777fac789dd974537eb54e07796827aec4c3a997883727e
Content-Type: application/octet-stream
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20240821T232356Z
X-Amz-Decoded-Content-Length: 0
Accept-Encoding: gzip

then the issue happens and I have HTTP/1.1 400 Bad Request reply.

[zentavr]

The workaround now is to create the prefix-folder manually, then also patch manually the PVC and create the volume manually as well.