[Bug]: milvus-backup check failed for Azure Blob Storage Account

[vavrap99]

Current Behavior

milvus-backup --config ./backup2az_rg-tmw4xrkn.yaml check
config:./backup2az_rg-tmw4xrkn.yaml
Failed to copy file from milvus storage to backup storage
Milvus version: v2.3.3
Storage:
milvus-bucket: milvus-bucket
milvus-rootpath: file
backup-bucket: milvus-bucket-backup
backup-rootpath: file-backup
storage: azure copy from url PUT https://my_storage_account_name.blob.core.windows.net:443/milvus-bucket-backup/file-backup/milvus_backup_check_2024-03-08 10:08:53.753750625 +0100 CET m=+0.093754724
--------------------------------------------------------------------------------
RESPONSE 401: 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
ERROR CODE: CannotVerifyCopySource
--------------------------------------------------------------------------------
<?xml version="1.0" encoding="utf-8"?><Error><Code>CannotVerifyCopySource</Code><Message>Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:xxxxxx
Time:2024-03-08T09:08:53.8029740Z</Message></Error>
--------------------------------------------------------------------------------
 abort previous %!w(*fmt.wrapError=&{storage: azure get properties HEAD https://my_storage_account_name.blob.core.windows.net:443/milvus-bucket-backup/file-backup/milvus_backup_check_2024-03-08 10:08:53.753750625 +0100 CET m=+0.093754724
--------------------------------------------------------------------------------
RESPONSE 404: 404 The specified blob does not exist.
ERROR CODE: BlobNotFound
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------

Expected Behavior

No response

Steps To Reproduce

Deploy Milvus on Azure with AKS with Azure Blob Storage Account ready as external object storage for Milvus as described at https://milvus.io/docs/azure.md :
...
minio:
  enabled: false

externalS3:
  enabled: true
  host: core.windows.net
  port: 443
  rootPath: "file"
  bucketName: "milvus-bucket" # the storage account container name
  cloudProvider: azure
  useSSL: true
  accessKey: "my_storage_account_name" # the storage account name
  secretKey: "my_storage_account_key1" 

Check Milvus blobs in Azure SA container ...

Configure backup2az_rg-tmw4xrkn.yaml for milvus-backup :
http:
  simpleResponse: true

milvus:
  address: 10.45.255.4
  port: 19530
  authorizationEnabled: true
  tlsMode: 0
  user: "root"
  password: "rootpass"

minio:
  storageType: "azure"
  address: "core.windows.net" # Address of MinIO/S3
  port: 443   # Port of MinIO/S3
  accessKeyID: "my_storage_account_name"  # accessKeyID of MinIO/S3
  secretAccessKey: "my_storage_account_key1" # MinIO/S3 encryption string
  useSSL: true # Access to MinIO/S3 with SSL
  useIAM: false
  iamEndpoint: ""

  bucketName: "milvus-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "file" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  backupAccessKeyID: "my_storage_account_name"  # the storage account name
  backupSecretAccessKey: "my_storage_account_key1" # the storage account key1

  backupBucketName: "milvus-bucket-backup"
  backupRootPath: "file-backup"


backup:
  maxSegmentGroupSize: 2G

  parallelism:
    backupCollection: 4
    copydata: 128
    restoreCollection: 2

  keepTempFiles: false

  gcPause:
    enable: false

Run:
milvus-backup --config ./backup2az_rg-tmw4xrkn.yaml check

Environment

Milvus version: v2.3.3
milvus-backup 0.4.11

Anything else?

In log file:
[2024/03/08 10:08:53.680 +01:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/03/08 10:08:53.680 +01:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup2az_rg-tmw4xrkn.log]
[2024/03/08 10:08:53.680 +01:00] [DEBUG] [core/backup_context.go:67] ["Start Milvus client"] [endpoint=10.45.255.4:19530]
[2024/03/08 10:08:53.691 +01:00] [DEBUG] [core/backup_context.go:91] ["Start minio client"] [address=core.windows.net:443] [bucket=milvus-bucket] [backupBucket=milvus-bucket-backup]
[2024/03/08 10:08:53.741 +01:00] [INFO] [storage/azure_chunk_manager.go:65] ["Azure chunk manager init success."]
[2024/03/08 10:08:53.798 +01:00] [ERROR] [storage/azure_chunk_manager.go:84] ["copyObject error"] [srcObjectKey="file/milvus_backup_check_2024-03-08 10:08:53.753750625 +0100 CET m=+0.093754724"] [dstObjectKey="file-backup/milvus_backup_check_2024-03-08 10:08:53.753750625 +0100 CET m=+0.093754724"] [error="storage: azure copy from url PUT https://my_storage_account_name.blob.core.windows.net:443/milvus-bucket-backup/file-backup/milvus_backup_check_2024-03-08 10:08:53.753750625 +0100 CET m=+0.093754724
--------------------------------------------------------------------------------
RESPONSE 401: 401 Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
ERROR CODE: CannotVerifyCopySource
--------------------------------------------------------------------------------
<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>CannotVerifyCopySource</Code><Message>Server failed to authenticate the request. Please refer to the information in the www-authenticate header.
RequestId:482806ea-b01e-0065-0638-710660000000
Time:2024-03-08T09:08:53.8029740Z</Message></Error>
--------------------------------------------------------------------------------
 abort previous %!w(*fmt.wrapError=&{storage: azure get properties HEAD https://my_storage_account_name.blob.core.windows.net:443/milvus-bucket-backup/file-backup/milvus_backup_check_2024-03-08 10:08:53.753750625 +0100 CET m=+0.093754724
--------------------------------------------------------------------------------
RESPONSE 404: 404 The specified blob does not exist.
ERROR CODE: BlobNotFound
--------------------------------------------------------------------------------
Response contained no body
--------------------------------------------------------------------------------
 0xc0012ede40})"] [stack="github.com/zilliztech/milvus-backup/core/storage.(*AzureChunkManager).Copy
        /home/runner/work/milvus-backup/milvus-backup/core/storage/azure_chunk_manager.go:84
github.com/zilliztech/milvus-backup/core.(*BackupContext).Check
        /home/runner/work/milvus-backup/milvus-backup/core/backup_context.go:583
github.com/zilliztech/milvus-backup/cmd.glob..func1
        /home/runner/work/milvus-backup/milvus-backup/cmd/check.go:25
github.com/spf13/cobra.(*Command).execute
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:876
github.com/spf13/cobra.(*Command).ExecuteC
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:990
github.com/spf13/cobra.(*Command).Execute
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:918
github.com/zilliztech/milvus-backup/cmd.Execute
        /home/runner/work/milvus-backup/milvus-backup/cmd/root.go:28
main.main
        /home/runner/work/milvus-backup/milvus-backup/main.go:24
runtime.main
        /opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"]

[gifi-siby]

Why this issue is closed?
I am also facing the same issue

[umar1997]

Facing the same issue. Is there any solution available?

[gifi-siby]

No idea. I have modified the code to resolve the backup issue, but it wasn't right I think. Because restore fails now.

[umar1997]

What changes did you make in the code above to make the backup start working?

[gifi-siby]

Inside AzureObjectStorage CopyObject, blobCli.CopyFromURL was failing since blob file path and folder path was coming in and it was failing for folders. I have added an extra function to check if path is folder or blob and will proceed to blobCli.CopyFromURL only for blob files.

[gifi-siby]

Issue raised again here: #446

[nisharyan]

Why this issue is closed? I am also facing the same issue

+1