[Bug]: 使用脚本备份和恢复没有报错但是恢复的表格没有数据

[yuan8180]

Current Behavior

在milvus2.3.1版本，使用milvus-backup进行备份和还原，日志无报错，可以还原新表结构，但是新表无数据

Expected Behavior

恢复的表中有数据

Steps To Reproduce

备份：
./milvus-backup create -n backup20241105_8 -c focuspasserbydetails
0.4.26 (Built on 2024-10-30T09:28:19Z from Git SHA dd4515017a2d2f7c9493070fe321156a6c9126d4)
config:backup.yaml
[2024/11/05 17:57:40.295 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/11/05 17:57:40.295 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/11/05 17:57:40.303 +08:00] [INFO] [core/backup_impl_create_backup.go:32] ["receive CreateBackupRequest"] [requestId=64d823c0-9b5c-11ef-aed3-eeeeeeeeeeee] [backupName=backup20241105_8] [collections="[focuspasserbydetails]"] [databaseCollections=] [async=false] [force=false] [metaOnly=false]
[2024/11/05 17:57:40.303 +08:00] [INFO] [core/backup_context.go:117] ["{Base:0xc0002aedc0 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:false GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091}"]
[2024/11/05 17:57:40.303 +08:00] [INFO] [core/backup_context.go:118] ["{Base:0xc0002aedc0 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/11/05 17:57:40.307 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=milvus-bucket] [root=file]
[2024/11/05 17:57:40.318 +08:00] [INFO] [core/backup_impl_create_backup.go:695] ["collections to backup"] [collections="[default.focuspasserbydetails]"]
[2024/11/05 17:57:40.318 +08:00] [INFO] [core/backup_impl_create_backup.go:231] ["start backup collection"] [db=default] [collection=focuspasserbydetails]
[2024/11/05 17:57:40.321 +08:00] [INFO] [core/backup_impl_create_backup.go:274] ["try to get index"] [collection_name=focuspasserbydetails]
[2024/11/05 17:57:40.328 +08:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=focuspasserbydetails] [field_name=PasserbyKey] ["index info"="[]"]
[2024/11/05 17:57:40.334 +08:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=focuspasserbydetails] [field_name=IndexObjectId] ["index info"="[]"]
[2024/11/05 17:57:40.338 +08:00] [INFO] [core/backup_impl_create_backup.go:294] ["field index"] [collection_name=focuspasserbydetails] [field_name=IndexOdEmbedding] ["index info"="[{}]"]
[2024/11/05 17:57:40.345 +08:00] [INFO] [core/backup_impl_create_backup.go:380] ["GetPersistentSegmentInfo before flush from milvus"] [databaseName=default] [collectionName=focuspasserbydetails] [segmentNumBeforeFlush=1]
[2024/11/05 17:57:40.349 +08:00] [INFO] [core/backup_impl_create_backup.go:408] ["flush segments"] [databaseName=default] [collectionName=focuspasserbydetails] [newSealedSegmentIDs="[]"] [flushedSegmentIDs="[453712021875091680]"] [timeOfSeal=1730800660] [BackupTimestamp=0] [channelCPs={}]
[2024/11/05 17:57:40.352 +08:00] [INFO] [core/backup_impl_create_backup.go:430] ["GetPersistentSegmentInfo after flush from milvus"] [databaseName=default] [collectionName=focuspasserbydetails] [segmentIDsBeforeFlush="[453712021875091680]"] [segmentIDsAfterFlush="[453712021875091680]"] [newL0SegmentsIDs="[]"]
[2024/11/05 17:57:40.352 +08:00] [INFO] [core/backup_impl_create_backup.go:525] ["finish build partition info"] [collectionName=focuspasserbydetails] [partitionNum=1] [l0SegmentsNum=0]
[2024/11/05 17:57:40.352 +08:00] [INFO] [core/backup_impl_create_backup.go:729] ["Finish prepare all collections meta"]
[2024/11/05 17:57:40.352 +08:00] [INFO] [core/backup_impl_create_backup.go:734] ["before backupCollectionExecute"] [collectionID=452761217100312709] [collection=focuspasserbydetails]
[2024/11/05 17:57:40.353 +08:00] [INFO] [core/backup_impl_create_backup.go:540] [backupCollectionExecute] [collectionMeta="id:\"64d823c0-9b5c-11ef-aed3-eeeeeeeeeeee\" start_time:1730800660 collection_id:452761217100312709 db_name:\"default\" collection_name:\"focuspasserbydetails\" schema:<name:\"focuspasserbydetails\" description:\"db passerby collection, store index od embedding\" fields:<fieldID:100 name:\"PasserbyKey\" is_primary_key:true data_type:VarChar type_params:<key:\"max_length\" value:\"255\" > > fields:<fieldID:101 name:\"IndexObjectId\" data_type:Int64 > fields:<fieldID:102 name:\"IndexOdEmbedding\" data_type:FloatVector type_params:<key:\"dim\" value:\"512\" > > > shards_num:1 consistency_level:Bounded has_index:true index_infos:<field_name:\"IndexOdEmbedding\" index_name:\"_default_idx_102\" index_type:\"IVF_FLAT\" params:<key:\"index_type\" value:\"IVF_FLAT\" > params:<key:\"metric_type\" value:\"L2\" > params:<key:\"params\" value:\"{\\\"nlist\\\":\\\"1024\\\"}\" > > load_state:\"Loaded\" backup_physical_timestamp:1730800660 "]
[2024/11/05 17:57:40.358 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=milvus-bucket] [root=file]
[2024/11/05 17:57:40.397 +08:00] [INFO] [core/backup_impl_create_backup.go:575] ["Begin copy data"] [dbName=default] [collectionName=focuspasserbydetails] [collectionID=452761217100312709] [partitionID=452761217100312710]
[2024/11/05 17:57:40.397 +08:00] [INFO] [core/backup_impl_create_backup.go:826] ["copy segment"] [collection_id=452761217100312709] [partition_id=452761217100312710] [segment_id=453712021875091680] [group_id=1] [backupBinlogPath=file/backup20241105_8/binlogs]
[2024/11/05 17:57:40.447 +08:00] [INFO] [core/backup_impl_create_backup.go:615] ["Finish copy data"] [dbName=default] [collectionName=focuspasserbydetails]
[2024/11/05 17:57:40.447 +08:00] [INFO] [core/backup_meta_manager.go:501] ["Get backup"] [state=BACKUP_SUCCESS] [backupedSize=94981] [totalSize=94981] [progress=100]
[2024/11/05 17:57:40.448 +08:00] [INFO] [core/backup_impl_create_backup.go:780] ["Final backupInfo"] [backupInfo="id:\"64d823c0-9b5c-11ef-aed3-eeeeeeeeeeee\" state_code:BACKUP_SUCCESS start_time:1730800660314 end_time:1730800660447 progress:100 name:\"backup20241105_8\" collection_backups:<id:\"64d823c0-9b5c-11ef-aed3-eeeeeeeeeeee\" start_time:1730800660 end_time:1730800660 collection_id:452761217100312709 db_name:\"default\" collection_name:\"focuspasserbydetails\" schema:<name:\"focuspasserbydetails\" description:\"db passerby collection, store index od embedding\" fields:<fieldID:100 name:\"PasserbyKey\" is_primary_key:true data_type:VarChar type_params:<key:\"max_length\" value:\"255\" > > fields:<fieldID:101 name:\"IndexObjectId\" data_type:Int64 > fields:<fieldID:102 name:\"IndexOdEmbedding\" data_type:FloatVector type_params:<key:\"dim\" value:\"512\" > > > shards_num:1 consistency_level:Bounded partition_backups:<partition_id:452761217100312710 partition_name:\"_default\" collection_id:452761217100312709 segment_backups:<segment_id:453712021875091680 collection_id:452761217100312709 partition_id:452761217100312710 num_of_rows:39 binlogs:<binlogs:<log_path:\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/0/453712021875091684\" log_size:553 > > binlogs:<fieldID:1 binlogs:<log_path:\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/1/453712021875091685\" log_size:579 > > binlogs:<fieldID:100 binlogs:<log_path:\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/100/453712021875091681\" log_size:486 > > binlogs:<fieldID:101 binlogs:<log_path:\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/101/453712021875091682\" log_size:789 > > binlogs:<fieldID:102 binlogs:<log_path:\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/102/453712021875091683\" log_size:92574 > > deltalogs:<> size:94981 group_id:453712021875091680 backuped:true > size:94981 load_state:\"Loaded\" > size:94981 has_index:true index_infos:<field_name:\"IndexOdEmbedding\" index_name:\"_default_idx_102\" index_type:\"IVF_FLAT\" params:<key:\"index_type\" value:\"IVF_FLAT\" > params:<key:\"metric_type\" value:\"L2\" > params:<key:\"params\" value:\"{\\\"nlist\\\":\\\"1024\\\"}\" > > load_state:\"Loaded\" backup_physical_timestamp:1730800660 > size:94981 milvus_version:\"v2.3.1\" "]
[2024/11/05 17:57:40.448 +08:00] [INFO] [core/backup_impl_create_backup.go:783] ["collection meta"] [value="{\"infos\":[{\"id\":\"64d823c0-9b5c-11ef-aed3-eeeeeeeeeeee\",\"state_code\":0,\"start_time\":1730800660,\"end_time\":1730800660,\"progress\":0,\"collection_id\":452761217100312709,\"db_name\":\"default\",\"collection_name\":\"focuspasserbydetails\",\"schema\":{\"name\":\"focuspasserbydetails\",\"description\":\"db passerby collection, store index od embedding\",\"fields\":[{\"fieldID\":100,\"name\":\"PasserbyKey\",\"is_primary_key\":true,\"data_type\":21,\"type_params\":[{\"key\":\"max_length\",\"value\":\"255\"}]},{\"fieldID\":101,\"name\":\"IndexObjectId\",\"data_type\":5},{\"fieldID\":102,\"name\":\"IndexOdEmbedding\",\"data_type\":101,\"type_params\":[{\"key\":\"dim\",\"value\":\"512\"}]}]},\"shards_num\":1,\"consistency_level\":2,\"partition_backups\":[{\"partition_id\":452761217100312710,\"partition_name\":\"_default\",\"collection_id\":452761217100312709,\"segment_backups\":[{\"segment_id\":453712021875091680,\"collection_id\":452761217100312709,\"partition_id\":452761217100312710,\"num_of_rows\":39,\"binlogs\":[{\"binlogs\":[{\"log_path\":\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/0/453712021875091684\",\"log_size\":553}]},{\"fieldID\":1,\"binlogs\":[{\"log_path\":\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/1/453712021875091685\",\"log_size\":579}]},{\"fieldID\":100,\"binlogs\":[{\"log_path\":\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/100/453712021875091681\",\"log_size\":486}]},{\"fieldID\":101,\"binlogs\":[{\"log_path\":\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/101/453712021875091682\",\"log_size\":789}]},{\"fieldID\":102,\"binlogs\":[{\"log_path\":\"file/insert_log/452761217100312709/452761217100312710/453712021875091680/102/453712021875091683\",\"log_size\":92574}]}],\"deltalogs\":[{}],\"size\":94981,\"group_id\":453712021875091680,\"backuped\":true}],\"size\":94981,\"load_state\":\"Loaded\"}],\"size\":94981,\"has_index\":true,\"index_infos\":[{\"field_name\":\"IndexOdEmbedding\",\"index_name\":\"_default_idx_102\",\"index_type\":\"IVF_FLAT\",\"params\":{\"index_type\":\"IVF_FLAT\",\"metric_type\":\"L2\",\"params\":\"{\\\"nlist\\\":\\\"1024\\\"}\"}}],\"load_state\":\"Loaded\",\"backup_physical_timestamp\":1730800660}]}"]
[2024/11/05 17:57:40.463 +08:00] [INFO] [core/backup_impl_create_backup.go:813] ["finish writeBackupInfoMeta"] [path=file/backup20241105_8/] [backupName=backup20241105_8] ["backup meta"="{\"id\":\"64d823c0-9b5c-11ef-aed3-eeeeeeeeeeee\",\"state_code\":2,\"start_time\":1730800660314,\"end_time\":1730800660447,\"progress\":100,\"name\":\"backup20241105_8\",\"size\":94981,\"milvus_version\":\"v2.3.1\"}"]
[2024/11/05 17:57:40.463 +08:00] [INFO] [core/backup_impl_create_backup.go:770] ["finish backup all collections"] [requestId=64d823c0-9b5c-11ef-aed3-eeeeeeeeeeee] [backupName=backup20241105_8] [collections="[focuspasserbydetails]"] [async=false]
success
duration:0 s

还原：
./milvus-backup restore -n backup20241105_8 -s _recover
[2024/11/05 17:57:41.192 +08:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/11/05 17:57:41.192 +08:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/11/05 17:57:41.193 +08:00] [INFO] [cmd/restore.go:47] ["restore cmd input args"] [args="[]"]
[2024/11/05 17:57:41.201 +08:00] [INFO] [core/backup_impl_restore_backup.go:31] ["receive RestoreBackupRequest"] [requestId=65611923-9b5c-11ef-bc97-eeeeeeeeeeee] [backupName=backup20241105_8] [onlyMeta=false] [restoreIndex=false] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=false] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=_recover] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=] [skipDiskQuotaCheck=false] [maxShardNum=0]
[2024/11/05 17:57:41.202 +08:00] [INFO] [core/backup_context.go:117] ["{Base:0xc00001cdc0 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:false GcPauseSeconds:7200 GcPauseAddress:http://localhost:9091}"]
[2024/11/05 17:57:41.202 +08:00] [INFO] [core/backup_context.go:118] ["{Base:0xc00001cdc0 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/11/05 17:57:41.202 +08:00] [INFO] [core/backup_context.go:308] ["receive GetBackupRequest"] [requestId=65626ff9-9b5c-11ef-bc97-eeeeeeeeeeee] [backupName=backup20241105_8] [backupId=] [bucketName=] [path=]
[2024/11/05 17:57:41.204 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=milvus-bucket] [root=file]
[2024/11/05 17:57:41.210 +08:00] [INFO] [core/backup_context.go:385] ["finish GetBackupRequest"] [requestId=65626ff9-9b5c-11ef-bc97-eeeeeeeeeeee] [backupName=backup20241105_8] [backupId=] [bucketName=] [path=]
[2024/11/05 17:57:41.210 +08:00] [INFO] [core/backup_impl_restore_backup.go:202] ["Collections to restore"] [collection_num=1]
[2024/11/05 17:57:41.222 +08:00] [INFO] [core/backup_impl_restore_backup.go:358] ["Start collection level restore pool"] [parallelism=2]
[2024/11/05 17:57:41.224 +08:00] [INFO] [core/backup_impl_restore_backup.go:362] ["executeRestoreBackupTask start"] [backup_name=backup20241105_8] [backupBucketName=milvus-bucket] [backupPath=file/backup20241105_8]
[2024/11/05 17:57:41.224 +08:00] [INFO] [core/backup_impl_restore_backup.go:373] ["start restore"] [backup_db_name=default] [backup_collection_name=default] [target_db_name=default] [target_collection_name=focuspasserbydetails_recover] [skipDiskQuotaCheck=false] [maxShardNum=0] [backupBucketName=milvus-bucket] [backupPath=file/backup20241105_8]
[2024/11/05 17:57:41.224 +08:00] [INFO] [core/backup_impl_restore_backup.go:373] ["collection schema"] [backup_db_name=default] [backup_collection_name=default] [target_db_name=default] [target_collection_name=focuspasserbydetails_recover] [skipDiskQuotaCheck=false] [maxShardNum=0] [fields="[{\"ID\":100,\"Name\":\"PasserbyKey\",\"PrimaryKey\":true,\"AutoID\":false,\"Description\":\"\",\"DataType\":21,\"TypeParams\":{\"max_length\":\"255\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0,\"DefaultValue\":null,\"Nullable\":false},{\"ID\":101,\"Name\":\"IndexObjectId\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0,\"DefaultValue\":null,\"Nullable\":false},{\"ID\":102,\"Name\":\"IndexOdEmbedding\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"\",\"DataType\":101,\"TypeParams\":{\"dim\":\"512\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0,\"DefaultValue\":null,\"Nullable\":false}]"]
[2024/11/05 17:57:41.327 +08:00] [INFO] [core/backup_impl_restore_backup.go:373] ["create collection"] [backup_db_name=default] [backup_collection_name=default] [target_db_name=default] [target_collection_name=focuspasserbydetails_recover] [skipDiskQuotaCheck=false] [maxShardNum=0] [hasPartitionKey=false]
[2024/11/05 17:57:41.334 +08:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=milvus-bucket] [root=file]
[2024/11/05 17:57:41.334 +08:00] [INFO] [core/backup_impl_restore_backup.go:373] ["start restore partition"] [backup_db_name=default] [backup_collection_name=default] [target_db_name=default] [target_collection_name=focuspasserbydetails_recover] [skipDiskQuotaCheck=false] [maxShardNum=0] [partition=_default]
[2024/11/05 17:57:41.339 +08:00] [INFO] [core/backup_impl_restore_backup.go:898] [getBackupPartitionPaths] [bucketName=milvus-bucket] [backupPath=file/backup20241105_8] [partitionID=452761217100312710] [groupId=453712021875091680]
[2024/11/05 17:57:41.341 +08:00] [INFO] [core/backup_impl_restore_backup.go:791] ["execute bulk insert"] [db=default] [collection=focuspasserbydetails_recover] [partition=_default] [files="[file/backup20241105_8/binlogs/insert_log/452761217100312709/452761217100312710/453712021875091680/,]"] [endTime=0]
[2024/11/05 17:57:41.351 +08:00] [INFO] [core/backup_impl_restore_backup.go:842] ["bulkinsert task state"] [id=453712021875690010] [state=2] [state="{\"ID\":453712021875690010,\"State\":2,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"backup\":\"true\",\"collection\":\"focuspasserbydetails_recover\",\"failed_reason\":\"\",\"files\":\"file/backup20241105_8/binlogs/insert_log/452761217100312709/452761217100312710/453712021875091680/,\",\"partition\":\"_default\",\"skip_disk_quota_check\":\"false\"},\"CollectionID\":453712021875690004,\"SegmentIDs\":null,\"CreateTs\":1730800661}"] [progress=0] [currentTimestamp=1730800661] [lastUpdateTime=1730800661]
[2024/11/05 17:57:46.356 +08:00] [INFO] [core/backup_impl_restore_backup.go:842] ["bulkinsert task state"] [id=453712021875690010] [state=6] [state="{\"ID\":453712021875690010,\"State\":6,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"backup\":\"true\",\"collection\":\"focuspasserbydetails_recover\",\"failed_reason\":\"\",\"files\":\"file/backup20241105_8/binlogs/insert_log/452761217100312709/452761217100312710/453712021875091680/,\",\"partition\":\"_default\",\"persist_cost\":\"0.01\",\"progress_percent\":\"100\",\"skip_disk_quota_check\":\"false\"},\"CollectionID\":453712021875690004,\"SegmentIDs\":null,\"CreateTs\":1730800661}"] [progress=100] [currentTimestamp=1730800666] [lastUpdateTime=1730800661]
[2024/11/05 17:57:46.357 +08:00] [INFO] [core/backup_impl_restore_backup.go:373] ["start restore l0 segments"] [backup_db_name=default] [backup_collection_name=default] [target_db_name=default] [target_collection_name=focuspasserbydetails_recover] [skipDiskQuotaCheck=false] [maxShardNum=0] [global_l0_segment_num=0] [partition_l0_segment_num=0]
[2024/11/05 17:57:46.357 +08:00] [INFO] [core/backup_impl_restore_backup.go:382] ["finish restore collection"] [db_name=default] [collection_name=focuspasserbydetails_recover] [size=94981]
[2024/11/05 17:57:46.358 +08:00] [INFO] [core/backup_impl_restore_backup.go:399] ["finish restore all collections"] [backupName=backup20241105_8] [collections=1] [taskID=restore_2024_11_05_09_57_41_210988948] ["duration in seconds"=5]
success
duration:5 s

Environment

milvus 2.3.1
milvus-backup 0.4.26

Anything else?

备份之后在minio中没有看到binlogs目录，只有meta目录

配置文件如下
backup.yaml.txt
check结果如下：

[yuan8180]

把milvus-backup降级到0.4.8之后，备份的minio改成milvus自己的minio之后就成功了，新的backup.yaml如下
new_backup.yaml.txt

[huanghaoyuanhhy]

可能是配置文件里 minio.rootPath 配置不正确，未来的版本里会在备份时加入相关检查。