transfering my collections

[MaLo94]

How to transfer my collections from one server to another without any loss of data or conflicts in indexing or query nodes?
A description of the way I did once and worked for me:
I launched milvus and use it on my server with docker images in docker-compose I made as below:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.0
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://*.*.*.*:****--data-dir /etcd

minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2022-03-17T06-34-49Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "****:****"
volumes:
- minio:/minio_data
command: minio server /minio_data --console-address ":****"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:****/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3

standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.2.0
command: ["milvus", "run", "standalone"]
environment:
ETCD_ENDPOINTS: etcd:****
MINIO_ADDRESS: minio:****
volumes:
- milvus:/var/lib/milvus
ports:
- "****:****"
depends_on:
- "etcd"
- "minio"

after my milvus collections were ready, I made copy of my etcd, minio, and milvus volumes and transferred them to another server, then stopped and removed related containers and then created them again to restore the collection I had on the previous server.
This solution worked for me, but in the past week when my collections in another server have lots of records (one of them has about 3 million objects) and I wanted to do the same approach as before, the unknown problem occurred! I checked the Attu that the collection loaded successfully, but when I want to check the data preview in Attu, I faced this error :
"fail to search on all shard leaders, err=fail to Query, QueryNode ID = 9, reason=Query 9 failed, reason Head "http://minio:9000/a-bucket/files/insert_log/..."
I do not know why this happened, cause I made tar files from my volumes to prevent corruption or losing data in transferring process, but the conflict just happened and I don't know how to fix it.
do you have any recommendation or solution to fix this flow for me?! I investigated docs and found out that I can use Milvus-tools like migrating my collection, but I think there is no support for milvus 2.2.0.

should i change the code of Milvus-service (i made) in the way to load collections at first?

thanks in advance

[yhmo]

For standalone, there are three volumes to store milvus data:

1. volume of etcd to store metadata
2. volume of minio to store data files
3. volume of milvus server to store un-consumed data

To start milvus server successfully, the metadata and all data files must be strongly consist.
If milvus server is still running when you copy the volumes, the metadata and data files may be inconsistent. So, it is better to stop the entire service before you backup the volumes.

[yhmo]

"NoSuchKey" indicates the metadata and data files is inconsistent. The etcd stores a meta point to "files/insert_log/43911.." but the data file doesn't exist on minio.
So, you restarted the server in this sequence:

1. stop milvus-etcd
2. stop milvus-minio
3. stop milvus-standalone
4. copy the three volumes
5. restart the containsers
6. insert new records

Seems is a bug if this steps can reproduce the error. I am not sure the root cause, maybe related to this issue: #20769

I will report this problem to core dev team, wait our response.

[MaLo94]

I did the same approach you mentioned, but the error keeps happening...
thanks in advance

[yhmo]

So, let's do a clean test by v2.2.0.
Firstly, prepare the docker-compose.yml file:

version: '3.5'

services:
  etcd:
    container_name: milvus-etcd
    image: quay.io/coreos/etcd:v3.5.0
    environment:
      - ETCD_AUTO_COMPACTION_MODE=revision
      - ETCD_AUTO_COMPACTION_RETENTION=1000
      - ETCD_QUOTA_BACKEND_BYTES=4294967296
      - ETCD_SNAPSHOT_COUNT=50000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
    command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2022-03-17T06-34-49Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  standalone:
    container_name: milvus-standalone
    image: milvusdb/milvus:v2.2.0
    command: ["milvus", "run", "standalone"]
    environment:
      ETCD_ENDPOINTS: etcd:2379
      MINIO_ADDRESS: minio:9000
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
    ports:
      - "19530:19530"
      - "9091:9091"
    depends_on:
      - "etcd"
      - "minio"

networks:
  default:
    name: milvus

And a python script to insert vectors:

import random
import uuid

from pymilvus import (
    connections,
    FieldSchema, CollectionSchema, DataType,
    Collection,
    utility
)


_HOST = '127.0.0.1'
_PORT = '19530'

# Const names
_COLLECTION_NAME = 'demo'

# Vector parameters
_DIM = 128

# Index parameters
_METRIC_TYPE = 'L2'
_INDEX_TYPE = 'IVF_FLAT'
_NLIST = 1024


# Create a Milvus connection
def create_connection():
    print(f"\nCreate connection...")
    connections.connect(host=_HOST, port=_PORT)
    print(f"\nList connections:")
    print(connections.list_connections())


def insert_items(collection, items:[]):
    cnt = len(items)
    data = [
        [items[i]["Object_id"] for i in range(cnt)],
        [items[i]["Post_id"] for i in range(cnt)],
        [items[i]["pk"] for i in range(cnt)],
        [items[i]["Embedding"] for i in range(cnt)],
    ]
    collection.insert(data)
    print(cnt, "items inserted")


def gen_data():
    if utility.has_collection(_COLLECTION_NAME):
        collection = Collection(_COLLECTION_NAME)
    else:
        field1 = FieldSchema(name="Object_id", dtype=DataType.VARCHAR, max_length=512, is_primary=True)
        field2 = FieldSchema(name="Post_id", dtype=DataType.VARCHAR, max_length=512)
        field3 = FieldSchema(name="pk", dtype=DataType.INT64)
        field4 = FieldSchema(name="Embedding", dtype=DataType.FLOAT_VECTOR, dim=_DIM)
        schema = CollectionSchema(fields=[field1, field2, field3, field4])
        collection = Collection(name=_COLLECTION_NAME, data=None, schema=schema)
        print("\ncollection created:", _COLLECTION_NAME)

        index_param = {
            "index_type": _INDEX_TYPE,
            "params": {"nlist": _NLIST},
            "metric_type": _METRIC_TYPE}
        collection.create_index("Embedding", index_param)


    items = []
    batch_cnt = 10000
    for i in range(100000):
        item = {
            "Object_id": str(uuid.uuid1()),
            "Post_id": str(uuid.uuid1()),
            "pk": i,
            "Embedding": [random.random() for _ in range(_DIM)],
        }
        items.append(item)
        if len(items) >= batch_cnt:
            insert_items(collection, items)
            items = []

    if len(items) > 0:
        insert_items(collection, items)

    collection.flush()
    print("row count:", collection.num_entities)



if __name__ == '__main__':
    create_connection()
    gen_data()

Now begin test:

1. copy the docker-compose.yml to a folder, and start the milvus server by this command:
   docker-compose up -d

2. run the python script to insert some rows

3. stop the milvus server with this command:
   docker-compose stop

4. copy the volumes folder to another place:
   cp -r ./volumes ../volumes

5. restart the milvus server:
   docker-compose start

6. run the python script again to insert some rows

I test the steps on my local, it works fine.

[MaLo94]

Oh, yes, I think in this test everything works fine after taking backups and transferring volumes to the other server, but one important thing is that my docker-compose file does not contain just services related to my Milvus DB ( etcd, minio, standalone ). Also, I can not stop all services to transfer the Milvus backup, so I need to stop just the 3 services I mentioned and then take my backup. I did this :
docker stop Hash id related to standalone
docker stop Hash id related to etcd
docker stop Hash id related to mini

then copied related volumes and pasted them into another server.
then started the 3 containers in this way: minio -> etcd -> standalone
is this true?

[yhmo]

Oh, yes, I think in this test everything works fine after taking backups and transferring volumes to the other server, but one important thing is that my docker-compose file does not contain just services related to my Milvus DB ( etcd, minio, standalone ). Also, I can not stop all services to transfer the Milvus backup, so I need to stop just the 3 services I mentioned and then take my backup. I did this : docker stop Hash id related to standalone docker stop Hash id related to etcd docker stop Hash id related to mini

then copied related volumes and pasted them into another server. then started the 3 containers in this way: minio -> etcd -> standalone is this true?

I also tried this case:

1. docker-compose up -d
2. insert some rows
3. docker stop [minio]
4. docker stop [etcd]
5. docker stop [milvus]
6. copy the volumes folder to another place
7. docker start [minio]
8. docker start [etcd]
9. docker start [milvus]
10. insert some rows and search

I tested the steps several times, seems no problem. Just need to wait a moment when the milvus restart because it restored data from minio/etcd.

[edencorbin]

What about if you are upgrading your Milvus version, would this make these data folders from an earlier version bad /stop working?

[yhmo]

For minor version upgrading, just change the version in the yaml file, then restart the service. The new version service will reuse the existing data folders.

[imneonizer]

Any way to recover entities from minio bucket?

[yhmo]

The data files in minio are in a special format. We don't have a tool to parse data from minio.Theoretically, you can build a tool to parse the files according to the data management logic of milvus source code.

There is a tool milvus-backup to backup/restore data between milvus instances. This tool doesn't parse the files by itself, it copies the files to milvus storage and lets the milvus server to parse them.