Skip to content
This repository has been archived by the owner on Nov 13, 2024. It is now read-only.

Latest commit

 

History

History
127 lines (93 loc) · 3.82 KB

README.md

File metadata and controls

127 lines (93 loc) · 3.82 KB
Raw

Milvus Migration: migrate data to Milvus 2.x

Deprecated

This Tool is currently deprecated, please use VTS (Vector Transport Service) instead. VTS is a new tool that supports migrating data from various data sources to Milvus 2.x. It is more powerful and flexible than milvus-migration.

  • Milvus
  • Pinecone
  • Qdrant
  • Elasticsearch
  • PostgreSQL
  • Tencent Cloud VectorDB
  • .etc

Overview

milvus-migration is a data migration tool for Milvus that supports importing Milvus 0.9.x ~ 1.x / faiss / elasticsearch7.x+ / Milvus2.3+ datas to milvus 2.x.

Architecture

Getting started

Prerequisites

  • Operating System Requirements
Operating System Supported Versions
CentOS 7.5 or later
Ubuntu LTS 18.04 or later
  • Software Requirements
Software Version
Milvus 0.9.x, 1.x or 2.x
Elasticsearch 7.x or 8.x
go 1.22.2 or later
  • Data Format Support
Source Data Type Target Data Type
Milvus 0.9.x - 1.x Milvus 2.x
Elasticsearch 7.x-8.x Milvus 2.x
Faiss (Beta) Milvus 2.x
Milvus 2.3 + Milvus 2.x

How to use this tool?

  1. Download the source code and build it:
git clone [email protected]:zilliztech/milvus-migration.git
go build
  1. After building, you will obtain the milvus-migration binary file. Place this file and the migration.yaml file in the directory structure like below:
directory
---- milvus-migration
---- configs/migration.yaml
  1. Milvus1.x or Faiss to Milvus2.x migration

Run the command dump to dump the source data to numpy:

./milvus-migration dump

Finally, load the numpy files to Milvus 2.x successfully by using the load command:

./milvus-migration load
  1. Elasticsearch to Milvus2.x migration:

Run the command start to migration es->2.x. here start == dump && load cmd

./milvus-migration start
  1. Milvus 2.3+ to Milvus2.x migration:
./milvus-migration start

Migration Examples (migration.yaml details)

how to learn more about using migration tool, see examples doc below:

  1. es -> milvux2.x : migrate_es_doc.
  2. milvus1.x -> milvux2.x : migrate_1.x_doc.
  3. faiss -> milvux2.x ( Beta) : migrate_faiss_doc.
  4. milvus2.x -> milvux2.x : migrate_milvus2x_doc.

How to verify migration result

When migration finished, you can use visual tool Attu or use Milvus SDK verify your new collection data rows.

Build Index and search

After the Milvus collection Data migration is completed, we can use SDK or Attu to create index and load collection for the next search operation.

Future work

  • Support Redis to Milvus 2.x
  • Support Mongodb to Milvus 2.x
  • Support others datasource to Milvus 2.x
  • Supports binary vectors

Contributing

Contributions to milvus-migration are welcome from everyone. See Contributing for details of the contribution workflow.