diff --git a/v2.5.x/site/en/about/overview.md b/v2.5.x/site/en/about/overview.md index 4d8a5b2d6f..136f86cde0 100644 --- a/v2.5.x/site/en/about/overview.md +++ b/v2.5.x/site/en/about/overview.md @@ -140,4 +140,4 @@ In addition to primitive data types, Milvus supports various advanced data types Milvus has planned a set of connectors for you to seamlessly integrate Milvus with third-party tools, such as Apache Spark. Currently, you can use our Spark Connector to feed your Milvus data to Apache Spark for machine-learning processing. For details, refer to [Spark-Milvus Connector](integrate_with_spark.md). - Vector Transmission Services (VTS) - Milvus provides a set of tools for you to transfer your data between a Milvus instance and a bunch of data sources, including Zilliz clusters, Elasticsearch, Postgres (PgVector), and another Milvus instance. For details, refer to [Migrations](migrate_overview.md). + Milvus provides a set of tools for you to transfer your data between a Milvus instance and a bunch of data sources, including Zilliz clusters, Elasticsearch, Postgres (PgVector), and another Milvus instance. For details, refer to [VTS](https://github.com/zilliztech/vts). diff --git a/v2.5.x/site/en/adminGuide/backup/h2m.md b/v2.5.x/site/en/adminGuide/backup/h2m.md index 4faec67183..9e18ee4dac 100644 --- a/v2.5.x/site/en/adminGuide/backup/h2m.md +++ b/v2.5.x/site/en/adminGuide/backup/h2m.md @@ -8,11 +8,11 @@ deprecate: true # Migrate Data from HDF5 to Milvus -This topic describes how to import data in HDF5 files into Milvus using [MilvusDM](migrate_overview.md), an open-source tool specifically designed for Milvus data migration. +This topic describes how to import data in HDF5 files into Milvus using [VTS](https://github.com/zilliztech/vts), an open-source tool specifically designed for Milvus data migration. ## Prerequisites -You need to [install MilvusDM](milvusdm_install.md) before migrating Milvus data. +You need to install MilvusDM before migrating Milvus data. ## 1. Download YAML file @@ -91,8 +91,8 @@ $ milvusdm --yaml H2M.yaml ## What's next - If you are interested in migrating data in other forms into Milvus, - - Learn how to [Migrate Data from Faiss to Milvus](f2m.md). + - Learn how to Migrate Data from Faiss to Milvus. - If you are looking for information about how to migrate data from Milvus 1.x to Milvus 2.0, - - Learn [version migration](m2m.md). + - Learn version migration. - If you are interested in learning more about the data migration tool, - - Read the overview of [MilvusDM](migrate_overview.md). + - Read the overview of [VTS](https://github.com/zilliztech/vts). diff --git a/v2.5.x/site/en/adminGuide/backup/m2h.md b/v2.5.x/site/en/adminGuide/backup/m2h.md index 87f7f1f9ea..4196feeb6d 100644 --- a/v2.5.x/site/en/adminGuide/backup/m2h.md +++ b/v2.5.x/site/en/adminGuide/backup/m2h.md @@ -7,7 +7,7 @@ deprecate: true --- # Migrate Data from Milvus to HDF5 -You can save data in Milvus as HDF5 files using [MilvusDM](migrate_overview.md). +You can save data in Milvus as HDF5 files using [VTS](https://github.com/zilliztech/vts). 1. Download **M2H.yaml**: diff --git a/v2.5.x/site/en/adminGuide/m2m_legacy.md b/v2.5.x/site/en/adminGuide/m2m_legacy.md index 92ab8ee0e1..93d6384320 100644 --- a/v2.5.x/site/en/adminGuide/m2m_legacy.md +++ b/v2.5.x/site/en/adminGuide/m2m_legacy.md @@ -7,7 +7,7 @@ deprecate: true --- # Version Migration -This topic describes how to migrate data from Milvus 1.x to Milvus 2.0 using [MilvusDM](migrate_overview.md), an open-source tool specifically designed for Milvus data migration. +This topic describes how to migrate data from Milvus 1.x to Milvus 2.0 using [VTS](https://github.com/zilliztech/vts), an open-source tool specifically designed for Milvus data migration.
MilvusDM does not support migrating data from Milvus 2.0 standalone to Milvus 2.0 cluster. @@ -16,7 +16,7 @@ MilvusDM does not support migrating data from Milvus 2.0 standalone to Milvus 2. ## Prerequisites -You need to [install MilvusDM](milvusdm_install.md) before migrating Milvus data. +You need to install MilvusDM before migrating Milvus data. ## 1. Download YAML file @@ -92,7 +92,7 @@ $ milvusdm --yaml M2M.yaml ## What's next - If you are interested in migrating data in other forms into Milvus, - - Learn how to [Migrate Data from Faiss to Milvus](f2m.md). - - Learn how to [Migrate from HDF5 to Milvus](h2m.md). + - Learn how to Migrate Data from Faiss to Milvus. + - Learn how to Migrate from HDF5 to Milvus. - If you are interested in learning more about the data migration tool, - - Read the overview of [MilvusDM](migrate_overview.md). + - Read the overview of [VTS](mhttps://github.com/zilliztech/vts). diff --git a/v2.5.x/site/en/getstarted/offline/install_offline-helm.md b/v2.5.x/site/en/getstarted/offline/install_offline-helm.md index 87c21a0142..8dbcaa4a4c 100644 --- a/v2.5.x/site/en/getstarted/offline/install_offline-helm.md +++ b/v2.5.x/site/en/getstarted/offline/install_offline-helm.md @@ -98,5 +98,4 @@ Having installed Milvus, you can: - [Upgrade Milvus Using Helm Chart](upgrade_milvus_cluster-helm.md). - [Scale your Milvus cluster](scaleout.md). -- Explore [MilvusDM](migrate_overview.md), an open-source tool designed for importing and exporting data in Milvus. - [Monitor Milvus with Prometheus](monitor.md). diff --git a/v2.5.x/site/en/menuStructure/en.json b/v2.5.x/site/en/menuStructure/en.json index 1aebed380f..f60a29ea8a 100644 --- a/v2.5.x/site/en/menuStructure/en.json +++ b/v2.5.x/site/en/menuStructure/en.json @@ -956,50 +956,6 @@ } ] }, - { - "label": "Milvus Migration", - "id": "milvus_migration", - "isMenu": true, - "order": 6, - "children": [ - { - "label": "Overview", - "id": "migrate_overview.md", - "order": 0, - "children": [] - }, - { - "label": "Install Migration Tool", - "id": "milvusdm_install.md", - "order": 1, - "children": [] - }, - { - "label": "From Elasticsearch", - "id": "es2m.md", - "order": 2, - "children": [] - }, - { - "label": "From Faiss", - "id": "f2m.md", - "order": 3, - "children": [] - }, - { - "label": "From Milvus 1.x", - "id": "m2m.md", - "order": 4, - "children": [] - }, - { - "label": "From Milvus 2.3.x", - "id": "from-m2x.md", - "order": 5, - "children": [] - } - ] - }, { "label": "Administration Guide", "id": "admin_guide", @@ -1653,10 +1609,17 @@ } ] }, + { + "label": "Milvus VTS", + "id": "milvus_vts", + "order": 6, + "externalLink": "https://github.com/zilliztech/vts", + "children": [] + }, { "label": "Milvus Sizing Tool", "id": "sizing", - "order": 6, + "order": 7, "externalLink": "https://milvus.io/tools/sizing/", "children": [] }, diff --git a/v2.5.x/site/en/migrate/es2m.md b/v2.5.x/site/en/migrate/es2m.md deleted file mode 100644 index e521541a76..0000000000 --- a/v2.5.x/site/en/migrate/es2m.md +++ /dev/null @@ -1,175 +0,0 @@ ---- -id: es2m.md -summary: This guide provides a comprehensive, step-by-step process for migrating data from Elasticsearch to Milvus 2.x. -title: From Elasticsearch ---- - -# From Elasticsearch - -This guide provides a comprehensive, step-by-step process for migrating data from Elasticsearch to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance. - -## Prerequisites - -- **Software versions**: - - Source Elasticsearch: 7.x or 8.x - - Target Milvus: 2.x - - For installation details, refer to [Installing Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html) and [Install Milvus](https://milvus.io/docs/install_standalone-docker.md). -- **Required tools**: - - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). -- **Supported data types for migration**: The fields to migrate from the source Elasticsearch index are of the following types - [dense_vector](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/dense-vector.html#dense-vector), [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/keyword.html#keyword-field-type), [text](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/text.html#text-field-type), [long](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [integer](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [double](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [float](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [boolean](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/boolean.html), [object](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/object.html). Data types not listed here are currently not supported for migration. Refer to [Field mapping reference](#field-mapping-reference) for detailed information on data mappings between Milvus collections and Elasticsearch indexes. -- **Elasticsearch index requirements**: - - The source Elasticsearch index must contain a vector field of the `dense_vector` type. Migration cannot start without a vector field. - -## Configure the migration file - -Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. - -```yaml -dumper: # configs for the migration job. - worker: - workMode: "elasticsearch" # operational mode of the migration job. - reader: - bufferSize: 2500 # buffer size to read from Elasticsearch in each batch. A value ranging from 2000 to 4000 is recommended. -meta: # meta configs for the source Elasticsearch index and target Milvus 2.x collection. - mode: "config" # specifies the source for meta configs. currently, onlly `config` is supported. - version: "8.9.1" - index: "qatest_index" # identifies the Elasticsearch index to migrate data from. - fields: # fields within the Elasticsearch index to be migrated. - - name: "my_vector" # name of the Elasticsearch field. - type: "dense_vector" # data type of the Elasticsearch field. - dims: 128 # dimension of the vector field. required only when `type` is `dense_vector`. - - name: "id" - pk: true # specifies if the field serves as a primary key. - type: "long" - - name: "num" - type: "integer" - - name: "double1" - type: "double" - - name: "text1" - maxLen: 1000 # max. length of data fields. required only for `keyword` and `text` data types. - type: "text" - - name: "bl1" - type: "boolean" - - name: "float1" - type: "float" - milvus: # configs specific to creating the collection in Milvus 2.x - collection: "Collection_01" # name of the Milvus collection. defaults to the Elasticsearch index name if not specified. - closeDynamicField: false # specifies whether to disable the dynamic field in the collection. defaults to `false`. - shardNum: 2 # number of shards to be created in the collection. - consistencyLevel: Strong # consistency level for Milvus collection. -source: # connection configs for the source Elasticsearch server - es: - urls: - - "http://10.15.1.***:9200" # address of the source Elasticsearch server. - username: "" # username for the Elasticsearch server. - password: "" # password for the Elasticsearch server. -target: - mode: "remote" # storage location for dumped files. valid values: `remote` and `local`. - remote: # configs for remote storage - outputDir: "migration/milvus/test" # output directory path in the cloud storage bucket. - cloud: "aws" # cloud storage service provider. Examples: `aws`, `gcp`, `azure`, etc. - region: "us-west-2" # region of the cloud storage; can be any value if using local Minio. - bucket: "zilliz-aws-us-****-*-********" # bucket name for storing data; must align with configs in milvus.yaml for Milvus 2.x. - useIAM: true # whether to use an IAM Role for connection. - checkBucket: false # checks if the specified bucket exists in the storage. - milvus2x: # connection configs for the target Milvus 2.x server - endpoint: "http://10.102.*.**:19530" # address of the target Milvus server. - username: "****" # username for the Milvus 2.x server. - password: "******" # password for the Milvus 2.x server. -``` - -The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Elasticsearch to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_ES.md#migrationyaml-reference). - -- `dumper` - - | Parameter | Description | - | --- | --- | - | `dumper.worker.workMode` | The operational mode of the migration job. Set to `elasticsearch` when migrating from Elasticsearch indexes. | - | `dumper.worker.reader.bufferSize` | Buffer size to read from Elasticsearch in each batch. Unit: KB. | - -- `meta` - - | Parameter | Description | - | --- | --- | - | `meta.mode` | Specifies the source for meta configs. Currently, only `config` is supported. | - | `meta.index` | Identifies the Elasticsearch index to migrate data from. | - | `meta.fields` | Fields within the Elasticsearch index to be migrated. | - | `meta.fields.name` | Name of the Elasticsearch field. | - | `meta.fields.maxLen` | Maximum length of the field. This parameter is required only when `meta.fields.type` is `keyword` or `text`. | - | `meta.fields.pk` | Specifies if the field serves as the primary key. | - | `meta.fields.type` | Data type of the Elasticsearch field. Currently, the following data types in Elasticsearch are supported: [dense_vector](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/dense-vector.html#dense-vector), [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/keyword.html#keyword-field-type), [text](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/text.html#text-field-type), [long](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [integer](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [double](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [float](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/number.html), [boolean](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/boolean.html), [object](https://www.elastic.co/guide/en/elasticsearch/reference/8.13/object.html). | - | `meta.fields.dims` | Dimension of the vector field. This parameter is required only when `meta.fields.type` is `dense_vector`. | - | `meta.milvus` | Configs specific to creating the collection in Milvus 2.x. | - | `meta.milvus.collection` | Name of the Milvus collection. Defaults to the Elasticsearch index name if not specified. | - | `meta.milvus.closeDynamicField` | Specifies whether to disable the dynamic field in the collection. Defaults to `false`. For more information on dynamic fields, refer to [Enable Dynamic Field](https://milvus.io/docs/enable-dynamic-field.md#Enable-Dynamic-Field). | - | `meta.milvus.shardNum` | Number of shards to be created in the collection. For more information on shards, refer to [Terminology](https://milvus.io/docs/glossary.md#Shard). | - | `meta.milvus.consistencyLevel` | Consistency level for the collection in Milvus. For more information, refer to [Consistency](https://milvus.io/docs/consistency.md). | - -- `source` - - | Parameter | Description | - | --- | --- | - | `source.es` | Connection configs for the source Elasticsearch server. | - | `source.es.urls` | Address of the source Elasticsearch server. | - | `source.es.username` | Username for the Elasticsearch server. | - | `source.es.password` | Password for the Elasticsearch server. | - -- `target` - - | Parameter | Description | - | --- | --- | - | `target.mode` | Storage location for dumped files. Valid values:
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | - | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | - | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | - | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | - | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | - | `target.remote.useIAM` | Whether to use an IAM Role for connection. | - | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | - | `target.milvus2x` | Connection configs for the target Milvus 2.x server. | - | `target.milvus2x.endpoint` | Address of the target Milvus server. | - | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | - | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | - -## Start the migration task - -Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. - -```bash -./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml -``` - -The following is an example of a successful migration log output: - -```bash -[task/load_base_task.go:94] ["[LoadTasker] Dec Task Processing-------------->"] [Count=0] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304] -[task/load_base_task.go:76] ["[LoadTasker] Progress Task --------------->"] [fileName=testfiles/output/zwh/migration/test_mul_field4/data_1_1.json] [taskId=442665677354739304] -[dbclient/cus_field_milvus2x.go:86] ["[Milvus2x] begin to ShowCollectionRows"] -[loader/cus_milvus2x_loader.go:66] ["[Loader] Static: "] [collection=test_mul_field4_rename1] [beforeCount=50000] [afterCount=100000] [increase=50000] -[loader/cus_milvus2x_loader.go:66] ["[Loader] Static Total"] ["Total Collections"=1] [beforeTotalCount=50000] [afterTotalCount=100000] [totalIncrease=50000] -[migration/es_starter.go:25] ["[Starter] migration ES to Milvus finish!!!"] [Cost=80.009174459] -[starter/starter.go:106] ["[Starter] Migration Success!"] [Cost=80.00928425] -[cleaner/remote_cleaner.go:27] ["[Remote Cleaner] Begin to clean files"] [bucket=a-bucket] [rootPath=testfiles/output/zwh/migration] -[cmd/start.go:32] ["[Cleaner] clean file success!"] -``` - -## Verify the result - -Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). - -## Field mapping reference - -Review the table below to understand how field types in Elasticsearch indexes are mapped to field types in Milvus collections. - -For more information on supported data types in Milvus, refer to [Supported data types](https://milvus.io/docs/schema.md#Supported-data-types). - -| Elasticsearch Field Type | Milvus Field Type | Description | -| --- | --- | --- | -| dense_vector | FloatVector | Vector dimensions remain unchanged during migration. | -| keyword | VarChar | Set Max Length (1 to 65,535). Strings exceeding the limit can trigger migration errors. | -| text | VarChar | Set Max Length (1 to 65,535). Strings exceeding the limit can trigger migration errors. | -| long | Int64 | - | -| integer | Int32 | - | -| double | Double | - | -| float | Float | - | -| boolean | Bool | - | -| object | JSON | - | \ No newline at end of file diff --git a/v2.5.x/site/en/migrate/f2m.md b/v2.5.x/site/en/migrate/f2m.md deleted file mode 100644 index 77534510d5..0000000000 --- a/v2.5.x/site/en/migrate/f2m.md +++ /dev/null @@ -1,135 +0,0 @@ ---- -id: f2m.md -title: From Faiss -related_key: Faiss, migrate, import -summary: Learn how to migrate Faiss data to Milvus. ---- - -# From Faiss - -This guide provides a comprehensive, step-by-step process for migrating data from Faiss to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance. - -## Prerequisites - -- **Software versions**: - - Source Faiss - - Target Milvus: 2.x - - For installation details, see [Installing Faiss](https://github.com/facebookresearch/faiss/blob/main/INSTALL.md) and [Install Milvus](https://milvus.io/docs/install_standalone-docker.md). -- **Required tools**: - - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). - -## Configure the migration - -Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. - -```yaml -dumper: # configs for the migration job. - worker: - limit: 2 - workMode: faiss # operational mode of the migration job. - reader: - bufferSize: 1024 - writer: - bufferSize: 1024 -loader: - worker: - limit: 2 -source: # configs for the source Faiss index. - mode: local - local: - faissFile: ./testfiles/faiss/faiss_ivf_flat.index - -target: # configs for the target Milvus collection. - create: - collection: - name: test1w - shardsNums: 2 - dim: 256 - metricType: L2 - - mode: remote - remote: - outputDir: testfiles/output/ - cloud: aws - endpoint: 0.0.0.0:9000 - region: ap-southeast-1 - bucket: a-bucket - ak: minioadmin - sk: minioadmin - useIAM: false - useSSL: false - checkBucket: true - milvus2x: - endpoint: localhost:19530 - username: xxxxx - password: xxxxx - -``` - -The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Faiss to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_FAISS.md#migrationyaml-reference). - -- `dumper` - - | Parameter | Description | - | --- | --- | - | `dumper.worker.limit` | The concurrency of dumper threads. | - | `dumper.worker.workMode` | The operational mode of the migration job. Set to faiss when migrating from Faiss indexes. | - | `dumper.worker.reader.bufferSize` | Buffer size to read from Faiss in each batch. Unit: KB. | - | `dumper.worker.writer.bufferSize` | Buffer size to write to Milvus in each batch. Unit: KB. | - -- `loader` - - | Parameter | Description | - | --- | --- | - | `loader.worker.limit` | The concurrency of loader threads. | - -- `source` - - | Parameter | Description | - | --- | --- | - | `source.mode` | Specifies where the source files are read from. Valid values:
- `local`: reads files from a local disk.
- `remote`: reads files from remote storage. | - | `source.local.faissFile` | The directory path where the source files are located. For example, `/db/faiss.index`. | - -- `target` - - | Parameter | Description | - | --- | --- | - | `target.create.collection.name` | Name of the Milvus collection. | - | `target.create.collection.shardsNums` | Number of shards to be created in the collection. For more information on shards, refer to [Terminology](https://milvus.io/docs/glossary.md#Shard). | - | `target.create.collection.dim` | Dimension of the vector field. | - | `target.create.collection.metricType` | Metric type used to measure similarities between vectors. For more information, refer to [Terminology](https://milvus.io/docs/glossary.md#Metric-type). | - | `target.mode` | Storage location for dumped files. Valid values:
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | - | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | - | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | - | `target.remote.endpoint` | Endpoint of Milvus 2.x storage. | - | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | - | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | - | `target.remote.ak` | Access key for Milvus 2.x storage. | - | `target.remote.sk` | Secret key for Milvus 2.x storage. | - | `target.remote.useIAM` | Whether to use an IAM Role for connection. | - | `target.remote.useSSL` | Whether to enable SSL when connecting to Milvus 2.x. For more information, refer to [Encryption in Transit](https://milvus.io/docs/tls.md#Encryption-in-Transit). | - | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | - | `target.milvus2x.endpoint` | Address of the target Milvus server. | - | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | - | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | - -## Start the migration task - -1. Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. - - ```bash - ./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml - ``` - - The command above converts the Faiss index data into NumPy files, and then uses the [bulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation to write the data to the target bucket. - -2. Once NumPy files are generated, import these files into Milvus 2.x with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. - - ```bash - ./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml - ``` - - -## Verify the result - -Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). diff --git a/v2.5.x/site/en/migrate/from-m2x.md b/v2.5.x/site/en/migrate/from-m2x.md deleted file mode 100644 index d9ac26bd74..0000000000 --- a/v2.5.x/site/en/migrate/from-m2x.md +++ /dev/null @@ -1,148 +0,0 @@ ---- -id: from-m2x.md -summary: This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above. -title: From Milvus 2.3.x ---- - -# From Milvus 2.3.x - -This guide provides a comprehensive, step-by-step process for migrating data from Milvus 2.3.x to Milvus 2.3.x or above. - -## Prerequisites - -- **Software versions**: - - Source Milvus: 2.3.0+ (The tool uses the iterator to fetch source collection data, requiring source Milvus to be version 2.3.0 or above.) - - Target Milvus: 2.3.0+ -- **Required tools**: - - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). -- **Data preparation**: - - Ensure that the source Milvus collection is loaded and ready for data export. - - If the target Milvus does not contain a collection corresponding to the source collection, the [milvus-migration](https://github.com/zilliztech/milvus-migration) tool will automatically create it. Note that after migration, the target collection will not be indexed, and you must manually index the collection afterward. - -## Configure the migration file - -Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. - -```yaml -dumper: - worker: - workMode: milvus2x - reader: - bufferSize: 500 - -meta: - mode: config - version: 2.3.0 - collection: src_table_name - -source: - milvus2x: - endpoint: {milvus2x_domain}:{milvus2x_port} - username: xxxx - password: xxxxx - -target: - milvus2x: - endpoint: {milvus2x_domain}:{milvus2x_port} - username: xxxx - password: xxxxx -``` - -The following table describes the parameters in the example config file. For more information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x). - -- `dumper` - - | Parameter | Description | - | --- | --- | - | `dumper.worker.workMode` | The operational mode of the migration job. Set to milvus2x when migrating from Milvus 2.x. | - | `dumper.worker.reader.bufferSize` | Buffer size to read from Milvus 2.x in each batch. | - -- `meta` - - | Parameter | Description | - | --- | --- | - | `meta.mode` | Specifies where the meta file is read from. Set to config, indicating that the meta config can be obtained from this migration.yaml file. | - | `meta.version` | Source Milvus version. Set to 2.3.0 or above. | - | `meta.collection` | Source collection name. | - -- `source` - - | Parameter | Description | - | --- | --- | - | `source.milvus2x.endpoint` | Address of the source Milvus server. | - | `source.milvus2x.username` | Username for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | - | `source.milvus2x.password` | Password for the source Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | - -- `target` - - | Parameter | Description | - | --- | --- | - | `target.milvus2x.endpoint` | Address of the target Milvus server. | - | `target.milvus2x.username` | Username for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | - | `target.milvus2x.password` | Password for the target Milvus server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](authenticate.md). | - -## Start the migration task - -You have two options to start the migration task - using CLI or making API requests. Choose the one that best fits your needs. - -### Option 1: Using CLI - -Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. - -```bash -./milvus-migration start --config=/{YourConfigFilePath}/migration.yaml -``` - -Monitor the logs for progress updates. Successful migration logs should include entries like: - -```bash -[INFO] [migration/milvus2x_starter.go:79] ["=================>JobProcess!"] [Percent=100] -[INFO] [migration/milvus2x_starter.go:27] ["[Starter] migration Milvus2x to Milvus2x finish!!!"] [Cost=94.877717375] -[INFO] [starter/starter.go:109] ["[Starter] Migration Success!"] [Cost=94.878243583] -``` - -### Option 2: Making API requests - -You can also use the Restful API to execute the migration. Start the API server with: - -```bash -./milvus-migration server run -p 8080 -``` - -Once the server starts successfully, place the `migration.yaml` file in the `configs/` directory of the project and start the migration using: - -```bash -curl -XPOST http://localhost:8080/api/v1/start -``` - -## Verify the result - -After the migration task is completed, use Attu to view the number of entities migrated. Additionally, you can create indexes and load collections in Attu. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). - -## Additional configuration options - -In addition to the basic configurations mentioned above, you can also add additional settings based on your specific requirements. - -- **Selective field migration**: If you need to migrate only specific fields in a collection rather than all fields, specify the fields to be migrated in the `meta` section of the `migration.yaml` file. - - ```yaml - meta: - fields: - - name: id - - name: title_vector - - name: reading_time - ``` - -- **Custom target collection**: To customize the properties of the target collection, add the related configurations in the `meta` section of the `migration.yaml` file. - - ```yaml - meta: - milvus: - collection: target_collection_name - shardNum: 2 - closeDynamicField: false - consistencyLevel: Customized - ``` - - -For detailed information, refer to [Milvus Migration: Milvus2.x to Milvus2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_2X.md#milvus-migration-milvus2x-to-milvus2x). diff --git a/v2.5.x/site/en/migrate/m2m.md b/v2.5.x/site/en/migrate/m2m.md deleted file mode 100644 index caddd36d1c..0000000000 --- a/v2.5.x/site/en/migrate/m2m.md +++ /dev/null @@ -1,156 +0,0 @@ ---- -id: m2m.md -summary: This guide provides a comprehensive, step-by-step process for migrating data from Milvus 1.x (including 0.9.x and above) to Milvus 2.x. -title: From Milvus 1.x ---- - -# From Milvus 1.x - -This guide provides a comprehensive, step-by-step process for migrating data from Milvus 1.x (including 0.9.x and above) to Milvus 2.x. By following this guide, you will be able to efficiently transfer your data, leveraging Milvus 2.x advanced features and improved performance. - -## Prerequisites - -- **Software versions**: - - Source Milvus: 0.9.x to 1.x - - Target Milvus: 2.x -- **Required tools**: - - [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool. For installation details, refer to [Install Migration Tool](milvusdm_install.md). - -## Export metadata of the source Milvus installation - -To prepare migration data for Milvus 0.9.x through 1.x, stop the source Milvus or at least stop performing any DML operations in it. - -1. Export metadata of the source Milvus installation to `meta.json`. - - - For those installations using MySQL as the backend, run - - ```bash - ./milvus-migration export -m "user:password@tcp(adderss)/milvus?charset=utf8mb4&parseTime=True&loc=Local" -o outputDir - ``` - - - For those installations using SQLite as the backend, run - - ```bash - ./milvus-migration export -s /milvus/db/meta.sqlite -o outputDir - ``` - -2. Copy the `tables` folder of your Milvus installation, then move both `meta.json` and the `tables` folder to an empty folder. - - Once this step is done, the structure of the empty folder should look like this: - - ``` - migration_data - ├── meta.json - └── tables - ``` - -3. Upload the folder prepared in the preceding step to an S3 block storage bucket or directly use this local folder in the next section. - -## Configure the migration file - -Save the example migration config file as `migration.yaml` and modify the configs based on your actual conditions. You are free to put the config file in any local directory. - -```yaml -dumper: - worker: - limit: 2 - workMode: milvus1x - reader: - bufferSize: 1024 - writer: - bufferSize: 1024 -loader: - worker: - limit: 16 -meta: - mode: local - localFile: /outputDir/test/meta.json -source: - mode: local - local: - tablesDir: /db/tables/ -target: - mode: remote - remote: - outputDir: "migration/test/xx" - ak: xxxx - sk: xxxx - cloud: aws - region: us-west-2 - bucket: xxxxx - useIAM: true - checkBucket: false - milvus2x: - endpoint: "{yourMilvus2_xServerAddress}:{port}" - username: xxxx - password: xxxx -``` - -The following table describes the parameters in the example config file. For a full list of configs, refer to [Milvus Migration: Milvus1.x to Milvus 2.x](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md#migrationyaml-reference). - -- `dumper` - - | Parameter | Description | - | --- | --- | - | `dumper.worker.limit` | The concurrency of dumper threads. | - | `dumper.worker.workMode` | The operational mode of the migration job. Set to `milvus1x` when migrating from Milvus 1.x. | - | `dumper.worker.reader.bufferSize` | Buffer size to read from Milvus 1.x in each batch. Unit: KB. | - | `dumper.worker.writer.bufferSize` | Buffer size to write to Milvus 2.x in each batch. Unit: KB. | - -- `loader` - - | Parameter | Description | - | --- | --- | - | `loader.worker.limit` | The concurrency of loader threads. | - -- `meta` - - | Parameter | Description | - | --- | --- | - | `meta.mode` | Specifies where the meta file meta.json is read from. Valid values: `local`, `remote`, `mysql`, `sqlite`. | - | `meta.localFile` | Local directory path where the `meta.json` file resides. This config is used only when `meta.mode` is set to `local`. For other meta configs, refer to [README_1X](https://github.com/zilliztech/milvus-migration/blob/main/README_1X.md#meta). | - -- `source` - - | Parameter | Description | - | --- | --- | - | `source.mode` | Specifies where the source files are read from. Valid values:
- `local`: reads files from a local disk.
- `remote`: reads files from remote storage. | - | `source.local.tablesDir` | The directory path where the source files are located. For example, `/db/tables/`. | - -- `target` - - | Parameter | Description | - | --- | --- | - | `target.mode` | Storage location for dumped files. Valid values:
- `local`: Store dumped files on local disks.
- `remote`: Store dumped files on object storage. | - | `target.remote.outputDir` | Output directory path in the cloud storage bucket. | - | `target.remote.ak` | Access key for Milvus 2.x storage. | - | `target.remote.sk` | Secret key for Milvus 2.x storage. | - | `target.remote.cloud` | Cloud storage service provider. Example values: `aws`, `gcp`, `azure`. | - | `target.remote.region` | Cloud storage region. It can be any value if you use local MinIO. | - | `target.remote.bucket` | Bucket name for storing data. The value must be the same as the config in Milvus 2.x. For more information, refer to [System Configurations](https://milvus.io/docs/configure_minio.md#miniobucketName). | - | `target.remote.useIAM` | Whether to use an IAM Role for connection. | - | `target.remote.checkBucket` | Whether to check if the specified bucket exists in object storage. | - | `target.milvus2x.endpoint` | Address of the target Milvus server. | - | `target.milvus2x.username` | Username for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | - | `target.milvus2x.password` | Password for the Milvus 2.x server. This parameter is required if user authentication is enabled for your Milvus server. For more information, refer to [Enable Authentication](https://milvus.io/docs/authenticate.md). | - -## Start the migration task - -1. Start the migration task with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. - - ```bash - ./milvus-migration dump --config=/{YourConfigFilePath}/migration.yaml - ``` - - The command above converts the source data in Milvus 1.x into NumPy files, and then uses the [bulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation to write the data to the target bucket. - -2. Once NumPy files are generated, import these files into Milvus 2.x with the following command. Replace `{YourConfigFilePath}` with the local directory where the config file `migration.yaml` resides. - - ```bash - ./milvus-migration load --config=/{YourConfigFilePath}/migration.yaml - ``` - - -## Verify the result - -Once the migration task is executed, you can make API calls or use Attu to view the number of entities migrated. For more information, refer to [Attu](https://github.com/zilliztech/attu) and [get_collection_stats()](https://milvus.io/api-reference/pymilvus/v2.4.x/MilvusClient/Collections/get_collection_stats.md). diff --git a/v2.5.x/site/en/migrate/migrate_overview.md b/v2.5.x/site/en/migrate/migrate_overview.md deleted file mode 100644 index 65a2d9be10..0000000000 --- a/v2.5.x/site/en/migrate/migrate_overview.md +++ /dev/null @@ -1,51 +0,0 @@ ---- -id: migrate_overview.md -summary: This article provides an overview of the Milvus-migration tool, including supported migrations, features, and architecture. -title: Milvus Migration Overview ---- - -# Milvus Migration Overview - -Recognizing the diverse needs of user base, Milvus has expanded its migration tools to not only facilitate upgrades from earlier Milvus 1.x versions but also to enable seamless integration of data from other systems like [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html) and [Faiss](https://github.com/facebookresearch/faiss). The [Milvus-migration](https://github.com/zilliztech/milvus-migration) project is designed to bridge the gap between these varied data environments and the latest advancements in Milvus technology, ensuring you can harness improved features and performance seamlessly. - -## Supported migrations - -The [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool supports a variety of migration paths to accommodate different user needs: - -- [Elasticsearch to Milvus 2.x](es2m.md): Enabling users to migrate data from Elasticsearch environments to take advantage of Milvus's optimized vector search capabilities. -- [Faiss to Milvus 2.x](f2m.md): Providing experimental support for transferring data from Faiss, a popular library for efficient similarity search. -- [Milvus 1.x to Milvus 2.x](m2m.md): Ensuring data from earlier versions is transitioned smoothly to the latest framework. -- [Milvus 2.3.x to Milvus 2.3.x or above](from-m2x.md): Providing a one-time migration path for users who have already migrated to 2.3.x. - -## Features - -Milvus-migration is designed with robust features to handle diverse migration scenarios: - -- Multiple interaction methods: You can perform migrations via a command line interface or through a Restful API, with flexibility in how migrations are executed. -- Support for various file formats and cloud storage: The [Milvus-migration](https://github.com/zilliztech/milvus-migration) tool can handle data stored in local files as well as in cloud storage solutions such as S3, OSS, and GCP, ensuring broad compatibility. -- Data type handling: [Milvus-migration](https://github.com/zilliztech/milvus-migration) is capable of dealing with both vector data and scalar fields, making it a versatile choice for different data migration needs. - -## Architecture - -The architecture of [Milvus-migration](https://github.com/zilliztech/milvus-migration) is strategically designed to facilitate efficient data streaming, parsing, and writing processes, enabling robust migration capabilities across various data sources. - -![Milvus-migration architecture](../../../assets/milvus-migration-architecture.jpeg) - -In the preceding figure: - -- **Data source**: [Milvus-migration](https://github.com/zilliztech/milvus-migration) supports multiple data sources including Elasticsearch via the [scroll API](https://www.elastic.co/guide/en/elasticsearch/reference/current/scroll-api.html), local or cloud storage data files, and Milvus 1.x databases. These are accessed and read in a streamlined manner to initiate the migration process. -- **Stream pipeline**: - - **Parse process**: Data from the sources is parsed according to its format. For example, for a data source from Elasticsearch, an Elasticsearch format parser is employed, while other formats use respective parsers. This step is crucial for transforming raw data into a structured format that can be further processed. - - **Convert process**: Following parsing, data undergoes conversion where fields are filtered, data types are converted, and table names are adjusted according to the target Milvus 2.x schema. This ensures that the data conforms to the expected structure and types in Milvus. -- **Data writing and loading**: - - **Write data**: The processed data is written into intermediate JSON or NumPy files, ready to be loaded into Milvus 2.x. - - **Load data**: Data is finally loaded into Milvus 2.x using the [BulkInsert](https://milvus.io/api-reference/pymilvus/v2.4.x/ORM/utility/do_bulk_insert.md) operation, which efficiently writes large volumes of data into Milvus storage systems, either cloud-based or filestore. - -## Future plans - -The development team is committed to enhancing [Milvus-migration](https://github.com/zilliztech/milvus-migration) with features such as: - -- **Support for more data sources**: Plans to extend support to additional databases and file systems, such as Pinecone, Chroma, Qdrant. If you need support for a specific data source, please submit your request through this [GitHub issue link](https://github.com/zilliztech/milvus-migration/issues). -- **Command simplification**: Efforts to streamline the command process for easier execution. -- **SPI parser** / **convert**: The architecture expects to include Service Provider Interface (SPI) tools for both parsing and converting. These tools allow for custom implementations that users can plug into the migration process to handle specific data formats or conversion rules. -- **Checkpoint resumption**: Enabling migrations to resume from the last checkpoint to enhance reliability and efficiency in case of interruptions. Save points will be created to ensure data integrity and are stored in databases such as SQLite or MySQL to track the progress of the migration process. diff --git a/v2.5.x/site/en/migrate/milvusdm_install.md b/v2.5.x/site/en/migrate/milvusdm_install.md deleted file mode 100644 index d3f16d4e4e..0000000000 --- a/v2.5.x/site/en/migrate/milvusdm_install.md +++ /dev/null @@ -1,49 +0,0 @@ ---- -id: milvusdm_install.md -summary: Learn how to install Milvus-Migration to migrate your data. -title: Install Migration Tool ---- - -# Install Migration Tool - -We support downloading the executable binary file or compiling the Milvus-migration tool from source. - -## Download the executable binary - -1. Download the latest release from the [Milvus-Migration GitHub repository](https://github.com/zilliztech/milvus-migration/tags). -2. Extract the downloaded file to obtain the `milvus-migration` executable binary. - -## Compile from source - -Alternatively, download and compile the source to obtain an executable binary file. - -1. Clone the Milvus-Migration repository: - - ```bash - # clone the source project - git clone https://github.com/zilliztech/milvus-migration.git - ``` - -2. Navigate to the project directory: - - ```bash - cd milvus-migration - ``` - -3. Compile the project to obtain the executable file: - - ```bash - # compile the project to obtain an executable file - go get & go build - ``` - - This will generate the `milvus-migration` executable in the project directory. - - -## What's next - -Having the Milvus-migration tool installed, you can migrate data from various sources: - -- [From Elasticsearch](es2m.md) -- [From Faiss](f2m.md) -- [From Milvus 1.x](m2m.md) \ No newline at end of file