Home
Caution
This documentation wiki is currently under construction for the upcoming release of Migration Assistant. While the contents can be used, please note that they will be changing rapidly until the release of Migration Assistant 2.0, which is targeted for October 2024.
This overview outlines the process for successfully performing an end-to-end, zero-downtime migration. The solution offered in this repository caters to several specific scenarios:
- Metadata Migration - Migrating cluster metadata, such as index settings, aliases, and templates.
- Backfill Migration - Migrating existing or historical data from a source to a target cluster.
- Live Traffic Migration - Replicating live ongoing traffic from source to target cluster.
- Comparative Tooling - Comparing the performance and behaviors of an existing cluster with a prospective new one.
In this guide, we focus on scenarios 1-3, guiding you through a backfill from a source cluster while concurrently handling live production traffic, which will be captured and replayed to a target cluster.
It's crucial to note that migration strategies are not universally applicable. This guide provides a detailed methodology, based on certain assumptions detailed throughout, emphasizing the importance of robust engineering practices to ensure a successful migration.
Your source cluster in this solution operates on Elasticsearch or OpenSearch, hosted on EC2 instances or similar computing environments. A proxy is set up to interact with this source cluster, either positioned in front of or directly on the coordinating nodes of the cluster.
A console that provides a migration-specific CLI and offers a variety of tools to streamline the migration process.
This component is designed for HTTP RESTful traffic, playing a dual role. It not only forwards traffic to the source cluster but also splits and channels this traffic to a stream-processing service for later playback.
Acting as a traffic simulation tool, the Traffic Replayer replays recorded request traffic to a target cluster, mirroring source traffic patterns. It links original requests and their responses to those directed at the target cluster, facilitating comparative analysis.
A tool integrated into the Migration CLI that can also be used independently to migrate cluster metadata, including index mappings, index configuration settings, templates, component templates, and aliases.
Reindexing data from an existing snapshot on Elastic Container Service (ECS) workers that coordinate the migration of documents from an existing snapshot, reindexing the documents in parallel to a target cluster.
The destination cluster for migration or comparison in an A/B test.
This architecture is based on the use of AWS cloud infrastructure, but most tools are designed to be cloud-independent. A local containerized version of this solution is also available.
The design deployed in AWS is as follows:
- Traffic is directed to the existing cluster.
- Traffic is redirected to an ALb with Traffic Capture Proxies replicating live traffic to be stored on the source cluster and in Amazon Managed Streaming for Apache Kafka (MSK) for playback.
- With continuous traffic capture in place, a backfill is initiated with a reindex-from-snapshot.
- Once a backfill has been completed, traffic captured is replayed by the user using a Traffic Replayer.
- The user evaluates the outcomes from routing traffic to both the original and the new cluster.
- After confirming the target cluster’s functionality meets expectations, the user dismantles all related stacks, retaining only the new cluster’s setup. Additionally, the user may retire and discard the old cluster’s legacy infrastructure.
Owner: @sumobrian