-
Notifications
You must be signed in to change notification settings - Fork 23
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
123 changed files
with
1,225 additions
and
1,321 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
55 changes: 55 additions & 0 deletions
55
docs/en-base/architecture-overview/architecture-details.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| This section describes the components and AWS services that make up this solution and the architecture details on how these components work together. | ||
|
|
||
|
|
||
| ## AWS services in this solution | ||
|
|
||
| The following AWS services are included in this solution: | ||
|
|
||
| | AWS service | Description | | ||
| | --- | --- | | ||
| | [Amazon CloudFront](https://aws.amazon.com/cloudfront/) | **Core**. To made available the static web assets (frontend user interface). | | ||
| | [AWS AppSync](https://aws.amazon.com/appsync/) | **Core**. To provide the backend APIs. | | ||
| | [AWS Lambda](https://aws.amazon.com/lambda/) | **Core**. To call backend APIs. | | ||
| | [Amazon ECS](https://aws.amazon.com/ecs/) | **Core**. To run the container images used by the plugin template. | | ||
| | [Amazon DynamoDB](https://aws.amazon.com/dynamodb/) | **Core**. To store a record with transfer status for each object. | | ||
| | [Amazon EC2](https://aws.amazon.com/ec2/) | **Core**. To consume the messages in Amazon SQS and transfer the object from the source bucket to the destination bucket. | | ||
| | [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/) | **Core**. Stores the credential for data transfer. | | ||
| | [AWS Step Functions](https://aws.amazon.com/step-functions/) | **Supporting**. To start or stop/delete the ECR or S3 plugin template. | | ||
| | [Amazon S3](https://aws.amazon.com/s3/) | **Supporting**. To store the static web assets (frontend user interface). | | ||
| | [Amazon Cognito](https://aws.amazon.com/cognito/) | **Supporting**. To authenticate users (in AWS Regions). | | ||
| | [Amazon ECR](https://aws.amazon.com/ecr/) | **Supporting**. To host the container images. | | ||
| | [Amazon SQS](https://aws.amazon.com/sqs/) | **Supporting**. To store the transfer tasks temporarily as a buffer. | | ||
| | [Amazon EventBridge](https://aws.amazon.com/eventbridge/) | **Supporting**. To invoke the transfer tasks regularly. | | ||
| | [Amazon SNS](https://aws.amazon.com/sns/) | **Supporting**. Provides topic and email subscription notifications for data transfer results. | | ||
| | [AWS CloudWatch](https://aws.amazon.com/cloudwatch/) | **Supporting**. To monitor the data transfer progress. | | ||
|
|
||
| ## How Data Transfer Hub works | ||
|
|
||
| This solution has three components: a web console, the Amazon S3 transfer engine, and the Amazon ECR transfer engine. | ||
|
|
||
| ### Web console | ||
| This solution provides a simple web console which allows you to create and manage transfer tasks for Amazon S3 and Amazon ECR. | ||
|
|
||
| ### Amazon S3 transfer engine | ||
| Amazon S3 transfer engine runs the Amazon S3 plugin and is used for transferring objects from their sources into S3 buckets. The S3 plugin supports the following features: | ||
|
|
||
| - Transfer Amazon S3 objects between AWS China Regions and AWS Regions | ||
| - Transfer objects from Alibaba Cloud OSS / Tencent COS / Qiniu Kodo to Amazon S3 | ||
| - Transfer objects from S3 Compatible Storage service to Amazon S3 | ||
| - Support near real time transfer via S3 Event | ||
| - Support transfer with object metadata | ||
| - Support incremental data transfer | ||
| - Support transfer from private payer request bucket | ||
| - Auto retry and error handling | ||
|
|
||
| ### Amazon ECR transfer engine | ||
|
|
||
| Amazon ECR engine runs the Amazon ECR plugin and is used for transferring container images from other container registries. The ECR plugin supports the following features: | ||
|
|
||
| - Transfer Amazon ECR images between AWS China Regions and AWS Regions | ||
| - Transfer from public container registry (such as Docker Hub, GCR.io, Quay.io) to Amazon ECR | ||
| - Transfer selected images to Amazon ECR | ||
| - Transfer all images and tags from Amazon ECR | ||
| The ECR plugin leverages [skopeo][skopeo] for the underlying engine. The AWS Lambda function lists images in their sources and uses Fargate to run the transfer jobs. | ||
|
|
||
| [skopeo]: https://github.com/containers/skopeo |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
46 changes: 46 additions & 0 deletions
46
docs/en-base/architecture-overview/design-considerations.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| This solution was designed with best practices from the [AWS Well-Architected Framework][well-architected-framework] which helps customers design and operate reliable, secure, efficient, and cost-effective workloads in the cloud. | ||
|
|
||
| This section describes how the design principles and best practices of the Well-Architected Framework were applied when building this solution. | ||
|
|
||
| ## Operational excellence | ||
| This section describes how the principles and best practices of the [operational excellence pillar][operational-excellence-pillar] were applied when designing this solution. | ||
|
|
||
| The Data Transfer Hub solution pushes metrics to Amazon CloudWatch at various stages to provide observability into the infrastructure, Lambda functions, Amazon EC2 transfer workers, Step Function workflow and the rest of the solution components. Data transferring errors are added to the Amazon SQS queue for retries and alerts. | ||
|
|
||
| ## Security | ||
| This section describes how the principles and best practices of the [security pillar][security-pillar] were applied when designing this solution. | ||
|
|
||
| - Data Transfer Hub web console users are authenticated and authorized with Amazon Cognito. | ||
| - All inter-service communications use AWS IAM roles. | ||
| - All roles used by the solution follows least-privilege access. That is, it only contains minimum permissions required so the service can function properly. | ||
|
|
||
| ## Reliability | ||
| This section describes how the principles and best practices of the [reliability pillar][reliability-pillar] were applied when designing this solution. | ||
|
|
||
| - Using AWS serverless services wherever possible (for example, Lambda, Step Functions, Amazon S3, and Amazon SQS) to ensure high availability and recovery from service failure. | ||
| - Data is stored in DynamoDB and Amazon S3, so it persists in multiple Availability Zones (AZs) by default. | ||
|
|
||
| ## Performance efficiency | ||
| This section describes how the principles and best practices of the [performance efficiency pillar][performance-efficiency-pillar] were applied when designing this solution. | ||
|
|
||
| - The ability to launch this solution in any Region that supports AWS services in this solution such as: AWS Lambda, AWS S3, Amazon SQS, Amazon DynamoDB, and Amazon EC2. | ||
| - Automatically testing and deploying this solution daily. Reviewing this solution by solution architects and subject matter experts for areas to experiment and improve. | ||
|
|
||
| ## Cost optimization | ||
| This section describes how the principles and best practices of the [cost optimization pillar][cost-optimization-pillar] were applied when designing this solution. | ||
|
|
||
| - Use Autoscaling Group so that the compute costs are only related to how much data is transferred. | ||
| - Using serverless services such as Amazon SQS and DynamoDB so that customers only get charged for what they use. | ||
|
|
||
| ## Sustainability | ||
| This section describes how the principles and best practices of the [sustainability pillar][sustainability-pillar] were applied when designing this solution. | ||
|
|
||
| - The solution‘s serverless design (using Lambda, Amazon SQS and DynamoDB) and the use of managed services (such as Amazon EC2) are aimed at reducing carbon footprint compared to the footprint of continually operating on-premises servers. | ||
|
|
||
| [well-architected-framework]:https://aws.amazon.com/architecture/well-architected/?wa-lens-whitepapers.sort-by=item.additionalFields.sortDate&wa-lens-whitepapers.sort-order=desc&wa-guidance-whitepapers.sort-by=item.additionalFields.sortDate&wa-guidance-whitepapers.sort-order=desc | ||
| [operational-excellence-pillar]:https://docs.aws.amazon.com/wellarchitected/latest/operational-excellence-pillar/welcome.html | ||
| [security-pillar]:https://docs.aws.amazon.com/wellarchitected/latest/security-pillar/welcome.html | ||
| [reliability-pillar]:https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html | ||
| [performance-efficiency-pillar]:https://docs.aws.amazon.com/wellarchitected/latest/performance-efficiency-pillar/welcome.html | ||
| [cost-optimization-pillar]:https://docs.aws.amazon.com/wellarchitected/latest/cost-optimization-pillar/welcome.html | ||
| [sustainability-pillar]:https://docs.aws.amazon.com/wellarchitected/latest/sustainability-pillar/sustainability-pillar.html |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| - Aiden Dai | ||
| - Eva Liu | ||
| - Kervin Hu | ||
| - Haiyun Chen | ||
| - Joe Shi | ||
| - Ashwini Rudra | ||
| - Jyoti Tyagi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| Use the following steps to deploy this solution on AWS. For detailed instructions, follow the links for each step. | ||
|
|
||
| Before you launch the solution, [review the cost](../../plan-deployment/cost), architecture, network security, and other considerations discussed in this guide. Follow the step-by-step instructions in this section to configure and deploy the solution into your account. | ||
|
|
||
|
|
||
| **Time to deploy**: Approximately 15 minutes | ||
|
|
||
| - Step 1. Launch the stack | ||
| - [(Option 1) Deploy the AWS CloudFormation template in AWS Regions](../deployment/#launch-cognito) | ||
| - [(Option 2) Deploy the AWS CloudFormation template in AWS China Regions](../deployment/#launch-openid) | ||
|
|
||
| - Step 2. [Launch the web console](../deployment/#launch-web-console) | ||
| - Step 3. [Create a transfer task](../deployment/#create-task) |
Oops, something went wrong.