Update to version v1.2.0

CHANGELOG.md

@@ -5,12 +5,29 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.2.0] - 2021-05-04
+
+### Added
+
+- Two stack deployment options that provision machine learning (ML) pipelines either in a single AWS account, or across multiple AWS accounts for development, staging/test, and production environments.
+- Ability to provide an optional AWS Key Management Service (KMS) key to encrypt captured data from the real-time Amazon SageMaker endpoint, output of batch transform and data baseline jobs, output of model monitor, and Amazon Elastic Compute Cloud (EC2) instance's volume used by Amazon SageMaker to run the solution's pipelines.
+- New pipeline to build and register Docker images for custom ML algorithms.
+- Ability to use an existing Amazon Elastic Container Registry (Amazon ECR) repository, or create a new one, to store Docker images for custom ML algorithms.
+- Ability to provide different input/output Amazon Simple Storage Service (Amazon S3) buckets per pipeline deployment.
+
+### Updated
+
+- The creation of Amazon SageMaker resources using AWS CloudFormation.
+- The request body of the solution's API calls to provision pipelines.
+- AWS SDK to use the solution's identifier to track requests made by the solution to AWS services.
+- AWS Cloud Development Kit (AWS CDK) and AWS Solutions Constructs to version 1.96.0.
+
 ## [1.1.1] - 2021-03-19
 
 ### Updated
 
 - AWS ECR image scan on push property's name from `scanOnPush` to `ScanOnPush` for image scanning based on the recently updated property name in AWS CloudFormation.
-- AWS ECR repository's name in the IAM policy's resource name from `<repository-name>*` to `<pipeline_stack_name>*-<repository-name>*` to accommodate recent repository name being prefixed with AWS CloudFormation stack name.
+- AWS ECR repository's name in the IAM policy's resource name from `<repository-name>*` to `*<repository-name>*` to accommodate recent repository name being prefixed with AWS CloudFormation stack name.
 
 ## [1.1.0] - 2021-01-26
 

README.md

@@ -2,38 +2,48 @@
 
 The machine learning (ML) lifecycle is an iterative and repetitive process that involves
 changing models over time and learning from new data. As ML applications gain popularity,
-organizations are building new and better applications for a wide range of use cases
-including optimized email campaigns, forecasting tools, recommendation engines, self-driving
-vehicles, virtual personal assistants, and more. While operational and pipelining
-processes vary greatly across projects and organizations, the processes contain
-commonalities across use cases.
-
-The AWS MLOps Framework solution helps you streamline and enforce architecture best
-practices for machine learning (ML) model productionization. This solution is an extendable
-framework that provides a standard interface for managing ML pipelines for AWS ML
-services and third-party services. The solution’s template allows customers to upload their
-trained models, configure the orchestration of the pipeline, trigger the start of the deployment
-process, move models through different stages of deployment, and monitor the successes
-and failures of the operations.
-
-You can use batch and real-time data inferences to configure the pipeline for your business
-context. You can also provision multiple model monitor pipelines to periodically monitor the quality of deployed Amazon SageMaker's ML models. This solution increases your team’s agility and efficiency by allowing them to
-repeat successful processes at scale.
+organizations are building new and better applications for a wide range of use cases including
+optimized email campaigns, forecasting tools, recommendation engines, self-driving vehicles,
+virtual personal assistants, and more. While operational and pipelining processes vary greatly
+across projects and organizations, the processes contain commonalities across use cases.
+
+The solution helps you streamline and enforce architecture best practices by providing an extendable
+framework for managing ML pipelines for Amazon Machine Learning (Amazon ML) services and third-party
+services. The solution’s template allows you to upload trained models, configure the orchestration of
+the pipeline, initiate the start of the deployment process, move models through different stages of
+deployment, and monitor the successes and failures of the operations. The solution also provides a
+pipeline for building and registering Docker images for custom algorithms that can be used for model
+deployment on an [Amazon SageMaker](https://aws.amazon.com/sagemaker/) endpoint.
+
+You can use batch and real-time data inferences to configure the pipeline for your business context.
+You can also provision multiple Model Monitor pipelines to periodically monitor the quality of deployed
+Amazon SageMaker ML models. This solution increases your team’s agility and efficiency by allowing them
+to repeat successful processes at scale.
+
+#### Benefits
+
+- **Leverage a pre-configured machine learning pipeline:** Use the solution's reference architecture to initiate a pre-configured pipeline through an API call or a Git repository.
+- **Automatically deploy a trained model and inference endpoint:** Use the solution's framework to automate the model monitor pipeline or the Amazon SageMaker BYOM pipeline. Deliver an inference endpoint with model drift detection packaged as a serverless microservice.
 
 ---
 
 ## Architecture
 
-The AWS CloudFormation template deploys a Pipeline Provisioning framework that
-provisions a machine learning pipeline (Bring Your Own Model for SageMaker). The
-template includes the AWS Lambda functions and AWS Identity and Access Management
-(IAM) roles necessary to set up your account, and it creates an Amazon Simple Storage
-Service (Amazon S3) bucket that contains the CloudFormation templates that set up the
-pipelines.The template also creates an Amazon API Gateway instance, an additional
-Lambda function, and an AWS CodePipeline instance.
-The provisioned pipeline includes four stages: source, build, deploy, and share.
+This solution is built with two primary components: 1) the orchestrator component, created by deploying the solution’s AWS CloudFormation template, and 2) the AWS CodePipeline instance deployed from either calling the solution’s API Gateway, or by committing a configuration file into an AWS CodeCommit repository. The solution’s pipelines are implemented as AWS CloudFormation templates, which allows you to extend the solution and add custom pipelines.
+
+To support multiple use cases and business needs, the solution provides two AWS CloudFormation templates: **option 1** for single account deployment, and **option 2** for multi-account deployment.
+
+### Template option 1: Single account deployment
+
+The solution’s single account architecture allows you to provision ML pipelines in a single AWS account.
+
+![architecture-option-1](source/architecture-option-1.png)
 
-![architecture](source/architecture.png)
+### Template option 2: Multi-account deployment
+
+The solution uses [AWS Organizations](https://aws.amazon.com/organizations/) and [AWS CloudFormation StackSets](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/what-is-cfnstacksets.html) to allow you to provision or update ML pipelines across AWS accounts. Using an administrator account (also referred to as the orchestrator account) allows you to deploy ML pipelines implemented as AWS CloudFormation templates into selected target accounts (for example, development, staging, and production accounts).
+
+![architecture-option-2](source/architecture-option-2.png)
 
 ---
 
@@ -117,7 +127,7 @@ aws s3 cp ./dist/ s3://my-bucket-name-<aws_region>/$SOLUTION_NAME/$VERSION/ --re
 
 ## Known Issues
 
-### Pipeline may fail in custom model container build due to Docker Hub rate limits
+### Image Builder Pipeline may fail due to Docker Hub rate limits
 
 When building custom model container that pulls public docker images from Docker Hub in short time period, you may occasionally face throttling errors with an error message such as:
 ` toomanyrequests You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit`
@@ -126,6 +136,16 @@ This is due to Docker Inc. [limiting the rate at which images are pulled under D
 
 For more information regarding this issue and short-term and long-term fixes, refer to this AWS blog post: [Advice for customers dealing with Docker Hub rate limits, and a Coming Soon announcement](https://aws.amazon.com/blogs/containers/advice-for-customers-dealing-with-docker-hub-rate-limits-and-a-coming-soon-announcement/)
 
+### Model Monitor Blueprint may fail in multi-account deployment option
+
+When using the blueprint for Model Monitor pipeline in multi-account deployment option, the deployment of the stack in the staging ("DeployStaging") account may fail with an error message:
+
+```
+Resource handler returned message: "Error occurred during operation 'CREATE'." (RequestToken:<token-id>, HandlerErrorCode: GeneralServiceException)
+```
+
+Workaround: there is no known workaround for this issue for the multi-account Model Monitor blueprint.
+
 ---
 
 Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.

deployment/build-s3-dist.sh

@@ -28,7 +28,7 @@
 set -e
 
 # Important: CDK global version number
-cdk_version=1.83.0
+cdk_version=1.96.0
 
 # Check to see if the required parameters have been provided:
 if [ -z "$1" ] || [ -z "$2" ] || [ -z "$3" ]; then
@@ -86,6 +86,10 @@ pip install -r ./lambdas/solution_helper/requirements.txt -t ./lambdas/solution_
 echo "pip install -r ./lib/blueprints/byom/lambdas/sagemaker_layer/requirements.txt -t ./lib/blueprints/byom/lambdas/sagemaker_layer/python/"
 pip install -r ./lib/blueprints/byom/lambdas/sagemaker_layer/requirements.txt -t ./lib/blueprints/byom/lambdas/sagemaker_layer/python/
 
+# setup crhelper for invoke lambda custom resource
+echo "pip install -r ./lib/blueprints/byom/lambdas/invoke_lambda_custom_resource/requirements.txt -t ./lib/blueprints/byom/lambdas/invoke_lambda_custom_resource/"
+pip install -r ./lib/blueprints/byom/lambdas/invoke_lambda_custom_resource/requirements.txt -t ./lib/blueprints/byom/lambdas/invoke_lambda_custom_resource/
+
 echo "------------------------------------------------------------------------------"
 echo "[Init] Install dependencies for the cdk-solution-helper"
 echo "------------------------------------------------------------------------------"
@@ -106,32 +110,39 @@ echo "npm install -g aws-cdk@$cdk_version"
 npm install -g aws-cdk@$cdk_version
 
 #Run 'cdk synth for BYOM blueprints
-echo "cdk synth BYOMRealtimeBuiltinStack > lib/blueprints/byom/byom_realtime_builtin_container.yaml"
-cdk synth BYOMRealtimeBuiltinStack > lib/blueprints/byom/byom_realtime_builtin_container.yaml
-echo "cdk synth BYOMRealtimeBuildStack > lib/blueprints/byom/byom_realtime_build_container.yaml"
-cdk synth BYOMRealtimeBuildStack > lib/blueprints/byom/byom_realtime_build_container.yaml
-echo "cdk synth BYOMBatchBuiltinStack > lib/blueprints/byom/byom_batch_builtin_container.yaml"
-cdk synth BYOMBatchBuiltinStack > lib/blueprints/byom/byom_batch_builtin_container.yaml
-echo "cdk synth BYOMBatchBuildStack > lib/blueprints/byom/byom_batch_build_container.yaml"
-cdk synth BYOMBatchBuildStack > lib/blueprints/byom/byom_batch_build_container.yaml
-echo "cdk synth ModelMonitorStack > lib/blueprints/byom/model_monitor.yaml"
-cdk synth ModelMonitorStack > lib/blueprints/byom/model_monitor.yaml
+echo "cdk synth ModelMonitorStack > lib/blueprints/byom/byom_model_monitor.yaml"
+cdk synth ModelMonitorStack > lib/blueprints/byom/byom_model_monitor.yaml
+echo "cdk synth SingleAccountCodePipelineStack > lib/blueprints/byom/single_account_codepipeline.yaml"
+cdk synth SingleAccountCodePipelineStack > lib/blueprints/byom/single_account_codepipeline.yaml
+echo "cdk synth MultiAccountCodePipelineStack > lib/blueprints/byom/multi_account_codepipeline.yaml"
+cdk synth MultiAccountCodePipelineStack > lib/blueprints/byom/multi_account_codepipeline.yaml
+echo "cdk synth BYOMRealtimePipelineStack > lib/blueprints/byom/byom_realtime_inference_pipeline.yaml"
+cdk synth BYOMRealtimePipelineStack > lib/blueprints/byom/byom_realtime_inference_pipeline.yaml
+echo "cdk synth BYOMCustomAlgorithmImageBuilderStack > lib/blueprints/byom/byom_custom_algorithm_image_builder.yaml"
+cdk synth BYOMCustomAlgorithmImageBuilderStack > lib/blueprints/byom/byom_custom_algorithm_image_builder.yaml
+echo "cdk synth BYOMBatchStack > lib/blueprints/byom/byom_batch_pipeline.yaml"
+cdk synth BYOMBatchStack > lib/blueprints/byom/byom_batch_pipeline.yaml
+
 # Replace %%VERSION%% in other templates
 replace="s/%%VERSION%%/$3/g"
-echo "sed -i -e $replace lib/blueprints/byom/byom_realtime_builtin_container.yaml"
-sed -i -e $replace lib/blueprints/byom/byom_realtime_builtin_container.yaml
-echo "sed -i -e $replace lib/blueprints/byom/byom_realtime_build_container.yaml"
-sed -i -e $replace lib/blueprints/byom/byom_realtime_build_container.yaml
-echo "sed -i -e $replace lib/blueprints/byom/byom_batch_builtin_container.yaml"
-sed -i -e $replace lib/blueprints/byom/byom_batch_builtin_container.yaml
-echo "sed -i -e $replace lib/blueprints/byom/byom_batch_build_container.yaml"
-sed -i -e $replace lib/blueprints/byom/byom_batch_build_container.yaml
-echo "sed -i -e $replace lib/blueprints/byom/model_monitor.yaml"
-sed -i -e $replace lib/blueprints/byom/model_monitor.yaml
-
-# Run 'cdk synth' for main template to generate raw solution outputs
-echo "cdk synth aws-mlops-framework --output=$staging_dist_dir"
-cdk synth aws-mlops-framework --output=$staging_dist_dir
+echo "sed -i -e $replace lib/blueprints/byom/byom_model_monitor.yaml"
+sed -i -e $replace lib/blueprints/byom/byom_model_monitor.yaml
+echo "sed -i -e $replace lib/blueprints/byom/byom_realtime_inference_pipeline.yaml"
+sed -i -e $replace lib/blueprints/byom/byom_realtime_inference_pipeline.yaml
+echo "sed -i -e $replace lib/blueprints/byom/single_account_codepipeline.yaml"
+sed -i -e $replace lib/blueprints/byom/single_account_codepipeline.yaml
+echo "sed -i -e $replace lib/blueprints/byom/multi_account_codepipeline.yaml"
+sed -i -e $replace lib/blueprints/byom/multi_account_codepipeline.yaml
+echo "sed -i -e $replace lib/blueprints/byom/byom_custom_algorithm_image_builder.yaml"
+sed -i -e $replace lib/blueprints/byom/byom_custom_algorithm_image_builder.yaml
+echo "sed -i -e $replace lib/blueprints/byom/byom_batch_pipeline.yaml"
+sed -i -e $replace lib/blueprints/byom/byom_batch_pipeline.yaml
+
+# Run 'cdk synth' for main templates to generate raw solution outputs
+echo "cdk synth aws-mlops-single-account-framework --output=$staging_dist_dir"
+cdk synth aws-mlops-single-account-framework --output=$staging_dist_dir
+echo "cdk synth aws-mlops-multi-account-framework --output=$staging_dist_dir"
+cdk synth aws-mlops-multi-account-framework --output=$staging_dist_dir
 
 # Remove unnecessary output files
 echo "cd $staging_dist_dir"
@@ -171,14 +182,20 @@ cd $template_dist_dir
 echo "Updating code source bucket in template with $1"
 replace="s/%%BUCKET_NAME%%/$1/g"
 
-echo "sed -i -e $replace $template_dist_dir/aws-mlops-framework.template"
-sed -i -e $replace $template_dist_dir/aws-mlops-framework.template
+echo "sed -i -e $replace $template_dist_dir/aws-mlops-single-account-framework.template"
+sed -i -e $replace $template_dist_dir/aws-mlops-single-account-framework.template
+echo "sed -i -e $replace $template_dist_dir/aws-mlops-multi-account-framework.template"
+sed -i -e $replace $template_dist_dir/aws-mlops-multi-account-framework.template
 replace="s/%%SOLUTION_NAME%%/$2/g"
-echo "sed -i -e $replace $template_dist_dir/aws-mlops-framework.template"
-sed -i -e $replace $template_dist_dir/aws-mlops-framework.template
+echo "sed -i -e $replace $template_dist_dir/aws-mlops-single-account-framework"
+sed -i -e $replace $template_dist_dir/aws-mlops-single-account-framework.template
+echo "sed -i -e $replace $template_dist_dir/aws-mlops-multi-account-framework.template"
+sed -i -e $replace $template_dist_dir/aws-mlops-multi-account-framework.template
 replace="s/%%VERSION%%/$3/g"
-echo "sed -i -e $replace $template_dist_dir/aws-mlops-framework.template"
-sed -i -e $replace $template_dist_dir/aws-mlops-framework.template
+echo "sed -i -e $replace $template_dist_dir/aws-mlops-single-account-framework.template"
+sed -i -e $replace $template_dist_dir/aws-mlops-single-account-framework.template
+echo "sed -i -e $replace $template_dist_dir/aws-mlops-multi-account-framework.template"
+sed -i -e $replace $template_dist_dir/aws-mlops-multi-account-framework.template
 
 
 echo "------------------------------------------------------------------------------"