Update to version v2.0.0 (#186)

* Update to version v2.0.0 * fix: fix the gh-page workflow
aws-solutions · Aug 23, 2023 · 82cc883 · 82cc883
1 parent d1efa83
commit 82cc883
Show file tree

Hide file tree

Showing 807 changed files with 66,934 additions and 43,265 deletions.
diff --git a/.github/workflows/gh-page.yml b/.github/workflows/gh-page.yml
@@ -17,14 +17,14 @@ jobs:
       - name: Setup Python
         uses: actions/setup-python@v2
         with:
-          python-version: '3.8'
+          python-version: '3.9'
 
       - name: Install dependencies
         run: |
-          python3 -m pip install mkdocs            # install mkdocs
-          python3 -m pip install mkdocs-material   # install material theme
-          python3 -m pip install mkdocs-macros-plugin  # install macros plugin
-          python3 -m pip install mkdocs-include-markdown-plugin # install include-markdown
+          python3 -m pip install mkdocs==1.3.1            # install mkdocs
+          python3 -m pip install mkdocs-material==8.5.3   # install material theme
+          python3 -m pip install mkdocs-macros-plugin==0.7.0  # install macros plugin
+          python3 -m pip install mkdocs-include-markdown-plugin==3.8.1 # install include-markdown
 
       - name: Build mkdocs
         run: |

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,21 +1,78 @@
 # Change Log
+
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [1.0.3] - 2023-06-28
+## [2.0.0] - 2023-08-22
+
+### Added
+
+- Log ingestion from S3 bucket to support more log sources #89
+- Show logs and metrics of the log analytics pipelines #112
+- Quickly enable alarms on log ingestion pipeline #113
+- Show the AWS resource changes when importing AOS using automatic networking mode #53
+- Log Agent Installation: Support of agent installation on AL2023 instances #88
+- Support of multi-AZ standby enabled OpenSearch cluster when creating log pipelines #170
+- Instance Group: Show error message on the console when the installation of log agent fails #169
+- Support same index name in different OpenSearch clusters #166
+- Installation: refresh the aws-exports.json once update the CloudFormation input parameters #161
+- Application log pipeline: Add a step to choose Log Config during the application pipeline creation steps #159
+- Log Agent: Auto rotation of Fluent Bit log file #158
+- Instance group: Add an option to attach IAM policies to Instance Group managed EC2 instances automatically #151
+- Domain management: Check the prerequisites of OpenSearch clusters before import OpenSearch clusters #148
+- Support ingest WAF (associate with CloudFront) sampled logs to OpenSearch in other regions except us-east-1 #129
+
+### Fixed
+
+- Log Config: Time key in Fluent Bit config for Spring Boot should be time type instead of None #71
+- EventBridge will be disabled automatically if deleting instances in instance group #164
+- Log Config should not be created without Regex/Log Format #163
+- Lack of region check before creating WAF log pipeline #162
+- The Fluent bit configuration file generated in sidecar deployment option has a wrong shared volume #160
+- S3 access log dashboard: 5xx Code description is covered #157
+- S3 access log dashboard: The Average Time Unit should be milliseconds #155
+- Cross-account: Unable to get instance list and create instance group in CN region #156
+- The OpenSearch information (e.g., version, data nodes) is not updated automatically after customer upgraded the cluster #150
+- Cannot differentiate the Lambda for different AWS Service log pipeline based on Lambda description #146
+- Fix data lost when cannot find the location with IP address using MaxMind database #126
+- Syslog: Fix port conflict when adding & deleting new log source in parallel #174
+
+### Changed
+
+- Minimize the permissions of EC2 log ingestion IAM role #154
+- Minimize the privileges of cross-account access role #153
+- Soft delete when removing OpenSearch domain #152
+- Save ALB access logs of Nginx based proxy to S3 bucket #149
+- Code refactor: DynamoDB design optimization and GraphQL API design optimization #147
+- Minimize security group egress of the provisioned ECS #145
+- WAF dashboard: Cannot filter results using `nonTerminatingMatchingRules.action` field #144
+
+### Removed
+
+- Domain management: Remove the support of Elasticsearch engine #176
+
+## [1.0.3] - 2023-06-27
+
 ### Fixed
-- Fix the processor Lambda function urllib3 version issue
+
+- Fix the processor Lambda function urllib3 version issue #138
 
 ## [1.0.2] - 2023-06-21
+
 ### Fixed
-- Fix the EKS Fluent-Bit deployment configuration generation issue
+
+- Support generation of Kubernetes YAML configuration file for EKS 1.24~1.27 #133
 
 ## [1.0.1] - 2023-04-17
+
 ### Fixed
+
 - Fix deployment failure due to S3 ACL changes
 
 ## [1.0.0] - 2023-03-16
+
 ### Added
+
 - All files, initial version
diff --git a/NOTICE.txt b/NOTICE.txt
@@ -11,11 +11,116 @@ THIRD PARTY COMPONENTS
 **********************
 This software includes third party software subject to the following copyrights:
 
-AWS SDK under the Apache License Version 2.0
-user-agents under the Apache License Version 2.0
-maxminddb under the Apache License Version 2.0
-requests-aws4auth under the Apache License Version 2.0
-urllib3 under the MIT License
-requests under the Apache License Version 2.0
+awscli under the Apache License Version 2.0
 boto3 under the Apache License Version 2.0
-botocore under the Apache License Version 2.0
+botocore under the Apache License Version 2.0
+defusedxml under the Python Software Foundation License
+py-serializable under the Apache License Version 2.0
+pathable under the Apache License Version 2.0 (details in this link: https://github.com/p1c2u/pathable/blob/master/LICENSE)
+Jinja2 under the BSD-3-Clause
+MarkupSafe under the BSD-3-Clause
+Werkzeug under the BSD-3-Clause
+attrs under the MIT License
+cffi under the MIT License
+colorama under the BSD License
+coverage under the Apache License Version 2.0
+cryptography under the Apache License Version 2.0 or OR BSD-3-Clause
+docker under the Apache License Version 2.0
+docutils under BSD License, GNU General Public License (GPL), Python Software Foundation License, Public Domain (public domain, Python, BSD-2-Clause, GPL 3(see https://docutils.sourceforge.io/COPYING.txt))
+flake8 under the MIT License
+iniconfig under the MIT License
+ipaddr under the Apache License Version 2.0
+jmespath under the MIT License
+jsonschema-spec under the Apache License Version 2.0
+lazy-object-proxy under the BSD-2-Clause
+maxminddb under the Apache License Version 2.0
+mccabe under the MIT License
+moto under the Apache License Version 2.0 
+openapi-schema-validator under the BSD-3-Clause
+pluggy under the MIT License
+pyOpenSSL under the Apache License Version 2.0
+pyasn1 under the BSD-2-Clause
+pycodestyle under the MIT License
+pycparser under the BSD License
+pydantic under the MIT License
+pyflakes under the MIT License
+pyrsistent under the MIT License
+pytest under the MIT License
+pytest-cov under the MIT License 
+pytest-mock under the MIT License
+python-dateutil under Apache Software License, BSD License (Dual License)
+requests-aws4auth under the MIT License
+requests-mock under the Apache License Version 2.0 
+responses under the Apache License Version 2.0
+rfc3339-validator under the MIT License
+rsa under the Apache License Version 2.0
+s3transfer under the Apache License Version 2.0 
+types-PyYAML under the Apache License Version 2.0
+typing_extensions under Python Software Foundation License
+ua-parser under the Apache License Version 2.0
+user-agents under the MIT License
+websocket-client under the Apache License Version 2.0 
+xmltodict under the MIT License
+@aws-cdk/aws-appsync-alpha under the Apache License Version 2.0
+@aws-cdk/aws-kinesisfirehose-alpha under the Apache License Version 2.0
+@aws-cdk/aws-kinesisfirehose-destinations-alpha under the Apache License Version 2.0
+@aws-solutions-constructs/aws-cloudfront-s3 under the Apache License Version 2.0 
+@typescript-eslint/eslint-plugin under the MIT License
+@typescript-eslint/parser under the BSD-2-Clause
+eslint under the MIT License
+eslint-config-prettier under the MIT License
+eslint-import-resolver-node under the MIT License
+eslint-import-resolver-typescript under the ISC License
+eslint-plugin-import under the MIT License
+eslint-plugin-prettier under the MIT License 
+aws-cdk under the Apache License Version 2.0 
+aws-cdk-lib under the Apache License Version 2.0
+cdk-nag under the Apache License Version 2.0 
+source-map-support under the MIT License
+fs under the MIT License
+exceptiongroup under the MIT License
+tomli under MIT License
+@apollo/client under the MIT License
+@aws-amplify/ui-components under the Apache License Version 2.0
+@aws-amplify/ui-react under the Apache License Version 2.0
+@material-ui/core under the MIT License
+@material-ui/icons under the MIT license
+@material-ui/lab under the MIT License
+@testing-library/jest-dom under the MIT License
+@testing-library/react under the MIT License
+@testing-library/user-event under the MIT License
+@types/jest under the MIT License
+@types/node under the MIT License
+@types/react under the MIT License
+@types/react-dom under the MIT License
+amplify under the MIT License
+apexcharts under the MIT License
+apollo-link under the MIT License
+aws-amplify under the Apache License Version 2.0
+aws-appsync-auth-link under the Apache License Version 2.0
+aws-appsync-subscription-link under the Apache License Version 2.0
+axios under the MIT License
+classnames under the MIT License
+date-fns under the MIT License
+graphql-tag under the MIT License
+i18next under the MIT License
+i18next-browser-languagedetector under the MIT License
+i18next-http-backend under the MIT License
+lodash.clonedeep under the MIT License
+moment under the MIT License
+node-sass under the Apache License Version 2.0
+oidc-client-ts under the Apache License Version 2.0
+react under the MIT License
+react-apexcharts under the MIT License 
+react-copy-to-clipboard under the MIT License
+react-dom under the MIT License
+react-i18next under the MIT License 
+react-minimal-datetime-range under the MIT License
+react-oidc-context under the MIT License
+react-redux under the MIT License
+react-router-dom under the MIT License
+redux under the MIT License
+sweetalert2 under the MIT License
+typescript under the Apache License Version 2.0
+web-vitals under the Apache License Version 2.0
+notice-js under the MIT-0
diff --git a/README.md b/README.md
@@ -4,11 +4,13 @@ The Centralized Logging with OpenSearch solution provides comprehensive log mana
 
 ## Table of content
 
-- [Solution Overview](#solution-overview)
-- [Architecture](#architecture)
-- [Deployment](#deployment)
-- [Customization](#customization)
-- [License](#License)
+- [Centralized Logging with OpenSearch](#centralized-logging-with-opensearch)
+  - [Table of content](#table-of-content)
+  - [Solution Overview](#solution-overview)
+  - [Architecture](#architecture)
+  - [Deployment](#deployment)
+  - [Customization](#customization)
+  - [Collection of operational metrics](#collection-of-operational-metrics)
 
 
 ## Solution Overview
@@ -17,9 +19,9 @@ The solution has the following features:
 
 - **All-in-one log ingestion**: provides a single web console to ingest both application logs and AWS service logs into the Amazon OpenSearch (AOS) domains.
 
-- **Codeless log processor**: supports log processor plugins developed by AWS. You are allowed to enrich the raw log data through a few clicks on the web console. 
+- **Codeless log processor**: supports log processor plugins developed by AWS. You are allowed to enrich the raw log data through a few clicks on the web console.
 
-- **Out-of-box dashboard template**: offers a collection of reference designs of visualization templates, for both commonly used software such as Nginx and Apache HTTP Server, and AWS services such as Amazon S3 and Amazon CloudTrail. 
+- **Out-of-box dashboard template**: offers a collection of reference designs of visualization templates, for both commonly used software such as Nginx and Apache HTTP Server, and AWS services such as Amazon S3 and Amazon CloudTrail.
 
 
 
@@ -40,7 +42,11 @@ Please follow the [Implementation Guide](https://docs.aws.amazon.com/solutions/l
 Please follow the [Customization Guide](CUSTOM_BUILD.md) for custom build.
 
 
-## License
+## Collection of operational metrics
+
+This solution collects anonymous operational metrics to help AWS improve the quality and features of the solution. For more information, including how to disable this capability, please see the [implementation guide](https://docs.aws.amazon.com/solutions/latest/centralized-logging-with-opensearch/collection-of-operational-metrics.html).
+
+***
 
 Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 

diff --git a/deployment/build-s3-dist.sh b/deployment/build-s3-dist.sh
@@ -276,31 +276,6 @@ echo "${bold}[Create] Templates${normal}"
 echo "------------------------------------------------------------------------------"
 
 if fn_exists create_template_${template_format}; then
-    rm -vf ./lambda/api/app_pipeline/common.py
-    rm -vf ./lambda/api/app_log_ingestion/common.py
-    rm -vf ./lambda/api/app_log_ingestion/util/aws_svc_mgr.py
-    rm -vf ./lambda/api/app_log_ingestion/aws_svc_mgr.py
-    rm -vf ./lambda/api/pipeline/aws_svc_mgr.py
-    rm -vf ./lambda/api/log_agent_status/aws_svc_mgr.py
-    rm -vf ./lambda/api/instance_meta/aws_svc_mgr.py
-    rm -vf ./lambda/api/instance_group/aws_svc_mgr.py
-    rm -vf ./lambda/api/resource/aws_svc_mgr.py
-    rm -vf ./lambda/api/eks_cluster/aws_svc_mgr.py
-    rm -vf ./lambda/main/cfnHelper/aws_svc_mgr.py
-
-    cp -vf ./lambda/pipeline/service/log-processor/../../common/custom-resource/boto3_client.py ./lambda/pipeline/service/log-processor/boto3_client.py
-    cp -vf ./lambda/api/app_pipeline/../common/common.py ./lambda/api/app_pipeline/common.py
-    cp -vf ./lambda/api/app_log_ingestion/../common/common.py ./lambda/api/app_log_ingestion/common.py
-    cp -vf ./lambda/api/pipeline/../common/aws_svc_mgr.py ./lambda/api/pipeline/aws_svc_mgr.py
-    cp -vf ./lambda/api/instance_group/../common/aws_svc_mgr.py ./lambda/api/instance_group/aws_svc_mgr.py
-    cp -vf ./lambda/api/app_log_ingestion/../common/aws_svc_mgr.py ./lambda/api/app_log_ingestion/aws_svc_mgr.py
-    cp -vf ./lambda/api/log_agent_status/../common/aws_svc_mgr.py ./lambda/api/log_agent_status/aws_svc_mgr.py
-    cp -vf ./lambda/api/instance_meta/../common/aws_svc_mgr.py ./lambda/api/instance_meta/aws_svc_mgr.py
-    cp -vf ./lambda/api/resource/../common/aws_svc_mgr.py ./lambda/api/resource/aws_svc_mgr.py
-    cp -vf ./lambda/api/eks_cluster/../common/aws_svc_mgr.py ./lambda/api/eks_cluster/aws_svc_mgr.py
-    cp -vf ./lambda/main/cfnHelper/../../api/common/aws_svc_mgr.py ./lambda/main/cfnHelper/aws_svc_mgr.py
-
-
     create_template_${template_format}
 else
     echo "Invalid setting for \$template_format: $template_format"

diff --git a/deployment/cdk-solution-helper/package.json b/deployment/cdk-solution-helper/package.json
@@ -1,6 +1,8 @@
 {
   "name": "cdk-solution-helper",
+  "description": "cdk solution helper",
   "version": "0.1.0",
+  "license": "Apache-2.0",
   "devDependencies": {
     "fs": "0.0.1-security"
   },

diff --git a/deployment/run-unit-tests.sh b/deployment/run-unit-tests.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+#
+# You can remove this script if you do NOT have unit test.
+#
+# This script should be run from the repo's deployment directory
+# cd deployment
+# ./run-unit-tests.sh
+#
+source_template_dir="$PWD"
+cd $source_template_dir/../source
+./run-all-tests.sh
diff --git a/docs/en/images b/docs/en/images
diff --git a/docs/en/images b/docs/en/images
@@ -0,0 +1 @@
+../images
diff --git a/docs/en/implementation-guide/alarm.md b/docs/en/implementation-guide/alarm.md
@@ -0,0 +1,37 @@
+There are different types of log alarms: log processor alarms, buffer layer alarms, and source alarms (only for application log pipeline). The alarms will be triggered when the defined condition is met.  
+
+| Log alarm type                    | Log alarm condition                                                               | Description                                                                                                                                                         |
+| -------------------------------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Log processor alarms                         | Error invocation # >= 10 for 5 minutes, 1 consecutive time    | When the number of log processor Lambda error calls is greater than or equal to 10 within 5 minutes (including 5 minutes), an email alarm will be triggered.                                                                                                     |
+| Log processor alarms                         | Failed record # >= 1 for 1 minute, 1 consecutive time                                       | When the number of failed records is greater than or equal to 1 within a 1-minute window, an alarm will be triggered.         |
+| Log processor alarms                         | Average execution duration in last 5 minutes >= 60000 milliseconds                                       | In the last 5 minutes, when the average execution time of log processor Lambda is greater than or equal to 60 seconds, an email alarm will be triggered.         |
+| Buffer layer alarms                         | SQS Oldest Message Age >= 30 minutes                                        | When the age of the oldest SQS message is greater than or equal to 30 minutes, it means that the message has not been consumed for at least 30 minutes, an email alarm will be triggered.         |
+| Source alarms (only for application log pipeline)                         | Fluent Bit output_retried_record_total >= 100 for last 5 minutes                                        | When the total number of retry records output by Fluent Bit in the past 5 minutes is greater than or equal to 100, an email alarm will be triggered.        |  
+
+You can choose to enable log alarms or disable them according to your needs.
+
+## Enable log alarms
+
+1. Sign in to the Centralized Logging with OpenSearch console.
+
+2. In the left navigation bar, under **Log Analytics Pipelines**, choose **AWS Service Log** or **Application Log**.
+
+3. Select the log pipeline created and choose **View details**.
+
+4. Select the **Alarm** tab.
+
+5. Switch on **Alarms** if needed and select an exiting SNS topic.
+
+6. If you choose **Create a new SNS topic**, you need to provide email address for the newly-created SNS topic to notify.
+
+## Disable log alarms
+
+1. Sign in to the Centralized Logging with OpenSearch console.
+
+2. In the left navigation bar, under **Log Analytics Pipelines**, choose **AWS Service Log** or **Application Log**.
+
+3. Select the log pipeline created and choose **View details**.
+
+4. Select the **Alarm** tab.
+
+5. Switch off **Alarms**.