diff --git a/docs/ecs_guide.md b/docs/ecs_guide.md index 863fb370..33bcfccb 100644 --- a/docs/ecs_guide.md +++ b/docs/ecs_guide.md @@ -11,11 +11,9 @@ ECS (Elastic Container Service) tasks are a good option for executing Prefect 2 ## ECS Flow Run Execution -To set up Prefect flow execution in your cloud infrastructure, you need to understand two key Prefect concepts: - -- a [worker](https://docs.prefect.io/latest/concepts/work-pools/#worker-overview) to poll Prefect's API (specifically that worker's [work queue](https://docs.prefect.io/latest/concepts/work-pools/#work-queues)) -- and the emphemeral [infrastructure](https://docs.prefect.io/2.10.11/concepts/infrastructure/) that this worker will spin up for each scheduled [flow run](https://docs.prefect.io/2.10.11/concepts/flows/#flow-runs). +Prefect enables remote flow execution via [workers](https://docs.prefect.io/2.11.1/concepts/work-pools/#worker-overview) and [work pools](https://docs.prefect.io/2.11.1/concepts/work-pools/#work-pool-overview). To learn more about these concepts please see our [deployment tutorial](https://docs.prefect.io/2.11.1/tutorial/deployments/). +For details on how workers and work pools are implemented for ECS, see the diagram below: #### Architecture Diagram ```mermaid graph TB @@ -84,7 +82,7 @@ Before you begin, make sure you have: - An AWS account with permissions to create ECS services and IAM roles. - The AWS CLI installed on your local machine. You can [download it from the AWS website](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). -- An [ECS Cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html) to host both the worker and the flow runs it submits. +- An [ECS Cluster](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/clusters.html) to host both the worker and the flow runs it submits. Follow [this guide](https://docs.aws.amazon.com/AmazonECS/latest/userguide/create_cluster.html) to create an ECS cluster or simply use the default cluster. - A [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) configured for your ECS tasks. A VPC is a good idea if using EC2 and required if using Fargate. ### Step 1: Set Up an ECS work pool @@ -108,9 +106,8 @@ Configuring custom fields is easiest from the UI. You need to have a VPC specified for your work pool if you are using AWS Fargate. ![Launch](img/LaunchType_UI.png) -Pictures of UI Cropped -Next, set up an Prefect ECS worker that will discover and pull work from this work pool. +Next, set up a Prefect ECS worker that will discover and pull work from this work pool. ### Step 2: Start a Prefect worker in your ECS cluster. @@ -165,84 +162,105 @@ To create an [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_role Now, you have a role named `ecsTaskExecutionRole` that you can assign to your ECS tasks. This role has the necessary permissions to pull container images and publish logs to CloudWatch. -4. **Create a task definition** - -Next, create an ECS task definition that specifies the Docker image for the Prefect worker, the resources it requires, and the command it should run. In this example, the command to start the worker is `prefect worker start --pool my-ecs-pool`. +4. **Launch an ECS Service to host the worker** -Here are the steps: + Next, create an ECS task definition that specifies the Docker image for the Prefect worker, the resources it requires, and the command it should run. In this example, the command to start the worker is `prefect worker start --pool my-ecs-pool`. -1. Create a JSON file with the following contents: + **Create a JSON file with the following contents:** -```json -{ - "family": "prefect-worker-task", - "networkMode": "awsvpc", - "taskRoleArn": "", - "executionRoleArn": "", - "containerDefinitions": [ + ```json { - "name": "prefect-worker", - "image": "prefecthq/prefect:2-latest", - "cpu": 512, - "memory": 1024, - "essential": true, - "command": [ - "pip", - "install", - "prefect-aws", - "&&", - "prefect", - "worker", - "start", - "--pool", - "my-ecs-pool" - ], - "environment": [ - { - "name": "PREFECT_API_URL", - "value": "https://api.prefect.cloud/api/accounts//workspaces/" - }, - { - "name": "PREFECT_API_KEY", - "value": "" - } - ] + "family": "prefect-worker-task", + "networkMode": "awsvpc", + "requiresCompatibilities": [ + "FARGATE" + ], + "cpu": "512", + "memory": "1024", + "executionRoleArn": "", + "taskRoleArn": "", + "containerDefinitions": [ + { + "name": "prefect-worker", + "image": "prefecthq/prefect", + "cpu": 512, + "memory": 1024, + "essential": true, + "command": [ + "pip", + "install", + "prefect-aws", + "&&", + "prefect", + "worker", + "start", + "--pool", + "my-ecs-pool", + "--type", + "ecs" + ], + "environment": [ + { + "name": "PREFECT_API_URL", + "value": "https://api.prefect.cloud/api/accounts//workspaces/" + }, + { + "name": "PREFECT_API_KEY", + "value": "" + } + ] + } + ] } - ] -} -``` + ``` -- Use `prefect config view` to view the `PREFECT_API_URL` for your current Prefect profile. + - Use `prefect config view` to view the `PREFECT_API_URL` for your current Prefect profile. Use this to replace both `` and ``. -- For the `PREFECT_API_KEY`, individuals on the organization tier can create a [service account](https://docs.prefect.io/latest/cloud/users/service-accounts/) for the worker. If on a personal tier, you can pass a user’s API key. + - For the `PREFECT_API_KEY`, individuals on the organization tier can create a [service account](https://docs.prefect.io/latest/cloud/users/service-accounts/) for the worker. If on a personal tier, you can pass a user’s API key. -- Replace `` with the ARN of the IAM role you created in Step 1, and `` with the URI of the Docker image you pushed to Amazon ECR. + - Replace `` with the ARN of the IAM role you created in Step 1. -- Notice that the CPU and Memory allocations are relatively small. The worker's main responsibility is to submit work, _not_ to execute your Prefect flow code. + - Notice that the CPU and Memory allocations are relatively small. The worker's main responsibility is to submit work through API calls to AWS, _not_ to execute your Prefect flow code. -!!! tip - To avoid hardcoding your API key into the task definition JSON see [how to add environment variables to the container definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html#secrets-envvar-secrets-manager-update-container-definition). + !!! tip + To avoid hardcoding your API key into the task definition JSON see [how to add environment variables to the container definition](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/secrets-envvar-secrets-manager.html#secrets-envvar-secrets-manager-update-container-definition). The API key must be stored as plain text, not the key-value pair dictionary that it is formatted in by default. + +6. **Register the task definition:** + + Before creating a service, you first need to register a task definition. You can do that using the `register-task-definition` command in the AWS CLI. Here is an example: + + ```bash -### Step 3: Create an ECS service to host your worker + aws ecs register-task-definition --cli-input-json file://task-definition.json + ``` -Finally, create a service that will manage your Prefect worker: -Open a terminal window and run the following command to create an ECS Fargate service: -```bash -aws ecs create-service \ - --service-name prefect-worker-service \ - --cluster \ - --task-definition file://.json \ - --launch-type FARGATE \ - --desired-count 1 \ - --network-configuration "awsvpcConfiguration={subnets=[],securityGroups=[]}" -``` + Replace `task-definition.json` with the name of your JSON file. -Replace `` with the name of your ECS cluster, `` with the path to the JSON file you created in Step 2, `` with a comma-separated list of your VPC subnet IDs, and `` with a comma-separated list of your VPC security group IDs. +7. **Create an ECS service to host your worker:** -!!! tip "Sanity check" - The work pool page in the Prefect UI allows you to check the health of your workers - make sure your new worker is live! + Finally, create a service that will manage your Prefect worker: + + Open a terminal window and run the following command to create an ECS Fargate service: + + ```bash + aws ecs create-service \ + --service-name prefect-worker-service \ + --cluster \ + --task-definition \ + --launch-type FARGATE \ + --desired-count 1 \ + --network-configuration "awsvpcConfiguration={subnets=[],securityGroups=[]}" + ``` + + - Replace `` with the name of your ECS cluster. + - Replace `` with the path to the JSON file you created in Step 2, `` with a comma-separated list of your VPC subnet IDs. Ensure that these subnets are aligned with the vpc specified on the work pool in step 1. + - Replace `` with a comma-separated list of your VPC security group IDs. + - Replace `` with the ARN of the task definition you just registered. + + !!! tip "Sanity check" + The work pool page in the Prefect UI allows you to check the health of your workers - make sure your new worker is live! ### Step 4: Pick up a flow run with your new worker! @@ -262,44 +280,13 @@ Replace `` with the name of your ECS cluster, `` with the name of your ECS cluster, `=0.2.0 - image_name: my-repo/my-image - tag: my-tag - dockerfile: auto - push: true + - prefect.deployments.steps.run_shell_script: + id: get-commit-hash + script: git rev-parse --short HEAD + stream_output: false + - prefect.deployments.steps.run_shell_script: + id: ecr-auth-step + script: aws ecr get-login-password --region | docker login --username + AWS --password-stdin <>.dkr.ecr..amazonaws.com + stream_output: false + - prefect_docker.deployments.steps.build_docker_image: + requires: prefect-docker>=0.3.0 + image_name: .dkr.ecr.us-east-2.amazonaws.com/ + tag: '{{ get-commit-hash.stdout }}' + dockerfile: auto + push: true ```