Skip to content

Latest commit

 

History

History
188 lines (131 loc) · 7.92 KB

README.md

File metadata and controls

188 lines (131 loc) · 7.92 KB
Raw

HUAWEI CLOUD OBS-FGS-CCI

This repository contains a minimal reproducible example for asynchronous Object Storage Service (OBS) file processing with FunctionGraph (FGS) and Cloud Container Instance (CCI) services from HUAWEI CLOUD.

Architecture and process

OBS-FGS-CCI architecture

  1. File is saved to OBS bucket in input_folder;
  2. The ObjectCreated event triggers the FunctionGraph function;
  3. The function uses the CCI API to start a Job, passing the input file path and other data through environment variables, and then stops running;
  4. CCI pulls the container image from SoftWare Repository for Container (SWR);
  5. The CCI Job/container downloads the input file and processes it;
  6. The CCI Job/container saves the output file back to OBS and then stops running.

Preparation

Identity and Access Management (IAM)

Create IAM User and download access key

Create an IAM User to be used exclusively for this pipeline. For Access Type, select only "Programatic Access". Do not assign to any user group.

In the "Download Access Key" message, click "OK" to download credentials.csv file, which contains the AK/SK for this IAM User. This file will have the following content:

User Name,Access Key Id,Secret Access Key
username,1FAF***7UUO,IvRd******rm8t

On the second line, the second value is the Access Key (AK), and the third value is the Secret Access Key (SK). We will use them later.

Assign permissions to IAM User

Assign permissions to the IAM User created. In the IAM User list, click on "Authorize" in the Operation column.

In "Authorization Method", click on "Select permissions", search for "CCI CommonOperations" and select it.

Object Storage Service (OBS)

Create the OBS bucket

If you don't need to mount the OBS bucket in an ECS, for example, Create an OBS bucket in the desired region. Select an unique bucket name, use the "Standard" storage class, and use "Private" bucket policy.

If you need to mount the OBS bucket in an ECS and use it as a file system, for example, Create a Parallel File System (a special type of OBS bucket) in the desired region. Select an unique file system name and use "Private" policy.

Create input folder

In the OBS bucket console, create a folder named "input_folder". Files uploaded in this folder will trigger the FunctionGraph function.

SoftWare Repository for Containers (SWR)

SWR is used to host container images on Huawei Cloud. Create an organization in the desired region to host the image that will be used in CCI.

Container image and CCI

With Docker installed in your machine, open the terminal in the cci_image directory and build the container image, using the SWR path (with your organization name) as tag:

docker build -t swr.{region}.myhuaweicloud.com/{SWR organization}/{image name}:{image version}

Go to the SWR Console and get a temporary login command by clicking on "Generate Login Command". Copy the command and execute in your terminal.

docker login -u {region}@0M***5Q -p 46***1d swr.{region}.myhuaweicloud.com

Then, push your image to SWR:

docker push swr.{region}.myhuaweicloud.com/{SWR organization}/{image name}:{image version}

Finally, create a Namespace in CCI. This namespace will be used to create the Job using FunctionGraph.

FunctionGraph

Make a zip file containing the three files in the functiongraph folder (do not compress the whole folder).

Create a Event Function from scratch, selecting"Python 3.10" as Runtime.

In the Code view, click on "Upload" on the top right corner, select "Local ZIP" and upload the zip file with the code.

FunctionGraph code

Create an OBS Trigger for the function, with the following configuration:

  • Trigger Type: Object Storage Service (OBS)
  • Bucket Name: name of OBS bucket
  • Events: ObjectCreated
  • Event Notification Name: choose an unique name or use the default one
  • Prefix: input_folder/ (created previously)
  • Suffix: .txt (only .txt files will be processed in our example)

OBS trigger in FunctionGraph

Configure the following environment variables:

  • ak: Access Key (AK) from credentials.csv
  • sk: Secret Access Key (SK) from credentials.csv
  • obs_endpoint: OBS endpoint (example: obs.sa-brazil-1.myhuaweicloud.com)
  • output_folder: where output file will be written (example: output_folder)
  • job_name: base name CCI job (example: cci-job), which will have the current timestamp appended to make the final CCI job name
  • cci_namespace: which namespace will be used to create the CCI job
  • cci_endpoint: CCI endpoint (example: cci.sa-brazil-1.myhuaweicloud.com)
  • swr_image: path to SWR image (example: swr.sa-brazil-1.myhuaweicloud.com/myorg/myimage:0.1.0)

FunctionGraph environment variables

Test

Create a simple text file, save with .txt extension, and upload to the input_folder in the OBS bucket.

OBS input file

In the FunctionGraph Console, Monitoring tab, Logs view, click in the Request ID to see the execution logs.

FunctionGraph Request logs

In the CCI Console, in the Workloads > Jobs page, check that the Job was created and executed successfully.

CCI job executing

Click on the Job name. After some seconds, check that the Execution completed successfully.

CCI job completed

In the Pod list, click in the "View Logs" operation to check the container output, saved by Application Operations Management (AOM) service.

Container logs (AOM)

Check the output folder of OBS and see that the output file was saved successfully.

OBS output file

References