Example of CI/CD with saagieapi (Python)

Description

This project contains an example to use github action to deploy jobs inside Saagie Platform using saagieapi.

You can use the following actions to interact with Saagie platform: package_job, update_job, run_job, update_pipeline and run_pipeline. Packaging only available for python package code.

In this repository, we have 4 Python jobs and 2 Bash job. For each python job, we have the source code inside code, and for the bash job, we only have the job configuration file, for example: /saagie/job/model_deployment.json. In fact, we don't need to create a zip for bash job.

Prerequisite

Global prerequisite

• You have an editor access to Saagie Platform
• You have to create at least one project in Saagie Platform (in the example, we have one in dev, another in prod)
• git clone this repo
• Create the following environment variables:
  • SAAGIE_URL
  • SAAGIE_USER
  • SAAGIE_PWD
  • SAAGIE_REALM
• If you have only one environment in Saagie, you can remove the dev directory inside saagie/envs/, and you create a json file with your environment name: your_env_name.json
• In your saagie/envs/your_env_name.json, you have to specify 3 key values:
  • platform_id
  • project_id
  • project_name

Local prerequisite

• Install Python (at least 3.8) and all libraries in requirements.txt

Configuration

If you want to use the CI for your own jobs, make sure that the respect of the following file organisation. In this repository, we have 2 directories:

• code:
  • each directories inside code is a job in Saagie
• saagie:
  • envs:
    • each json file is a configuration file for an environment
  • jobs:
    • each json file is a configuration file for a job, the name of the file should be the same as the name of job directory in code
  • pipelines:
    • each json file is a configuration file for a pipeline

You can also modify github workflows. For now, each push on a branch will trigger a check to see which configuration files have been modified. Based on this, it will trigger actions such as updating a job, or updating a pipeline.

How to use it in local

• To package a job, you can run the following command by replacing your_job_name by the job that you want to package: python __main__.py --action package_job --job_name your_job_name. It suppose that your job code is in code/your_job_name

• To update a job,

  • For windows user, use the following command line by replacing your_job_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action update_job --job_name your_job_name --saagie_url "%SAAGIE_URL%" --saagie_user "%SAAGIE_USER%" --saagie_pwd "%SAAGIE_PWD%" --saagie_realm "%SAAGIE_REALM%" --saagie_env your_env_name

  • For Linux user, use the following command line by replacing your_job_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action update_job --job_name your_job_name --saagie_url "$SAAGIE_URL" --saagie_user "$SAAGIE_USER" --saagie_pwd "$SAAGIE_PWD" --saagie_realm "$SAAGIE_REALM" --saagie_env your_env_name

• To run a job, same thing as update, by using --action run,

  • For windows user, use the following command line by replacing your_job_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action run_job --job_name your_job_name --saagie_url "%SAAGIE_URL%" --saagie_user "%SAAGIE_USER%" --saagie_pwd "%SAAGIE_PWD%" --saagie_realm "%SAAGIE_REALM%" --saagie_env your_env_name

  • For Linux user, use the following command line by replacing your_job_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action run_job --job_name your_job_name --saagie_url "$SAAGIE_URL" --saagie_user "$SAAGIE_USER" --saagie_pwd "$SAAGIE_PWD" --saagie_realm "$SAAGIE_REALM" --saagie_env your_env_name

• To update a pipeline,

  • For windows user, use the following command line by replacing your_pipeline_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action update_pipeline --pipeline_name your_pipeline_name --saagie_url "%SAAGIE_URL%" --saagie_user "%SAAGIE_USER%" --saagie_pwd "%SAAGIE_PWD%" --saagie_realm "%SAAGIE_REALM%" --saagie_env your_env_name

  • For Linux user, use the following command line by replacing your_job_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action update_pipeline --pipeline_name your_pipeline_name --saagie_url "$SAAGIE_URL" --saagie_user "$SAAGIE_USER" --saagie_pwd "$SAAGIE_PWD" --saagie_realm "$SAAGIE_REALM" --saagie_env your_env_name

• To run a pipeline,

  • For windows user, use the following command line by replacing your_pipeline_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action run_pipeline --pipeline_name your_pipeline_name --saagie_url "%SAAGIE_URL%" --saagie_user "%SAAGIE_USER%" --saagie_pwd "%SAAGIE_PWD%" --saagie_realm "%SAAGIE_REALM%" --saagie_env your_env_name

  • For Linux user, use the following command line by replacing your_job_name and dev if you want to use another environment: python cicd_saagie_tool/__main__.py --action run_pipeline --pipeline_name your_pipeline_name --saagie_url "$SAAGIE_URL" --saagie_user "$SAAGIE_USER" --saagie_pwd "$SAAGIE_PWD" --saagie_realm "$SAAGIE_REALM" --saagie_env your_env_name

Environment configuration file

Each json file inside /saagie/envs has following schema:

Environment

An environment inside Saagie

Type: object

Properties

• platform_id required
  • ID of Saagie platform
  • Type: string
  • Example: "1"
• project_id required
  • ID of the Saagie project
  • Type: UUID
  • Example: "5d714745-1fea-4b30-ab5c-a9965a028355"
• project_name required
  • Name of the Saagie project
  • Type: string
  • Example: "My project"

Job configuration file

Each json file inside /saagie/jobs has following schema:

Job

A job inside Saagie

Type: object

Properties

• job_name required
  • Job name
  • Type: string
  • Example: Job extraction
• file_path required
  • File path of the code
  • Type: string
  • Example: ./path/to/code/artefact.zip
• description optional
  • Description for your job. If not filled or not present, defaults to current value
  • Type: string
  • Example: This is my description
• category optional
  • Category of the job, If not filled or not present, defaults to current value
  • Type: string
  • Must be Extraction, Processing or Smart App
• technology optional
  • Technology label of the job. Only required for the creation, and it's not possible to change this value for a job
  • Type: string
  • Example: Python
• technology_catalog optional
  • Technology catalog containing the technology to use for this job. Only required for the creation, and it's not possible to change this value for a job
  • Type: string
  • Example: Saagie
• runtime_version optional
  • Technology version of the job, the ID of the context, If not filled or not present, defaults to current value
  • Type: string
  • Example: 3.10
• command_line optional
  • Command line of the job, If not filled or not present, defaults to current value
  • Type: string
  • Example: python {file}
• extra_technology optional
  • Extra technology when needed (spark jobs). If not needed, leave to empty string or the request will not work
  • Type: string
• extra_technology_version optional
  • Version of the extra technology. Leave to empty string when not needed
  • Type: string

Pipeline

Each file inside /saagie/pipelines has following schema:

A pipeline inside Saagie

Type: object

Properties

• file_path required
  • The path of pipeline artefact
  • Type: string

A pipeline artefact have the following schema, it can be a json or yaml file.

A pipeline inside Saagie Type: object

Properties

• env required

  • Dict that contains all environment where you want to deploy job
  • Type: object
  • environment_name required
    • Group all variables needed by each environment inside one dict
    • Type: object
    • Example: "dev": {"graph_pipeline": {"job_nodes": [], "condition_nodes": []}}
    • Properties:
      • graph_pipeline
        • Graph pipeline
        • Type: object
        • Properties:
          • job_nodes
            • List of job nodes
            • Type: object
            • job:
              • Properties:
                • id
                  • An unique string to identify this job
                  • Type: UUID
                  • Example: 0c83c5ab-2987-45cd-b91b-c63434f49ed7
                • job_name
                  • Name of the job
                  • Type: string
                  • Example: my awesome job 1
                • job_id
                  • Job ID on Saagie's platform
                  • Type: string
                  • Example: 6bb1b3b2-dd21-4d63-9661-ef2ad7308728
                • next_nodes
                  • Next nodes of this job, each value in the list should be an ID of job or condition
                  • Type: List[UUID]
                  • Example: ["7165e9cb-3a05-4b9f-b2ac-cdd7ec29c949"]
          • condition_nodes
            • List of condition nodes
            • Type: object
            • condition:
              • Properties:
                • id
                  • An unique string to identify this condition
                  • Type: string
                  • Example: 7165e9cb-3a05-4b9f-b2ac-cdd7ec29c9498
                • condition_type
                  • Type of the condition, should be status or expression
                  • Type: string
                  • Example: status
                • value
                  • The value of the condition.If it's a condition on status, the value should be AllSuccess or AllSuccessOrSkipped or AtLeastOneSuccess . If it's a condition on expression, make sure the value is a Common Expression Language (CEL) expression
                  • Type: string
                  • Example: AllSuccessOrSkipped
                • next_nodes_success
                  • Next nodes of this job in case of success, each value in the list should be an ID of job or condition
                  • Type: List[string]
                  • Example: ["fca09324-a52e-4121-a9d3-b04f1c03dd28"]
                • next_nodes_failure
                  • Next nodes of this job in case of failure, each value in the list should be an ID of job or condition
                  • Type: List[string]
                  • Example: ["5cc506e4-f5cb-4ed3-810b-d79b8484b857"]

• has_execution_variables_enabled optional

  • Whether to activate the execution variables. If not filled or not present, defaults to current value
  • Type: bool
  • Example: True

• description optional

  • Description for your pipeline. If not filled or not present, defaults to current value
  • Type: string
  • Example: This is my description