Saagie Technologies

This repository contains all certified technologies used in Saagie. It also contains some experimental technologies before being certified.

For information about using these technologies with your Saagie platform, refer to Saagie's official documentation.

CONTRIBUTING

All contributions are made via Github Pull Requests.

Build

The build is using Gradle

Launch the Gradle task localBuildModifiedJobs or localBuildModifiedApps to build locally the technology modified locally:

./gradlew localBuildModifiedJobs
./gradlew localBuildModifiedApps

CI

The build is running using a Github Action workflow (build only modified). It builds only technologies modified and generate a pre release containing assets. The name of the pre release = current version + name of the branch.

Structure

The build is using the Gradle plugin for Saagie technologies repository. It expects a specific directory structure:

• technologies: the required base directory. It contains sub directories for each type of technology metadata: -- app: contains folders of each app technology -- job: contains folders of each job technology -- connectiontype: contains folders of each connection type (deprecated) -- scripts is a sub directory dedicated to external technologies which share the same javascript files (deprecated)

Each metadata folder must then contain a metadata.yaml file. For job and app technology metadata, the build is automatically generating them from the technology.yaml file and each context.yaml found in the sub folders.

NOTE: Generated files (metadata.yaml, js files, etc..) need to be checked in git, since the build is trying to avoid rebuilding technologies which didn't change.

Docker based technologies

For each context of a job (non external, based on an Docker image) or of an app technology, the folder dedicated to the context must then contain:

• build.gradle.kts which apply the SaagieTechnologiesGradlePlugin and the DockerRemoteApiPlugin.
• Dockerfile which declares how to build the Docker image
• image_test.yml: if present, the generated Docker image will be tested with the GoogleContainerTools/container-structure-test

External technologies

For contexts of an external job technology, the context.yaml file will reference javascript files, relatively to its location. The referenced javascript files should be generated by a dedicated javascript build.

The build of such javascript files can be setup anywhere in the directory structure, as long as the relative path is proprely filled in the context.yaml file.

The Saagie Gradle plugin can be used to trigger the build of the javascript file in the CI. It needs:

• a build.gradle.kts which apply the SaagieTechnologiesGradlePlugin and the NodePlugin.
• a package.json with a build and test scripts.

The Saagie Gradle plugin will launch Yarn to install depednencies and run the scripts to build and test the javascript files.

Developping an External Job Technology

It is highly recommended to use the Saagie SDK when developping an external job technology.

In technologies which doesn't share scripts with other technologies, it should be already setup. For instance, in the technology folder of dataiku, just run yarn dev and the build of the javascript files will be running with the watch mode, and the SDK webapp will start.

For technologies which share javascript files, like aws, gcp or azure, the setup is split in two parts. In the folder dedicated to the shared scripts, run yarn dev to run the build with the watch mode. Then in the folder dedicated to the technology to develop, like aws-lambda, run npx @saagie/sdk start to start the SDK webapp.

Promotion

When the pull-request is merged in master, another Github action (running a gradle task) starts. It will retag docker images with branch name into a "production" name and generate a real release (and delete the pre release)