From 041120ee4a70ba735aee92e895760a07b497b91a Mon Sep 17 00:00:00 2001 From: ajuvercr Date: Tue, 12 Jul 2022 16:49:34 +0200 Subject: [PATCH] update readme --- README.md | 202 +++++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 185 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 0875961..6629eb2 100644 --- a/README.md +++ b/README.md @@ -1,22 +1,24 @@ -# nautirust +# Nautirust -An orchestrator for the connector architecture. -This architecture defines processors that take input. -Input can be many things, mostly configuration stuff, streamReaders and streamWriters. +Nautirust is a cli-utility program designed to assist the user in configuring and starting a pipeline of processes (called steps). +Nautirust does not restrict on programming language or startup sequence. -Readers and writers are connectors that use a channel to forward messages in some serialization (json, turtle, xml, plain). -What channel is used, usually doesn't really matter, they all have the same interface. +## Intro -Nautirust is primariy used to configure pipelines that consist of steps. -Each step corresponds to a processor and is configured with a step configuration file denoting the expected runner, runner specific arguments (source file, source function) and arguments to be configured. -Nautirust configures the parameters used for the processors per step. To do this it mostly asks the user the actual _implementations_ of the arguments. -Nautirust understands that readers and writers have to be linked up to function (the same channel configuration), this way it can guide the user in creating the correct pipeline. +Nautirust consists of three concepts, a step, a runner and a channel. A pipeline is a sequence of steps. -After configuring the steps of a pipeline a pipeline config is generated. This config contains all the actual arguments (including channel configuration). +Runners are designed to start a step and can require steps to define some additional properties. +For example a runner that executes a step written in javascript requires the location of the main source file and the name of the function that should be executed. + +Each step is programming language independent, but has to specify what runner must be used to start the step. +A step also specifies what arguments should be provided to that step, including the type. + +The type of these arguments is loose but there are two special cases: stream reader and stream writer. +Readers and writers connect two processes together with _a_ channel. Nautirust helps connecting these channels. + +The goal of the runner is not only to start up a step, but also to abstract away the underlying channel. This eases the implementation of steps. +Runners specify what channels they support. Not all runners can support all channels, nautirust uses this information to only suggest plausible channels. -When Nautirust executes a configured pipeline, it executes a specific _runner_ for a specific step. -For example, if a step consists of a JS function, then a JSRunner is used to actually execute this function. -Ideally the runner starts up the channel and provides the step with a instance of a reader or writer and the configured arguments. ## Usage @@ -40,13 +42,13 @@ Example step file (for more details see later) Generate a plan and save it to plan.json: ``` -cargo run -- generate -o plan.json [...steps] +nautirust generate -o plan.json [...steps] ``` Execute the plan: ``` -cargo run -- run plan.json +nautirust run plan.json ``` @@ -61,6 +63,7 @@ runners = "configs/runners/*/runner.json" This configuration file will look for any channel defined inside `configs/channels` and will look for runners defined in `configs/runners` that have a `runner.json` file. + ### Channel configuration Example channel configuration: @@ -80,7 +83,8 @@ Example channel configuration: ``` Here a channel is defined called `file` and specifies two fields are required: `path` and `onReplace`. -It also defines some options for the orchestrator and user to choose from. +It also defines some options for the nautirust. The user can choose between these options. + ### Runner configuration @@ -136,3 +140,167 @@ Example step configuration: A step is a processor. The processor specifies that it can be executed with the `JsRunner` runner. It also specifies what arguments have to be defined before being able to execute. + +## Functionality + +```sh +$ nautirust -h +nautirust 0.1.1 + +USAGE: + nautirust [OPTIONS] + +OPTIONS: + -c, --channels Glob to indicate channels locations + --config Location of a config file + -h, --help Print help information + -r, --runners Glob to indicate runners locations + -V, --version Print version information + +SUBCOMMANDS: + docker Create a docker-compose file from a nautirust pipeline + generate Generate a pipeline of steps + help Print this message or the help of the given subcommand(s) + prepare Prepares the execution pipeline by starting the required channels/runner + run Run a configured pipeline + stop Gracefully stop the runners and channels specified in the config + validate Validate configureations for runners and channels +``` + +You can specify (by glob) where to find channels and runners with `--channels` and `--runners` respectively. +Nautirust also supports toml file that specifies these properties (`--config`). + + +### generate + +```sh +$ nautirust generate -h +nautirust-generate +Generate a pipeline of steps + +USAGE: + nautirust generate [OPTIONS] [STEPS]... + +ARGS: + ... Steps to include in the pipeline (ordered) + +OPTIONS: + -a, --automatic Try infer basic configurations details + -h, --help Print help information + -o, --output Output location of the generated pipeline file +``` + +Nautirust takes multiple steps to create a pipeline configuration file. +Interactively the user is asked questions about the required arguments, taking special care of stream readers and stream writers. + +These questions can add up, use the `-a` flag to let nautirust infer some basic configuration consisting of: +- automatic linking of stream readers and writers with the same name +- automatically choosing a channel configuration when the channel type is specified + +`-o` takes a filename to store the generated configuration (default is stdout). + +### run +```sh +$ nautirust run -h +nautirust-run +Run a configured pipeline + +USAGE: + nautirust run [OPTIONS] + +ARGS: + Config file + +OPTIONS: + -h, --help Print help information + -t, --tmp-dir temporary directory to put step configuration files +``` + +Nautirust runs a generated configuration file. + +Each runners takes in a configuration file that specifies the steps that should be executed, with `-t` you can specify the location of these configuration files. + + +### prepare +```sh +$ nautirust prepare -h +nautirust-prepare +Prepares the execution pipeline by starting the required channels/runner + +USAGE: + nautirust prepare + +ARGS: + Config file + +OPTIONS: + -h, --help Print help information +``` + +Nautirust takes a generated configuration file, and prepares the used steps, runners and channels. +This can be used to run a build script, start a docker-compose instance, ... + + +### stop +```sh +$ nautirust stop -h +nautirust-stop +Gracefully stop the runners and channels specified in the config + +USAGE: + nautirust stop + +ARGS: + Config file + +OPTIONS: + -h, --help Print help information +``` + +Same as prepare, but in reverse. + + +### docker + +**EXPERIMENTAL** +```sh +$ nautirust docker -h +nautirust-docker +Create a docker-compose file from a nautirust pipeline + +USAGE: + nautirust docker [OPTIONS] + +ARGS: + Config file + +OPTIONS: + -h, --help Print help information + -o, --output + -t, --tmp-dir temporary directory to put step configuration files +``` + +Does the same as `nautirust run` but generates a docker-compose file that when executed start the pipeline. +This only works when the runners have specified a docker command. +This docker command is expected to generate a Dockerfile somewhere that executes the step. +This command also prints to stdout the required contents for this step (at least specify the generated Dockerfile and set up the correct docker context). + +The used channels also return a part of the docker-compose file (if anything). + + +### validate + +```sh +$ nautirust validate -h +nautirust-validate +Validate configureations for runners and channels + +USAGE: + nautirust validate + +OPTIONS: + -h, --help Print help information +``` + +Validates the specified channels and runners. +