-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #506 from viash-io/develop
Viash 0.7.5 (2023-08-11): Minor breaking changes and new features
- Loading branch information
Showing
187 changed files
with
6,167 additions
and
3,672 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| --- | ||
| title: CLI | ||
| listing: | ||
| - id: cli | ||
| contents: . | ||
| template: template.ejs | ||
| order: 10 | ||
| --- | ||
|
|
||
| These are the available commands available on the Command Line Interface: | ||
|
|
||
| :::{#cli} | ||
| ::: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| ```{=html} | ||
| <ul> | ||
| <% for (const item of items) { %> | ||
| <li> | ||
| <a href="<%- item.path %>"><%= item.title %></a>: <%= item.description %> | ||
| </li> | ||
| <% } %> | ||
| </ul> | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| --- | ||
| title: Dynamic Config Modding | ||
| order: 50 | ||
| --- | ||
|
|
||
| Viash can modify a [viash config](/reference/config/index.html) at runtime using a custom Domain Specific Language (DSL). This allows making dynamic changes to your components or projects. | ||
| All Viash subcommands have support for the DSL through the `-c|--config_mod` parameter. The format for these is as follows: | ||
|
|
||
| ```bash | ||
| viash COMMAND -c '.SECTION.PROPERTY := VALUE' | ||
| ``` | ||
|
|
||
| Multiple config mods can be added by adding more `-c|--config_mod` parameters: | ||
|
|
||
| ```bash | ||
| viash COMMAND \ | ||
| -c '.SECTION.PROPERTY := VALUE' \ | ||
| -c '.SECTION.PROPERTY := VALUE' | ||
| ``` | ||
|
|
||
| ## Examples | ||
|
|
||
| Change the version of a component: | ||
|
|
||
| ```bash | ||
| viash build -c '.functionality.version := "0.3.0"' | ||
| ``` | ||
|
|
||
| Change the registry of a docker container: | ||
|
|
||
| ```bash | ||
| viash build -c \ | ||
| '.platforms[.type == "docker"].container_registry := "url-to-registry"' | ||
| ``` | ||
|
|
||
| Add an author to the list: | ||
|
|
||
| ```bash | ||
| viash build -c '.functionality.authors += { name: "Mr. T", role: "sponsor" }' | ||
| ``` | ||
|
|
||
| You can use dynamic config modding to alter the config of multiple components at once: | ||
|
|
||
| ```bash | ||
| viash ns build \ | ||
| -c '.functionality.version := "0.3.0"' \ | ||
| -c '.platforms[.type == "docker"].container_registry := "url-to-registry"' \ | ||
| -c '.functionality.authors += { name: "Mr. T", role: "sponsor" }' | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| --- | ||
| title: Import a VDSL3 module | ||
| --- | ||
|
|
||
| A VDSL3 module is a Nextflow module generated by Viash. See the [guide](/guide/nextflow_vdsl3/introduction.qmd) for a more in-depth explanation on how to create Nextflow workflows with VDSL3 modules. | ||
|
|
||
| ## Importing a VDSL3 module | ||
|
|
||
|
|
||
| After building a VDSL3 module from a component, the VDSL3 module can be imported just like any other Nextflow module. | ||
|
|
||
| **Example:** | ||
|
|
||
| ```groovy | ||
| include { mymodule } from 'target/nextflow/mymodule/main.nf' | ||
| ``` | ||
|
|
||
| ## VDSL3 module interface | ||
|
|
||
| VDSL3 modules are actually workflows which take one channel and emit one channel. It expects the channel events to be tuples containing an 'id' and a 'state': `[id, state]`, where `id` is a unique String and `state` is a `Map[String, Object]`. The resulting channel then consists of tuples `[id, new_state]`. | ||
|
|
||
| **Example:** | ||
|
|
||
| ```groovy | ||
| workflow { | ||
| Channel.fromList([ | ||
| ["myid", [input: file("in.txt")]] | ||
| ]) | ||
| | mymodule | ||
| } | ||
| ``` | ||
|
|
||
| :::{.callout-note} | ||
| If the input tuple has more than two elements, the elements after the second element are passed through to the output tuple. | ||
| That is, an input tuple `[id, input, ...]` will result in a tuple `[id, output, ...]` after running the module. | ||
| For example, an input tuple `["foo", [input: file("in.txt")], "bar"]` will result in an output tuple `["foo", [output: file("out.txt")], "bar"]`. | ||
| ::: | ||
|
|
||
| ## Customizing VDSL3 modules on the fly | ||
|
|
||
| Usually, Nextflow processes are quite static objects. For example, changing its directives can be quite tricky. | ||
|
|
||
| The `.run()` function is a unique feature for every VDSL3 module which allows dynamically altering the behaviour of a module from within the pipeline. For example, we use it to set the `publishDir` directive to `"output/"` so the output of that step in the pipeline will be stored as output. | ||
|
|
||
| **Example:** | ||
|
|
||
| ```groovy | ||
| workflow { | ||
| Channel.fromList([ | ||
| ["myid", [input: file("in.txt")]] | ||
| ]) | ||
| | mymodule.run( | ||
| args: [k: 10], | ||
| directives: [cpus: 4, memory: "16 GB"] | ||
| ) | ||
| } | ||
| ``` | ||
|
|
||
| ### Arguments of `.run()` | ||
|
|
||
| - `key` (`String`): A unique key used to trace the process and help make names of output files unique. Default: the name of the Viash component. | ||
|
|
||
| - `args` (`Map[String, Object]`): Argument overrides to be passed to the module. | ||
|
|
||
| - `directives` (`Map[String, Object]`): Custom directives overrides. See the Nextflow documentation for a list of available directives. | ||
|
|
||
| - `auto` (`Map[String, Boolean]`): Whether to apply certain automated processing steps. Default values are inherited from the [Viash config](/reference/config/platforms/nextflow/auto.qmd). | ||
|
|
||
| - `auto.simplifyInput`: If `true`, if the input tuple is a single file and if the module only has a single input file, the input file will be passed the module accordingly. Default: `true` (inherited from Viash config). | ||
|
|
||
| - `auto.simplifyOutput`: If `true`, if the output tuple is a single file and if the module only has a single output file, the output map will be transformed into a single file. Default: `true` (inherited from Viash config). | ||
|
|
||
| - `auto.publish`: If `true`, the output files will be published to the `params.publishDir` folder. Default: `false` (inherited from Viash config). | ||
|
|
||
| - `auto.transcript`: If `true`, the module's transcript will be published to the `params.transcriptDir` folder. Default: `false` (inherited from Viash config). | ||
|
|
||
| - `map` (`Function`): Apply a map over the incoming tuple. Example: `{ tup -> [ tup[0], [input: tup[1].output] ] + tup.drop(2) }`. Default: `null`. | ||
|
|
||
| - `mapId` (`Function`): Apply a map over the ID element of a tuple (i.e. the first element). Example: `{ id -> id + "_foo" }`. Default: `null`. | ||
|
|
||
| - `mapData` (`Function`): Apply a map over the data element of a tuple (i.e. the second element). Example: `{ data -> [ input: data.output ] }`. Default: `null`. | ||
|
|
||
| - `mapPassthrough` (`Function`): Apply a map over the passthrough elements of a tuple (i.e. the tuple excl. the first two elements). Example: `{ pt -> pt.drop(1) }`. Default: `null`. | ||
|
|
||
| - `filter` (`Function`): Filter the channel. Example: `{ tup -> tup[0] == "foo" }`. Default: `null`. | ||
|
|
||
| - `fromState`: Fetch data from the state and pass it to the module without altering the current state. `fromState` should be `null`, `List[String]`, `Map[String, String]` or a function. | ||
|
|
||
| - If it is `null`, the state will be passed to the module as is. | ||
| - If it is a `List[String]`, the data will be the values of the state at the given keys. | ||
| - If it is a `Map[String, String]`, the data will be the values of the state at the given keys, with the keys renamed according to the map. | ||
| - If it is a function, the tuple (`[id, state]`) in the channel will be passed to the function, and the result will be used as the data. | ||
|
|
||
| Example: `{ id, state -> [input: state.fastq_file] }` | ||
| Default: `null` | ||
|
|
||
| - `toState`: Determine how the state should be updated after the module has been run. `toState` should be `null`, `List[String]`, `Map[String, String]` or a function. | ||
|
|
||
| - If it is `null`, the state will be replaced with the output of the module. | ||
| - If it is a `List[String]`, the state will be updated with the values of the data at the given keys. | ||
| - If it is a `Map[String, String]`, the state will be updated with the values of the data at the given keys, with the keys renamed according to the map. | ||
| - If it is a function, a tuple (`[id, output, state]`) will be passed to the function, and the result will be used as the new state. | ||
|
|
||
| Example: `{ id, output, state -> state + [counts: state.output] }` | ||
| Default: `{ id, output, state -> output }` | ||
|
|
||
| - `debug`: Whether or not to print debug messages. Default: `false`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| --- | ||
| title: Nextflow VDSL3 | ||
| order: 35 | ||
| --- | ||
|
|
||
| Viash supports creating Nextflow workflows in multiple ways. | ||
|
|
||
| * [Run a module as a standaline pipeline](run_module.qmd) | ||
| * [Import a VDSL3 module](import_module.qmd) | ||
| * Create a Nextflow workflow with dependencies |
Oops, something went wrong.