Merge pull request #535 from viash-io/develop

Release 0.8.0-RC1

.github/workflows/ns_test.yml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
 
     - name: Set up sbt
       uses: actions/setup-java@v3

.github/workflows/sbt_test.yml

@@ -19,7 +19,7 @@ jobs:
           - { ver: '20', run_nextflow: false, run_coverage: false }
 
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
 
     - uses: viash-io/viash-actions/update-docker-engine@v4
       if: runner.os == 'Linux'

.gitignore

@@ -34,3 +34,6 @@ work
 
 # MacOS
 .DS_Store
+
+# ignore automatically generated WorkflowHelper.nf
+src/main/resources/io/viash/platforms/nextflow/WorkflowHelper.nf

CHANGELOG.md

@@ -1,3 +1,74 @@
+# Viash 0.8.0-RC1 (2023-10-02): Nextflow workflows and dependencies
+
+Nextflow workflows definitions are picked up by Viash and assembled into a functional Nextflow workflow, reducing the amount of boilerplate code needed to be written by the user.
+We added new 'dependencies' functionality to allow for more advanced functionality to be offloaded and re-used in components and workflows.
+
+## BREAKING CHANGES
+
+* `NextflowPlatform`: Changed the default value of `auto.simplifyOutput` from `true` to `false` (#522, PR #518). With `simplifyOutput` set to `true`, the resulting Map could be simplified into a `File` or a `List[File]` depending on the number of outputs. To replicate the previous behaviour, add the following config mod to `_viash.yaml`:
+
+  ```yaml
+  config_mods: |
+    .platforms[.type == 'nextflow'].auto.simplifyOutput := true
+  ```
+
+* `VDSL3Helper.nf`: Removed from the Viash jar file (PR #518). Its functions have been moved to `WorkflowHelper.nf`.
+
+* `DataflowHelper.nf`: Added deprecation warning to functions from this file (PR #518).
+
+* `preprocessInputs()` in `WorkflowHelper.nf`: Added deprecation warning to `preprocessInputs()` because this function causes a synchronisation event (PR #518).
+
+* `author.props`: Removed deprecated `props` field (PR #536). Deprecated since 0.7.4.
+
+## NEW FUNCTIONALITY
+
+* `dependencies`: Add `dependencies` and `repositories` to `functionality` (PR #509). 
+  The new functionality allows specifying dependencies and where to retrieve (repositories) them in a component, and subsequentially allows advanced functionality to be offloaded and re-used in scripts and projects. This is alike e.g. `npm`, `pip` and many others. A big difference is that we aim to provide the needed boilerplate code to ease the usage of the dependencies in scripts, workflows and pipelines.
+  Note that the dependency is required to be a built Viash component or project and not a random file or code project found externally. This is needed to provide the necessary background information to correctly link dependencies into a component.
+
+* `NextflowScript` & `NextflowPlatform`: Merged code for merging the `main.nf` files for VDSL3 components and wrapped Nextflow workflows (PR #518).
+  By aligning the codebase for these two, wrapped Nextflow workflows are more similar to VDSL3 components. For example, you can override the behaviour of a
+  wrapped Nextflow workflow using the `.run()` method. Status of a workflows `.run()` arguments:
+
+  - Works as intended: `auto.simplifyInput`, `auto.simplifyOutput`, `fromState`, `toState`, `map`, `mapData`, `mapPassthrough`, `filter`, `auto.publish = "state"`
+  - Does not work (yet): `auto.transcript`, `auto.publish = true`, `directives`, `debug`.
+
+  In a next PR, each of the dependencies will have their values overridden by the arguments of the `.run`.
+
+* `NextflowPlatform`: The data passed to the input of a component and produced as output by the component are now validated against the arguments defined in the Viash config (PR #518).
+
+* `NextflowPlatform`: Use `stageAs` to allow duplicate filenames to be used automatigically (PR #518).
+
+* `NextflowPlatform`: When wrapping Nextflow workflows, throw an error if the IDs of the output channel doesn't match the IDs of the input channel (PR #518).
+  If they don't, the workflow should store the original ID of the input tuple in the in the `_meta.join_id` field inside the state as follows:
+  Example input event: `["id", [input: file(...)]]`,
+  Example output event: `["newid", [output: file(...), _meta: [join_id: "id"]]]`
+
+
+## MAJOR CHANGES
+
+* `WorkflowHelper.nf`: The workflow helper was split into different helper files for each of the helper functions (PR #518).
+  For now, these helper files are pasted together to recreate the `WorkflowHelper.nf`.
+  In Viash development environments, don't forget to run `./configure` to start using the updated Makefile.
+
+* `NextflowPlatform`: Set default tag to `"$id"` (#521, PR #518).
+
+## MINOR CHANGES
+
+* `NextflowPlatform`: Throw error when unexpected keys are passed to the `.run()` method (#512, PR #518).
+
+* `NextflowPlatform`: Use `moduleDir` instead of `projectDir` to determine the resource directory.
+
+## DOCUMENTATION
+
+* Minor fixes to VDSL3 reference documentation (PR #508).
+
+## BUG FIXES
+
+* `WorkflowHelper.nf`: Only set default values of output files which are **not already set**, and if the output file argument is **not required** (PR #514).
+
+* `NextflowPlatform`: When using `fromState` and `toState`, do not throw an error when the state or output is missing an optional argument (PR #515).
+
 # Viash 0.7.5 (2023-08-11): Minor breaking changes and new features
 
 This release contains minor breaking change due to deprecated or outdated functionality being removed.

RELEASE_GUIDE.md

@@ -32,6 +32,11 @@ Add the files `bin/viash` and `bin/viash_install`.
 The source repository is [here](https://github.com/viash-io/website).
 Follow the information in the README.md to get up and running if needed.
 
+## Update _viash.yaml
+
+Update the viash version to the latest Viash version.
+This is needed to get updated information about the CLI and JSON schema.
+
 ## Rerender Website
 
 Short summary if you already have an operational envirionment:

build.sbt

@@ -1,6 +1,6 @@
 name := "viash"
 
-version := "0.7.5"
+version := "0.8.0-RC1"
 
 scalaVersion := "2.13.10"
 
@@ -10,7 +10,9 @@ libraryDependencies ++= Seq(
   "org.rogach" %% "scallop" % "5.0.0",
   "org.scala-lang" % "scala-reflect" % scalaVersion.value,
   "org.scala-lang.modules" %% "scala-parser-combinators" % "2.1.1",
-  "org.scala-lang.modules" %% "scala-parallel-collections" % "1.0.4"
+  "org.scala-lang.modules" %% "scala-parallel-collections" % "1.0.4",
+  "com.github.julien-truffaut" %% "monocle-core"  % "2.1.0",
+  "com.github.julien-truffaut" %% "monocle-macro" % "2.1.0"
 )
 
 val circeVersion = "0.14.1"
@@ -35,3 +37,39 @@ licenses += "GPL-3.0-or-later" -> url("https://www.gnu.org/licenses/gpl-3.0.html
 
 
 // Test / parallelExecution := false
+
+lazy val generateWorkflowHelper = taskKey[Unit]("Generates WorkflowHelper.nf")
+
+generateWorkflowHelper := {
+  import sbt._
+  import java.nio.file._
+
+  val basePath = (Compile / resourceDirectory).value / "io" / "viash" / "platforms" / "nextflow"
+  val wfHelper = Paths.get(basePath.toString, "WorkflowHelper.nf")
+
+  // brute force recursive file listing instead of using a glob that skips files in the root directory
+  def recursiveListFiles(f: File): Array[File] = {
+    val these = f.listFiles
+    these.filter(_.isFile()) ++ these.filter(_.isDirectory).flatMap(recursiveListFiles)
+  }
+  // skip files in the root directory
+  val files = basePath.listFiles().filter(p => p.isDirectory()).map(f => recursiveListFiles(f)).flatten
+
+  Files.deleteIfExists(wfHelper)
+  Files.write(wfHelper, 
+    """////////////////////////////
+      |// VDSL3 helper functions //
+      |////////////////////////////
+      |""".stripMargin.getBytes(),
+    StandardOpenOption.CREATE)
+
+  files.sorted.foreach { path =>
+    Files.write(wfHelper, s"\n// helper file: '${path}'\n".getBytes(), StandardOpenOption.APPEND)
+    Files.write(wfHelper, Files.readAllBytes(path.toPath()), StandardOpenOption.APPEND)
+  }
+
+  streams.value.log.info(s"Generated WorkflowHelper.nf at ${wfHelper.toAbsolutePath}")
+}
+
+assembly := ((assembly) dependsOn generateWorkflowHelper).value
+Test / testOptions := ((Test / testOptions) dependsOn generateWorkflowHelper).value

docs/reference/nextflow_vdsl3/import_module.qmd

@@ -2,7 +2,7 @@
 title: Import a VDSL3 module
 ---
 
-A VDSL3 module is a Nextflow module generated by Viash. See the [guide](/guide/nextflow_vdsl3/introduction.qmd) for a more in-depth explanation on how to create Nextflow workflows with VDSL3 modules.
+A VDSL3 module is a Nextflow module generated by Viash. See the [guide](/guide/nextflow_vdsl3/index.qmd) for a more in-depth explanation on how to create Nextflow workflows with VDSL3 modules.
 
 ## Importing a VDSL3 module
 

docs/reference/nextflow_vdsl3/run_module.qmd

@@ -51,7 +51,7 @@ arg2: 5
 
 You can run the pipeline on the list of parameters using the `-params-file` parameter.
 
-```{bash}
+```bash
 nextflow run target/main.nf -params-file params.yaml --publish_dir output2
 ```
 

src/main/resources/io/viash/helpers/bashutils/ViashFindTargetDir.sh

@@ -0,0 +1,11 @@
+# ViashFindTargetDir: return the path of the '.build.yaml' file, following symlinks
+# usage   : ViashFindTargetDir 'ScriptPath'
+# $1      : The location from where to start the upward search
+# returns : The absolute path of the '.build.yaml' file
+function ViashFindTargetDir {
+  SOURCE="$1"
+  while [[ "$SOURCE" != "" && ! -e "$SOURCE/.build.yaml" ]]; do
+    SOURCE=${SOURCE%/*}
+  done
+  echo $SOURCE
+}

src/main/resources/io/viash/platforms/nextflow/DataflowHelper.nf

@@ -1,3 +1,13 @@
+// This helper file will be deprecated soon
+dataflowHelperDeprecationWarningPrinted = false
+
+def dataflowHelperDeprecationWarning() {
+  if (!dataflowHelperDeprecationWarningPrinted) {
+    dataflowHelperDeprecationWarningPrinted = true
+    System.err.println("Warning: the functions in the DataflowHelper.nf (setWorkflowArguments, getWorkflowArguments) are set to be deprecated Viash 0.9.0.")
+  }
+}
+
 /* usage:
 | setWorkflowArguments(
   pca: [ "input": "input", "obsm_output": "obsm_pca" ]
@@ -8,6 +18,8 @@
 */
 
 def setWorkflowArguments(Map args) {
+  dataflowHelperDeprecationWarning()
+
   def wfKey = args.key != null ? args.key : "setWorkflowArguments"
   args.keySet().removeAll(["key"])
 
@@ -68,6 +80,8 @@ def setWorkflowArguments(Map args) {
 
 
 def getWorkflowArguments(Map args) {
+  dataflowHelperDeprecationWarning()
+
   def inputKey = args.inputKey != null ? args.inputKey : "input"
   def wfKey = "getWorkflowArguments_" + args.key
 
@@ -108,6 +122,8 @@ def getWorkflowArguments(Map args) {
 
 
 def strictMap(Closure clos) {
+  dataflowHelperDeprecationWarning()
+
   def numArgs = clos.class.methods.find{it.name == "call"}.parameterCount
 
   workflow strictMapWf {
@@ -131,6 +147,8 @@ def strictMap(Closure clos) {
 }
 
 def passthroughMap(Closure clos) {
+  dataflowHelperDeprecationWarning()
+
   def numArgs = clos.class.methods.find{it.name == "call"}.parameterCount
 
   workflow passthroughMapWf {
@@ -152,6 +170,8 @@ def passthroughMap(Closure clos) {
 }
 
 def passthroughFlatMap(Closure clos) {
+  dataflowHelperDeprecationWarning()
+
   def numArgs = clos.class.methods.find{it.name == "call"}.parameterCount
 
   workflow passthroughFlatMapWf {
@@ -177,6 +197,8 @@ def passthroughFlatMap(Closure clos) {
 }
 
 def passthroughFilter(Closure clos) {
+  dataflowHelperDeprecationWarning()
+
   def numArgs = clos.class.methods.find{it.name == "call"}.parameterCount
 
   workflow passthroughFilterWf {