[FEATURE] Add stub support to every module
#4570
Comments
|
Hi @ewels This is such a good idea and can be immensely helpful when implementing dataflows. Here is something that I have been wondering about and I thought I'll share it here to get your feedback. With nf-test snapshot feature, it may be possible to partially automate If there is a function in nf-test ( touch ${prefix}.gff3
touch ${prefix}.report.html
...versions.yml...It won't be able to tackle complex situations but can provide starter code. Moreover, the Here is a pull request where I have tried to implement something similar but manually: #4627 |
|
That would be great 👌🏻 @mashehu / @mirpedrol - what do you think: could we make a #tools command to scaffold a |
|
I think if |
|
I have tried to implement something similar in logic to Each test creates a sorted list of outputs which should be matched by the same test with the {
assert snapshot(
(
[process.out.reads[0][0].toString()] + // meta
process.out.reads.collect { file(it[1]).getName() } +
process.out.json.collect { file(it[1]).getName() } +
process.out.html.collect { file(it[1]).getName() } +
process.out.log.collect { file(it[1]).getName() } +
process.out.reads_fail.collect { file(it[1]).getName() } +
process.out.reads_merged.collect { file(it[1]).getName() }
).sort()
).match("test_fastp_single_end-for_stub_match")
} |
|
On @sateeshperi suggestion, also raised it on nf-test askimed/nf-test#168 |
|
Can we create a nice comprehensive list where the stub is still missing (such as was done for other batch changes)? |
|
Good idea @famosab As of https://github.com/nf-core/modules/tree/d5b47a24314cab9f64593f29cf97a64b0acc7dce, there are 512 modules without mods=($(find ./modules/nf-core -name main.nf))
for file in $mods; do grep -q 'stub:' "$file" || sed -n 's/process \(.*\) {/- \1/p' "$file"; done | sort -V
|
|
I've been updating stub for modules in rnaseq cf nf-core/rnaseq#1335
If this is not properly done, then subworkflows stubs are not working properly either, and pipeline stubs are just a pipe dream. Which lead to the creation of this nf-test issue askimed/nf-test#227 Do we just want to test the names and the files created, and thus the channel logic when chaining modules. Or do we use stub as well to check that the virtual env/containers are working? |
|
I updated @GallVp list with a task list icon so that we can keep track of which modules have been already added the |
|
Also, if someone is working on a module please assign it to your self e.g.:
So that we can avoid duplicating the work. |
Using command
stubblocks in Nextflow is useful when developing syntax and quickly testing pipelines. However, its use within nf-core is limited because the vast majority of nf-core modules do not havestubblocks.My suggestion is to require that every module should have a
stubblock. All 1095 of themWe can lint for the presence of this in linting and also potentially run with
-stub-runin the module CI.Although adding a stub block for one or two modules is not difficult, adding it for all will be a significant effort. This could be a nice project focus for a future hackathon(s).
Tasks
The text was updated successfully, but these errors were encountered: