Add Fairy #6647

SantaMcCloud · 2024-12-20T00:50:19Z

FOR CONTRIBUTOR:

I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
License permits unrestricted use (educational + commercial)
This PR adds a new tool or tool collection
This PR updates an existing tool or tool collection
This PR does something else (explain below)

SantaMcCloud · 2024-12-20T00:52:46Z

I know that there will be an error because of the format bcsp. Is it possible to add this format into the galaxy datatype?

I can do it but i have to read and maybe ask question about it. I tried to use the tabular format but with this the tool had always erros possible because the bcsp is a binary file.

SantaMcCloud · 2024-12-20T01:08:25Z

i also get this error when i try to test fairy_cov.xml

(planemo) (fairy_env) sf373@LAPTOP-7RMLPR2D:~/sf373$ planemo test tools-iuc/tools/fairy/fairy_cov.xml 
Traceback (most recent call last):
  File "/home/sf373/sf373/planemo/bin/planemo", line 8, in <module>
    sys.exit(planemo())
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/cli.py", line 103, in handle_blended_options
    return f(*args, **kwds)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/commands/cmd_test.py", line 76, in cli
    return_value = test_runnables(ctx, runnables, original_paths=uris, **kwds)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/engine/test.py", line 13, in test_runnables
    test_data = engine.test(runnables, test_timeout=kwds.get("test_timeout"))
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/engine/interface.py", line 79, in test
    test_cases = [t for tl in map(cases, runnables) for t in tl]
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/engine/interface.py", line 79, in <listcomp>
    test_cases = [t for tl in map(cases, runnables) for t in tl]
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/planemo/runnable.py", line 261, in cases
    test_dicts = tool_source.parse_tests_to_dict()
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 649, in parse_tests_to_dict
    tests.append(_test_elem_to_dict(test_elem, i, profile))
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 693, in _test_elem_to_dict
    outputs=__parse_output_elems(test_elem),
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 718, in __parse_output_elems
    name, file, attributes = __parse_output_elem(output_elem)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 728, in __parse_output_elem
    file, attributes = __parse_test_attributes(output_elem, attrib, parse_discovered_datasets=True)
  File "/home/sf373/sf373/planemo/lib/python3.7/site-packages/galaxy/tool_util/parser/xml.py", line 781, in __parse_test_attributes
    attributes["object"] = json.loads(attrib.pop("value_json"))
  File "/home/sf373/anaconda3/envs/comebin_env/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/sf373/anaconda3/envs/comebin_env/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/sf373/anaconda3/envs/comebin_env/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I dont know how this error happen but i will have a look at this the next couple of days. Maybe i can fix it somehow

bernt-matthias · 2024-12-20T07:45:31Z

I know that there will be an error because of the format bcsp. Is it possible to add this format into the galaxy datatype?

Yes. Is they documentation available?

bernt-matthias · 2024-12-20T07:50:37Z

I dont know how this error happen but i will have a look at this the next couple of days. Maybe i can fix it somehow

I think value_json="normal_test.tsv" should be value="normal_test.tsv".

SantaMcCloud · 2024-12-20T13:07:26Z

I know that there will be an error because of the format bcsp. Is it possible to add this format into the galaxy datatype?

Yes. Is they documentation available?

Yes and i found the right tab to it. I will read it and try it out!

SantaMcCloud · 2024-12-20T13:07:50Z

I dont know how this error happen but i will have a look at this the next couple of days. Maybe i can fix it somehow

I think value_json="normal_test.tsv" should be value="normal_test.tsv".

yes i didnt saw this typo yesterday thank you for the hint :)

SantaMcCloud · 2024-12-20T15:23:39Z

@bernt-matthias i have a question about adding a datatype. Do i have to add a sniffer or is this an optional function?

SantaMcCloud · 2024-12-20T18:56:18Z

galaxyproject/galaxy#19365

bernt-matthias · 2024-12-22T10:20:19Z

@bernt-matthias i have a question about adding a datatype. Do i have to add a sniffer or is this an optional function?

Optional, I would say.

bernt-matthias

Added a few more comments.

bernt-matthias · 2024-12-22T10:24:02Z

tools/fairy/fairy_cov.xml

+    <command detect_errors="exit_code">
+        <![CDATA[
+
+            ln -s '$contig' '$contig.element_identifier' &&


It's not a good idea to just use the element identifiers, since they might contain dangerous or forbidden characters. We usually solve this like so

tools-iuc/tools/coverm/macros.xml

Line 55 in 295026b

#set $fn = 'bam/' + re.sub('[^\s\w\-\\.]', '_', str($mapped.mode.bam_files.element_identifier))

Did change it on both wrappers

bernt-matthias · 2024-12-22T10:25:14Z

tools/fairy/fairy_cov.xml

+            fairy coverage
+            '$contig.element_identifier'
+            '$bcsp_file.element_identifier'
+            -t \${GALAXY_SLOTS:-8}


Double quotes around bash variables are often a good idea.

should have done if i did understand it correct

bernt-matthias · 2024-12-22T10:26:12Z

tools/fairy/fairy_cov.xml

+            -m ${minimum_ani}
+            -M ${min_number_kmers}
+            -c ${c}
+            -k ${k.value}


This just $k

bernt-matthias · 2024-12-22T10:28:28Z

tools/fairy/fairy_cov.xml

+            -k ${k.value}
+            --min-spacing ${min_spacing}
+            ${full_contig_name}
+            #if $output_type.value == 'semi':


Also here no .value needed.

You could simplify this, by using the command line parameter as value in the options and then just use $output_value here.

Change it and there is no need for the value. I just forgot it to remove it since it did work with it

bernt-matthias · 2024-12-22T10:30:04Z

tools/fairy/fairy_cov.xml

+        <param name="contig" type="data" format="fasta,fasta.gz" label="Input fasta contig file" help="Input the raw fasta contig file. It can be gzip!"/>
+        <param name="bcsp_file" type="data" format="bcsp" label="Input the pre-sketched file (.bcsp file)" help="This file will be generated with the fairy sketch tool."/>
+        <param argument="--minimum-ani" type="integer" optional="true" min="0" max="100" value="95" label="Set minimum ANI" help="Set the minimum adjusted ANI for the coverage calculation. CARE: only adjust it when you know what it does!"/>
+        <param argument="--min-number-kmers" type="integer" value="8" optional="true" label="Genome filter" help="Set the number for exclude genomes with less than this number of sampled k-mers."/>


Don't undersand this parameter. For the default sequences with less than 8 kmers are removed?

I did rewrote this help, maybe it is better now to understand but yes this paramtere should be a filter to filter out genomes with less k-mers then this paramter

I still do not understand this parameter. What does it mean if a genome has x k-mers?

It means that a genome have less then x (the value of the parameter) k-mers it will execlude from the algorithm. It should not appear in the coverage file generated by Fairy. It this more cleare? And sorry for this missunderstanding

Unfortunately no. I just do not understand what it means that a genome has a kmer. A genome of length n has n-k+1 kmers. There must be something special about the x kmers the help text is talking about.

This x stands for a lower border to filter out genomes with less then x k-mers to not appear in the coverage file.

For an example there are 3 genome in a contig. The first hast 5 k-mers, the second has 10 k-mers and the last one has 8 k-mers. We set this paramter to a 7 which means that the first genome with 5 k-mers will not be included since it hast less then 7 k-mers. When we set this to 2 then there will be no excluding of a genome since all 3 has more then 2 k-mers.

Is this explenation better?

bernt-matthias · 2024-12-22T10:31:40Z

tools/fairy/fairy_sketch.xml

+                <param argument="--reads" type="data" format="fastqsanger,fasta,fastq,fasta.gz,fastq.gz" label="Input single-end reads"/>
+            </when>
+            <when value="pair">
+                <param argument="--first_pairs" type="data" format="fastqsanger,fasta,fastq,fasta.gz,fastq.gz" label="Input first paired-end reads"/>


I would prefer input as paired collection, at least add it as option.

Add this option with a test!

SantaMcCloud · 2024-12-23T00:52:05Z

Will still fail since the datatype is not in galaxy yet

SantaMcCloud added 2 commits December 20, 2024 01:44

Add Fairy

2617364

remove uneeded file

f93520a

fix typo but test will still error since new format

9071a35

fix test. Now evreything is done just wait till datatype is up

6a9629b

SantaMcCloud mentioned this pull request Dec 20, 2024

Add new datatype bcsp galaxyproject/galaxy#19363

Closed

4 tasks

bernt-matthias reviewed Dec 22, 2024

View reviewed changes

add review changes

fecf9b3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Fairy #6647

Add Fairy #6647

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

bernt-matthias commented Dec 20, 2024

bernt-matthias commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

bernt-matthias commented Dec 22, 2024

bernt-matthias left a comment

bernt-matthias Dec 22, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 22, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 22, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 22, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 22, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 23, 2024

SantaMcCloud Dec 23, 2024

bernt-matthias Dec 24, 2024

SantaMcCloud Dec 27, 2024 •

edited

Loading

bernt-matthias Dec 22, 2024

SantaMcCloud Dec 23, 2024

SantaMcCloud commented Dec 23, 2024

Add Fairy #6647

Are you sure you want to change the base?

Add Fairy #6647

Conversation

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 • edited Loading

SantaMcCloud commented Dec 20, 2024 • edited Loading

bernt-matthias commented Dec 20, 2024

bernt-matthias commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024

SantaMcCloud commented Dec 20, 2024 • edited Loading

bernt-matthias commented Dec 22, 2024

bernt-matthias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SantaMcCloud Dec 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SantaMcCloud commented Dec 23, 2024

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud commented Dec 20, 2024 •

edited

Loading

SantaMcCloud Dec 27, 2024 •

edited

Loading