correctly embed reference in bam/cram

[alpapan]

currently minimap2's bam/cram output stores a tmp file as the reference fasta. This prevents other galaxy tools from using it as the spec expects that the stored reference is a real file. This patch simply copies existing code over to ensure the real full path to the reference is used. It won't solve all cases (e.g. when reference file is deleted) but it will solve the other cases.

FOR CONTRIBUTOR:

• I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
• License permits unrestricted use (educational + commercial)
• This PR adds a new tool or tool collection
• This PR updates an existing tool or tool collection
• This PR does something else (explain below)

[alpapan]

example error that this patch fixes

tool: CONVERTER_cram_to_bam_0

SAMtools 1.17
[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::cram_decode_slice] Unable to fetch reference #0:5264-59085

[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::cram_decode_slice] Unable to fetch reference #0:1-23670

[E::cram_next_slice] Slice decode failure
samtools sort: truncated file. Aborting
[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::cram_decode_slice] Unable to fetch reference #0:192066-249955

[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::cram_decode_slice] Unable to fetch reference #0:122987-171298

[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::cram_decode_slice] Unable to fetch reference #0:332137-380341

[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::cram_decode_slice] Unable to fetch reference #0:257327-310839

[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::fai_build3_core] Failed to open the file /galaxy/database/jobs_directory/022/22894/working/reference.fa
[E::refs_load_fai] Failed to open reference file '/galaxy/database/jobs_directory/022/22894/working/reference.fa'
[W::cram_get_ref] Failed to populate reference for id 0
[E::cram_decode_slice] Unable to fetch reference #0:393134-445372

[mvdbeek]

Can you bump the version ? As you said, this is just as broken in anything but the most basic scenario where inputs are assumed to be available on a shared filesystem. As soon as there's any sort of staging involved this will break. Wasn't there a cram mode that didn't use reference-guided compression ?

[bernt-matthias]

The change still does not convince me. If I got the discussion in #6523 right then you need to supply the reference in downstream tools anyway if the linked one is not available.

I think this (i.e. the reference not being available) will happen in many/most cases (would also be my interpretation of the comment of @mvdbeek). Thus IMO the more important part of the solution should be that we always allow to supply the reference in all downstream tools that process CRAM.

Wondering how much space is saved in this mode vs the no-ref mode.

To me all this makes CRAM rather user unfriendly and unusable in a HPC/cloud environment like Galaxy. But I might be completely wrong - as so often :)

[bernt-matthias]

The code block should be a macro token.