Two bugs in remote test data handling

[bernt-matthias]

Describe the bug

1. if there are multiple test files with the same "basename" then in the command section the value of all input variables (i.e. the path) is the same. this hits us for instance for zenodo URLs which all end with "/content" .. see test2
2. if there are multiple test files with value and location, where for both the file is locally present then for the second test input location is used.. see test4

https://github.com/bernt-matthias/galaxy/tree/location_errors

bernt-matthias@fb4175e

Any idea @davelopez ?

Galaxy Version and/or server at which you observed the bug
Galaxy Version: 24.1 (but probably earlier .. 23.1 #16311)

To Reproduce

Run the test tool in planemo

Expected behavior

Test succeed.

[davelopez]

Thanks for the tests to reproduce! I will investigate and try to fix it 👍

[davelopez]

I've been investigating these 2 issues (on the 24.0 branch):

• For the first one, what do you think would be the best way to fix this? Maybe using a hash of the location URL instead of the basename in the URL? 🤔 As a workaround for now, you can provide a "value" that is different for each input so it doesn't default to the basename "content" in your case:

  <test>
      <param name="input" value="bed1" location="https://raw.githubusercontent.com/galaxyproject/planemo/refs/heads/master/tests/data/1.bed"/>
      <param name="input2" value="bed2" location="https://raw.githubusercontent.com/galaxyproject/galaxy/refs/heads/dev/test-data/1.bed"/>
      <assert_stdout>
          <has_n_lines n="2"/>
      </assert_stdout>
  </test>

• I'm a bit confused about this second test:

  <test>
      <!-- When the `value` and the `location` are defined, if the local file already exists, the location will be ignored. -->
      <param name="input" value="simple_line.txt" location="https://raw.githubusercontent.com/galaxyproject/planemo/7be1bf5b3971a43eaa73f483125bfb8cabf1c440/tests/data/hello.txt"/>
      <param name="input" value="simple_line_alternative.txt" location="https://raw.githubusercontent.com/galaxyproject/planemo/master/tests/data/not_hello.txt"/>
      <output name="output">
          <assert_contents>
              <has_line line="This is a line of text."/>
          </assert_contents>
      </output>
      <output name="output">
          <assert_contents>
              <has_line line="This is a different line of text."/>
          </assert_contents>
      </output>
  </test>

  Since the same input is defined twice, I would expect only one output with the contents of the last input ("simple_line_alternative.txt"). However, the test is expecting two outputs with the same reference name (output), one for each input content. Is this correct?
  If I replace the expected outputs with:

  <test>
      <!-- When the `value` and the `location` are defined, if the local file already exists, the location will be ignored. -->
      <param name="input" value="simple_line.txt" location="https://raw.githubusercontent.com/galaxyproject/planemo/7be1bf5b3971a43eaa73f483125bfb8cabf1c440/tests/data/hello.txt"/>
      <param name="input" value="simple_line_alternative.txt" location="https://raw.githubusercontent.com/galaxyproject/planemo/master/tests/data/not_hello.txt"/>
      <output name="output">
          <assert_contents>
              <has_line line="This is a different line of text."/>
          </assert_contents>
      </output>
  </test>

  Then the test passes. But surprisingly, only when I run the tests using galaxy-tool-test and not with planemo test. If I run planemo test, the test fails with the following error:

  Output output:  different than expected
  Expected line 'This is a different line of text.' in output ('Not Hello World!
  ')
  

  So indeed, with planemo it tries to download the file from the location, but with galaxy-tool-test (on a running 24.0 galaxy) it uses the local file correctly. Maybe planemo couldn't find the local file because it was looking for it in the wrong place? This completely threw me off the track. 😅

I'll keep investigating but I'm not sure if I'll be able to find a fix before the winter break 😞

[bernt-matthias]

Thanks for the workaround. Using a hash seems a good option?

Sorry for the second test. Stupid copy paste error. But still stage that it passes with Galaxy-tool-test. Not sure if it's worth to investigate this further. Intuitively: No.

not sure if I'll be able to find a fix before the winter break

No worries. Have a great time!