Test result attached to a different workflow that is being run concurrently

[lgolez]

Describe the bug

• I have two separate workflows with different goals. One to test Android, the other IOS
• Both of these workflows are being run together when a PR is created.
• Both of these workflows utilise a composite step where it prints out the results e.g.

name: "Print test results"
description: "Prints test results into github actions"

runs:
  using: "composite"
  steps:
      - name: Test Report
        uses: github-actions/[email protected]
        if: success() || failure()   
        with:
          name: Test results            
          path: ./results/*.xml    # Path to test results
          reporter: jest-junit

To Reproduce

Steps to reproduce the behavior:

1. Create two separate workflows that run tests and produce an XML test result
2. Both workflows should be triggered at the same time e.g. both should have a similar trigger like on push
3. At the end of the two workflows, add a step that reads those result
4. Check result

Expected behavior

Job should be attached to their respective workflows

Screenshots

Two different workflows

Last one does not contain the test result

First workflow to be triggered contained both the results

Additional context

I attempted to separate test results without using the custom action and explicitly added the test report script into separate workflows but still got the same issue.

[i2van]

• No tests under Jobs: https://github.com/i2van/hourglass/actions/runs/10468117626

• Two tests under Jobs: https://github.com/i2van/hourglass/actions/runs/10468062689

[peterbrendel]

Edit

Actually the workflow that it got attached to wasn't even running, I noticed this now because it is skipped for that PR, so it attached to a finished (skipped) workflow.

---

I experienced the same issue with different steps to reproduce, the other workflow doesn't use this action:

To Reproduce

Steps to reproduce the behavior:

1. Have one or more workflows running
2. Generate test report

[alevincenzi]

Hello
I saw the same behavior too. We fire two workflows at the PR and sometimes it's appended to the other one.
But isn't this problem that we all see related to the github limitation mentioned in the readme?

https://github.com/dorny/test-reporter?tab=readme-ov-file#github-limitations

I found other github actions for TRX reports and they all mention the same. Only GitHub today "is allowed to create a check suite for each workwflow run and there is no public Check Suite API" [cit].

Take it or leave it ...

[alevincenzi]

Check this line https://github.com/dorny/test-reporter/blob/main/src/main.ts#L178

where the action uses the octokit client to create the check using the sha of the commit.

Here the official API called to create the run: https://docs.github.com/en/rest/checks/runs?apiVersion=2022-11-28#create-a-check-run

In the body there is no mention "of the workflow" and only name and head_sha are required.
If it goes to another workflow, it's probably GitHub's fault.

[alevincenzi]

Also, this has never been solved on GitHub side: https://github.com/orgs/community/discussions/24616

[peterbrendel]

I can't say for sure, but I might have a workaround, give it a try and see if it works:

I added another step after the report step and now GHA is attaching to the correct workflow, it worked 3 times in a row, could be coincidence tho.

Example

- name: Test Report
  uses: dorny/test-reporter@v1
  with:
    name: Unit Tests
    path: 'test-results/*.xml'
    reporter: java-junit
    list-tests: none
    max-annotations: '50'
- name: Random step
  run: echo "Test"

[rlordcardano]

@peterbrendel - that didn't work for me. In our setup the master build runs on a cron job, and the test reports get appended to the original build that was merged back into master.

[ccpp]

[EDIT] Sorry, I just found tat you already have several duplicate issues to this one, like #67 and #224, so you know this problem well...
Anyways, here's my observation which might still be an issue...

---

I have the same issue and I suspect that the problem is somewhere here. The github API seems not used correctly as far as I can judge:

export function getCheckRunContext(): {sha: string; runId: number} {
  if (github.context.eventName === 'workflow_run') {
    // should be "event_name" ?
    core.info('Action was triggered by workflow_run: using SHA and RUN_ID from triggering workflow')
    const event = github.context.payload
    if (!event.workflow_run) {
      throw new Error("Event of type 'workflow_run' is missing 'workflow_run' field")
    }
    return {
      sha: event.workflow_run.head_commit.id,
      runId: event.workflow_run.id
    }
  }

  const runId = github.context.runId
  // should be "run_id" ?
  if (github.context.payload.pull_request) {
    core.info(`Action was triggered by ${github.context.eventName}: using SHA from head of source branch`)
    const pr = github.context.payload.pull_request as PullRequest
    return {sha: pr.head.sha, runId}
  }

  return {sha: github.context.sha, runId}
}

[lneuhaus]

@ccpp are you able to confirm that this the workflow is the correct one with your patch?

[ccpp]

No, sorry, this is untested! @lneuhaus

I even cannot reproduce the problem at the moment, probably since I switched to the artifact configuration. now the report is shown in the summary of the correct workflow