Being able to upload test replay only for failing test cases

[leroydev]

What would you like?

I'd like to be able to configure Cypress test replays to only be uploaded when a test fails.
I couldn't find a way to accomplish this myself , after:spec does work for this in the case of videos but I haven't found a way to influence test replays in this way.

This might be a quick fix to alleviate #27818 partly, as we're also heavily impacted by that. I can however imagine that it's only possible to prevent test replay uploads (instead of also preventing creation) for succeeding testcases and the performance impact is mostly in the creation of test replays, so I'm not entirely sure whether this will help with the test replay performance impact.

Why is this needed?

Test replays aren't that interesting for tests that succeed.

[jennifer-shehane]

@leroydev Can you share the Test Replay link where you’re seeing the slow performance? We’d like to prioritize a fix for that.

We plan to add some reporting on the time to upload Test Replays so everyone can see how much time that’s adding and we could also optimize it. For now, I do suspect it’s as you said, that the capturing for Test Replay is likely more of the slowdown than the upload time.

[leroydev]

@jennifer-shehane My apologies, I just noticed you replied.
Run with test replay, takes 59:13: https://cloud.cypress.io/projects/pswr65/runs/931/overview?roarHideRunsWithDiffGroupsAndTags=1
Run without test replay on the same commit, takes 22:41: https://cloud.cypress.io/projects/pswr65/runs/933/overview?roarHideRunsWithDiffGroupsAndTags=1

[mattleonowicz]

Since videoUploadOnPasses property was removed from configuration, I suppose you're not willing to add something similar for replays. Is it possible to remove replays of successful tests before the uploading begins, like presented here?

[jennifer-shehane]

In 13.6.0 we added logging of upload times for Test Replay so that users can see the time spent on upload. We're also intending to internally look at this data to make more informed decisions on next steps on how to handle uploads. There are other options like uploading in the background that could be considered.

[jennifer-shehane]

Hi everyone. We've looked at some of the upload timing that we've been seeing for Test Replay upload. The majority of uploads for Test Replay are happening under 300ms, with the slowest time (p99) being 4 seconds. Due to the upload of the Protocol not being a large amount of time spent and the fact that we've fixed A LOT of perf issues in v13 where we're often seeing those runs faster now, we are not considering adding this as a feature.

Please take a look at the upload time for your Test Replay and let us know if there's a long upload time that you think is an issue.

Terminal Output