Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: .actual.png images do not have FRSOURCE_CPVRD_V set, resulting in confusing git diffs when used to update baselines #322

Open
2 of 3 tasks
mwtrew opened this issue Oct 7, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@mwtrew
Copy link

mwtrew commented Oct 7, 2024

Describe the Bug

Hi, first of all huge thanks for your work on this great plugin.

I've noticed [...].actual.png files don't have FRSOURCE_CPVRD_V set, which causes problems when they're used to update the baseline images.

In our workflow we mostly use Cypress on the command line (cypress run). We "accept" a new baseline image by overwriting the old baseline with the new [...].actual.png file (see "Additional context" for why we do this).

This mostly works fine, but this .actual.png file doesn't have the FRSOURCE_CPVRD_V field set. This means that when we run the same tests later, FRSOURCE_CPVRD_V gets set by the plugin, and therefore there is a diff according to git, but not visually. This can show up later in the diff for an unrelated change, which is difficult for a reviewer to understand, and will happen repeatedly as baselines are updated over time.

Steps to reproduce

Developer A:

  1. Create a test and run it, generating an initial baseline
  2. Modify the component under test such that the test will fail
  3. Run the test, which fails, producing .actual.png and .diff.png files
  4. Rename .actual.png to overwrite the baseline
  5. Commit, push etc.

Developer B:

  1. Run the test, which passes, but which automatically updates the baseline to include FRSOURCE_CPVRD_V
  2. Now any commits/PR etc. this developer makes may include a confusing rewrite of that image
  3. Any subsequent baseline updates (performed like in step 4) will unset FRSOURCE_CPVRD_V and cause this cycle to happen again!

Expected behavior

I would expect to see no diff after step 6. The implication of step 8 is that seemingly random results are introduced into diffs, including PRs. I think setting FRSOURCE_CPVRD_V when generating the .actual.png files would solve the issue.

Additional context

I'm aware that updateImages does not have this problem, but it doesn't work for our GHA workflow: we run the tests and would like indicate which are failing (not possible with updateImages) and then automatically open a PR with new baselines. It is these PRs which contain many irrelevant diffs.

Despite our (potentially unusual) use case, I imagine all users who rename .actual.png to update their baselines are running into similar problems.

Let me know if I'm missing anything, I could look into opening a PR otherwise.

Package version

3.3.10

Device

Any

OS and version

Any

Browser and version

Any

Node version

Any

I can work on it by myself

  • Yes
  • No

Code of Conduct

  • I agree to follow this project's Code of Conduct
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant