Best Practices on Maintaining Snapshot Tests

[jwedel]

Hi,

we are using snapshots a lot as we have an interactive canvas based chart. We currently have round about 50 snapshot images that we test.

It gets increasingly difficult to manage that in comparison to unit or e2e tests so I wanted to ask if others have developed some best practices maintaining a bigger number of snapshots.

Here are a couple it issues and how we solved them

Issue	Current Solution
Identifying The Test Case	We have one snapshot per it block.
Linking the test and image	We added #XY to the describe and it block to have an ordered list of images in the snapshots directory. This has the downside that when inserting new tests, we have to rename the others to keep the numbering
Identifying what the image should look like	In the test, we need to click on x/y position on the canvas and thus it is very hard to get the information about what is actually done in the test case. Therefore we added a very detailed description to the test case name like it('should highlight the temperature curve in blue when hovered over the second item and hide the pressure curve'). This is very verbose but we couldn't find a better way. This is also used to identify the should behavior when a new feature breaks all existing tests and we need to re-evaulate if it is breaking because of a regression bug or because it is supposed to be changed due to the new feature.

I would appreciate some insights on how other are dealing with that.

[simonsmith]

Linking the test and image

You can pass a file name to matchImageSnapshot. I encourage this practice on the projects I work on so they are separate from the test name, and each image can have a descriptive file name

Identifying what the image should look like

Bit puzzled on your approach here. The snapshot that is checked in is how it should look. In my experience the snapshot images should be reviewed as part of the change request, and this serves as a referece to other engineers how it should look

Then any failures are scrutinised

[jwedel]

You can pass a file name to matchImageSnapshot.

Nice, didn't know that!

Bit puzzled on your approach here. The snapshot that is checked in is how it should look. In my experience the snapshot images should be reviewed as part of the change request, and this serves as a referece to other engineers how it should look

I get your point. So yes, we use the existing image to as a reference and if the test breaks, we can see what changes. IN the case of a new feature it occurs very often that ALL tests fail. So we need to got one by one to figure our if they broke due to a regression indroduced by the new feature or of the new features was supposed to change that. To verify that, we use expressive test case names that explains on a domain level what we do and what we should see.

[simonsmith]

Hmm, but don't you run the risk of expressive names being like code comments that can go out of date?

I think there needs to be an expectation that if you make a large change to the codebase (like the body text colour for example) then there will be many failures. This is a feature, and should be embraced. Usually the engineer performing the change would be aware they are making a widespread change and can review each snapshot to ensure it looks correct.

Essentially the snapshots checked into master should always reflect exactly what the UI should look like at that time. That is more accurate and helpful than a descriptive test name/comment.

If the question is, how do we know what it should look like? I'd say the engineer and/or designer of the change should know what the intended UI will look like and then the snapshots should reflect it accordingly. I can't see a scenario where I wouldn't know what my change was supposed to look like, because I'm the one making it.

If you are having problems with a lot of unrelated changes then that might signal that the snapshots are too broad.
An example of this might be a snapshot of a Modal that fails a test because the text size in the body underneath the mask was changed in an unrelated ticket, but the snapshot can still "see" that change. In cases like this it helps to be more precise with your snapshots and only snapshot a particular element (in this case the actual Modal element) instead.

[simonsmith]

We have one snapshot per it block.

I would also recommend against this. As seen in the Cypress best practices document you should avoid writing unit style tests.

In specs that I work on it's common to have multiple calls to matchImageSnapshot in one it test. You might for example:

• open a modal
• snapshot the modal
• interact with an item inside the modal
• snapshot the modal again with the new state

Combining matchImageSnapshot with a custom filename (now that you know about it 😄 ) makes this easier

[jwedel]

Thanks for your feedback.

I know that it's the recommendation and I had a lengthy discussion about that in the Cypress Discord channel. For us, the downsides of having completely independent it blocks are a lot higher compared to the benefits. But this is probably another discussion we should not have here :)

Your suggestions are highly appreciated, we will try to incorporate them.

[simonsmith]

Cool. Interested to hear how it goes. Some form of best practices page/document might be a good addition to this repo

[jwedel]

Yes. I also like the Cypress documentation very much because it provides guidance on how it was supposed to be used and what works well. Would be great to see this added here.