Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v13.4.0 Release #5551

Merged
merged 5 commits into from
Oct 30, 2023
Merged

v13.4.0 Release #5551

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
49a4bb4
Experimental retries (#5536)
ryanpei Oct 30, 2023
aa0f943
docs: minor updates
emilyrohrbough Oct 30, 2023
d3556fd
add v13.4.0 changelog
emilyrohrbough Oct 30, 2023
8da1684
relative links
emilyrohrbough Oct 30, 2023
f5a5008
anchor link in changelog
emilyrohrbough Oct 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/guides/cloud/flaky-test-management.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,8 @@ tests by Cypress Cloud. Therefore, **enabling
[test retries](/guides/guides/test-retries#Configure-Test-Retries) is required**
to take advantage of any flaky test management feature.

If you want more control over the pass/fail result of any test deemed flaky by test retries, you may want to use [experimental test retries](/guides/references/experiments#Experimental-Flake-Detection-Features), which is available as of Cypress `13.4.0`.

:::

### Flagging Flaky Tests
Expand Down
12 changes: 10 additions & 2 deletions docs/guides/guides/test-retries.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,13 +26,21 @@ conditions that could result in unreliable tests include:
- Resource dependencies availability
- Network issues

With test retries, Cypress is able to retry failed tests to help reduce test
flakiness and continuous integration (CI) build failures. By doing so, this will
With test retries, Cypress is able to retry failed tests to help [detect test
flakiness](/guides/cloud/flaky-test-management#Flake-Detection) and continuous integration (CI) build failures. By doing so, this will
save your team valuable time and resources so you can focus on what matters most
to you.

## How It Works

:::info

You may want to use the [experimental retries](/guides/references/experiments#Experimental-Retries) feature which offers more options for [detecting flaky tests](/guides/cloud/flaky-test-management#Flake-Detection). This experimental feature is available as of Cypress `13.4.0`.

Test retries were originally intended to give failing tests more chances to "pass" (for instance, if CI environments for testing are unreliable), while still being determined as flaky if they do eventually pass. However this may not be the desired result in all cases. Experimental retries give you control over the conditions of the pass or fail result.

:::

By default, tests will not retry when they fail. You will need to
[enable test retries in your configuration](#Configure-Test-Retries) to use this
feature.
Expand Down
12 changes: 12 additions & 0 deletions docs/guides/references/changelog.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,18 @@
title: Changelog
---

## 13.4.0

_Released 10/30/2023_

**Features:**

- Introduced experimental configuration options for advanced retry logic: adds `experimentalStrategy` and `experimentalOptions` keys to the `retry` configuration key. See [Experimental Flake Detection Features](/guides/references/experiments/#Experimental-Flake-Detection-Features) for more information. Addressed in [#27930](https://github.com/cypress-io/cypress/pull/27930).

**Bugfixes:**

- Fixed a regression in [`13.3.2`](#13.3.2) where Cypress would crash with 'Inspected target navigated or closed' or 'Session with given id not found'. Fixes [#28141](https://github.com/cypress-io/cypress/issues/28141) and [#28148](https://github.com/cypress-io/cypress/issues/28148).

## 13.3.3

_Released 10/24/2023_
Expand Down
39 changes: 20 additions & 19 deletions docs/guides/references/configuration.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ default values.
| `redirectionLimit` | `20` | The number of times that the application under test can redirect before erroring. |
| `reporter` | `spec` | The [reporter](/guides/tooling/reporters) used during `cypress run`. |
| `reporterOptions` | `null` | The [reporter options](/guides/tooling/reporters#Reporter-Options) used. Supported options depend on the reporter. |
| `retries` | `{ "runMode": 0, "openMode": 0 }` | The number of times to retry a failing test. Can be configured to apply to `cypress run` or `cypress open` separately. See [Test Retries](/guides/guides/test-retries) for more information. |
| `retries` | `{ "runMode": 0, "openMode": 0 }` | The number of times to retry a failing test. Can be configured to apply to `cypress run` or `cypress open` separately. As of Cypress `13.4.0`, the experimental Flake Detection strategy can also be configured. See [Test Retries](/guides/guides/test-retries) for more information. |
| `watchForFileChanges` | `true` | Whether Cypress will watch and restart tests on test file changes. |

### Timeouts
Expand Down Expand Up @@ -716,24 +716,25 @@ DEBUG=cypress:cli,cypress:server:specs

## History

| Version | Changes |
| --------------------------------------------- | ------------------------------------------------------------------------------------- |
| [13.0.0](/guides/references/changelog#13-0-0) | Removed `nodeVersion` option. |
| [13.0.0](/guides/references/changelog#13-0-0) | Removed `videoUploadOnPasses` option. |
| [11.0.0](/guides/references/changelog#11-0-0) | Removed `e2e.experimentalSessionAndOrigin` option. |
| [10.4.0](/guides/references/changelog#10-4-0) | Added `e2e.testIsolation` option. |
| [10.0.0](/guides/references/changelog#10-0-0) | Reworked page to support new `cypress.config.js` and deprecated `cypress.json` files. |
| [8.7.0](/guides/references/changelog#8-7-0) | Added `slowTestThreshold` option. |
| [8.0.0](/guides/references/changelog#8-0-0) | Added `clientCertificates` option and removed `firefoxGcInterval` configuration. |
| [7.0.0](/guides/references/changelog#7-0-0) | Added `e2e` and `component` options. |
| [7.0.0](/guides/references/changelog#7-0-0) | Added `redirectionLimit` option. |
| [6.1.0](/guides/references/changelog#6-1-0) | Added `scrollBehavior` option. |
| [5.2.0](/guides/references/changelog#5-2-0) | Added `includeShadowDom` option. |
| [5.0.0](/guides/references/changelog#5-0-0) | Added `retries` configuration. |
| [5.0.0](/guides/references/changelog#5-0-0) | Renamed `blacklistHosts` configuration to `blockHosts`. |
| [4.1.0](/guides/references/changelog#4-12-0) | Added `screenshotOnRunFailure` configuration. |
| [4.0.0](/guides/references/changelog#4-0-0) | Added `firefoxGcInterval` configuration. |
| [3.5.0](/guides/references/changelog#3-5-0) | Added `nodeVersion` configuration. |
| Version | Changes |
| --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| [13.4.0](/guides/references/changelog#13-4-0) | Added support for configuring the Experimental Flake Detection strategy via `retries.experimentalStrategy` and `retries.experimentalOptions`. |
| [13.0.0](/guides/references/changelog#13-0-0) | Removed `nodeVersion` option. |
| [13.0.0](/guides/references/changelog#13-0-0) | Removed `videoUploadOnPasses` option. |
| [11.0.0](/guides/references/changelog#11-0-0) | Removed `e2e.experimentalSessionAndOrigin` option. |
| [10.4.0](/guides/references/changelog#10-4-0) | Added `e2e.testIsolation` option. |
| [10.0.0](/guides/references/changelog#10-0-0) | Reworked page to support new `cypress.config.js` and deprecated `cypress.json` files. |
| [8.7.0](/guides/references/changelog#8-7-0) | Added `slowTestThreshold` option. |
| [8.0.0](/guides/references/changelog#8-0-0) | Added `clientCertificates` option and removed `firefoxGcInterval` configuration. |
| [7.0.0](/guides/references/changelog#7-0-0) | Added `e2e` and `component` options. |
| [7.0.0](/guides/references/changelog#7-0-0) | Added `redirectionLimit` option. |
| [6.1.0](/guides/references/changelog#6-1-0) | Added `scrollBehavior` option. |
| [5.2.0](/guides/references/changelog#5-2-0) | Added `includeShadowDom` option. |
| [5.0.0](/guides/references/changelog#5-0-0) | Added `retries` configuration. |
| [5.0.0](/guides/references/changelog#5-0-0) | Renamed `blacklistHosts` configuration to `blockHosts`. |
| [4.1.0](/guides/references/changelog#4-12-0) | Added `screenshotOnRunFailure` configuration. |
| [4.0.0](/guides/references/changelog#4-0-0) | Added `firefoxGcInterval` configuration. |
| [3.5.0](/guides/references/changelog#3-5-0) | Added `nodeVersion` configuration. |

## See also

Expand Down
Loading