J^e⁄_ust Retry

Motivation

E2E test failures due to reasons outside the domain of source code, including:

1. Network timeouts
2. Flaky endpoints etc.
3. Login issues
4. Flaky selenium selectors (bad code which is hard to diagnose)
5. Whatever other mishaps, etc.

Setup

Create a run-config to pass into jest-retry

Example run

require('jest-retry')({
    debug: false,
    testDir: 'src/tests',
    testFilter: 'table-tests',
    // jest-junit is currently the only option
    outputTestResults: true,
    testResultsOutput: '.',

    /* Test retry options */
    flakyTestMock: false,
    flakyTestMockDir: 'src/mocks',
    flakyFailureMessages: ['Network timeout'],
    
    // Matches to test failures `testFilePath` 
    // These are still marked as fail, but does not contribute to jest exiting with failure status
    knownIssues: ['src/test/known-issue.test'],
    
    flakyNumRetries: 2, // 0 for disable retry pattern or Infinity for continue running until getting same results
       
    flakyMarkAll: false,
    flakyWaitBeforeRerun: 1000,
    // Jest options
    setupTestFrameworkScriptFile: 'jest-config.js',
    maxWorkers: 1,
    noStackTrace: false,
    silent: false,
});

Test

npm run test runs a short 2 test suite in which a failure occurs and is forced to pass on the second retry

Test output

☁  jest-retry [master] npm run test

> [email protected] test /code/jest-retry
> ./test/run.js

 PASS  test/always-pass.test.js
 FAIL  test/conditional-fail.test.js
  ● mocks/conditional-fail › conditionally fails

    expect(received).toBe(expected) // Object.is equality

    Expected value to be:
      false
    Received:
      true

      1 | describe('mocks/conditional-fail', async () => {
      2 |   it('conditionally fails', async () => {
    > 3 |     if (!process.env.SKIP) expect(true).toBe(false);
      4 |   });
      5 | });
      6 |

      at Object.it (test/conditional-fail.test.js:3:41)

Test Suites: 1 failed, 1 passed, 2 total
Tests:       1 failed, 1 passed, 2 total
Snapshots:   0 total
Time:        0.28s, estimated 1s
Ran all test suites.

flakyDictionaryCount: {
  "Expected value to be": 1
}

Retrying the following test suites:  [
  "/code/jest-retry/test/conditional-fail.test.js"
]
===============================================

 PASS  test/conditional-fail.test.js
  mocks/conditional-fail
    ✓ conditionally fails

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        0.099s, estimated 1s
Ran all test suites.

All failures have now passed

Test retries result: true