Filter or grep specific tests to run (like mocha grep pattern)

[hannah06]

Current behavior:

Run specific files but cannot choose specific single/multi tests.

Desired behavior:

Choose expected tests to run, like mocha grep pattern.
Though we put a group of tests in a spec file, but sometimes we just want to run parts of them, so we need filter them out.

[jennifer-shehane]

From mocha docs:

-g, --grep <pattern> only run tests matching <pattern>

The --grep option when specified will trigger mocha to only run tests matching the given pattern which is internally compiled to a RegExp.

Suppose, for example, you have “api” related tests, as well as “app” related tests, as shown in the following snippet; One could use --grep api or --grep app to run one or the other. The same goes for any other part of a suite or test-case title, --grep users would be valid as well, or even --grep GET.

described ('api', function() {
  describe('GET /api/users', function() {
    it('respond with an array of users', function() {
      // ...
    });
  });
});

describe('app', function() {
  describe('GET /users', function() {
    it('respond with an array of users', function() {
      // ...
    });
  });
});

[kuceb]

I personally like the idea of tagging tests, because you don't have to worry about organization as much. It would be nice to find all the tests related to something using only the concept of Tagging, as explained in the mocha wiki below:

https://github.com/mochajs/mocha/wiki/Tagging

Tagging

visionmedia edited this page on Oct 10, 2012 · 2 revisions
Mocha's --grep feature may be used both on the client (via ?grep=) and server-side. Recent releases of Mocha allow you to also click on the suite or test-case names in the browser to automatically grep them. The concept of Tagging utilizes regular grepping, however may be a useful way to keep related tests in the same spot, while still conditionally executing them.

A good example of this is if you wanted to run slow tests only before releasing, or periodically. You could use any sequence of characters you like, perhaps #slow, @slow to tag them as shown here:

describe('app', function(){
  describe('GET /login', function(){
    it('should respond with the login form @fast', function(){
      
    })
  })

  describe('GET /download/:file', function(){
    it('should respond with the file @slow', function(){
      
    })
  })
})

To execute fast tests only then you may do --grep @fast. Another alternative is to only tag @slow, and utilize --grep @slow --invert to invert the grep expression.

[MaaikeFox]

+1!
We would really be helped with this feature. We don't use cucumber so we can not use the tag feature for that. And we want to stay close to the original Cypress without hacking things in ourselves.

We have a complex system with several services and applications working together. Cypress covers the End-to-end tests on all of this together. So we want to be able to select groups of tests on CI (Gitlab) to not have to run all tests every time but only a relevant selection of the tests.

[fakiolinho]

This would be a super useful feature especially when we are testing a big app so we want to split our e2e tests somehow into suites after tagging them accordingly.

[jesusochoahelp]

This would be really useful. Being able to only run smoke test TCs from different files makes organization easier.

[rachelsevier]

This would be so useful!

[ziyailkemerogul]

I've been searching for this nice feature in Cypress and found out this post, thanks to @hannah06 .

Here are my thoughts on this feature:

I have lots of test cases with different priorities and I want to tag them according to their priorities. Actually, it's not for just a priority like; "Critical", "High", "Medium", "Low", I can use tags like "Functional", "Non-Functional", "Smoke", "UI", etc. So, I can group and run my tests, how I want or how I structure. Sometimes I just want to be able to run only the "Critical" cases, or "Critical" and "UI" cases. By this way, we will be able to structure more dynamic tests.

[xeger]

I'm using cypress-cucumber-preprocessor which translates Gherkin to JS and supports Cucumber tag expressions. It relies on the TAGS env variable to provide a tag filter.

For those using pure JS, you could emulate this behavior by wrapping the Mocha it method:

// check Cypress.env('TAGS') which might be e.g. "@foo and not @baz"
withTags('@foo @bar').it('works well', () => { cy... } )

The cucumber-js tag helper API does all of the hard parsing/matching and could be plugged into this solution.

If you wanted something more magical feeling, you could write a preprocessor plugin that scans the file for magic comment lines or special describe blocks or something. (Probably overkill!)

[jweingarten]

Here is my workaround:

Test example:

import TestFilter from '../../support/util/test-filter';

describe('Dashboard', function() {
  TestFilter.any(['smoke', 'pr'], () => {
    it('loads and validates dashboard page', function() {
      cy.dspVisit('/app/dashboard/list');
      cy.PageObjects.DashboardPage().validate();
    });
  });
});

TestFilter class:

const TestFilter = {
  any(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (found) {
      testFn();
    }
  },

  notIn(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (!found) {
      testFn();
    }
  },

  all(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');

    if (
      tags
        .map((tag) => {
          return tags.length === selectedTags.length && selectedTags.indexOf(tag) !== -1;
        })
        .reduce((acc, cur) => acc && cur, true)
    ) {
      testFn();
    }
  }
};

export default TestFilter;

And you run your tests by passing in TEST_TAGS.

[wjcheng1]

any updates on this? would love to have this. thanks!

[StormPooper]

I ended up with this after experimenting with it today, coincidentally. It will look through the entire suite and skip anything that doesn't match your tags, including the parents if there's no tests that need to run - plus it doesn't need anything importing in individual tests, which is nice too. I've simplified it slightly from our production code, so let me know if there are any mistakes.

Install the cucumber-tag-expressions node module and add or import this in your /support/index.js:

import { TagExpressionParser } from 'cucumber-tag-expressions';
const tagParser = new TagExpressionParser();

before(function() {
	this.test.parent.suites.forEach(checkSuite);
});

const shouldSkip = test => {
	const tags = Cypress.env('tags');
        if(!tags) return;
	const tagger = tagParser.parse(tags);
	return !tagger.evaluate(test.fullTitle());
};

const checkSuite = suite => {
	if (suite.pending) return;
	if (shouldSkip(suite)) {
		suite.pending = true;
		return;
	}
	(suite.tests || []).forEach(test => {
		if (shouldSkip(test)) test.pending = true;
	});
	(suite.suites || []).forEach(checkSuite);
};

Then you can use it in your tests by setting a tags environment variable in Cypress (I do this with npm-run-all and it's argument placeholders within our script definitions, but you can use any method you like to set the tags). The variable should be a string using the Cucumber tag expressions syntax format, so "@foo and @bar" for example.

You use the tags in your test names, like so:

describe('set of tests @foo', () => {
	it('test @bar', () => {...});
	it('another test @baz', () => {...});
	describe('sub-suite @xyzzy', () => {...});
});

[bahmutov]

if you use JavaScript specs, take a look at the plugin https://github.com/bahmutov/cypress-select-tests that allows one to select tests to run by filename or by substring

 ## run tests with "works" in their full titles
 $ npx cypress open --env grep=works
 ## runs only specs with "foo" in their filename
 $ npx cypress run --env fgrep=foo
 ## runs only tests with "works" from specs with "foo"
 $ npx cypress run --env fgrep=foo,grep=works
 ## runs tests with "feature A" in the title
 $ npx cypress run --env grep='feature A'

[x-yuri]

@jennifer-shehane or anybody else, how do I make use of the --grep switch?

$ ./node_modules/.bin/cypress run -g 'some pattern'                                                                             
                                                                                                                
  error: unknown option: -g
...

[jennifer-shehane]

@x-yuri It is not supported. This issue is requesting this to be added as a feature. You could use this plugin today as a workaround: https://github.com/bahmutov/cypress-select-tests

[deshdeep-airwallex]

Here is my workaround:

Test example:

import TestFilter from '../../support/util/test-filter';

describe('Dashboard', function() {
  TestFilter.any(['smoke', 'pr'], () => {
    it('loads and validates dashboard page', function() {
      cy.dspVisit('/app/dashboard/list');
      cy.PageObjects.DashboardPage().validate();
    });
  });
});

TestFilter class:

const TestFilter = {
  any(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (found) {
      testFn();
    }
  },

  notIn(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');
    const found = tags.some((r) => selectedTags.indexOf(r) >= 0);

    if (!found) {
      testFn();
    }
  },

  all(tags, testFn) {
    const selectedTags = (Cypress.env('TEST_TAGS') || '').split(';');

    if (
      tags
        .map((tag) => {
          return tags.length === selectedTags.length && selectedTags.indexOf(tag) !== -1;
        })
        .reduce((acc, cur) => acc && cur, true)
    ) {
      testFn();
    }
  }
};

export default TestFilter;

And you run your tests by passing in TEST_TAGS.

This looks good and works. How I can write a CLI script for it to filter in Jenkins job
"cypress-webpr": "cypress run --record --key $CYPRESS_KEY --parallel --spec \"cypress/integration/**/!(*.wip).spec.js\" --env TEST_TAGS=pr -b chrome"
Does this looks right? it seems to ignore the --env variable TEST_TAGS in CI

[smart625]

@jennifer-shehane any updates on this? It's very helpful for our project. Thanks!

[tomardern]

I've managed to do this in a simpler way:

  on('file:preprocessor', file => {
    if (config.env.specname && file.filePath.indexOf(config.integrationFolder) > -1) {
      const contents = fs.readFileSync(file.filePath, 'utf8');
      const modified = contents.replace(
        /(^\s.?)(it)\((.*$)/gim,
        (a, b, c, d) => `${b}it${RegExp(config.env.specname, 'gmi').test(d) ? '' : '.skip'}(${d}`
      );
      fs.writeFileSync(file.outputPath, modified, 'utf8');
    }
    return Promise.resolve(file.outputPath);
  });

To use:
cypress run --env specname='NAME HERE'

This will add '.skip' to any tests which do not include the specname defined in the environment variable

[x-yuri]

If you want to temporarily focus on a couple of tests:

describe('...', function() {
    specify('test1', () => {
        console.log('test1');
    });
    specify('test2', () => {
        console.log('test2');
    });
    const t3 = specify('test3', () => {
        console.log('test3');
    });
    console.log(t3.fullTitle());
    mocha.grep(/test[12]/);
});

[lulubobst]

Please add tagging

[Rulexec]

My current solution without wrapping test code at all:

// support/index.js

let testFilter;

global._testFilter = function(x) {
	if (typeof x === 'function') {
		testFilter = x;
	} else {
		// `x` must be RegExp
		testFilter = function({ fullTitle }) {
			return x.test(fullTitle);
		};
	}
};

// Replace `context`/`it` to collect full title and filter them

let suiteNames = [];

let oldContext = global.context;
global.context = function(name) {
	suiteNames.push(name);

	let result = oldContext.apply(this, arguments);

	suiteNames.pop();

	return result;
};

let oldIt = global.it;
global.it = function(name) {
	if (!testFilter) return oldIt.apply(this, arguments);

	let fullTitle = suiteNames.join(' ') + ' ' + name;

	if (!testFilter({ fullTitle })) return;

	return oldIt.apply(this, arguments);
};

Then in the beginning of the test file simply add something like _testFilter(/user should be/);. It is possible to read regexp from environment variable in the support file, if you want.

But there anyway will be empty contexts.

[bahmutov]

Proposal

When Cypress runner finishes collecting tests its creates a single object and then starts running the tests. We can insert an async operation into this gap. We can pass the tree of collected tests to the user's async function in the plugins file. The user can filter the tests by name in any way desired: using CLI arguments or by looking up which tests to run via API requests, or by reading the names of the tests from a file.

// plugins file
// run all tests
on('filter:tests', (rootSuite) => {
  // rootSuite
  //   tests: [test objects]
  //   suites: [recursive suites]
  // each test object has "title"
  return Promise.resolve(rootSuite)
})

or run just the first test from the root suite

on('filter:tests', (rootSuite) => {
  rootSuite.suites.length = 0
  rootSuite.tests.length = 1
  return Promise.resolve(rootSuite)
})

The runner code on receiving an object from 'filter:tests' callback with filtered tests will go through its normalized tree of tests and remove any tests that are NOT in the returned tree of tests.

Inspiration: run-time filtering of Mocha's tests https://glebbahmutov.com/blog/filter-mocha-tests/

[jimpriest]

Would like to see this as well. I'm converting some of my Robot Framework tests over and am missing this feature. I don't think it should be tied to a file name as I want to be able to change 'tags' without renaming things.

[thviQit]

@UncleGus
We use gulp to compile the TypeScript into javascript, and the script is written to be used with types. Going to try the new functionality in Cypress 4.4, where it's not necessary to compile to javascript, it's going to be nice!

Anyway. The transpiled version looks like this:

// support/filterTestByTags.js
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
(function () {
    var _a, _b;
    if (!Cypress.env('tags') && !Cypress.env('exclTags')) {
        return;
    }
    var envTags = (_a = Cypress.env('tags')) !== null && _a !== void 0 ? _a : '';
    var envExclTags = (_b = Cypress.env('exclTags')) !== null && _b !== void 0 ? _b : '';
    var hasTags = envTags !== '';
    var hasExclTags = envExclTags !== '';
    // Don't filter if both is defined. We do not know what is right
    if (hasTags && hasExclTags) {
        console.log('Both tags and excluding tags has been defined. Not filtering');
        return;
    }
    var tags = hasTags ? envTags.split(' ') : [];
    var exclTags = hasExclTags ? envExclTags.split(' ') : [];
    var orgIt = it;
    var filterFunction = hasTags ? onlyWithTags : onlyWithoutTags;
    var filteredIt = filterFunction;
    filteredIt.skip = orgIt.skip;
    filteredIt.only = orgIt.only;
    filteredIt.retries = orgIt.retries;
    it = filteredIt;
    function onlyWithTags(title, fn) {
        if (tags.find(function (t) { return title.indexOf(t) === -1; })) {
            fn = null;
        }
        orgIt(title, fn);
    }
    function onlyWithoutTags(title, fn) {
        if (exclTags.find(function (t) { return title.indexOf(t) !== -1; })) {
            fn = null;
        }
        orgIt(title, fn);
    }
})();

// support/index.js
"use strict";
Object.defineProperty(exports, "__esModule", { value: true });
require("./commands");
require("./filterTestsByTags");

If you are trying to avoid the types, your code should probably look something like that.
Remember to add the tags in the --env parameter when running Cypress as shown in #1865 (comment) or nothing happens.

[UncleGus]

Hmm, no luck with the transpiled code. It is still just executing all the tests. And switching back to Typescript in v4.4.1 still gets the same error:
Error: Cannot find module '/filterTestsByTags' from '/home/.../cypress/support'

[UncleGus]

I've just converted my test spec file to Typescript, added the tsconfig.json and all that, and it's still not working. I must be missing something. I also have a number of errors/warnings in the filter file, because the two functions' parameters are not typed, but I don't think that's a significant issue.

[kirill-golovan]

@jennifer-shehane does Cypress dashboard count skipped tests as a record?

[KMKoushik]

@jennifer-shehane Is someone working on this feature?

[jennifer-shehane]

@kirill-golovan No, skipped tests are not counted as part of your Billing.

This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

[wilsoncd35]

Thanks for this. Sharing my use case, if it helps.

We tag our mocha tests that are proving known bugs. We want to keep the test but still allow the pipeline to pass.

# Run tests except known bugs and improvements.
mocha --grep '(@bug|@improve)' --invert test/foo.test.js

[jimpriest]

Ugh. Still no work on this? What do people do in CI to filter their tests? Really missing this after moving from Robot Framework :(

[x-yuri]

#1865 (comment)
there are probably other solutions in the issue

[filiphric]

Ugh. Still no work on this? What do people do in CI to filter their tests? Really missing this after moving from Robot Framework :(

@jimpriest I have written up a solution that I use for grepping my tests, maybe it’ll help.

[jimpriest]

i may have found a solution using cypress-tags - https://github.com/annaet/cypress-tags

[meseguerfantasy]

any update @jennifer-shehane ?

[VickyLund]

This feature will add a lot of value to us. If we are able to add multiple tags then each of our team can slice the tests based on their requirements.

[midleman]

@thviQit thanks for sharing your example. it works for me... except if i have a before (or after) block in the spec file, it will still run the code in the before block even if all it blocks are being skipped (aka not tagged). is there a possible workaround/solution for this?

[marsimeau]

For anyone using @thviQit's workaround, if you want to use it.only and it.skip you have to rewrite those functions as they use the global it function internally, but it will break because it's not the original one.

After looking at Mocha's source, they're pretty straight forward to replicate:

filteredIt.skip = (title) => originalIt(title);
filteredIt.only = (title, fn) => {
  const test = originalIt(title, fn);
  test.parent.appendOnlyTest(test);
};

This has worked pretty well for me.

[sharmilajesupaul]

I don't mean to nit pick here, but I'd like to disambiguate between test tags and grep. For grep, which mocha supports using --grep & what seems what this plugin does too: https://github.com/bahmutov/cypress-select-tests parsing the test title/filename is all fine and good. It would be great if Cypress could support this! That would be a great help.

Exposing a similar --grep flag in Cypress could be great. It could be used like a tag if it were something like this:

// npx cypress run --grep=#desktop

it('does something #desktop', () => {...}) // should run
it('does something else', () => {...}) // should not run

But for tagging or specifically using test tags (I often compare this to the rspec implementation of test tagging). It would be great if there was an API that didn't need to rely on parsing test title strings. This is so that in a large repo with many thousands of tests, we don't run the risk of accidentally grabbing things that might look like tags but are not intended to be.

If the test block itself could support an array / object of tags, eg.

// npx cypress run --tag desktop

// the API is clear about what those strings are intended for
it('does something', () => {}, { tags: ['desktop', 'slow' ] })

// or alternatively
it('does something', () => {}, { tags: {  desktop: true, slow: true } })

[iljapavlovs]

Considering that this is issue is from 2018 and Cypress is a paid tool as well, how come this was not yet implemented with so many upvotes??

[bahmutov]

@iljapavlovs the Cypress test runner is free open-source MIT-licensed tool https://docs.cypress.io/faq/questions/general-questions-faq#Is-Cypress-free-and-open-source

Right now this is not part of the core, but I think https://github.com/bahmutov/cypress-grep is doing pretty much everything one needs to grep by test title or tags. Please try using this plugin, and if the feedback is positive we can think about moving this into the core.

# run all tests with "hello" in their title
$ npx cypress run --env grep=hello
# run all tests with "hello world" in their title
$ npx cypress run --env grep="hello world"
# run all tests WITHOUT "hello world" in their title
$ npx cypress run --env grep="-hello world"

# enable the tests with tag "one" or "two"
$ npx cypress run --env grepTags="one two"
# enable the tests with both tags "one" and "two"
$ npx cypress run --env grepTags="one+two"
# enable the tests with "hello" in the title and tag "smoke"
$ npx cypress run --env grep=hello,grepTags=smoke

[jimpriest]

Nice! I haven't looked at this plugin in awhile and didn't know it now supported tags!
+1 for adding this to the core!

FWIW Think it would be useful to edit the initial description to mention tags.

[danmooney]

+1 running a single it without any code modification should be supported natively

[petergeczi931]

+1 upvote for this to be supported natively

[danmooney]

It's really hard to fathom this isn't supported. When you work on adding a test to an existing file, you have to run the whole file to verify the result? Anyone who uses or works on Cypress knows this is a huge miss.