Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wpt run should be able to run only tests in Interop #48603

Open
gsnedders opened this issue Oct 14, 2024 · 1 comment
Open

wpt run should be able to run only tests in Interop #48603

gsnedders opened this issue Oct 14, 2024 · 1 comment
Labels
wptrunner The automated test runner, commonly called through ./wpt run

Comments

@gsnedders
Copy link
Member

gsnedders commented Oct 14, 2024

We should add a number of command line arguments to wpt run to make it possible to select the subset of tests in Interop.

There's a few potential variants here:

  • Running tests with a given label,
  • Running tests in a given Interop category (which is formed of a number of labels),
  • Running all tests in Interop 20xx.

The data is basically split between https://github.com/web-platform-tests/results-analysis/blob/main/interop-scoring/category-data.json and https://github.com/web-platform-tests/wpt.fyi/blob/main/webapp/static/interop-data.json, and we can get all the labels from https://wpt.fyi/api/metadata?includeTestLevel=true&product=chrome rather than manually re-parsing all of https://github.com/web-platform-tests/wpt-metadata.

(Arguably this can already be done by writing a script to enumerate all the tests in one of those variants and put them in a file and then just run with --include-file, but it would be easier to just have this all integrated into wpt run.)

@gsnedders gsnedders added the wptrunner The automated test runner, commonly called through ./wpt run label Oct 14, 2024
@WeizhongX
Copy link
Contributor

I would prefer to check in the list of tests from interop then use --include-file. I think Wptrunner's command line needs some cleanup, instead of adding more, :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wptrunner The automated test runner, commonly called through ./wpt run
Projects
None yet
Development

No branches or pull requests

2 participants