Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interop score for interop-2023-motion [stable] seems nonsense #201

Open
gsnedders opened this issue Nov 15, 2023 · 1 comment
Open

Interop score for interop-2023-motion [stable] seems nonsense #201

gsnedders opened this issue Nov 15, 2023 · 1 comment
Assignees

Comments

@gsnedders
Copy link
Member

Currently the Motion Path [stable] graph looks like:

Screenshot 2023-11-15 at 11 50 18

At the beginning of the year, the Interop score was higher than the Firefox or Chrome score, which seems impossible—54.7% of tests cannot pass in all three browsers if one browser only passes 44.3%.

@DanielRyanSmith
Copy link
Contributor

I just did some investigating on this. It is caused by the number of tests in the focus area fluctuating over time (presumably adding new tests later in the year). The experimental chart shows the same strange interop score, which corrects itself sometime in May. Before that time, the focus area was scored on 73 individual tests. And after May, the focus area moved to using 93 tests, which is nearly the number that is used today.

The likely way to circumvent this is to calculate the interop scores at the end of the entire yearly calculation, averaging by the number of tests present in the final runs, rather than the tests present in the runs of that day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants