Add automatic triage utility #793
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
A different test case: |
gspschmid
reviewed
Oct 7, 2024
There was a problem hiding this comment.
Looks good overall, a few comments.
Algorithmic parts seem reasonable to me, but would nonetheless likely benefit from some tests :-)
Fwiw, I think this is a good target for property-based testing -- the idea being that we can randomly sample (or enumerate) some small scenarios and designate an XLA or JAX commit from which the CI will fail. So we'd mock container_exists, check_container and build_and_test and then test the logic using a model like
# Fabricate XLA and JAX commits for days a/b/c
def fake_commits(n_commits, n_days, rng):
dt = datetime.datetime('2024-10-01')
delta = datetime.timedelta(days=n_days) / n_commits
commits: list[tuple[bool, datetime, str]] = []
for i in range(n_commits):
dt += delta
is_jax = rng.coinflip()
commit = '{0:09d}'.format(i) # not really hex, but whatever
commits.append((is_jax, dt, commit))
return commits
def test_one_scenario():
commits = fake_commits(100, 10, rng(seed=123))
for bad_jax, bad_dt, bad_commit in commits:
def build_and_test(xla_commit, jax_commit):
return jax_commit >= bad_commit if bad_jax else xla_commit >= bad_commit
def check_container(date):
return ... # build_and_test(earliest xla commit >= date, earliest jax commit >= xla commit)
assert triage(commits, check_container, build_and_test) == (bad_jax, bad_commit)
olupton
force-pushed
the
olupton/triage
branch
from
561adde
to
19bc222
Compare
This is a small Python program that implements a two-stage bisection, identifying a commit in JAX or XLA that caused a test case to start failing.
olupton
force-pushed
the
olupton/triage
branch
from
f0ce788
to
62dccd1
Compare
gspschmid
previously approved these changes
Oct 9, 2024
|
My case here: And this 28a4ebf1369ce34c50dfcbdf9ec5bf5acdfa4e22 is the culprit commit! |
gpupuck
approved these changes
Oct 11, 2024
olupton
added a commit
that referenced
this pull request
Oct 22, 2024
Improve documentation of the triage tool added in #793.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a small Python program that implements a two-stage bisection, identifying a commit in JAX or XLA that caused a test case to start failing.
An example use-case was the
JetTest.test_dottest, which had been failing in the JAX unit tests on A100 for an extended period.pointing the finger at openxla/xla@a580700. The failure is worked around in jax-ml/jax#21035.