Improve CI Reliability of Generator Specs #546
Conversation
WalkthroughThe pull request introduces a configuration change in the GitHub Actions workflow file for the generator specifications. By adding Changes
Suggested Reviewers
Possibly related PRs
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (19)
Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
.github/workflows/generator.yml (1)
22-22: Consider monitoring failed combinations.The
continue-on-error: truesetting is appropriate for handling transient failures. However, to maintain visibility into these failures:
- Consider adding a post-job step to aggregate and report failed combinations
- Set up monitoring to track the frequency of failures to identify patterns
Here's a verification script to analyze the current failure patterns:
#!/bin/bash # Description: Check recent workflow runs to understand failure patterns # Get the last 20 workflow runs for the generator specs gh run list --workflow=generator.yml --limit 20 --json conclusion,createdAt,url | \ jq 'map(select(.conclusion != "success")) | map({conclusion, created_at: .createdAt, url})'
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
.github/workflows/generator.yml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (18)
- GitHub Check: Testing (ubuntu-latest, 3.4, gemfiles/Gemfile-rails.8.0.x)
- GitHub Check: Testing (ubuntu-latest, 3.3, gemfiles/Gemfile-rails.8.0.x)
- GitHub Check: Testing (ubuntu-latest, 3.3, gemfiles/Gemfile-rails.7.0.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.2, gemfiles/Gemfile-rails.7.1.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.2, gemfiles/Gemfile-rails.7.0.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.1, gemfiles/Gemfile-rails.7.2.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.1, gemfiles/Gemfile-rails.7.1.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.1, gemfiles/Gemfile-rails.7.0.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.1, gemfiles/Gemfile-rails.6.1.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.0, gemfiles/Gemfile-rails.7.1.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.0, gemfiles/Gemfile-rails.7.0.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.0, gemfiles/Gemfile-rails.6.1.x)
- GitHub Check: Generator specs (ubuntu-latest, 3.0, gemfiles/Gemfile-rails.6.0.x)
- GitHub Check: Generator specs (ubuntu-latest, 2.7, gemfiles/Gemfile-rails.7.1.x)
- GitHub Check: Generator specs (ubuntu-latest, 2.7, gemfiles/Gemfile-rails.7.0.x)
- GitHub Check: Generator specs (ubuntu-latest, 2.7, gemfiles/Gemfile-rails.6.1.x)
- GitHub Check: test
- GitHub Check: Generator specs (ubuntu-latest, 2.7, gemfiles/Gemfile-rails.6.0.x)
🔇 Additional comments (1)
.github/workflows/generator.yml (1)
18-21: Well-documented explanation for the change.The comments clearly explain the problem (random failures due to network requests) and the solution (allowing individual matrix combinations to fail without affecting others).
Generator specs experience random failures likely due to numerous network requests to the node registry. To mitigate this, `continue-on-error: true` is set, allowing the current combination to fail without halting other matrix combinations.
tagliala
force-pushed
the
chore/generator-specs-continue-on-error
branch
from
f0793b7 to
cfddf35
Compare
tagliala
changed the title
|
I don't quite see this being a particularly good idea. Most of the errors we've been seeing have been widespread so failing fast worked quite well. Current issues are widespread and not just affecting us actions/runner-images#3737 and we've not seen anything to that extent before. There might be cases down the line where some old Ruby/Rails started falling and it could be more obvious with other jobs succeeding that this is isolated, but that's not enough benefit IMO to warrant this change. |
Yes, in this particular execution a lot of failures are happening, in my tests in the fork just 2 or 3 over 24 jobs failed |
Generator specs experience random failures likely due to numerous
network requests to the node registry.
To mitigate this,
continue-on-error: trueis set, allowing the currentcombination to fail without halting other matrix combinations.
Summary by CodeRabbit