You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It is sometimes the case that a user will want to customize the validation done by the scripts when running in an automated environment to something other than executed_cacheable_tasks == 0.
A common use case is the user has one or more known cache misses they want to exclude from the assertion. More recently we had a case where someone wanted to fail Gradle experiment 1 if any tasks re-executed, but this isn't available out-of-the-box.
What I recommend doing in these scenarios is:
Do not include --fail-if-not-fully-cacheable in the validation workflows
Add a follow up step to the job that calls the API and performs the desired validation using a tool like jq to parse the response
This isn't complicated, but does require the user to parse the CSV in the .data folder for the Build Scan ID, build the request (including authorization if necessary), and make the HTTP request.
However, the scripts already call the API to grab the attributes and build cache performance build models to do its own assertions. We should consider additionally writing these responses out to the corresponding experiment directory under the .data folder such that anyone requiring a tailored validation could do so more easily.
We should also consider doing #619 at the same time as these both make the custom validation use case easier.
Finally, we should use this ticket to keep track of any other use cases.
The text was updated successfully, but these errors were encountered:
It is sometimes the case that a user will want to customize the validation done by the scripts when running in an automated environment to something other than
executed_cacheable_tasks == 0
.A common use case is the user has one or more known cache misses they want to exclude from the assertion. More recently we had a case where someone wanted to fail Gradle experiment 1 if any tasks re-executed, but this isn't available out-of-the-box.
What I recommend doing in these scenarios is:
--fail-if-not-fully-cacheable
in the validation workflowsjq
to parse the responseThis isn't complicated, but does require the user to parse the CSV in the
.data
folder for the Build Scan ID, build the request (including authorization if necessary), and make the HTTP request.However, the scripts already call the API to grab the attributes and build cache performance build models to do its own assertions. We should consider additionally writing these responses out to the corresponding experiment directory under the
.data
folder such that anyone requiring a tailored validation could do so more easily.We should also consider doing #619 at the same time as these both make the custom validation use case easier.
Finally, we should use this ticket to keep track of any other use cases.
The text was updated successfully, but these errors were encountered: