Update yml file to run 8b tests on presubmit and 70b and 405b tests nightly

[aviator19941]

Updates yml file to run 8b tests on each pull_request and 70b and 405b tests nightly.

[ScottTodd]

Changing which tests a workflow runs based on the event trigger feels like a recipe for confusion.

• What happens with the workflow_dispatch trigger?
• Will developers know that https://github.com/nod-ai/SHARK-Platform/actions/workflows/ci-llama.yaml?query=event%3Apull_request and https://github.com/nod-ai/SHARK-Platform/actions/workflows/ci-llama.yaml?query=event%3Aschedule are testing entirely different things?

Workflow should be as simple and predictable as possible - don't overengineer this.

I would split into one workflow that runs on push and pull_request then a separate workflow that runs on schedule (and both should support workflow_dispatch for debugging).

If you're concerned about duplicating boilerplate... that needs to be fixed too. The nightly workflow should be using nightly release packages, and we need to roll out the new packaging into these workflows. Lines 59-76 should be condensed down to one line.

[aviator19941]

alright sounds good, will split into two workflow files

[ScottTodd]

If the tests fit nicely alongside other sharktank tests, whatever runs on push and pull_request could be part of https://github.com/nod-ai/SHARK-Platform/blob/main/.github/workflows/ci-sharktank.yml.

What were the original reasons for creating a separate workflow?

[aviator19941]

The original reason for creating a separate workflow was to be able to have a few of the small llama tests (8b) run on pull_request in order to test any regressions in sharktank with all the changes coming in. The larger llama tests (70b and 405b) take much longer, so would just be run nightly.

CC @saienduri

[saienduri]

Yeah let's go with two workflows due to the machine constraint (won't be able to slot nicely into ci-sharktank).

[saienduri]

The main thing is the hardcoded paths, so it would have to be it's own step that only runs when the matrix.runs_on is the mi300x machine, which I guess is fine. For the long tests, we only want to run that on a schedule/workflow_dispatch, so having that in its own workflow file makes the most sense.

[saienduri]

So, Avi we can probably add the 8b testing to the ci-sharktank.yml and the full llama testing to a new workflow file ci-sharktank-nightly.yml

[ScottTodd]

Well we shouldn't have any hardcoded paths. That needs to be fixed. Write tests first for developers to use (on any system that has the right hardware, enough disk space, etc.), then teach the CI to run them.

[saienduri]

@aviator19941 is it possible to have the 8b files in huggingface?

[saienduri]

Fully agree, but looks like there are too many moving parts/files/configurations right now (model size, data type, batch size, TP sharding, decomposed vs decodeposed attention) which makes it hard to move to huggingface quickly. Let's go with separate workflows for now (just so it doesn't halt testing coverage for llama), but the end goal here should be moving to huggingface and the ci-sharktank.yml.

[saienduri]

Can you update the reasons appropriately for all the tests?

[saienduri]

LGTM

[ScottTodd]

In the future, when making substantial changes to a pull request after obtaining approval, please re-request review. I wasn't paying attention here because there were no new comments on the review thread and the commit history is confusing (32 commits with multiple rounds of force pushing over may days). A PR with +539 −384 lines changed should at a minimum have a longer summary / commit message and go through closer review.