ci: enables ollama integration tests

[codefromthecrypt]

This adds CI for the existing ollama tests.

The model we use, currently mistral-nemo is somewhat large, so not helpful to run in docker for a couple reasons. Instead, we nohup ollama directly.

• slower execution through containerization
• more complicated cache setup, as you need to manage the model, which is pulled in docker

To reduce problems, pulling and smoke testing the model are done in a separate step, logging on failure. This gives us a high confidence that when the tests actually run, failures are related to tests, not slow first-time model startup hitting some timeout. All that said, the tests themselves do take time as it is using real inference now.

[codefromthecrypt]

Lemme know if we are brave enough to run this in CI. One reason not to is that it could be flakey.

[michaelneale]

always brave enough for CI.

[codefromthecrypt]

Made tests conditional on OLLAMA_HOST and pull the model we need.

Note: docker/testcontainers is not a viable option as the models are slow to start (especially in docker which disallows GPU access), and also eat a lot of memory. I think if we want CI, we should just start ollama in github action. If sounds good, 👍 and I'll give it a go.

[codefromthecrypt]

oops sorry about red I'll sort it later.

[codefromthecrypt]

🤞 on CI, I am only running it in latest python, after normal unit tests.

[codefromthecrypt]

last try on CI today 🤞

[codefromthecrypt]

notes for tomorrow:

• 1 of 4 passed. so look more closely at why that could be the case.
• double check the cache actually works. i used the key based on ollama defaults

[codefromthecrypt]

put back to draft until I sort out what's going on in CI. Feel free to edit this PR if someone wants to finish before I respond again

[codefromthecrypt]

@k33g has some interesting options in https://github.com/parakeet-nest/awesome-slms trying again with testcontainers may not be a terrible idea. I'm out of time for the moment so anyone can give a go if they like.

[codefromthecrypt]

@anuraaga if you want to do another drive-by review, more than welcome!

[codefromthecrypt]

confusing part is the log here ends up saying "hey I'm running" when well it isn't ;) I used act locally to figure it out.

act pull_request -P ubuntu-latest=ghcr.io/catthehacker/ubuntu:act-latest --container-architecture linux/amd64

[codefromthecrypt]

been pretty consistently 5m all in

[anuraaga]

This is pretty awesome

[codefromthecrypt]

thanks again for the feedback @anuraaga this is better

[anuraaga]

LGTM (seems possible for repo to disable grey check drive by approvals, just learned that 😀)

[michaelneale]

I like this as focused and self contained (and means can use the check as a matrix for ollama support)

[codefromthecrypt]

PTAL, I added a commit I can revert that makes the test a lot faster by using qwen2.5's smallest model. I did this in a way that leaves the default mistral. Idea being that we don't spend 5mins for the few tests we have now. Let's see.

[codefromthecrypt]

down to 1m20s

As the default model is for goose anyway, my thinking is that we only test with a model needed to pass tests here. In goose, when we make an integration test, we can use the larger model as well. The main tradeoff is someone could add a typo to the default model, and the PR check wouldn't know about it, but that's the case in the other providers anyway.

[codefromthecrypt]

I'm also happy to make a constant in ollama.py OLLAMA_TEST_MODEL, re-used for these two tests. That way we don't need to define it in CI. OTOH, I didn't do that so far because I think ad-hoc can still use the large model, and that's a good thing.