Add llama-run #452

ericcurtin · 2024-11-13T19:23:28Z

And update llama.cpp and whisper.cpp.

sourcery-ai · 2024-11-13T19:23:33Z

Reviewer's Guide by Sourcery

This pull request adds a llama-cpp-python server to the container images. It updates the build scripts to install the server and modifies the model execution command to use the new server. It also updates the llama.cpp and whisper.cpp dependencies.

Sequence diagram for model execution flow changes

sequenceDiagram
    participant User
    participant CLI
    participant Server
    participant Model

    User->>CLI: Execute command
    CLI->>CLI: Parse arguments
    CLI->>Server: Initialize llama-cpp-python server
    Note over Server: New server component
    Server->>Model: Load model
    Server->>Model: Process prompt
    Model-->>Server: Generate response
    Server-->>CLI: Return results
    CLI-->>User: Display output

File-Level Changes

Change	Details	Files
Added `llama-cpp-python` server installation to build scripts.	Added `pip install \"llama-cpp-python[server]\"` to install the server. Set the `CMAKE_ARGS` environment variable to configure the build. Added a function to set the install prefix based on the container type. Updated the cmake steps to use the common flags and install prefix. Added submodule update for llama.cpp and whisper.cpp	`container-images/scripts/build_llama_and_whisper.sh`
Modified model execution command to use `llama-run`.	Replaced `llama-cli` with `llama-run`. Removed unnecessary arguments like `--in-prefix`, `--in-suffix`, `-p`, and `-cnv`. Added model path and prompt as arguments to `llama-run`. Updated the dryrun test to reflect the new command.	`ramalama/model.py` `test/system/030-run.bats`
Updated llama.cpp and whisper.cpp dependencies.	Updated the `LLAMA_CPP_SHA` and `WHISPER_CPP_SHA` arguments to the latest commit hashes.	`container-images/cuda/Containerfile` `container-images/asahi/Containerfile` `container-images/ramalama/Containerfile` `container-images/rocm/Containerfile` `container-images/vulkan/Containerfile`
Removed unnecessary redirection of stderr.	Removed the code block that redirects stderr to /dev/null.	`ramalama/common.py`
Removed conditional check for GGUF extension.	Removed the check for `.GGUF` or `.gguf` extension before setting the `exec_model_path` in the `serve` function.	`ramalama/model.py`
Added a helper script to replace SHA values.	Created a new script `replace-shas.sh` to replace the SHA values in the container files. Added functions to find and replace the SHA values using `grep`, `xargs`, and `sed`.	`scripts/replace-shas.sh`
Minor updates and cleanup.	Simplified the dnf install command by removing redundant epel-release installation. Sourced the gcc-toolset enable script using dot instead of source. Removed default prompt in `run` function. Used `inp` instead of `input` as variable name for stdin input. Added a new line after version argument in `configure_arguments` function.	`container-images/scripts/build_llama_and_whisper.sh` `ramalama/model.py` `ramalama/cli.py`

Tips and commands

Interacting with Sourcery

Trigger a new review: Comment @sourcery-ai review on the pull request.
Continue discussions: Reply directly to Sourcery's review comments.
Generate a GitHub issue from a review comment: Ask Sourcery to create an
issue from a review comment by replying to it.
Generate a pull request title: Write @sourcery-ai anywhere in the pull
request title to generate a title at any time.
Generate a pull request summary: Write @sourcery-ai summary anywhere in
the pull request body to generate a PR summary at any time. You can also use
this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

Enable or disable review features such as the Sourcery-generated pull request
summary, the reviewer's guide, and others.
Change the review language.
Add, remove or edit custom review instructions.
Adjust other review settings.

Getting Help

Contact our support team for questions or feedback.
Visit our documentation for detailed guides and information.
Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

sourcery-ai

Hey @ericcurtin - I've reviewed your changes - here's some feedback:

Overall Comments:

Please provide justification for changing the default runtime from llama.cpp to llama-cpp-python. What are the benefits that led to this decision?
Consider updating any additional documentation to reflect the new runtime option and explain the differences between the available runtimes

Here's what I looked at during the review

🟢 General issues: all looks good
🟢 Security: all looks good
🟢 Testing: all looks good
🟢 Complexity: all looks good
🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

sourcery-ai

We have skipped reviewing this pull request. It looks like we've already reviewed the commit 47fdf9a in this pull request.

ericcurtin · 2024-11-15T17:27:01Z

@cooktheryan @lsm5 @mrunalp @slp @rhatdan @tarilabs @umohnani8 @ygalblum PTAL

ygalblum

LGTM for the code.
But, I think I'm missing something. Is llama-cpp-python already a dependency?

ericcurtin · 2024-11-15T18:39:15Z

@ygalblum we probably need to push some container images before merging this, but when we do that, we should be all good.

rhatdan · 2024-12-18T20:34:09Z

ramalama/cli.py

@@ -103,7 +103,7 @@ def load_and_merge_config():
        )

    ramalama_config['carimage'] = ramalama_config.get('carimage', "registry.access.redhat.com/ubi9-micro:latest")
-    ramalama_config['runtime'] = ramalama_config.get('runtime', 'llama.cpp')
+    ramalama_config['runtime'] = ramalama_config.get('runtime', 'llama-cpp-python')


Looks like we have a different runtime for ramalama serve versus ramalama run?

For ramalama run, the only commands that are available to do that in a single process are the llama.cpp ones, like llama-cli (and llama-run which I'll be adding in this PR now that the last few bits were merged a couple of hours ago). There are other possibilities for chatbots, but the other require us to set up a server process and a client process. I think llama-run is cleaner than setting up a client and server.

As regards server there are 3 clear options, vLLM, llama.cpp and llama-cpp-python (which is just a python wrapper around llama.cpp). Although vLLM is the best enterprise option today for a server in my opinion, it lacks the diverse hardware support for local development, which leaves us the other two. llama.cpp server is a very minimal implementation of an OpenAI-compatible server as in it is barely OpenAI-compatible, last I tried it, it didn't work with OpenAI-compatible clients like: https://github.com/open-webui/open-webui

llama.cpp server might catch up as it is actively being contributed to, but it's not there today. One key feature it is missing is multi-model support, so the client normally specifies the model it wants to use.

llama-cpp-python is further ahead in terms of it's OpenAI-compatible server implementation, that's why I'm proposing we switch in this PR.

My opinion here is choice is bad. We should only offer llama.cpp and if the server tool llama-cpp-python is better, then default to using that, if a better one shows up in the future, then we could move to it, as long as it implements the Same API as llama-cpp-python. We are doing the same thing on the run side moving from llama-client to llama-run.

Bottom line is keep the selection to llama.cpp and vLLM (And whisper.cpp) and we can modify the tools used to work with the libraries under the covers. Eventually we could allow specifying the commands to run within the containers for run and serve, but that is a ways a way.

ericcurtin · 2024-12-19T14:01:55Z

Adding temperature option to llama-run here:

ggerganov/llama.cpp#10899

rhatdan · 2024-12-19T14:08:28Z

ramalama/cli.py

@@ -205,8 +205,8 @@ def configure_arguments(parser):
    parser.add_argument(
        "--runtime",
        default=config.get("runtime"),
-        choices=["llama.cpp", "vllm"],
-        help="specify the runtime to use; valid options are 'llama.cpp' and 'vllm'",
+        choices=["llama-cpp-python", "llama.cpp", "vllm"],


As stated above, do not give this choice, just stick with llama.cpp and use llama-cpp-python server for serve

ericcurtin · 2025-01-02T10:29:44Z

I reduced the scope of this. One of the main drivers for llama-cpp-python server is to attempt to find a server that's compatible with:

https://github.com/open-webui/open-webui

we probably need to explore a few options like llama-server, llama-cpp-python and https://github.com/mostlygeek/llama-swap

sourcery-ai

Hey @ericcurtin - I've reviewed your changes - here's some feedback:

Overall Comments:

The removal of stderr redirection capability in exec_cmd() could make debugging more difficult. Consider keeping this functionality unless there's a specific reason for removing it.

Here's what I looked at during the review

🟢 General issues: all looks good
🟢 Security: all looks good
🟢 Testing: all looks good
🟢 Complexity: all looks good
🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

And update llama.cpp and whisper.cpp. Signed-off-by: Eric Curtin <[email protected]>

ericcurtin · 2025-01-02T23:47:34Z

This is ready to go @rhatdan

rhatdan · 2025-01-03T18:12:38Z

LGTM

ericcurtin self-assigned this Nov 13, 2024

sourcery-ai bot reviewed Nov 13, 2024

View reviewed changes

ericcurtin marked this pull request as draft November 13, 2024 19:24

ericcurtin marked this pull request as ready for review November 15, 2024 13:02

sourcery-ai bot reviewed Nov 15, 2024

View reviewed changes

ericcurtin force-pushed the llama-cpp-python branch 5 times, most recently from c4f91bc to fe9c0ca Compare November 15, 2024 16:57

ygalblum reviewed Nov 15, 2024

View reviewed changes

ericcurtin force-pushed the llama-cpp-python branch 2 times, most recently from 1d79a3c to 4c35af3 Compare December 18, 2024 18:54

rhatdan reviewed Dec 18, 2024

View reviewed changes

ericcurtin force-pushed the llama-cpp-python branch 2 times, most recently from acb465f to 678ca99 Compare December 19, 2024 13:09

rhatdan reviewed Dec 19, 2024

View reviewed changes

ericcurtin force-pushed the llama-cpp-python branch 9 times, most recently from 07202c5 to ebf4d34 Compare December 20, 2024 12:30

ericcurtin marked this pull request as draft December 20, 2024 12:56

ericcurtin force-pushed the llama-cpp-python branch from ebf4d34 to 4a5e4d8 Compare January 2, 2025 10:26

ericcurtin force-pushed the llama-cpp-python branch 3 times, most recently from 19f64aa to 9ba28af Compare January 2, 2025 16:06

ericcurtin marked this pull request as ready for review January 2, 2025 16:06

sourcery-ai bot reviewed Jan 2, 2025

View reviewed changes

ericcurtin force-pushed the llama-cpp-python branch 3 times, most recently from 13ed742 to cba1655 Compare January 2, 2025 21:35

ericcurtin changed the title ~~Add llama-cpp-python server~~ Add llama-run Jan 2, 2025

ericcurtin force-pushed the llama-cpp-python branch 2 times, most recently from 3c26215 to 6624389 Compare January 2, 2025 22:46

Add llama-run

fb45f23

And update llama.cpp and whisper.cpp. Signed-off-by: Eric Curtin <[email protected]>

ericcurtin force-pushed the llama-cpp-python branch from 6624389 to fb45f23 Compare January 2, 2025 23:16

ericcurtin mentioned this pull request Jan 3, 2025

Update README and model.py for Nvidia GPU support #530

Merged

rhatdan merged commit 4a6ac32 into main Jan 3, 2025
12 checks passed

ericcurtin deleted the llama-cpp-python branch January 3, 2025 22:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add llama-run #452

Add llama-run #452

ericcurtin commented Nov 13, 2024 •

edited

Loading

sourcery-ai bot commented Nov 13, 2024 •

edited

Loading

Interacting with Sourcery

Customizing Your Experience

Getting Help

sourcery-ai bot left a comment

sourcery-ai bot left a comment

ericcurtin commented Nov 15, 2024

ygalblum left a comment

ericcurtin commented Nov 15, 2024

rhatdan Dec 18, 2024

ericcurtin Dec 19, 2024 •

edited

Loading

rhatdan Dec 19, 2024

ericcurtin commented Dec 19, 2024

rhatdan Dec 19, 2024

ericcurtin commented Jan 2, 2025

sourcery-ai bot left a comment

ericcurtin commented Jan 2, 2025

rhatdan commented Jan 3, 2025

Add llama-run #452

Add llama-run #452

Conversation

ericcurtin commented Nov 13, 2024 • edited Loading

sourcery-ai bot commented Nov 13, 2024 • edited Loading

Reviewer's Guide by Sourcery

Sequence diagram for model execution flow changes

File-Level Changes

Interacting with Sourcery

Customizing Your Experience

Getting Help

sourcery-ai bot left a comment

Choose a reason for hiding this comment

sourcery-ai bot left a comment

Choose a reason for hiding this comment

ericcurtin commented Nov 15, 2024

ygalblum left a comment

Choose a reason for hiding this comment

ericcurtin commented Nov 15, 2024

rhatdan Dec 18, 2024

Choose a reason for hiding this comment

ericcurtin Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

rhatdan Dec 19, 2024

Choose a reason for hiding this comment

ericcurtin commented Dec 19, 2024

rhatdan Dec 19, 2024

Choose a reason for hiding this comment

ericcurtin commented Jan 2, 2025

sourcery-ai bot left a comment

Choose a reason for hiding this comment

ericcurtin commented Jan 2, 2025

rhatdan commented Jan 3, 2025

ericcurtin commented Nov 13, 2024 •

edited

Loading

sourcery-ai bot commented Nov 13, 2024 •

edited

Loading

ericcurtin Dec 19, 2024 •

edited

Loading