Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LLM Inference] MNN-LLM: llm on device inference framework #73

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[LLM Inference] MNN-LLM: llm on device inference framework #73

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -994,6 +994,12 @@
![](https://img.shields.io/github/stars/bentoml/OpenLLM.svg)
* 简介：支持将要部署的LLM连接多个适配器，可以实现只使用一个底座模型来执行多个特定的任务；支持量化推理和LangChain集成。但对批处理和分布式推理的支持相对不友好。

* MNN LLM

* 地址：https://github.com/alibaba/MNN
![](https://github.com/alibaba/MNN.svg)
* 简介：主要支持不同端侧设备上高效推理，包括移动设备（iOS或Android设备）、PC设备等的高效推理。支持CPU/GPU加速，支持加载多个Lora模型。使用教程：https://mnn-docs.readthedocs.io/en/latest/transformers/llm.html

* MLC LLM

* 地址：https://github.com/mlc-ai/mlc-llm
Expand Down