Langchain Support for Onnx Llama

[Anindyadeep]

Feature request

Recently Microsoft have announced their first iteration of running Llama using Onnx format. Hence it will be awesome if LangChain comes up with an early support for Onnx runtime models.

Motivation

There are two reasons for this.

1. Onnx has always been the standard for running inference in CPU / GPU (Onnx GPU), so this idea of providing LLMs supported for Onnx runtime format will move forward fast
2. Current implementations of running the same is an overhead, LangChain can provide the abstraction easily.

Your contribution

I can try starting out to experiment with this of whether we can implement or not by using the existing LLM interface. However the bottleneck here becomes the .onnx format weights which are to be requested to Microsoft. I filled out the application, waiting for approval. Let me know if we can work on this issue.

[clemlesne]

Strongly approve :)

[Anindyadeep]

Awesome, although facing a common issue right now of onnxruntime.capi.onnxruntime_pybind11_state.InvalidProtobuf: error. Raised an issue here. Hopefully will get an head's up there soon.

[dosubot]

Hi, @Anindyadeep! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you opened this issue requesting support for Onnx runtime models in LangChain. You mentioned that you are willing to experiment with implementing this feature, but you are waiting for approval to access the .onnx format weights from Microsoft. Clemlesne has strongly approved the request. Additionally, you raised another issue regarding an error you are facing and are hoping for a response soon.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding, and we appreciate your contribution to the LangChain project. Let us know if you have any further questions or concerns.

[Anindyadeep]

Hi, @Anindyadeep! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you opened this issue requesting support for Onnx runtime models in LangChain. You mentioned that you are willing to experiment with implementing this feature, but you are waiting for approval to access the .onnx format weights from Microsoft. Clemlesne has strongly approved the request. Additionally, you raised another issue regarding an error you are facing and are hoping for a response soon.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding, and we appreciate your contribution to the LangChain project. Let us know if you have any further questions or concerns.

Yes, this issue is still relevant, as I seen in their repo, this is not been implemented. The reason is same, there are some problems people are facing with onnx. Raised in this issue. So I am gonna try myself to see if there is any workaround. If not then I will wait for them to fix that issue.

[dosubot]

@baskaryan Could you please help @Anindyadeep with this issue? They have indicated that the issue is still relevant and they are planning to try implementing a workaround. They have also mentioned a related issue with onnx that they are facing. Thank you!