Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install backends from WebUI #3953

Open
mudler opened this issue Oct 24, 2024 · 0 comments
Open

Install backends from WebUI #3953

mudler opened this issue Oct 24, 2024 · 0 comments
Labels
enhancement New feature or request roadmap

Comments

@mudler
Copy link
Owner

mudler commented Oct 24, 2024

Is your feature request related to a problem? Please describe.
We have a huge matrix of Docker images and Python images are quite big.

Describe the solution you'd like
A way to pull backends in runtime, ideally like installing models (from webUI and API), but for backends (e.g. parler-tts, etc)
This would allow us to release binaries or smaller container images without any embeded backend. At the same time it would allow to pull backends during the installation of the models too.
We could serve backends as container images that could be pulled with the same code that pulls models, but would just overlay backends in a defined folder.

There are challenges to this:

  • container images would still depend on underlying drivers (e.g. nvidia, hipblas, etc)
  • CI changes to the release process (build each backend in a container image and publish it separately)

Describe alternatives you've considered
Keep things as-is

Additional context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request roadmap
Projects
None yet
Development

No branches or pull requests

1 participant