Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[META] Streaming Model/Agent prediction/execution APIs #3328

Open
6 tasks
jngz-es opened this issue Jan 2, 2025 · 0 comments
Open
6 tasks

[META] Streaming Model/Agent prediction/execution APIs #3328

jngz-es opened this issue Jan 2, 2025 · 0 comments
Labels
enhancement New feature or request untriaged

Comments

@jngz-es
Copy link
Collaborator

jngz-es commented Jan 2, 2025

Is your feature request related to a problem?
The meta issue to track the streaming feature progress.

What solution would you like?

  • Model predict rest api based on transport-reactor-netty4 plugin.
  • Agent execute rest api based on transport-reactor-netty4 plugin.
  • Remote model streaming predict api by supporting sse (sever-sent events) connection with LLMs.
  • MLModelTool streaming run api.
  • Agent streaming execute api with streaming supported tools.
  • Agent streaming execute api with non-streaming tools.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

@jngz-es jngz-es added enhancement New feature or request untriaged labels Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request untriaged
Projects
None yet
Development

No branches or pull requests

1 participant