New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve task reporting for lower latency and support for larger files #1049

Open

seriousben opened this issue Nov 20, 2024 · 0 comments

Labels

Member

seriousben commented Nov 20, 2024 •

edited

Loading

In order to support multiple gigabyte sized files and lots of different function outputs, we need to change how Task Reporting is done.

A temporary fix was introduced in #1048.

Approaches brought up so far:

WebSocket based approach: previously implemented in a previous version of indexify https://github.com/tensorlakeai/indexify/blob/v1/src/ingest_extracted_content.rs
Using pre-signed URLs to upload files
Two phase API approach allowing uploading individual files and then finalizing the task separately
Support two approaches, the existing ingest API and another more performant one to keep the executor simple to configure by default.

seriousben added the enhancement label

seriousben mentioned this issue

bug(sdk): increase task reporting timeout from 5s to 5m #1048

Merged

3 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment