fix(server): stop returning diagnostic messages of all invocations being placed

[seriousben]

Context

If lots of invocations / tasks are being placed at the same time by the scheduler, it will broadcast placement issues for all invocations to all callers using block_until_done.

This results in leaking data about other tasks and can also result in lots of bandwidth being spent across all clients and tasks.

For example: a simple task can easily get megabytes of data.

What

Only returning diagnostic data related to the current invocation.
In the future, we should consider adding a way to toggle diagnostic messages in the server config.

Testing

Using:

image = (
    Image()
    .base_image("tensorlake/default-base-image")
    .name("tensorlake/WRONG!")
)


@indexify_function(input_encoder="json", output_encoder="json", image=image)
def smoke_test_node(text: str) -> str:
    return text

I invoke the graph multiple times from multiple terminals to confirm each terminal was getting the scheduler diagnostic only once instead of once per pending tasks:

scheduler diagnostic: executor mJ2RNNR5VSpTJhEBoQoBL, image name: tensorlake/indexify-executor-default does not match function image name tensorlake/WRONG!. Make sure the executor is running the latest
image

Contribution Checklist

• If the python-sdk was changed, please run make fmt in python-sdk/.
• If the server was changed, please run make fmt in server/.
• Make sure all PR Checks are passing.

[diptanu]

Let’s merge this. I was working on reworking how diagnostic messages are created and stored. But had too many unrelated changes, I will start fresh from main again.