feat(server): prevent allocating to unregistered executors

[seriousben]

Context

Timing issues can cause tasks to be allocated to an unregistered executor.

What

Check in transaction for the existence of the executor before allocation.

Testing

Contribution Checklist

• If the python-sdk was changed, please run make fmt in python-sdk/.
• If the server was changed, please run make fmt in server/.
• Make sure all PR Checks are passing.

[diptanu]

@seriousben If we place a task on an executor which is removed, assuming the server processes the executor removed event, the task is going to be put back on the unallocated tasks queue right?

With this change, the scheduler state change request is not going to have an affect, and the event will be marked as processed. Unless there is another event which triggers the scheduler, this task is going to remain in the unplaced tasks.

[seriousben]

@seriousben If we place a task on an executor which is removed, assuming the server processes the executor removed event, the task is going to be put back on the unallocated tasks queue right?

With this change, the scheduler state change request is not going to have an affect, and the event will be marked as processed. Unless there is another event which triggers the scheduler, this task is going to remain in the unplaced tasks.

Yeah, I think we might need a state change for scheduling errors in order to look at unplaced tasks.

I also want to split task creation from scheduling so that they can each react to different state changes and have their own error handling.

[seriousben]

Closing in favor of a refactor getting rid of this and other timing issue edge cases caused by concurrent data updates.