Allow a slave to "fail" a piece of work

[mvr]

In Kesha, some of the zones might be way harder than others. If a zone takes too long, we want to "fail", and send that piece of work to the end of the queue to be tried again later. This way we can get most of the zones done early.

[jamiecook]

why would we want to repeat the work?

[mvr]

The idea is we would kill a particular bit of work after some preset timeout, then put it at the end of the queue to be done for real after everything else.

[jamiecook]

To what end? Are we streaming the results back so that partial results are useful?

[mvr]

Yeah, results get streamed back. The ruby code around Kesha does some post-processing that can happen as soon as each bit of work is done by a slave. Also, if for some reason one of the slaves hangs, we don't want it to be useless for there rest of a run.

[jamiecook]

Cool - just wanted to know the motivation