[Ingest] Callback not working for failed jobs

[mreyescdl]

An oversight on how job notifications are processed.
Email notifications work at the Batch level, however Callback notifications are at the Job level.
This is done so that as large batches are processed, the status of these jobs are quickly processed.

However, this requires a failed Job to be acquired by the Job notification daemon.
With a Failed state a daemon can not process these jobs.

2 possible solutions

• Process callbacks at the batch level. All jobs must complete before we start issuing callbacks
• Architect the processing of Failed jobs by the Notification daemon somehow.

[elopatin-uc3]

After some initial discussion, we're thinking that the appropriate way to address this is to add a new state.