BEAST WDL emit checkpoint and/or logs/trees in case of premature termination

[dpark01]

Currently, our BEAST WDL task dumps its primary outputs (BEAST log and trees files) to local files and then delocalizes them at the end by standard wdl File descriptors in the output block.

However, there are many times in which a BEAST run might prematurely terminate, especially since they tend to run so long. In these cases, we lose the entire (expensive) run.

Ideally it would be nice if we could somehow funnel the outputs through stdout instead. WDL allows for task outputs to be derived from stdout, and most cloud executors like PAPI2 and dnanexus make a point of checkpointing stdout on a frequent basis. This would mean that if a running beast job was prematurely terminated for any reason, we would still have a deserialized copy available in a bucket.

The main challenge though is to figure out how to get both of the critical output files (.trees and .log) through a single pipe (leaving stderr untouched for the usual messages).

[dpark01]

Brand new features!

On the cromwell/terra side:

• https://cromwell.readthedocs.io/en/develop/optimizations/CheckpointFiles/

On the beast side:

• https://beast.community/online_inference.html#generating-a-state-or-checkpoint-file-in-beast
• beast -save_every 500000 -save_state checkpoint.state input.xml
• beast -load_state checkpoint.state input.xml