[WIP] Quick band-aid fix for MPI termination

[jchodera]

This is a quick band-aid fix to ensure we don't leave straggler processes around after MPI processes receive a SIGABRT, SIGINT, or SIGTERM from Torque.

A much better approach would be to use context managers to ensure that all files are gracefully synced and closed and MPI is aborted as the program terminates, but this will require some refactoring/redesign. As a result, the current solution may result in some additional data corruption that is hopefully minimized by the fact that NetCDF syncs the files each iteration.

[jchodera]

Attempts to address issues in cBio/cbio-cluster#415

[jchodera]

Initial tests indicate the code just ignores the Torque kill signal.

[jchodera]

Fixes #356

[andrrizzi]

I'm not sure why this didn't work, but I wrote a function that configures MPI to Abort whenever it receives those 3 signals and when any exception is raised. I tested it, and it seems to work. At least, I can see a application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0 in the log when the job hits the wall time.

If you want I can open a pull request and check in that function.

[andrrizzi]

I also wanted to write a decorator for the function repex._write_iteration_netcdf() to not get interrupted by SIGTERM or SIGINT so that we won't risk data corruption.

[andrrizzi]

More specifically, I want to implement something like this: http://stackoverflow.com/a/21919644

[jchodera]

This is a great idea!

[jchodera]

Do you want to open your own version of this PR and we can close this one?

[andrrizzi]

Ok! I'll close this one when I've tested everything in mine.

[jchodera]

Closed in favor of #412.