Reapply reproducibility improvements, add more unit tests

[charleskawczynski]

This PR reapplies some of the reproducibility improvements, and adds many more unit tests.

I think there are two remaining pieces:

• Unit test data movement (reproducibility_tests/move_output.jl)
• Unit test repro tests (reproducibility_tests/test_mse.jl), which happens in a subsequent process (so that artifacts are uploaded to buildkite regardless of the outcome of the reproducibility tests)
• Rename reproducibility_tests/print_new_mse.jl -> reproducibility_tests/mse_summary.jl
• Rename compute_mse.jl -> reproducibility_tests/reproducibility_tools.jl
• Delete reproducibility_tests/test_reset.jl
• Use environment flags to simplify opting-in to reproducibility tests.

[charleskawczynski]

Ok, I think I realized the incremental path here. We can first add this infrastructure alongside our existing one & export the hdf5 data, then pull out the nc comparisons. That's what I'll do. I'll open a separate PR since this one is pretty entangled around the source code.

[charleskawczynski]

I think that our exported NC data is exporting data correctly, and only captures a subset of the domain (this was afterall developed for a single column in mind).

I don't think we can incrementally change to a system that uses HDF5 data-- we would first need to fix the exported NC data, which seems a bit laborious to me. We could make some other incremental changes, but most of those would be name improvements, but some of them only make sense in this new context.

So, I think the right course of action is to take a leap onto this new system, which is significantly more tested than the existing one. Since I'm on vacation tomorrow, I'll probably try to merge this on December 18, instead.

The last thing needed is to increment the reference counter.

[charleskawczynski]

I think I'll do the environment variable business in a subsequent PR, since it's somewhat orthogonal to the existing logic.