Releases: TyberiusPrime/pypipegraph2
Maintenance release
Breaking changes
* MultiFileGenerationJobs (MFG) and similar jobs now keep the order of filenames
* Minimum python version now 3.8
Bug fixes:
- JobGeneratingJobs no longer lead to lost history
- One more case of job outcomes (finished-aborted) is considered when building the history.
Other changes
* (ExternalJob)[https://tyberiusprime.github.io/pypipegraph2/docs/reference/jobs/externaljob/] introduced
* Better directory configuration API (ppg.new(dir_config="root_to_ppg_files") for the easy casei, a DirConfig object for the complex one.
* interactive works again
* interactive output sorted by wait/running
* improved error messages when MFG fails to create a file
* subsecond timestamps
* Temp jobs no longer rehashed if a dependency needs to restart
* hugo documentation
* runtimes file fixed with regard to correct newlines.
* 'history was different' on CLI output shortend.
* log 'deduplicates' code locations into a separate file.
* find cython-source works for newer cythons
* maturin update, abi3-py38 wheels that should be good for any python release >=3.8
* performance enhancements
Known Bugs:
- The deephash library we're using does not correctly hash the contents of pandas dataframes.
This has been fixed upstream and is waiting on teh release.
Bugfix release / JobGeneratingJobs + MultiFileGeneratingJos
A bug which prevented history from being saved correctly when a JobGeneratingJob created job dependend on a MultiFileGeneratingJob was fixed.
3.0.6: Bugfix release
This release fixes a "can't decide what to do next error".
There was a bug in propagating 'needs to be done' from
a changed job, crosstream to a series of chained ephemerals.
This would be detected, leading to an endlessly repeated 'Evaluator is not finished, reports no jobs ready to run, but no jobs currently running -> a bug in the state machine.' message.
The underlying bug is now fixed (and tested for),
and the engine will now exit uncleanly if it ever again get's into this
inconsistent state.
SharedMultiFileGeneratingJob locking fix
When two SharedMultiFileGeneratingJobs wanted to report their usage concurrently,
there was a race condition which this release fixes.
Bugfix: Abort robustness
I found a set of corner cases that can only
be triggered when a ppg2 is 'abort'ed, i.e. partially
evaluated with regards to the ephemeral jobs.
The problem happend when an Ephemeral was invalidated,
done, it's output changed, and the downstreams were only
partially run.
It would go into a state where it would complain
that the ephemeral had changed it's output, and that it mustn't do so,
because in the new ppg2 run it was considered 'validated'.
This is now fixed.
Bugfix for interactive mode.
We had an issue where using the interactive mode,
and giving any input (such as enter to get job status)
would cause all jobs to halt, as if they were suspended
by the terminal.
This was a side effect of our 'main process went away'
watchdog and was fixed in this release.
Rust engine
The inner core was completely rewritten in rust,
which allowed much more comprehensive testing
and should be more reliable.
Disadvantage is of course that you need a rust compiler,
or the binary wheels from Pypi.
(This is 3.0.0 apart from github release fiddeling).