Releases: m-lab/etl-gardener
Fix broken sidestream source bucket
Sidestream reprocessing should now pull from the unified archive in archive-mlab-oti. This updates the config accordingly.
This also fixes a bug with CopyConfig that was causing final copies to fail.
Note that, at time of release, the archive-mlab-oti change is not yet integrated into master.
Run Gardener in production with more datasets
This release of gardener includes recent changes, such as:
- Separate NDT, Sidestream, and Switch (DISCO) dataset configurations
- Update Datastore schema to partition 'tasks' by Experiment name
- Adds Copy&Replace with sanity checking in the Finalize stage
- Add contexts throughout
- Improve error handling with bqiface
- Handler additional task error states
- Other major code improvements
Production release - task based with restart
- There is now one goroutine per task (date*experiment). The config is set up only for sidestream, from 2017/06/01 forward.
- Task state is stored in datastore, and tasks are retrieved and restarted when deployment restarts.
Currently, partitions immediately overwrite mlab-oti:private.sidestream, so once we set up the public view, public data will be updated without prior sanity checking.
Production Release
Improvements in state tracking.
Most changes since last release are in preparation for switching to task state machines, but this is not enabled yet. There have been a handful of other changes that impact the active code in fairly minor ways, mostly improving logging, and recording of state changes in datastore.
Prerelease to process sidestream data for discard analysis
Adds nodeSelector.
Prerelease to process sidestream data for discard analysis
This is a temporary release intended only to process the full year of sidestream data for discard analysis.
Hack to reprocess sidestream for discard analysis
This is a temporary version, hacked to reprocess the sidestream data needed for the final discard analysis for a blog post.
Prerelease to test production cluster config.
Adds bug workaround for task queue statistics bug, and overrides BQ project from mlab-oti to measurement-lab
Prerelease to test production cluster config.
Adds bug workaround for task queue statistics bug.
Prerelease to test production cluster config.
Small code hack to select measurement-lab BQ project, instead of mlab-oti