Merge v0.2.2 dev branch (#326)

* Cache NUMBA kernels between CI runs (#279)

* Cache NUMBA kernels between CI runs

* Use actions/cache@v3

* Cache per python version

* runner.tmp -> runner.temp

* Debugging

* Fix

* Run entire test suite

* timestamp needed otherwise cache hit occurs and cache not updated

* Fix output

* Add revert_me.txt

* Use nearest-neighbour interpolation in regions where extrapolation is required. (#285)

* Fix version drift.

* Bump to 0.2.0

* Use nearest-neighbour interpolation for points requiring extrapolation.

* Utilise environment variable when dask.address is unset. (#288)

* Fix version drift.

* Bump to 0.2.0

* Inspect envvar for scheduler address when one isn't specified.

* Encode environment varraible as ascii.

* Simplify.

* Add plotting functionality (#290)

* Fix version drift.

* Bump to 0.2.0

* Initial commit of basic plotting functionality.

* Change naming convention.

* Improve transform argument.

* Simplify transform selection.

* Add rudimentary time and frequency selection.

* Checkpoint ploter changes. Can now handle scans and spws, but is very slow.

* More work on plotter - can now plot datasets in parallel.

* Some tidying.

* Slightly improve plot speed. Dominant cost is still saving the figures.

* Commit some minor changes which speed up figure saving.

* Lots of tiny fixes.

* Tiny cosmetic changes.

* Add custom tick formatter so that plots are the same size regardless.

* Add matplotlib dependency.

* Rework construction of plotting dictionary. Add a few utility functions which will likely be useful in other places in QC.

* Rename variable to avoid confusion.

* Fix bug affecting recursive grouping.

* Avoid copies in grouping code.

* Checkpoint work on extending functionality.

* Make plotter more powerful. Add colourization option. Begin simplifying interface.

* Allow user specification of colourmap.

* Add plotsize parameter.

* Fix #293 - OOB access caused by `output.subtract_directions`  (#294)

* Fix version drift.

* Bump to 0.2.0

* Fix #293.

* Namedbackups (#296)

* Fix version drift.

* Bump to 0.2.0

* Add optional label and single field selection to backup app

* remove item instead of pop@index

* do not .remove() from xds_list

* Simplify using some existing functionality.

---------

Co-authored-by: JSKenyon <[email protected]>
Co-authored-by: landmanbester <[email protected]>

* Selectively disable MAD flagging criteria (#298)

* Fix version drift.

* Bump to 0.2.0

* Setting MAD threshold to zero will disable flagging on a given statistic.

* Disable mad flagging on off-diagonals by default (#300)

* Fix version drift.

* Bump to 0.2.0

* Disable flagging based on off-diagonal correlations in the mad flagger by default. This should make the mad flagger less agressive on data with unmodelled polarised emission.

* Fix bug affecting non-standard columns in `input_ms.data_column` (#301)

* Fix version drift.

* Bump to 0.2.0

* Fix a bug afecting the use of non-standard columns in data column input.

* Don't allow restore app to overwrite metadata (#307)

* assign to ms to avoid over-writing metadata in restore app

* zip datasets in enumerate

* add comment to document failure case

* use backup_column_name in restore app

* Apply OCD.

---------

Co-authored-by: landmanbester <[email protected]>
Co-authored-by: JSKenyon <[email protected]>

* Fix for summary reporting SOURCE_ID as FIELD_ID (#309)

* Fix version drift.

* Bump to 0.2.0

* Make summary correctly report FIELD_ID and SOURCE_ID.

* Fix receptor summary (#310)

* Fix version drift.

* Bump to 0.2.0

* Fix incorrect assumption that FEED substable will always have 2 receptors.

* Fix similar problem affecting parallactic angle construction.

* Update missing column selection for compatibility with upsteam changes.

* Fix xarray dims (#318)

* Fix version drift.

* Bump to 0.2.0

* Move all usage of xds.dims[dim] to xds.sizes[dim] in preparation for change of return type in xds.dims.

* Fixes for changes relating to Numba error types. (#319)

* Move now-deprecated graph metrics function into the scheduler plugin code. (#320)

* Make small changes to enable 3.11 compatibilty. Requires changes in stimela + a release. (#321)

* Restringify keys in scheduler plugin. (#322)

* Attempt very dodgy solution to caching problem.

* Look for code in the correct place.

* Update pyproject.toml. Add poetry.lock. Update docs. (#323)

* Drop 3.8. Commit poetry lock file.

* Update stimela requirement.

* Update docs.

* Set min and max versions in pyproject.toml.

* Remove python3.8 from test matrix.

* Some debugging.

* Fix unsaved file.

* More debugging.

* Temporarily make test suite much smaller.

* Fix path.

* Actually fix path.

* Attempt at safer caching.

* More fiddling with paths.

* Fix bad tabbing.

* Try to find out where things are failing.

* More fiddling.

* More fiddling.

* More fiddling.

* Try restore time action.

* Tidy up caching approach. Use action. Restore matrix and test everything.

* Remove tmp file.

* Reword CI step name.

---------

Co-authored-by: JSKenyon <[email protected]>
Co-authored-by: Landman Bester <[email protected]>
Co-authored-by: JSKenyon <[email protected]>
Co-authored-by: landmanbester <[email protected]>

* Bump dask-ms and codex-africanus dependencies. Update lock.

---------

Co-authored-by: Simon Perkins <[email protected]>
Co-authored-by: Landman Bester <[email protected]>
Co-authored-by: landmanbester <[email protected]>

.github/workflows/ci.yaml

@@ -15,6 +15,9 @@ jobs:
     runs-on: ${{ matrix.os }}
     continue-on-error: true
 
+    env:
+      NUMBA_CACHE_DIR: /tmp/numba-cache
+
     strategy:
       matrix:
         os: [ubuntu-20.04, ubuntu-22.04]
@@ -38,7 +41,27 @@ jobs:
       - name: Checkout source
         uses: actions/checkout@v3
         with:
-          fetch-depth: 1
+          fetch-depth: 0  # Full fetch required to restore times.
+
+      - name: Restore repo times
+        uses: chetan/git-restore-mtime-action@v2
+
+      # Upgrade pip and setuptools
+      - name: Upgrade pip and setuptools
+        run: python -m pip install -U pip setuptools
+
+      - name: Create Key and Numba Cache Directory
+        id: numba-key
+        run: |
+          mkdir -p ${{ env.NUMBA_CACHE_DIR }}
+          echo "timestamp=$(/bin/date -u '+%Y%m%d%H%M%S')" >> $GITHUB_OUTPUT
+
+      - name: Cache Numba Kernels
+        uses: actions/cache@v3
+        with:
+          key: numba-cache-${{ matrix.python-version }}-${{ steps.numba-key.outputs.timestamp }}
+          restore-keys: numba-cache-${{ matrix.python-version }}-
+          path: ${{ env.NUMBA_CACHE_DIR }}
 
       - name: List the measures directory
         run: curl ftp://ftp.astron.nl/outgoing/Measures/ > measures_dir.txt