Merge pull request #62 from VERITAS-Observatory/1.1_rc

1.1 rc

.github/workflows/bump_version.yml

@@ -4,7 +4,8 @@ on:
   push:
     branches:
     # disabling it for now
-      - _main
+      - main
+      - *_rc
 
 jobs:
   bump_version:

CHANGELOG.md

@@ -1,31 +1,64 @@
-## Unreleased
-
-### Feat
-
-- **background_models**: adding a user defined smoothing sigma
+## 1.0.1 (2024-05-09)
 
 ### Fix
 
-- **gammapy_tools/analysis/rbm.py**: removing hard coded map size values
+- **pyproject.toml**: fixing versioning
+- **gammapy_tools/make_background/prepare_data.py**: adding overwrite option
 
 ### Refactor
 
-- **analysis_notebook**: updating example
-- **data_products**: removing plt.show
+- **gammapy_tools/__version__.py**: post test update
+- **__version__**: switching to __version__ for the version number
+
+## 1.0.0 (2024-05-07)
 
-## 0.1.3 (2024-02-07)
+### Feat
+
+- **utils/exclusion_finder.py**: adding exclusion finder
+- **gammapy_tools/fake_source_coordinates/process.py**: adding mimic_data
+- **make_background**: parallel reduction of background and mimic search
+- **gammapy_tools/make_background**: implementing a parallel + reduction method to spead up background generation
+- **make_background**: implementation of closed N background runs
+- **background_models**: adding a user defined smoothing sigma
 
 ### Fix
 
+- **pyproject.toml**: changing gammapy version
+- **gammapy_tools/make_background/background_models.py**: updating to gammapy1.2
+- **gammapy_tools/utils/exclusion_finder.py**: adding a check for gammacat and hawc
+- **gammapy_tools/utils/exclusion_finder.py**: correcting path
+- **gammapy_tools/utils/exclusion_finder.py**: correcting imports and file paths
+- **fake_source_coordinates/process.py**: adding safety check
+- **gammapy_tools/fake_source_coordinates/process.py**: fixing print and removing target
+- **process.py**: correcting background
+- **make_background**: storing kl_div table
+- **background_tools.py**: exposure searching
+- **prepare_data.py**: adding more info
+- **prepare_data.py**: raise error when no runs are found
+- **gammapy_tools/analysis/rbm.py**: removing hard coded map size values
+- **Hipparcos_MAG8_1997.dat**: updated hipparcos file to have correct colour column
 - **analysis**: update the analysis scripts to work with config files
 
 ### Refactor
 
+- **gammapy_tools/fake_source_coordinates/process.py**: adding more robust file finder
+- **fake_source_coordinates/process.py**: reincluding source when scrambling
+- **gammapy_tools/fake_source_coordinates/fake_location.py**: popping exisiting background
+- **make_background**: removing ignore warnings
+- **make_background.py**: removing additional background and allow previous method
+- **templates/config.py**: adding KL_DIV
+- **utils**: moving functions to utils
+- **analysis_notebook**: updating example
+- **data_products**: removing plt.show
 - **_version.**: bumping version
 - **analysis_notebook**: migrating analysis notebook
 - **fake_source_coordinates**: adding __all__
 - **pyproject.toml**: changing required to minimum python version
 
+### Perf
+
+- **gammapy_tools/utils/run_details.py**: Changed the mimic selection to perform a nested search
+
 ## 0.1.2 (2024-01-30)
 
 ### Fix

Dockerfile

@@ -1,7 +1,7 @@
 FROM jupyter/minimal-notebook AS base
 
 # Install gammapy
-RUN mamba install gcc jupyterlab --yes
+RUN mamba install gcc jupyterlab "gammapy==1.2" ipykernel --yes
 WORKDIR /gammapy-tools
 
 
@@ -28,18 +28,23 @@ COPY --from=base /opt/conda /opt/conda
 WORKDIR /gammapy-tools/tmp_build
 
 # RUN gammapy download datasets
-ENV GAMMAPY_DATA=/gammapy-tools/gammapy-datasets/1.1/
+ENV GAMMAPY_DATA=/gammapy-tools/gammapy-datasets/1.2/
 RUN mkdir -p $GAMMAPY_DATA
+WORKDIR /gammapy-tools/
+RUN gammapy download datasets
 
 # Add package
 ADD --chown=1000:100 . /gammapy-tools/tmp_build/gammapy-tools
 WORKDIR /gammapy-tools/tmp_build/gammapy-tools
 
 
+
 # RUN ls -lah
 RUN pip install .
 # RUN ./gammapy_tools/Hipparcos_MAG8_1997.dat $GAMMAPY_DATA/catalogs/
 RUN cp /opt/conda/lib/python3.11/site-packages/gammapy_tools/Hipparcos_MAG8_1997.dat  $GAMMAPY_DATA/
+RUN wget https://raw.githubusercontent.com/gammapy/gammapy/main/gammapy/datasets/map.py -O /opt/conda/lib/python3.11/site-packages/gammapy/datasets/map.py
+
 
 USER root
 RUN mkdir /local_data
@@ -48,4 +53,8 @@ RUN mkdir /local_data
 RUN rm -r /gammapy-tools/tmp_build
 USER jovyan
 RUN mamba clean -a --yes
-WORKDIR /local_data
+WORKDIR /local_data
+
+# Keep alive for Docker ipykernel usage
+RUN echo -e "#!/bin/bash\nwhile true; do sleep 5; done" >> /gammapy-tools/keep_alive.sh ; chmod a+x /gammapy-tools/keep_alive.sh 
+CMD ["/gammapy-tools/keep_alive.sh"]

docker-compose.yml

@@ -11,5 +11,5 @@ services:
       - JUPYTER_TOKEN=letmein
     volumes:
       # Replace with where your data is located
-      - /path/to/data:/local_data  
-      - $GAMMAPY_DATA:/gammapy-tools/gammapy-datasets/1.1/
+      - /home/obriens/DataAnalysis/Veritas/Gammapy:/local_data  
+      # - $GAMMAPY_DATA:/gammapy-tools/gammapy-datasets/1.1/

docs/containers.md

@@ -90,4 +90,59 @@ This will have access to data mounted in `/local_data` and run the python versio
 Finally, you can start an interactive shell using:
 ```
 singularity/apptainer shell -B /path/to/my/data:/local_data gammapy-tools.sif
-```
+```
+
+## Containerized Jupyter Kernel
+
+One can used a containerized Jupyter kernel through one's own python environment, regardless of the setup (venv, poetry, conda, mamba, etc). The benifit of this method is that all the python code is evaluated in a reproducable containerized environment, without needing to worry about file permission, mounts/binds, networking or any other boundries of when using Docker or Apptainer/Singularity. 
+
+To do this you need either Docker or Apptainer/Singularity installed and a working python install with ipykernel installed.
+First make sure you've build either the Docker or Apptainer/Singularity image using the instructions above. 
+Next create a new custom kernel:
+```
+python -m ipykernel install --user --name gammapy-kernel --display-name="gammapy-kernel"
+```
+This will create a new directory (for example):
+```
+Installed kernelspec custom-kernel in /home/obriens/.local/share/jupyter/kernels/gammapy-kernel
+```
+Navigate to the `/home/obriens/.local/share/jupyter/kernels/gammapy-kernel/` (correcting for your own install) and replace the `kernel.json` file with the `kernel.json` [file from this repository ](../kernel.json):
+```
+{
+ "argv": [
+     "/path/to/launch_kernel.sh",
+     "{connection_file}"
+ ],
+ "display_name": "gammapy-kernel",
+ "language": "python",
+ "metadata": {
+  "debugger": true
+ }
+}
+```
+Replace:
+```
+     "/path/to/launch_kernel.sh",
+```
+With the path to the `launch_kernel_apptainer.sh` or  `launch_kernel_docker.sh` file from this repository. 
+Make the `launch_kernel_apptainer.sh` or `launch_kernel_docker.sh` script executable, for example:
+```
+chmod +x launch_kernel_apptainer.sh 
+```
+Export the environmental variable `GAMMAPY_KERNEL_IMAGE` to:
+```
+export GAMMAPY_KERNEL_IMAGE=/path/to/gammapy-tools.sif
+```
+for Apptainer/Singularity, or
+```
+export GAMMAPY_KERNEL_IMAGE="docker_username/gammapy_tools:latest"
+```
+for Docker. Additionally for Docker, you must set the `GAMMAPY_WORK_DIR` environmental variable:
+```
+export GAMMAPY_WORK_DIR=/path/to/working/directory
+```
+This is needed for Docker, as, by default, no directories will be mounted within the container. Whereas Apptainer/Singularity will, by default (unless your sysadmin installed it differently), will have access to [a set of commonly used directories](https://apptainer.org/docs/user/main/bind_paths_and_mounts.html#system-defined-bind-paths). If you're using a location not bound by default, then you'll also need to set the `GAMMAPY_WORK_DIR` environmental variable when working with Apptainer/Singularity.
+
+Launch a Jupyter instances as you normally do from any envrionment. When creating a new notebook you'll now see the option to use the containerized `gammapy-kernel`.
+
+For a detailed explaination, see [this post](https://www.physics.mcgill.ca/~obriens/Tutorials/containerized_kernels/).