Merge pull request #887 from pypeit/staged

Merges develop into master (1.0.0 tag)
pypeit · Apr 7, 2020 · 63adbe6 · 63adbe6
2 parents 3fbac56 + b5a9447
commit 63adbe6
Show file tree

Hide file tree

Showing 263 changed files with 24,178 additions and 16,371 deletions.
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.13.2
+current_version = 1.0.0
 commit = False
 tag = False
 tag_name = {new_version}

diff --git a/.gitignore b/.gitignore
@@ -3,7 +3,7 @@
 
 # Compiled source #
 ###################
-*.c
+# *.c
 *.com
 *.class
 *.dll
@@ -48,6 +48,7 @@ Thumbs.db
 *.ipynb_checkpoints
 *.pytest_cache
 *.swp
+pypeit/_compiler.c
 pypeit/data/telluric/*.fits
 pypeit/data/telluric/*.fits.gz
 
diff --git a/.travis.yml b/.travis.yml
@@ -35,7 +35,7 @@ env:
         - MAIN_CMD='python setup.py'
         - SETUP_CMD='test'
         - CONDA_DEPENDENCIES='scipy h5py pyyaml matplotlib future numba scikit-learn configobj IPython'
-        - PIP_DEPENDENCIES=''
+        - PIP_DEPENDENCIES='extension_helpers'
         - EVENT_TYPE='pull_request push'
         - TASK='test'
 

diff --git a/CHANGES.rst b/CHANGES.rst
@@ -1,4 +1,72 @@
 
+1.0.0 (07 Apr 2020)
+-------------------
+
+- Replaces usage of the `tslits_dict` dictionary with
+  `pypeit.slittrace.SlitTraceSet` everywhere.  This `SlitTraceSet`
+  object is now the main master file used for passing around the slit
+  edges once the edges are determined by `EdgeTraceSet`.
+- Removes usage of `pypeit.pixels.tslits2mask` and replaces it with
+  `pypeit.slittrace.SlitTraceSet.slit_img`.
+- Significant changes to flat-fielding control flow.
+    - Added `rej_sticky`, `slit_trim`, `slit_pad`, `illum_iter`,
+      `illum_rej`, `twod_fit_npoly` parameters to FlatFieldPar.
+    - Illumination flat no longer removed if the user doesn't want to
+      apply it to the data.  The flat was always created, but all that
+      work was lost if the illumination correction wasn't requested.
+    - Replaced tweak edges method with a more direct algorithm.
+    - `pypeit.core.flat.fit_flat` moved to
+      `pypeit.flatfield.FlatField.fit`.
+- Reoriented trace images in the `EdgeTraceSet` QA plots.  Added the
+  sobel image to the ginga display.
+- Added `bspline_profile_qa` for generic QA of a bspline fit.
+- Eliminate MasterFrame class
+- Masks handled by a DataContainer
+- Move DetectorPar into a DataContainer (named DetectorContainer) which
+  enables frame-level construction
+- Advances to DataContainer (array type checking; nested DataContainers;
+  to_master_file)
+- Dynamic docs for calibration images
+- Every calibration output to disk is help within a DataContainer,
+  separate from previous classes.  Exception is WaveCalib (this needsd a
+  fit DataContainer first)
+- Substantial refactoring of Calibrations
+- Add MDM OSMOS spectrograph
+- Moved pypeit.core.pydl.bspline into its own module, `pypeit.bspline`
+- Introduced C backend functions to speed up bspline fitting
+    - now require `extension_helpers` package to build pypeit and
+      necessary files/code in `setup.py` to build the C code
+    - C functions will be used by default, but code will revert to pure
+      python, if there's some problem importing the C module
+    - Added tests and pre-cooked data to ensure identical behavior
+      between the pure python and C functions.
+- Moved some basis function builders to pypeit.core.basis
+- Release 1.0 doc
+- Lots of new docs
+- pypeit_chk_2dslits script
+- DataContainer's for specobj, bspline
+- Introduction of Spec2DObj, AllSpec2DObj, and OneSpec (for Coadd1D)
+- Added bitmask to SlitTraceSet
+- Introduced SlitTraceSet.spat_id and its usage throughout the code
+- Spatial flexure corrections
+    - Significant refactor of flatfield.BuildFlatField.fit()
+    - Spatial flexure measuring code
+    - PypeItPar control
+    - Modifications to SlitTraceSet methods
+    - Illumflat generated dynamically with different PypeIt control
+    - waveimage generated dynamicall and WaveImage deprecated
+- Moved RawImage into ProcessRawImage and renamed the latter to the
+  former
+- Continued refactoring of Calibrations
+- Initial code for syncing SpecObjs across exposures
+- Option to ignore profile masking during extraction
+- Additional code in DataContainer related to MasterFrames
+- Eliminated WaveImage
+- Updates to QL scripts
+- Lots of new tests
+
+
+
 0.13.2 (17 Mar 2020)
 --------------------
 
@@ -126,7 +194,7 @@
   of each slit/order.  For VLT XShooter NIR, this was needed to ensure
   the sigma calculation didn't include the off-order spectral positions.
 - Added a staticmethed to :class:`pypeit.edgetrace.EdgeTraceSet` that
-  construces a ``tslits_dict`` object directly from the Master file.
+  constructs a ``tslits_dict`` object directly from the Master file.
 
 0.11.0.1
 ---------

diff --git a/MANIFEST.in b/MANIFEST.in
@@ -18,3 +18,4 @@ prune docs/api
 #exclude astropy_helpers/.gitignore
 
 global-exclude *.pyc *.o
+
diff --git a/README.md b/README.md
@@ -32,17 +32,22 @@ Conduct](https://pypeit.readthedocs.io/en/latest/codeconduct.html).
 # Instruments Served
 * Gemini/GNIRS
 * Gemini/GMOS 
+* Gemini/FLAMINGOS 2 
 * Lick/Kast 
-* Lick/APF (planned)
 * Magellan/MagE
-* Keck/DEIMOS
+* Magellan/Fire
+* MDM/OSMOS
+* Keck/DEIMOS (600ZD, 830G, 1200G)
 * Keck/LRIS 
-* Keck/MOSFIRE (soon)
+* Keck/MOSFIRE  (J and Y gratings tested)
 * Keck/NIRES
 * Keck/NIRSPEC (low-dispersion)
 * VLT/X-Shooter
 * VLT/FORS2
 * WHT/ISIS
+* LBT/Luci-I, Luci-II
+* LBT/MODS (beta)
+* Lick/APF (planned)
 
 # Requirements
 

diff --git a/bin/pypeit_c_enabled b/bin/pypeit_c_enabled
@@ -0,0 +1,14 @@
+#!/usr/bin/env python
+
+"""
+Check that the installation of python is C enabled.
+"""
+
+# Try to load the bspline C utils
+try:
+    from pypeit.bspline import utilc
+except:
+    pass
+else:
+    print('Successfully imported bspline C utilities.')
+
diff --git a/bin/pypeit_chk_2dslits b/bin/pypeit_chk_2dslits
@@ -0,0 +1,9 @@
+#!/usr/bin/env python
+
+"""
+Print slit info summary to the screen for a given spec2D file
+"""
+
+from pypeit.scripts import chk_spec2d_slits
+if __name__ == '__main__':
+    chk_spec2d_slits.main(chk_spec2d_slits.parser())
diff --git a/bin/pypeit_chk_flats b/bin/pypeit_chk_flats
@@ -4,8 +4,6 @@
 Show the trace image in a Ginga window with edges
 """
 
-import pypeit.scripts.chk_flats as chk_flats
-
+from pypeit.scripts import chk_flats
 if __name__ == '__main__':
-    args = chk_flats.parser()
-    chk_flats.main(args)
+    chk_flats.main(chk_flats.parser())
diff --git a/bin/pypeit_find_objects b/bin/pypeit_find_objects
diff --git a/doc/Makefile b/doc/Makefile
@@ -56,9 +56,10 @@ clean:
 	rm -rf $(LOCALFILES)
 
 apirst:
-	SPHINX_APIDOC_OPTIONS=$(SPHINXAPIOPT) $(SPHINXAPI) --separate -o ./api ../pypeit ../pypeit/tests/* ../pypeit/scripts/*
+	SPHINX_APIDOC_OPTIONS=$(SPHINXAPIOPT) $(SPHINXAPI) --separate -o ./api ../pypeit ../pypeit/tests/* ../pypeit/scripts/* ../pypeit/deprecated/*
 	python3 ./scripts/build_par_rst.py
 	python3 ./scripts/build_specobj_rst.py
+	python3 ./scripts/build_calib_images_rst.py
 
 html:
 	make apirst

diff --git a/doc/_static/css/pypeit.css b/doc/_static/css/pypeit.css
@@ -0,0 +1,12 @@
+.wy-nav-content {
+    max-width: none;
+}
+
+.wy-table-responsive table td {
+    white-space: normal !important;
+}
+
+.wy-table-responsive {
+    overflow: visible !important;
+}
+
diff --git a/doc/_templates/layout.html b/doc/_templates/layout.html
@@ -0,0 +1,5 @@
+{% extends "!layout.html" %}
+{% block extrahead %}
+    <link href="{{ pathto("_static/css/pypeit.css", True) }}" rel="stylesheet" type="text/css">
+{% endblock %}
+
diff --git a/doc/api/pypeit.traceimage.rst → doc/api/pypeit.alignframe.rst b/doc/api/pypeit.traceimage.rst → doc/api/pypeit.alignframe.rst
@@ -1,7 +1,7 @@
-pypeit.traceimage module
+pypeit.alignframe module
 ========================
 
-.. automodule:: pypeit.traceimage
+.. automodule:: pypeit.alignframe
    :members:
    :private-members:
    :undoc-members:

diff --git a/doc/api/pypeit.arcimage.rst b/doc/api/pypeit.arcimage.rst
diff --git a/doc/api/pypeit.bspline.bspline.rst b/doc/api/pypeit.bspline.bspline.rst
@@ -0,0 +1,8 @@
+pypeit.bspline.bspline module
+=============================
+
+.. automodule:: pypeit.bspline.bspline
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.bspline.rst b/doc/api/pypeit.bspline.rst
@@ -0,0 +1,21 @@
+pypeit.bspline package
+======================
+
+Submodules
+----------
+
+.. toctree::
+
+   pypeit.bspline.bspline
+   pypeit.bspline.setup_package
+   pypeit.bspline.utilc
+   pypeit.bspline.utilpy
+
+Module contents
+---------------
+
+.. automodule:: pypeit.bspline
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.images.processrawimage.rst → doc/api/pypeit.bspline.setup_package.rst b/doc/api/pypeit.images.processrawimage.rst → doc/api/pypeit.bspline.setup_package.rst
@@ -1,7 +1,7 @@
-pypeit.images.processrawimage module
+pypeit.bspline.setup\_package module
 ====================================
 
-.. automodule:: pypeit.images.processrawimage
+.. automodule:: pypeit.bspline.setup_package
    :members:
    :private-members:
    :undoc-members:

diff --git a/doc/api/pypeit.bspline.utilc.rst b/doc/api/pypeit.bspline.utilc.rst
@@ -0,0 +1,8 @@
+pypeit.bspline.utilc module
+===========================
+
+.. automodule:: pypeit.bspline.utilc
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.bspline.utilpy.rst b/doc/api/pypeit.bspline.utilpy.rst
@@ -0,0 +1,8 @@
+pypeit.bspline.utilpy module
+============================
+
+.. automodule:: pypeit.bspline.utilpy
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.core.basis.rst b/doc/api/pypeit.core.basis.rst
@@ -0,0 +1,8 @@
+pypeit.core.basis module
+========================
+
+.. automodule:: pypeit.core.basis
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.core.flexure.rst b/doc/api/pypeit.core.flexure.rst
@@ -0,0 +1,8 @@
+pypeit.core.flexure module
+==========================
+
+.. automodule:: pypeit.core.flexure
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.core.gui.skysub_regions.rst b/doc/api/pypeit.core.gui.skysub_regions.rst
@@ -0,0 +1,8 @@
+pypeit.core.gui.skysub\_regions module
+======================================
+
+.. automodule:: pypeit.core.gui.skysub_regions
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.core.rst b/doc/api/pypeit.core.rst
@@ -15,10 +15,12 @@ Submodules
 .. toctree::
 
    pypeit.core.arc
+   pypeit.core.basis
    pypeit.core.coadd
    pypeit.core.combine
    pypeit.core.extract
    pypeit.core.flat
+   pypeit.core.flexure
    pypeit.core.flux_calib
    pypeit.core.framematch
    pypeit.core.load
@@ -31,7 +33,6 @@ Submodules
    pypeit.core.procimg
    pypeit.core.pydl
    pypeit.core.qa
-   pypeit.core.save
    pypeit.core.skysub
    pypeit.core.telluric
    pypeit.core.trace

diff --git a/doc/api/pypeit.images.buildimage.rst b/doc/api/pypeit.images.buildimage.rst
@@ -0,0 +1,8 @@
+pypeit.images.buildimage module
+===============================
+
+.. automodule:: pypeit.images.buildimage
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.images.calibrationimage.rst b/doc/api/pypeit.images.calibrationimage.rst
diff --git a/doc/api/pypeit.images.detector_container.rst b/doc/api/pypeit.images.detector_container.rst
@@ -0,0 +1,8 @@
+pypeit.images.detector\_container module
+========================================
+
+.. automodule:: pypeit.images.detector_container
+   :members:
+   :private-members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/doc/api/pypeit.images.scienceimage.rst → doc/api/pypeit.images.imagebitmask.rst b/doc/api/pypeit.images.scienceimage.rst → doc/api/pypeit.images.imagebitmask.rst
@@ -1,7 +1,7 @@
-pypeit.images.scienceimage module
+pypeit.images.imagebitmask module
 =================================
 
-.. automodule:: pypeit.images.scienceimage
+.. automodule:: pypeit.images.imagebitmask
    :members:
    :private-members:
    :undoc-members:

diff --git a/doc/api/pypeit.images.maskimage.rst b/doc/api/pypeit.images.maskimage.rst