Rcal 596 Add association processing to the pipeline code #802

ddavis-stsci · 2023-08-01T18:37:24Z

Closes #755

This PR adds the initial association processing to the pipeline. This is an initial implementation and once rcal-620 and rcal-621 are done we will use ModelContainer and the open function to load the association.
The current usage requires the use of disable-crds-steppar option but should be relaxed on the two Jira issues above are
addressed. To run the pipeline with an association the command is,
strun --disable-crds-steppar roman_elp r90001-a3001_image_001_asn.json

2023-08-01 13:51:57,520 - stpipe.ExposurePipeline - INFO - ExposurePipeline instance created.
2023-08-01 13:51:57,521 - stpipe.ExposurePipeline.dq_init - INFO - DQInitStep instance created.
2023-08-01 13:51:57,522 - stpipe.ExposurePipeline.saturation - INFO - SaturationStep instance created.
...
2023-08-01 14:11:21,769 - stpipe.ExposurePipeline.source_detection - INFO - Found 3869 sources.
2023-08-01 14:11:21,802 - stpipe.ExposurePipeline.source_detection - INFO - Step source_detection done
2023-08-01 14:11:21,803 - stpipe.ExposurePipeline - INFO - Roman exposure calibration pipeline ending...
2023-08-01 14:11:23,380 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI01_cal.asdf
2023-08-01 14:11:24,925 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI02_cal.asdf
2023-08-01 14:11:26,485 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI03_cal.asdf
2023-08-01 14:11:28,130 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI04_cal.asdf
2023-08-01 14:11:28,130 - stpipe.ExposurePipeline - INFO - Step ExposurePipeline done

Checklist

added entry in CHANGES.rst under the corresponding subsection
updated relevant tests
updated relevant documentation
updated relevant milestone(s)
added relevant label(s)

pytest romancal/tests/
=============================================================== test session starts ================================================================
platform darwin -- Python 3.11.0, pytest-7.4.0, pluggy-1.2.0
rootdir: /Users/ddavis/src/Roman/romancal
configfile: pyproject.toml
plugins: remotedata-0.4.0, asdf-2.15.0, hypothesis-6.82.0, doctestplus-0.13.0, cov-4.1.0, filter-subpackage-0.1.2, mock-3.11.1, astropy-header-0.2.2, astropy-0.10.0, ci-watson-0.6.1, openfiles-0.5.0, env-0.8.2, arraydiff-0.5.0
collected 92 items

romancal/tests/test_import.py ............................................................................................ [100%]

================================================================ 92 passed in 1.59s ================================================================
(rcal_dev) bash>pytest romancal/datamodels/tests/
=============================================================== test session starts ================================================================
platform darwin -- Python 3.11.0, pytest-7.4.0, pluggy-1.2.0
rootdir: /Users/ddavis/src/Roman/romancal
configfile: pyproject.toml
plugins: remotedata-0.4.0, asdf-2.15.0, hypothesis-6.82.0, doctestplus-0.13.0, cov-4.1.0, filter-subpackage-0.1.2, mock-3.11.1, astropy-header-0.2.2, astropy-0.10.0, ci-watson-0.6.1, openfiles-0.5.0, env-0.8.2, arraydiff-0.5.0
collected 42 items

romancal/datamodels/tests/test_datamodels.py ......................................... [ 97%]
romancal/datamodels/tests/test_filetype.py . [100%]

=============================================================== 42 passed in 23.78s

Since plwishmaster is down I ran the regression tests locally,
platform darwin -- Python 3.11.0, pytest-7.4.0, pluggy-1.2.0
rootdir: /Users/ddavis/src/Roman/romancal
configfile: pyproject.toml
plugins: remotedata-0.4.0, asdf-2.15.0, hypothesis-6.82.0, doctestplus-0.13.0, cov-4.1.0, filter-subpackage-0.1.2, mock-3.11.1, astropy-header-0.2.2, astropy-0.10.0, ci-watson-0.6.1, openfiles-0.5.0, env-0.8.2, arraydiff-0.5.0
collected 21 items

../../../romancal/romancal/regtest/test_dark_current.py .... [ 19%]
../../../romancal/romancal/regtest/test_jump_det.py . [ 23%]
../../../romancal/romancal/regtest/test_linearity.py .. [ 33%]
../../../romancal/romancal/regtest/test_ramp_fitting.py . [ 38%]
../../../romancal/romancal/regtest/test_refpix.py . [ 42%]
../../../romancal/romancal/regtest/test_tweakreg.py . [ 47%]
../../../romancal/romancal/regtest/test_wfi_dq_init.py .. [ 57%]
../../../romancal/romancal/regtest/test_wfi_flat_field.py ... [ 71%]
../../../romancal/romancal/regtest/test_wfi_photom.py . [ 76%]
../../../romancal/romancal/regtest/test_wfi_pipeline.py ... [ 90%]
../../../romancal/romancal/regtest/test_wfi_saturation.py .. [100%]
...
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================= 21 passed, 19 warnings in 7665.81s (2:07:45)

braingram · 2023-08-01T18:49:31Z

FYI: doc failure appears related to an old theme version, similar to: https://github.com/spacetelescope/jwst/pull/7787/files

codecov · 2023-08-01T18:50:30Z

Codecov Report

Patch coverage: 35.21% and project coverage change: -1.00% ⚠️

Comparison is base (0489537) 76.95% compared to head (7ba47e9) 75.96%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #802      +/-   ##
==========================================
- Coverage   76.95%   75.96%   -1.00%     
==========================================
  Files          94       96       +2     
  Lines        5690     5799     +109     
==========================================
+ Hits         4379     4405      +26     
- Misses       1311     1394      +83

Flag	Coverage Δ		*Carryforward flag
nightly	`63.95% <ø> (-0.18%)`	⬇️	Carriedforward from 0489537

*This pull request uses carry forward flags. Click here to find out more.

Files Changed	Coverage Δ
romancal/pipeline/exposure_pipeline.py	`40.36% <6.06%> (-43.85%)`	⬇️
romancal/associations/load_as_asn.py	`39.58% <39.58%> (ø)`
romancal/datamodels/filetype.py	`96.42% <96.42%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ddavis-stsci · 2023-08-01T19:43:25Z

Thanks Brett, that fixed the problem!

schlafly

Looks good to me. I left a few comments in line in the pipeline stage where I didn't understand the logic.

Do the regression tests really take 2 hours to run on your system? Is that just the difference between your system and plwishmaster (~20 min, IIRC?)?

Yes, Mostly downloading lots of large files.

schlafly · 2023-08-03T14:30:23Z

romancal/pipeline/exposure_pipeline.py

-        else:
-            log.info("Flat Field step is being SKIPPED")
-            result.meta.cal_step.flat_field = "SKIPPED"
+                return results


I'm confused about this. The old version had this line under the is_fully_saturated condition; here this is outside that condition, so if there is ' groupdq' in result.keys() we stop the pipeline? I think I would have guessed that this should be indented one block, and that rather than returning, we continue to the next SCA?

That looks like a rebase/merge issue and I'll fix that.

schlafly · 2023-08-03T14:31:56Z

romancal/pipeline/exposure_pipeline.py


-        if result.meta.exposure.type == "WFI_IMAGE":
+            result = self.photom(result)
            result = self.source_detection(result)


This might be a merge / rebase issue or something, but I think we want to keep this "only do source detection for images" condition?

Yes that must have been wiped out.
The check is also in the steps so it doesn't matter too much. The logic is here to avoid extra setup.

schlafly · 2023-08-03T14:32:46Z

romancal/pipeline/exposure_pipeline.py

+            self.setup_output(result)
+            log.info("Roman exposure calibration pipeline ending...")
+
+            self.output_use_model = True


I didn't see where this was used.

This is used in stpipe to set the output file names.

ddavis-stsci · 2023-08-03T18:26:02Z

Updates for logging and step skips for non WFI_IMAGE data. The pipeline now produces with imaging and spectral data,

2023-08-03 14:21:13,575 - stpipe.ExposurePipeline - INFO - Flat Field step is being SKIPPED
2023-08-03 14:21:13,576 - stpipe.ExposurePipeline - INFO - Photom and source detection steps are being SKIPPED
2023-08-03 14:21:13,576 - stpipe.ExposurePipeline - INFO - Roman exposure calibration pipeline ending...
2023-08-03 14:21:15,180 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI01_cal.asdf
2023-08-03 14:21:16,635 - stpipe.ExposurePipeline - INFO - Saved model in r0000201001001001003_01101_0001_WFI01_cal.asdf
2023-08-03 14:21:16,635 - stpipe.ExposurePipeline - INFO - Step ExposurePipeline done

and for the spectral data file

          assign_wcs	COMPLETE
          flat_field	SKIPPED
                dark	COMPLETE
             dq_init	COMPLETE
                jump	COMPLETE
           linearity	COMPLETE
              photom	SKIPPED
    source_detection	SKIPPED
            ramp_fit	COMPLETE
          saturation	COMPLETE

nden · 2023-08-04T13:21:51Z

romancal/associations/load_as_asn.py

+
+        Returns
+        -------
+        association : DMSLevel2bBase


Should we get away from JWST names and levels, since there's noLevel 2b for Roman?

That's fine. I did this for expediency and when the associations were much more poorly defined. Can this be a separate ticket to clean this up?

nden · 2023-08-04T13:22:38Z

romancal/associations/load_as_asn.py

+        if isinstance(obj, str):
+            file_name, file_ext = os_path.splitext(obj)
+
+            if file_ext == ".fits":


And what about not using FITS files or do you see this code being combined in the future with the code for JWST?

I kept this to cover a case if/when someone will give romancal a fits file. We can just give a warning and gracefully exit.
There is also the type error in roman_datamodels (TypeError: Roman datamodels does not accept FITS files or objects) should we stick with that?

nden · 2023-08-04T13:28:07Z

romancal/pipeline/exposure_pipeline.py

+            try:
+                input = rdm.open(input)
+            except TypeError:
+                log.debug("Error opening file:")


What should happen in the case of one asdf file which cannot be opened? Should it return?

I guess the question is, if any of the inputs in the association fail for any reason will the end user still want the results?
My choice would be to stop and let the user know. They can fix the error and rerun the pipeline. That way they make the decision if the results are valid without all the inputs. Not very pythonic of me.

nden · 2023-08-04T13:29:08Z

romancal/pipeline/exposure_pipeline.py

+            try:
+                asn = LoadAsLevel2Asn.load(input, basename=self.output_file)
+            except AssociationNotValidError:
+                log.debug("Error opening file:")


Similarly here - if an association is invalid shouldn't the code exit?

Is there a way to make sure it's a Level 2 asn, to prevent running the code on L2 products? Even if this is desirable I'm not asking to be done in this PR.

I left all that out. I'd like to address that once rcal-620 and 621 are in. I think that would be a better place to enforce validity.
But yes here the code should just exit.

ddavis-stsci · 2023-08-04T14:46:34Z

That is not actually true. In some cases you will only want one result. For this case you should get a list of results. Updated the code.

…

On 8/4/23 10:05 AM, Eddie Schlafly wrote: ***@***.**** commented on this pull request. ------------------------------------------------------------------------ In romancal/pipeline/exposure_pipeline.py <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*discussion_r1284469655__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!wHxyUKS7hhumUaUEsyUXPN-bGrPcBJD5lhUYk3VfIbwDViKm09zir4rctOTIrZkmz7842lOUAaqZMVpSX3CUER-i$>: > if is_fully_saturated(result): - # Set all subsequent steps to skipped - for step_str in [ - "assign_wcs", - "flat_field", - "photom", - "source_detection", - ]: - result.meta.cal_step[step_str] = "SKIPPED" - - # Set suffix for proper output naming - self.suffix = "cal" + log.info("All pixels are saturated. Returning a zeroed-out image.") + + # Return zeroed-out image file (stopping pipeline) + return self.create_fully_saturated_zeroed_image(result) I hear you that some of this could be a different PR, but at least the bit where we still want to process all of the images even if the first one is all saturated applies? i.e., replace return self.create_fully_saturated_zeroed_image with results.append(self.create_fully_saturated_zeroed_image) ; continue, right? The step should always return a list of results if fed an association, and never just one? — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*discussion_r1284469655__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!wHxyUKS7hhumUaUEsyUXPN-bGrPcBJD5lhUYk3VfIbwDViKm09zir4rctOTIrZkmz7842lOUAaqZMVpSX3CUER-i$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALXCXWPG36SQAZWFTNCB2BLXTT6UNANCNFSM6AAAAAA3AFONEE__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!wHxyUKS7hhumUaUEsyUXPN-bGrPcBJD5lhUYk3VfIbwDViKm09zir4rctOTIrZkmz7842lOUAaqZMVpSX1MBg03r$>. You are receiving this because you were assigned.Message ID: ***@***.***>

for more information, see https://pre-commit.ci

schlafly

I'm still confused about the intended behavior in the fully saturated case. I've added code suggestions below and am approving. Thanks!

schlafly · 2023-08-07T13:59:24Z

romancal/pipeline/exposure_pipeline.py

-        log.info("Roman exposure calibration pipeline ending...")
-
-        return result
+                return results


Suggested change

return results

continue

Indenting and changing return -> continue so other detectors will still process.

schlafly · 2023-08-07T13:59:56Z

romancal/pipeline/exposure_pipeline.py

+
+                    # Set suffix for proper output naming
+                    self.suffix = "cal"
+                    results.append(result)

                # Return fully saturated image file (stopping pipeline)


Suggested change

# Return fully saturated image file (stopping pipeline)

# Skip remaining steps on this detector.

Indenting this to go under the saturation check, and updating the comment.

ddavis-stsci · 2023-08-07T14:33:37Z

I don't understand your objections. When there is a fully saturated input file I get 2023-08-07 10:31:44,768 - stpipe.ExposurePipeline.source_detection - INFO - Step source_detection done 2023-08-07 10:31:44,768 - stpipe.ExposurePipeline - INFO - Roman exposure calibration pipeline ending... 2023-08-07 10:31:46,735 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI01_cal.asdf 2023-08-07 10:31:48,481 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI01_sat_cal.asdf 2023-08-07 10:31:50,110 - stpipe.ExposurePipeline - INFO - Saved model in r0000101001001001001_01101_0001_WFI02_cal.asdf 2023-08-07 10:31:50,110 - stpipe.ExposurePipeline - INFO - Step ExposurePipeline done So the unsaturated data is calibrated and output and the saturated data output is also output. and the saturated data file has 0's as expected. np.max(sat.data) <Quantity 0. electron / s>

…

On 8/7/23 10:03 AM, Eddie Schlafly wrote: ***@***.**** approved this pull request. I'm still confused about the intended behavior in the fully saturated case. I've added code suggestions below and am approving. Thanks! ------------------------------------------------------------------------ In romancal/pipeline/exposure_pipeline.py <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*discussion_r1285907756__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!x0q7ttMj6nmthy1hN0uNxFfvgHcpl4Xvy9DIr1eEe-7PWuI5BxeqNfApTwxdBO-62RN8nhxHQCk8ffyI7DOxjU8D$>: > - result.meta.cal_step.flat_field = "SKIPPED" - - result = self.photom(result) - - if result.meta.exposure.type == "WFI_IMAGE": - result = self.source_detection(result) - else: - log.info("Source Detection step is being SKIPPED") - result.meta.cal_step.source_detection = "SKIPPED" - - # setup output_file for saving - self.setup_output(result) - log.info("Roman exposure calibration pipeline ending...") - - return result + return results ⬇️ Suggested change - return results + continue Indenting and changing return -> continue so other detectors will still process. ------------------------------------------------------------------------ In romancal/pipeline/exposure_pipeline.py <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*discussion_r1285908434__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!x0q7ttMj6nmthy1hN0uNxFfvgHcpl4Xvy9DIr1eEe-7PWuI5BxeqNfApTwxdBO-62RN8nhxHQCk8ffyI7Bi5lkRq$>: > + + # Test for fully saturated data + if "groupdq" in result.keys(): + if is_fully_saturated(result): + # Set all subsequent steps to skipped + for step_str in [ + "assign_wcs", + "flat_field", + "photom", + "source_detection", + ]: + result.meta.cal_step[step_str] = "SKIPPED" + + # Set suffix for proper output naming + self.suffix = "cal" + results.append(result) # Return fully saturated image file (stopping pipeline) ⬇️ Suggested change - # Return fully saturated image file (stopping pipeline) + # Skip remaining steps on this detector. Indenting this to go under the saturation check, and updating the comment. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*pullrequestreview-1565395332__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!x0q7ttMj6nmthy1hN0uNxFfvgHcpl4Xvy9DIr1eEe-7PWuI5BxeqNfApTwxdBO-62RN8nhxHQCk8ffyI7DG95Bi4$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALXCXWKLFVKNCKKT4ZVQZ6DXUDYT3ANCNFSM6AAAAAA3AFONEE__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!x0q7ttMj6nmthy1hN0uNxFfvgHcpl4Xvy9DIr1eEe-7PWuI5BxeqNfApTwxdBO-62RN8nhxHQCk8ffyI7E2yn-9v$>. You are receiving this because you were assigned.Message ID: ***@***.***>

schlafly · 2023-08-07T15:11:13Z

Okay, I guess I'm misunderstanding the code, go ahead.

ddavis-stsci · 2023-08-07T15:48:48Z

Sure. If this is a corner case or something I'm missing let's just write a ticket for an improvement.

…

On 8/7/23 11:11 AM, Eddie Schlafly wrote: Okay, I guess I'm misunderstanding the code, go ahead. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*issuecomment-1668061033__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!xBKWSOfJOLg1pWDZlOF_yIKgb0590jFzvAuQ07SQ6xbdhBUX2Mkjbd1KVodUlg2VQ-zw_NnuyiSj7Zh66qlQJBKd$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALXCXWOW5T4TZP5MZWXJDXDXUEAR3ANCNFSM6AAAAAA3AFONEE__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!xBKWSOfJOLg1pWDZlOF_yIKgb0590jFzvAuQ07SQ6xbdhBUX2Mkjbd1KVodUlg2VQ-zw_NnuyiSj7Zh66qvVFTn2$>. You are receiving this because you were assigned.Message ID: ***@***.***>

braingram · 2023-08-07T20:19:17Z

pyproject.toml

-    'rad @ git+https://github.com/spacetelescope/rad.git@main',
-    # 'roman_datamodels >=0.15.0',
-    'roman_datamodels @ git+https://github.com/spacetelescope/roman_datamodels.git@main',
+    'rad == 0.17.0',


The devdeps job on plwishmaster is currently installing rad and roman_datamodels 0.17.0 instead of the dev versions:
https://plwishmaster.stsci.edu:8081/blue/organizations/jenkins/RT%2FRoman-devdeps/detail/Roman-devdeps/361/pipeline/176

I think this pin is causing the devdeps job on plwishmaster to pull the older, 0.17.0 rad (and similar for roman_datamodels) instead of testing against the 0.17.1 (or development version, which I think is the same at the moment).

@ddavis-stsci was this a requirement for this PR? If so, is there a follow-up PR or tracking issue to remove the pin?

ddavis-stsci · 2023-08-07T20:24:10Z

The pin is needed until we get new input files that match the latest RAD and roman_datamodels. Once we have that we'll switch back to the latest released version of both packages.

…

On 8/7/23 4:19 PM, Brett Graham wrote: ***@***.**** commented on this pull request. ------------------------------------------------------------------------ In pyproject.toml <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*discussion_r1286348990__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!zhaTuljbhJSosDCuhH6HTdov51V2a2_HOpD7OcFIYKFJ_d8tKiQz4hElhSwEP2Lrq9XwS7xd-C1XDuK3y-gJBLYy$>: > @@ -22,10 +22,11 @@ dependencies = [ 'photutils >=1.6.0', 'pyparsing >=2.4.7', 'requests >=2.22', - # 'rad >=0.15.0', - 'rad @ ***@***.***', - # 'roman_datamodels >=0.15.0', - 'roman_datamodels @ ***@***.***', + 'rad == 0.17.0', I think this pin is causing the devdeps job on plwishmaster to pull the older, 0.17.0 rad (and similar for roman_datamodels) instead of testing against the 0.17.1 (or development version, which I think is the same at the moment). @ddavis-stsci <https://urldefense.com/v3/__https://github.com/ddavis-stsci__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!zhaTuljbhJSosDCuhH6HTdov51V2a2_HOpD7OcFIYKFJ_d8tKiQz4hElhSwEP2Lrq9XwS7xd-C1XDuK3y2BxXenk$> was this a requirement for this PR? If so, is there a follow-up PR or tracking issue to remove the pin? — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/spacetelescope/romancal/pull/802*pullrequestreview-1566093493__;Iw!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!zhaTuljbhJSosDCuhH6HTdov51V2a2_HOpD7OcFIYKFJ_d8tKiQz4hElhSwEP2Lrq9XwS7xd-C1XDuK3y6qm5S7b$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ALXCXWPWR6KVYJQC7CJDBWTXUFEVFANCNFSM6AAAAAA3AFONEE__;!!CrWY41Z8OgsX0i-WU-0LuAcUu2o!zhaTuljbhJSosDCuhH6HTdov51V2a2_HOpD7OcFIYKFJ_d8tKiQz4hElhSwEP2Lrq9XwS7xd-C1XDuK3y-vSzd9E$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

braingram · 2023-08-07T20:27:02Z

Thanks! That explains why 22 of the 23 failing tests fixed themselves :)

WilliamJamieson · 2023-08-08T14:44:43Z

pyproject.toml

+    'rad == 0.17.0',
+#    'rad >= 0.16.0, < 0.17.1',
+#    'rad @ git+https://github.com/spacetelescope/rad.git@main',
+    'roman_datamodels ==0.17.0',
+#    'roman_datamodels @ git+https://github.com/spacetelescope/roman_datamodels.git@main',


There absolutely should not be exact pins for these dependencies. This will cause us major headaches in development.

…ope#802) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

ddavis-stsci added pipeline associations labels Aug 1, 2023

ddavis-stsci added this to the 23Q4_B11 milestone Aug 1, 2023

ddavis-stsci self-assigned this Aug 1, 2023

github-actions bot added the testing label Aug 1, 2023

ddavis-stsci force-pushed the rcal-596_dsd branch 2 times, most recently from f872848 to e965b24 Compare August 1, 2023 18:41

ddavis-stsci force-pushed the rcal-596_dsd branch from e1aa6f9 to 13869c6 Compare August 1, 2023 19:29

github-actions bot added the documentation Improvements or additions to documentation label Aug 1, 2023

ddavis-stsci force-pushed the rcal-596_dsd branch from 25951b7 to 13869c6 Compare August 1, 2023 19:45

ddavis-stsci marked this pull request as ready for review August 2, 2023 13:57

ddavis-stsci requested a review from a team as a code owner August 2, 2023 13:57

github-actions bot removed the documentation Improvements or additions to documentation label Aug 2, 2023

ddavis-stsci requested review from nden and schlafly August 2, 2023 13:58

schlafly reviewed Aug 3, 2023

View reviewed changes

ddavis-stsci force-pushed the rcal-596_dsd branch 2 times, most recently from d98fd5c to 9dae67e Compare August 3, 2023 18:23

ddavis-stsci added 8 commits August 3, 2023 14:47

rcal-596 cleanup pipeline code logs

691810a

rcal-596 intermediate results

a6b5445

rcal-596 update to the pipeline

604cbf9

RCAL-596 Updates for association processing

2ec5fbe

RCAL-596 pipeline updates

420405d

RCAL-596 test_filetype updates

c507cdc

RCAL-596 Update changes file

b4c256d

RCAL-596 pipeline code cleanup

e9764c9

RCAL-596 Fix roman_datamodels & RAD in pyproject

e9ddba6

ddavis-stsci force-pushed the rcal-596_dsd branch from d3d9677 to e9ddba6 Compare August 4, 2023 12:44

nden reviewed Aug 4, 2023

View reviewed changes

RCAL-596 Fix roman_datamodels & RAD in dev requirements

435b75c

ddavis-stsci force-pushed the rcal-596_dsd branch from 5e68e6b to 435b75c Compare August 4, 2023 13:46

RCAL-596 Update for more code review comments

11ff6ba

ddavis-stsci force-pushed the rcal-596_dsd branch from 9c7bfae to 11ff6ba Compare August 4, 2023 14:30

RCAL-596 Update for more code review comments

19496c9

ddavis-stsci force-pushed the rcal-596_dsd branch from 5b82460 to 19496c9 Compare August 4, 2023 14:41

RCAL-596 Fix for if any of the input files are saturated

5c316fd

ddavis-stsci force-pushed the rcal-596_dsd branch from 1477775 to 5c316fd Compare August 7, 2023 13:14

RCAL-596 Fix for if any of the input files are saturated

d4ed5f4

ddavis-stsci force-pushed the rcal-596_dsd branch from 0b94ad9 to d4ed5f4 Compare August 7, 2023 13:26

[pre-commit.ci] auto fixes from pre-commit.com hooks

7ba47e9

for more information, see https://pre-commit.ci

schlafly approved these changes Aug 7, 2023

View reviewed changes

ddavis-stsci merged commit df09aa5 into spacetelescope:main Aug 7, 2023
19 of 21 checks passed

ddavis-stsci deleted the rcal-596_dsd branch August 7, 2023 15:37

braingram reviewed Aug 7, 2023

View reviewed changes

WilliamJamieson reviewed Aug 8, 2023

View reviewed changes

WilliamJamieson mentioned this pull request Aug 8, 2023

Avoid exact pins for rad and roman_datamodels #807

Open

braingram mentioned this pull request Aug 9, 2023

ExposurePipeline uses meta.filename instead of actual filename #818

Open

mairanteodoro pushed a commit to mairanteodoro/romancal that referenced this pull request Aug 10, 2023

Rcal 596 Add association processing to the pipeline code (spacetelesc…

220e4ea

…ope#802) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

braingram mentioned this pull request Aug 29, 2023

use filename instead of meta.filename for exposure pipeline #850

Closed

5 tasks

braingram mentioned this pull request Nov 18, 2024

exposure pipeline stops processing association on first fully saturated input #1523

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rcal 596 Add association processing to the pipeline code #802

Rcal 596 Add association processing to the pipeline code #802

ddavis-stsci commented Aug 1, 2023 •

edited

Loading

braingram commented Aug 1, 2023

codecov bot commented Aug 1, 2023 •

edited

Loading

ddavis-stsci commented Aug 1, 2023

schlafly left a comment •

edited by ddavis-stsci

Loading

schlafly Aug 3, 2023

ddavis-stsci Aug 3, 2023

schlafly Aug 3, 2023

ddavis-stsci Aug 3, 2023

schlafly Aug 3, 2023

ddavis-stsci Aug 3, 2023

ddavis-stsci commented Aug 3, 2023 •

edited

Loading

nden Aug 4, 2023

ddavis-stsci Aug 4, 2023

nden Aug 4, 2023

ddavis-stsci Aug 4, 2023

nden Aug 4, 2023

ddavis-stsci Aug 4, 2023

nden Aug 4, 2023

nden Aug 4, 2023

ddavis-stsci Aug 4, 2023 •

edited

Loading

ddavis-stsci commented Aug 4, 2023 via email

schlafly left a comment

schlafly Aug 7, 2023

schlafly Aug 7, 2023

ddavis-stsci commented Aug 7, 2023 via email

schlafly commented Aug 7, 2023

ddavis-stsci commented Aug 7, 2023 via email

braingram Aug 7, 2023 •

edited

Loading

ddavis-stsci commented Aug 7, 2023 via email

braingram commented Aug 7, 2023

WilliamJamieson Aug 8, 2023

	# Return fully saturated image file (stopping pipeline)
	# Skip remaining steps on this detector.

Rcal 596 Add association processing to the pipeline code #802

Rcal 596 Add association processing to the pipeline code #802

Conversation

ddavis-stsci commented Aug 1, 2023 • edited Loading

braingram commented Aug 1, 2023

codecov bot commented Aug 1, 2023 • edited Loading

Codecov Report

ddavis-stsci commented Aug 1, 2023

schlafly left a comment • edited by ddavis-stsci Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddavis-stsci commented Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddavis-stsci Aug 4, 2023 • edited Loading

Choose a reason for hiding this comment

ddavis-stsci commented Aug 4, 2023 via email

schlafly left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ddavis-stsci commented Aug 7, 2023 via email

schlafly commented Aug 7, 2023

ddavis-stsci commented Aug 7, 2023 via email

braingram Aug 7, 2023 • edited Loading

Choose a reason for hiding this comment

ddavis-stsci commented Aug 7, 2023 via email

braingram commented Aug 7, 2023

Choose a reason for hiding this comment

ddavis-stsci commented Aug 1, 2023 •

edited

Loading

codecov bot commented Aug 1, 2023 •

edited

Loading

schlafly left a comment •

edited by ddavis-stsci

Loading

ddavis-stsci commented Aug 3, 2023 •

edited

Loading

ddavis-stsci Aug 4, 2023 •

edited

Loading

braingram Aug 7, 2023 •

edited

Loading