JP-3753: Refactor extract1d

[melanieclarke]

Resolves JP-3753

Significant refactor for extract1d, to improve code organization, maintainability, and efficiency, as well as set up for PSF-based extraction, to be added in the future.

Work in progress: more test coverage and documentation updates are needed.

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

• changes/<PR#>.general.rst: infrastructure or miscellaneous change
• changes/<PR#>.docs.rst
• changes/<PR#>.stpipe.rst
• changes/<PR#>.datamodels.rst
• changes/<PR#>.scripts.rst
• changes/<PR#>.fits_generator.rst
• changes/<PR#>.set_telescope_pointing.rst
• changes/<PR#>.pipeline.rst

stage 1

• changes/<PR#>.group_scale.rst
• changes/<PR#>.dq_init.rst
• changes/<PR#>.emicorr.rst
• changes/<PR#>.saturation.rst
• changes/<PR#>.ipc.rst
• changes/<PR#>.firstframe.rst
• changes/<PR#>.lastframe.rst
• changes/<PR#>.reset.rst
• changes/<PR#>.superbias.rst
• changes/<PR#>.refpix.rst
• changes/<PR#>.linearity.rst
• changes/<PR#>.rscd.rst
• changes/<PR#>.persistence.rst
• changes/<PR#>.dark_current.rst
• changes/<PR#>.charge_migration.rst
• changes/<PR#>.jump.rst
• changes/<PR#>.clean_flicker_noise.rst
• changes/<PR#>.ramp_fitting.rst
• changes/<PR#>.gain_scale.rst

stage 2

• changes/<PR#>.assign_wcs.rst
• changes/<PR#>.badpix_selfcal.rst
• changes/<PR#>.msaflagopen.rst
• changes/<PR#>.nsclean.rst
• changes/<PR#>.imprint.rst
• changes/<PR#>.background.rst
• changes/<PR#>.extract_2d.rst
• changes/<PR#>.master_background.rst
• changes/<PR#>.wavecorr.rst
• changes/<PR#>.srctype.rst
• changes/<PR#>.straylight.rst
• changes/<PR#>.wfss_contam.rst
• changes/<PR#>.flatfield.rst
• changes/<PR#>.fringe.rst
• changes/<PR#>.pathloss.rst
• changes/<PR#>.barshadow.rst
• changes/<PR#>.photom.rst
• changes/<PR#>.pixel_replace.rst
• changes/<PR#>.resample_spec.rst
• changes/<PR#>.residual_fringe.rst
• changes/<PR#>.cube_build.rst
• changes/<PR#>.extract_1d.rst
• changes/<PR#>.resample.rst

stage 3

• changes/<PR#>.assign_mtwcs.rst
• changes/<PR#>.mrs_imatch.rst
• changes/<PR#>.tweakreg.rst
• changes/<PR#>.skymatch.rst
• changes/<PR#>.exp_to_source.rst
• changes/<PR#>.outlier_detection.rst
• changes/<PR#>.tso_photometry.rst
• changes/<PR#>.stack_refs.rst
• changes/<PR#>.align_refs.rst
• changes/<PR#>.klip.rst
• changes/<PR#>.spectral_leak.rst
• changes/<PR#>.source_catalog.rst
• changes/<PR#>.combine_1d.rst
• changes/<PR#>.ami.rst

other

• changes/<PR#>.wfs_combine.rst
• changes/<PR#>.white_light.rst
• changes/<PR#>.cube_skymatch.rst
• changes/<PR#>.engdb_tools.rst
• changes/<PR#>.guider_cds.rst

[codecov]

Codecov Report

Attention: Patch coverage is 68.68327% with 88 lines in your changes missing coverage. Please review.

Project coverage is 65.50%. Comparing base (72cb6f0) to head (f2ad06a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
jwst/extract_1d/extract_1d_step.py	58.71%	45 Missing ⚠️
jwst/extract_1d/extract1d.py	69.17%	41 Missing ⚠️
jwst/extract_1d/ifu.py	93.93%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8961      +/-   ##
==========================================
+ Coverage   64.52%   65.50%   +0.97%     
==========================================
  Files         375      375              
  Lines       38739    37749     -990     
==========================================
- Hits        24997    24727     -270     
+ Misses      13742    13022     -720     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[melanieclarke]

Initial regression test run here:
https://github.com/spacetelescope/RegressionTests/actions/runs/11846882332
Total test time is about half an hour shorter than the last scheduled run on main.

There are differences in every x1d and x1dints product, but they are all minor:

• DQ array is now updated to set DO_NOT_USE where output flux is NaN. The previous version was not setting DQ in a useful way.
• Flux and background errors are different for apertures that include partial pixels. This is because the variance propagation in the current version is incorrect: variances in partial pixels are currently weighted by the pixel fraction, like the science data is, but they should be weighted by pixel fraction squared. The weights are fixed in this version.
• In x1dints files, extraction start/stop values are now added to the headers for spectra from all integrations, not just the first integration.
• Left and right extraction limit values are now directly set from the aperture edges, where they first go to zero, instead of from the input extraction parameters. This makes a one pixel difference to the MIRI LRS slitless EXTRYSTR/EXTRYSTP values (test_miri_lrs_slitless_tso_spec2[x1dints]).
• Approximate extraction start/stop values are added to the headers for spectra extracted with polynomial src_coeff parameters. Before, there was no information about the extraction location in this case (test_nirspec_bots_custom_extraction).

This small change also appears in these results, but probably shouldn't:

• Background values are usually zero if there is no background data, but they are now sometimes NaN at the edges of spectra, where they used to be zero.

I will modify the code to make sure all background values are finite. Test attempt 2 at the same link above incorporates this change.