8000 JP-3753: Refactor extract1d by melanieclarke · Pull Request #8961 · spacetelescope/jwst · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

JP-3753: Refactor extract1d #8961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 61 commits into from
Dec 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
20ff9ac
Start extract1d refactor
melanieclarke Oct 31, 2024
fffc5a3
Add test for custom extraction with polynomial limits
melanieclarke Nov 5, 2024
008ab50
Fix log message f-string
melanieclarke Nov 5, 2024
90be511
Deduplicate extraction interface
melanieclarke Nov 5, 2024
40e7c1e
Remove image reference type
melanieclarke Nov 6, 2024
a43bcef
Separate IFU from slit extraction
melanieclarke Nov 6, 2024
dd7ae7f
Remove NIRISS SOSS references from slit extraction code
melanieclarke Nov 6, 2024
fa073c3
Remove unnecessary middleman function; condense input type checking
melanieclarke Nov 6, 2024
28cdba4
Add helper functions for special extraction modes
melanieclarke Nov 7, 2024
287097e
Clean up messages
melanieclarke Nov 7, 2024
3d4cb24
First version incorporating new extraction engine
melanieclarke Nov 8, 2024
c992641
Fix extraction limit defaults and partial pixel weights
melanieclarke Nov 11, 2024
1690a2a
Npixels is sum of weights instead of sum of pixels with non-zero weight
melanieclarke Nov 11, 2024
a1c3670
Move profile and wavelength calculations out of integration loop
melanieclarke Nov 11, 2024
6a7e9eb
Tidying up
melanieclarke Nov 11, 2024
ca09589
Update unit tests for new extract1d
melanieclarke Nov 12, 2024
c7aec5d
Add left/right limit handling, fix partial pixel weights
melanieclarke Nov 12, 2024
cea2a60
Make polynomial coefficient definition backward compatible
melanieclarke Nov 12, 2024
2d80f63
Clean up aperture definition, fix edge cases
melanieclarke Nov 12, 2024
dffea4c
Coverage tests for step interface
melanieclarke Nov 13, 2024
e4517e8
Move aperture correction setup outside integration loop
melanieclarke Nov 13, 2024
d03d049
Return separate values from locn_from_wcs function
melanieclarke Nov 14, 2024
aef5fc4
Rename 'boxcar' extraction to 'box'
melanieclarke Nov 14, 2024
91c6bbd
Set up for future optimal extraction
melanieclarke Nov 14, 2024
49dad52
Tidy up code organization, add docstrings
melanieclarke Nov 14, 2024
485000f
Code style fixes
melanieclarke Nov 14, 2024
75c8722
Make sure background values are finite: they default to 0, not NaN
melanieclarke Nov 15, 2024
6d31859
More general check for unresampled nirspec data
melanieclarke Nov 18, 2024
f95ce81
More robust file saving for containers
melanieclarke Nov 19, 2024
a4db010
Add option to save spatial profile
melanieclarke Nov 19, 2024
9d92181
Add check for negative values in profile
melanieclarke Nov 20, 2024
69a5d0f
Add option to save scene model
melanieclarke Nov 20, 2024
5f42f84
Fix scene model default
melanieclarke Nov 21, 2024
f8388e4
Minor fixes for optimal extraction support
melanieclarke Nov 21, 2024
22faef6
Add check for missing variances
melanieclarke Nov 21, 2024
20fb4cb
Adding unit tests for extraction
melanieclarke Nov 21, 2024
687d14d
Add temporary fix for background fit test
melanieclarke Nov 21, 2024
0ed0d96
Coverage tests for extract1d
melanieclarke Nov 21, 2024
e47e853
Better npixels estimate for optimal extraction; fix background fit test
melanieclarke Nov 22, 2024
f71317d
More unit tests for extract_1d coverage
melanieclarke Nov 22, 2024
73f3649
Fix npixels for non-uniform weights
melanieclarke Nov 25, 2024
49b9e50
More coverage tests
melanieclarke Nov 25, 2024
82615f4
More unit tests for extract
melanieclarke Nov 25, 2024
74e2e6e
Tests for extract_one_slit
melanieclarke Nov 26, 2024
b660e81
Add change log fragment
melanieclarke Nov 26, 2024
4eb52d6
Make some parameters for create_extraction keywords instead of args
melanieclarke Nov 26, 2024
ec1c53c
Test coverage for create_extraction and run_extract1d
melanieclarke Nov 26, 2024
ee6ebc7
Reformat change note
melanieclarke Nov 27, 2024
7a68e62
Update docs for extract1d
melanieclarke Nov 27, 2024
a847dde
Add more API docs for extraction tools
melanieclarke Nov 27, 2024
55c5fd3
Make change notes fit on one line
melanieclarke Nov 27, 2024
83f27fd
Minor docs clarifications
melanieclarke Dec 3, 2024
40e94d7
Fixes and clarifications from PR review
melanieclarke Dec 4, 2024
6752d31
Refactor extract1d with helper functions for extraction cases
melanieclarke Dec 4, 2024
f24f9ea
Clarify partial pixel weights; check for finite wavelengths
melanieclarke Dec 5, 2024
61628fe
Fix wavelength check; revise aperture center
melanieclarke Dec 5, 2024
b6cd760
Don't override explicit user setting for use_source_posn
melanieclarke Dec 9, 2024
5d60520
Add guardrails for smoothing length
melanieclarke Dec 10, 2024
2144996
Add note about use_source_posn for extended sources
melanieclarke Dec 10, 2024
0a3dfad
Fix typo
melanieclarke Dec 10, 2024
67bef53
Reduce repeated code for copying keywords
melanieclarke Dec 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changes/8961.extract_1d.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Refactor the core extraction algorithm and aperture definition modules for slit and slitless extractions, for greater efficiency and maintainability. Extraction reference files in FITS format are no longer supported. Current behavior for extractions proceeding from extract1d reference files in JSON format is preserved, with minor improvements: DQ arrays are populated and error propagation is improved for some aperture types.
133 changes: 67 additions & 66 deletions docs/jwst/extract_1d/arguments.rst
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Arguments are reordered, with new section headings, to make it clearer which parameters work for which exposure types.

Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,33 @@ Step Arguments

The ``extract_1d`` step has the following step-specific arguments.

General Step Arguments
----------------------
The following arguments apply to all modes unless otherwise specified.

``--subtract_background``
Flag to specify whether the background should be subtracted. If None or True,
background subtraction will be performed if there are background regions
specified in the reference file. If False, no background subtraction will be
performed. Has no effect for NIRISS SOSS data.

``--apply_apcorr``
Switch to select whether or not to apply an APERTURE correction during the
Extract1dStep processing. Default is ``True``. Has no effect for NIRISS SOSS data.

Step Arguments for Slit and Slitless Spectroscopic Data
-------------------------------------------------------

``--use_source_posn``
Specify whether the target and background extraction
region locations specified in the :ref:`EXTRACT1D <extract1d_reffile>` reference
file should be shifted to account for the expected position of the source. If None (the default),
the step will decide whether to use the source position based
on the observing mode and the source type. By default, source position corrections
are attempted only for NIRSpec MOS and NIRSpec and MIRI LRS fixed-slit point sources.
Set to False to ignore position estimates for all modes; set to True to additionally attempt
source position correction for NIRSpec BOTS data or extended sources.

``--smoothing_length``
If ``smoothing_length`` is greater than 1 (and is an odd integer), the
image data used to perform background extraction will be smoothed in the
Expand All @@ -20,7 +47,7 @@ The ``extract_1d`` step has the following step-specific arguments.
``--bkg_fit``
The type of fit to perform to the background data in each image column
(or row, if the dispersion is vertical). There are four allowed values:
"poly", "mean", and "median", and None (the default value). If left as None,
"poly", "mean", "median", and None (the default value). If left as None,
the step will search the reference file for a value - if none is found,
``bkg_fit`` will be set to "poly". If set to "poly", the background
values for each pixel within all background regions in a given column (or
Expand All @@ -45,59 +72,28 @@ The ``extract_1d`` step has the following step-specific arguments.
0 for that particular column (or row). If "bkg_fit" is not "poly", this
parameter will be ignored.

``--bkg_sigma_clip``
The background values will be sigma-clipped to remove outlier values from
the determination of the background. The default value is a 3.0 sigma clip.

``--log_increment``
Most log messages are suppressed while looping over integrations, i.e. when
the input is a CubeModel or a 3-D SlitModel. Messages will be logged while
processing the first integration, but since they would be the same for
every integration, most messages will only be written once. However, since
there can be hundreds or thousands of integrations, which can take a long
time to process, it would be useful to log a message every now and then to
let the user know that the step is still running.

``log_increment`` is an integer, with default value 50. If it is greater
than 0, an INFO message will be printed every ``log_increment``
integrations, e.g. "... 150 integrations done".
For multi-integration extractions, if this parameter is set to a value greater
than zero, an INFO-level log message will be printed every `log_increment` integrations
to report on progress. Default value is 50.

``--subtract_background``
This is a boolean flag to specify whether the background should be
subtracted. If None, the value in the :ref:`EXTRACT1D <extract1d_reffile>`
reference file (if any) will be used. If not None, this parameter overrides
the value in the reference file.
``--save_profile``
Flag to enable saving the spatial profile representing the extraction aperture.
If True, the profile is saved to disk with suffix "profile".

``--use_source_posn``
This is a boolean flag to specify whether the target and background extraction
region locations specified in the :ref:`EXTRACT1D <extract1d_reffile>` reference
file should be shifted
to account for the expected position of the source. If None (the default),
the step will make the decision of whether to use the source position based
on the observing mode and the source type. The source position will only be
used for point sources and for modes where the source could be located
off-center due to things like nodding or dithering. If turned on, the position
of the source is used in conjunction with the World Coordinate System (WCS) to
compute t F438 he x/y source location. For NIRSpec non-IFU modes, the source position
is determined from the ``source_xpos/source_ypos`` parameters. For MIRI LRS fixed slit,
the dither offset is applied to the sky pointing location to determine source position.
All other modes use ``targ_ra/targ_dec``. If this parameter is specified in the
:ref:`EXTRACT1D <extract1d_reffile>` reference file, the reference file value will
override any automatic settings based on exposure and source type. As always, a value
given by the user as an argument to the step overrides all settings in the reference
file or within the step code.
``--save_scene_model``
Flag to enable saving a model of the 2D flux as defined by the extraction aperture.
If True, the model is saved to disk with suffix "scene_model".

Step Arguments for IFU Data
---------------------------

``--center_xy``
A list of two integer values giving the desired x/y location for the center
of the circular extraction aperture used for extracting spectra from 3-D
IFU cubes. Ignored for non-IFU modes and non-point sources. Must be given in
x,y order and in units of pixels along the x,y axes of the 3-D IFU cube, e.g.
``--center_xy="27,28"``. If given, the values override any position derived
from the use of the ``use_source_posn`` argument. Default is None.

``--apply_apcorr``
Switch to select whether or not to apply an APERTURE correction during the
Extract1dStep processing. Default is ``True``
IFU cubes. Must be given in x,y order and in units of pixels along the x,y
axes of the 3-D IFU cube, e.g. ``--center_xy="27,28"``.
Default is None.

``--ifu_autocen``
Switch to select whether or not to enable auto-centroiding of the extraction
Expand All @@ -106,6 +102,10 @@ The ``extract_1d`` step has the following step-specific arguments.
becomes extremely low) and using DAOStarFinder to locate the brightest
source in the field. Default is ``False``.

``--bkg_sigma_clip``
The background values will be sigma-clipped to remove outlier values from
the determination of the background. The default value is a 3.0 sigma clip.

``--ifu_rfcorr``
Switch to select whether or not to run 1d residual fringe correction on the
extracted 1d spectrum (MIRI MRS only). Default is ``False``.
Expand All @@ -128,65 +128,66 @@ The ``extract_1d`` step has the following step-specific arguments.
for covariance between adjacent spaxels in the IFU data cube. The default value is
1.0 (i.e., no correction) unless set by a user or a parameter reference file. This
parameter only affects MIRI and NIRSpec IFU spectroscopy.


Step Arguments for NIRISS SOSS Data
-----------------------------------

``--soss_atoca``
This is a NIRISS-SOSS algorithm-specific parameter; if True, use the ATOCA
algorithm to treat order contamination. Default is ``True``.
Flag to enable using the ATOCA algorithm to treat order contamination. Default is ``True``.

``--soss_threshold``
This is a NIRISS-SOSS algorithm-specific parameter; this sets the threshold
value for a pixel to be included when modelling the spectral trace. The default
Threshold value for a pixel to be included when modeling the spectral trace. The default
value is 0.01.

``--soss_n_os``
This is a NIRISS-SOSS algorithm-specific parameter; this is an integer that sets
An integer that sets
the oversampling factor of the underlying wavelength grid used when modeling the
trace. The default value is 2.

``--soss_estimate``
This is a NIRISS-SOSS algorithm-specific parameter; filename or SpecModel of the
estimate of the target flux. The estimate must be a SpecModel with wavelength and
flux values.

``--soss_wave_grid_in``
This is a NIRISS-SOSS algorithm-specific parameter; filename or SossWaveGridModel
Filename or SossWaveGridModel
containing the wavelength grid used by ATOCA to model each valid pixel of the
detector. If not given, the grid is determined based on an estimate of the flux
(soss_estimate), the relative tolerance (soss_rtol) required on each pixel model
and the maximum grid size (soss_max_grid_size).

``--soss_wave_grid_out``
This is a NIRISS-SOSS algorithm-specific parameter; filename to hold the wavelength
Filename to hold the wavelength
grid calculated by ATOCA, stored in a SossWaveGridModel.

``--soss_estimate``
Filename or SpecModel of the
estimate of the target flux. The estimate must be a SpecModel with wavelength and
flux values.

``--soss_rtol``
This is a NIRISS-SOSS algorithm-specific parameter; the relative tolerance needed on a
The relative tolerance needed on a
pixel model. It is used to determine the sampling of the soss_wave_grid when not
directly given. Default value is 1.e-4.

``--soss_max_grid_size``
This is a NIRISS-SOSS algorithm-specific parameter; the maximum grid size allowed. It is
The maximum grid size allowed. It is
used when soss_wave_grid is not provided to make sure the computation time or the memory
used stays reasonable. Default value is 20000.

``--soss_tikfac``
This is a NIRISS-SOSS algorithm-specific parameter; this is the regularization
This is the regularization
factor used in the SOSS extraction. If not specified, ATOCA will calculate a
best-fit value for the Tikhonov factor.

``--soss_width``
This is a NIRISS-SOSS algorithm-specific parameter; this specifies the aperture
This specifies the aperture
width used to extract the 1D spectrum from the decontaminated trace. The default
value is 40.0 pixels.

``--soss_bad_pix``
This is a NIRISS-SOSS algorithm-specific parameter; this parameter sets the method
This parameter sets the method
used to handle bad pixels. There are currently two options: "model" will replace
the bad pixel values with a modeled value, while "masking" will omit those pixels
from the spectrum. The default value is "model".

``--soss_modelname``
This is a NIRISS-SOSS algorithm-specific parameter; if set, this will provide
If set, this will provide
the optional ATOCA model output of traces and pixel weights, with the filename
set by this parameter. By default this is set to None and this output is
not provided.
Loading
Loading
0