JP-2250: Refactor refpix IRS2 to reduce memory usage #6356
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #6356 +/- ##
==========================================
- Coverage 77.68% 77.40% -0.28%
==========================================
Files 409 409
Lines 35099 35274 +175
==========================================
+ Hits 27265 27304 +39
- Misses 7834 7970 +136
*This pull request uses carry forward flags. Click here to find out more.
Continue to review full report at Codecov.
|
stscieisenhamer
approved these changes
Sep 16, 2021
loicalbert
pushed a commit
to talensgj/jwst
that referenced
this pull request
Nov 5, 2021
…6356) * JP-2250: Refactor refpix IRS2 to reduce memory usage * add change log entry
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #6282
Resolves JP-2250
Description
This PR refactors the parts of the refpix step used to correct NIRSpec exposures using the IRS2 readout mode, in order to reduce the memory usage. The major change is to the
subtract_referencefunction in irs2_subtract_reference.py to replace all of the repeated loops over "sectors" (regions of the data readout by the 4 different amps) with one big loop that computes and applies the correction to one "sector" at a time, so that you're not always carrying along the data for all 4 sectors all the time. Other more minor changes, such as moving the copy of input_model -> output_model to the end of the step, because only the SCI (data) array is needed for all of the computations, hence you're not carrying along 2 complete copies of the exposure for the entire duration, and making sure to explicitly delete the large objects as soon as they aren't needed anymore.Also did some general reorganization and clean-up, just to try to make the code more readable and manageable in the future (such as splitting up the previous single monolithic
subtract_referencefunction into several smaller pieces).For the dataset that initially triggered JP-2250, which uses NINTS=1 and NGROUPS=200, the old version of the step hit a peak memory usage of ~65GB. The new version peaks at around 35GB. Still quite big, but it's a really big exposure!
Checklist