Workable soss extraction #6467
Merged
Workable soss extraction #6467
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #6467 +/- ##
==========================================
- Coverage 78.33% 77.47% -0.86%
==========================================
Files 405 413 +8
Lines 34620 36685 +2065
==========================================
+ Hits 27119 28422 +1303
- Misses 7501 8263 +762
*This pull request uses carry forward flags. Click here to find out more.
Continue to review full report at Codecov.
|
hbushouse
reviewed
Jan 5, 2022
Closed
added 22 commits
January 5, 2022 22:24
…de them from the log-likelihood.
… pixels are reconstructed.
X, and Y offset boundaries are currently limited to +/-10 pixels and theta +/- 1 degree. Theta I think is good, but X/Y could potentially be shrunk if need be.
…s pulled out of model_image and placed in run_extract1d. Logic for handling order 1 and or 2 in cases of F277 or SUBSTRIP96 exposures tested.
…a_utils and soss_extract
…ixes to get a clean result in testing
…d pixel_mapping funcs
8e6f96e to
2b3d839
Compare
hbushouse
approved these changes
Jan 6, 2022
|
🎉 🚀 |
stscieisenhamer
pushed a commit
to stscieisenhamer/jwst
that referenced
this pull request
Jan 7, 2022
* Add prepared SOSS extraction code (engine will follow). * Add first pseudo-code for interaction with the pipeline. * Further pseudo-code for interaction with the pipeline. * Add code into the Extract1dStep to branch in case of NIRISS SOSS observation. * Add the partially reviewed extraction engine. * Change import statement. * Integrate the Engine into soss_extract.py and bugfixes in soss_solver.py * Add some logging messages. * Invert scimask and increase precision of scidata and scierr. * Added Tikhonov factor as a Step parameter. * Increase strength of background mask. * Set masked pixels to zero when computing the rebuilt trace, but exclude them from the log-likelihood. * Compute the log-likelihood of the best-fit trace model. * Added more SOSS parameters to the Extract1dStep. * Use a separate instance of the engine for evaluating the model so bad pixels are reconstructed. * Add 1D box extraction, uses external code that will soon be integrated. * Handle x, y, wavelength info correctly when performing the box extraction. * Added transform as a Step parameter. * Added number of pixels inside the photometric aperture to the box extraction. * Added checks for implemented filters/subarrays. * Set background to 0 instead of NaN for fully masked columns. * Various updated comments and TODO items. * Temporarily added devname to control development output files. * Change way out_table is constructed to fix number of fields bug. * Improved way the former border_mask, now refmask, is constructed. * Re-write solver to use clean parts of order 1 and 2 for determining a suitable transformation. * Split image modelling and box-extraction. This will make adding outputs and filter/subarray functionality easier. * Add box extraction code to the jwst repository. * Updating TODO items. * Re-factor and debug transforms, partially to fix some issues with tilt in orders 2/3. * Remove obsolete I/O function. * Fix double background subtraction when using the solver. * Add optional diagnostic figure to solve_transform(). * Bypass broken sanity check, fix later. * Remove changes made to Tikhonov, need to implement update. * Suppress understood warnings. * Extract on all columns by default. * Set the BACKGROUND field of the output spectra with the columnwise, per pixel values for now. * Add devtools file for now. * Comment out import so code will run * Remove outdated references to soss_read_refs Removed all edgetrig functions. verified that get_centroids_com still functions as expected, and that extract_1d.py runs properly. * remove redundant _plot_centroids function * Update docstrings * Add option to solver to not use rotation. * Fix docstrings * minor cleanup * Cleaner (in my opinion) way of implementing multiple models. Easier to extend in the future? * Implement new Tikhonov code, to be verified by Antoine. * Add first estimate of the tikhonov factor based on an extraction on the first order only * Minor corrections in soss_extract.py. Add possibility to choose the fill_value in Extraction Engine rebuild method. * Make a ExtractionEngine instance to model each order independently. * Add option to correct bad pixels using the trace models in soss_extract.extract_image * extract each orders in a for loop. * Add bad pixel handling parameter to Extract1dStep. * Add small corrections * Estimate empirically bad pixels errors using pixels with similar value. * Rename files to atoca * Update imports to new file names (atoca). * Add description in the code file header referencing to ATOCA. * Include a reference to ATOCA in the log. * Typo fix Totally trivial - I just noticed a typo in the ATOCA logging message when running run_extraction.py * first attempt - safe space * Remove debugging TODOs. * Update soss_centroids.py Update the bounds test re GJ's suggestion. Also reformat to match further checks. * Update soss_solver.py * Add spectral order and integration number to the 1d spectra outputs (MultiSpecModel or SpecModel) * Use proper inputs ref files from stsci * Work in progress * First implementation to enable F277W branching and to handle subarrays. * First implementation to enable F277W branching and to handle subarrays. * extract_1d_step.py and soss_extract.py Added transform = [0,0,0] for the F277W case. * Stable first pass at solution * cleanup solve_transform * Fix verbose mode. Minor debugs * extract_1d_step.py and soss_extract.py Fixed minor bug when not outputting the expected number of parameters. * Bug correction in WebbKernels class. Add minimum value to the kernels and explicit inputs when building the convolution matrix in the Engine. * Include WebbKernel modifications into soss_extract.py. Use directly the wavelength solution (1d) instead of the 2d wave map when calling WebbKernel. Use only the chi2 to determine the best tikhonov factor. * Make sure all wavelengths in wavelength maps are following the same direction in the dispersion axis. * soss_extract.py extract_1d_step.py Added temporary skip to the background subtraction step for testing performance. * Bugfix: The input wavemap was modified inside the function transform_wavemap. Not anymore. * Correct bug in check_dispersioon_direction. Clean up code. * Add options for fit with only rotation or only offsets * F277W filter compatibility in solver * Add soss_filter argument to model_image * Add boundaries in chi_squared minimization. X, and Y offset boundaries are currently limited to +/-10 pixels and theta +/- 1 degree. Theta I think is good, but X/Y could potentially be shrunk if need be. * soss_extract.py soss_solver.py Important revision. Solve_transform was pulled out of model_image and placed in run_extract1d. Logic for handling order 1 and or 2 in cases of F277 or SUBSTRIP96 exposures tested. * Change scale in error plot * soss_extract.py bug correction after merge * Remove debuging lines * remove more debug lines * remove refmask TODO * Rectify rebase * Rectify rebase * attempt a commit to existing PR, using soss_utils as test * docstrings and logging for syscor, move ref output out of default return * docstrings and logging for centroids and solver. removing plotting and empty main * cleaning up soss_devname, some TODOs, still work to do on atoca, atoca_utils and soss_extract * atoca_utils updates to L1900, logging to atoca * atoca_utils updates * atoca and atoca_utils updates * final pass through atoca_utils * test newton method over backwards-incompatible root_scalar * penultimate commit? soss_extract cleaned, adopting Howard's keyword fixes to get a clean result in testing * Final changes? Merged extract() into __call__, more docstrings, merged pixel_mapping funcs * put soss fork after modelcontainer check * rebase, changelog * rebase and changelog * flake8 Co-authored-by: Geert Jan Talens <talens@astro.umontreal.ca> Co-authored-by: radicamc <radica@astro.umontreal.ca> Co-authored-by: loicalbert <albert@astro.umontreal.ca> Co-authored-by: Tyler Pauly <tapastro@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This sugge
2D6E
stion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses JP-1550 and JP-201
Description
First workable version of the soss extraction algorithm (ATOCA).
Checklist