spacetelescope · emolter · Oct 22, 2024 · Oct 11, 2024 · Oct 11, 2024 · Oct 11, 2024
@@ -0,0 +1 @@
+reorganize outlier detection documentation
@@ -1,19 +1,53 @@
 .. _outlier_detection_step_args:
 
+Step Arguments
+==============
+
+The outlier detection step has the following optional arguments
+that control the behavior of the processing.
 For more details about step arguments (including datatypes, possible values
 and defaults) see :py:obj:`jwst.outlier_detection.OutlierDetectionStep.spec`.
 
-Step Arguments for Non-IFU data
-===============================
-The `outlier_detection` step for non-IFU data has the following optional arguments
-that control the behavior of the processing:
+
+General Step Arguments
+----------------------
+The following arguments apply to all modes unless otherwise specified:
+
+``--save_intermediate_results``
+  Specifies whether or not to save any intermediate products created
+  during step processing.
+
+``--good_bits``
+  The DQ bit values from the input image DQ arrays
+  that should be considered 'good'. Any pixel with a DQ value not included
+  in this value (or list of values) will be ignored when resampling and flagged
+  when building the weight mask. See DQ flag :ref:`dq_parameter_specification` for details.
+  Has no effect for IFU data.
+
+``--snr``
+  The signal-to-noise values to use for bad pixel identification.
+  Since cosmic rays often extend across several pixels, the user
+  must specify two cut-off values for determining whether a pixel should
+  be masked: the first for detecting the primary cosmic ray, and the
+  second (typically lower threshold) for masking lower-level bad pixels
+  adjacent to those found in the first pass.  Valid values are a pair of
+  floating-point values in a single string (for example "5.0 4.0").
+  Has no effect for IFU data.
+
+
+Step Arguments for Imaging and Slit-like Spectroscopic data
+-----------------------------------------------------------
 
 ``--weight_type``
-  The type of data weighting to use during resampling.
+  The type of data weighting to apply to the resampled data. Available options are ``ivm``
+  (default) to compute and use an inverse-variance map, and ``exptime`` to
+  weight by the exposure time.
 
 ``--pixfrac``
-  The pixel fraction used during resampling;
-  valid values go from 0.0 to 1.0.
+  The pixel fraction used during resampling; valid values go from 0.0 to 1.0.
+  Indicates the fraction by which input pixels are "shrunk" before being drizzled onto the
+  output image grid. This specifies the size of the footprint, or "dropsize", of a pixel
+  in units of the input pixel size.
 
 ``--kernel``
   The form of the kernel function used to distribute flux onto a
@@ -29,15 +63,6 @@ that control the behavior of the processing:
   The percent of maximum weight to use as lower-limit for valid data;
   valid values go from 0.0 to 1.0.
 
-``--snr``
-  The signal-to-noise values to use for bad pixel identification.
-  Since cosmic rays often extend across several pixels the user
-  must specify two cut-off values for determining whether a pixel should
-  be masked: the first for detecting the primary cosmic ray, and the
-  second (typically lower threshold) for masking lower-level bad pixels
-  adjacent to those found in the first pass.  Valid values are a pair of
-  floating-point values in a single string (for example "5.0 4.0").
-
 ``--scale``
   The scaling factor applied to derivative used to identify bad pixels.
   Since cosmic rays often extend across several pixels the user
@@ -50,59 +75,54 @@ that control the behavior of the processing:
 ``--backg``
   User-specified background value to apply to the median image.
 
-``--rolling_window_width``
-  Number of integrations over which to take the median when using rolling-window
-  median for TSO observations.
-
-``--save_intermediate_results``
-      Specifies whether or not to save any intermediate products created
-  during step processing.
-
 ``--resample_data``
   Specifies whether or not to resample the input images when
   performing outlier detection.
 
-``--good_bits``
-  The DQ bit values from the input image DQ arrays
-  that should be considered 'good' when building the weight mask. See
-  DQ flag :ref:`dq_parameter_specification` for details.
-
 ``--allowed_memory``
   Specifies the fractional amount of
   free memory to allow when creating the resampled image. If ``None``, the
   environment variable ``DMODEL_ALLOWED_MEMORY`` is used. If not defined, no
   check is made. If the resampled image would be larger than specified, an
   ``OutputTooLargeError`` exception will be generated.
-
   For example, if set to ``0.5``, only resampled images that use less than half
   the available memory can be created.
 
 ``--in_memory``
   Specifies whether or not to load and create all images that are used during
   processing into memory. If ``False``, input files are loaded from disk when
   needed and all intermediate files are stored on disk, rather than in memory.
-  This flag is superseded by the pipeline-level ``--in-memory`` flag, and thus
-  has no effect when running the full level 3 pipeline.
+  Has no effect for spectroscopic data. For imaging data this parameter is 
+  superseded by the pipeline-level ``in_memory`` parameter set by
+  ``calwebb_image3``.
+
 
 Step Arguments for IFU data
-===========================
-The `outlier_detection` step for IFU data has the following optional arguments
-that control the behavior of the processing:
+---------------------------
 
 ``--kernel_size``
   The size of the kernel to use to normalize the pixel differences. The kernel size
   must only contain odd values. Valid values are a pair of ints in a single string
-  (for example "7 7").
+  (for example "7 7", the default).
 
 ``--threshold_percent``
   The threshold (in percent) of the normalized minimum pixel difference used to identify bad pixels.
-  Pixels with   a normalized minimum pixel difference above this percentage are flagged as a outlier.
+  Pixels with a normalized minimum difference above this percentage are flagged as outliers.
 
-``--save_intermediate_results``
-  Specifies whether or not to save any intermediate products created
-  during step processing.
+``--ifu_second_check``
+  Perform a secondary check searching for outliers. This will set outliers
+  where ever the difference array of adjacent pixels is a Nan.
 
-``--in_memory``
-  Specifies whether or not to load and create all images that are used during
-  processing into memory. If ``False``, input files are loaded from disk when
-  needed and all intermediate files are stored on disk, rather than in memory.
+
+Step Arguments for TSO data
+---------------------------
+
+``--rolling_window_width``
+  Number of integrations over which to take the median when using rolling-window
+  median for TSO observations. The default is 25. If the number of integrations
+  is less than or equal to ``rolling_window_width``, a simple median is used instead.
+
+
+Step Arguments for Coronagraphic data
+-------------------------------------
+General step arguments apply to coronagraphic data. No additional arguments are used.
@@ -8,12 +8,12 @@ Outlier Detection
    :maxdepth: 2
 
    main.rst
+   reference_files.rst
    arguments.rst
-   outlier_detection_step.rst
    outlier_detection_imaging.rst
+   outlier_detection_spec.rst
    outlier_detection_coron.rst
    outlier_detection_tso.rst
    outlier_detection_ifu.rst
-   outlier_detection_spec.rst
 
 .. automodapi:: jwst.outlier_detection
@@ -1,58 +1,42 @@
-Description
-===========
+.. _outlier_design:
 
-:Classes: `jwst.outlier_detection.OutlierDetectionStep`
-:Aliases: outlier_detection
+Overview
+========
+
+This module provides the sole interface to all methods of performing outlier
+detection on JWST observations.
 
 Processing multiple datasets together allows for the identification of bad pixels
-or cosmic-rays that remain in each of the input images, many times at levels which
-were not detectable by the :ref:`jump <jump_step>` step. The ``outlier_detection`` step 
-implements the following algorithm to identify and flag any remaining cosmic-rays or
-other artifacts left over from previous calibrations:
-
-#. build a stack of input data
-
-   - all inputs will need to have the same WCS since outlier detection assumes 
-     the same flux for each point on the sky, and variations from one image to
-     the next would indicate a problem with the detector during readout of that pixel
-   - if needed, each input will be resampled to a common output WCS
-
-#. create a median image from the stack of input data
-
-   - this median operation will ignore any input pixels which have a weight 
-     which is too low (<70% max weight)
-
-#. create "blotted" data from the median image to exactly match each original
-   input dataset
-
-#. perform a statistical comparison (pixel-by-pixel) between the median blotted
-   data with the original input data to look for pixels with values that are  
-   different from the mean value by more than some specified sigma 
-   based on the noise model
-
-   - the noise model used relies on the error array computed by previous 
-     calibration steps based on the readnoise and calibration errors
-
-#. flag the DQ array for the input data for any pixel (or affected neighboring
-   pixels) identified as a statistical outlier
-
-The outlier detection step serves as a single interface to apply this general 
-process to any JWST data, with specific variations of this algorithm for each 
-type of data.  Sub-classes of the outlier detection algorithm have been developed
-specifically for:
-
-#. Imaging data
-#. IFU spectroscopic data
-#. TSO data
-#. coronagraphic data
-#. spectroscopic data
-
-This allows the outlier_detection step to be tuned to the variations in each type 
-of JWST data.
-
-Reference Files
-===============
-
-The ``outlier_detection`` step uses the PARS-OUTLIERDETECTIONSTEP parameter reference file.
-
-.. include:: ../references_general/pars-outlierdetectionstep_reffile.inc
+or cosmic rays that remain in each of the input images, often at levels which
+were not detectable by the :ref:`jump <jump_step>` step.
+The ``outlier_detection`` step supports multiple
+algorithms and determines the appropriate algorithm for the type of observation
+being processed.  This step supports:
+
+* **Image modes**: 'FGS_IMAGE', 'MIR_IMAGE', 'NRC_IMAGE', 'NIS_IMAGE'
+   - See :ref:`outlier-detection-imaging` for algorithm details
+* **Slit-like Spectroscopic modes**: 'MIR_LRS-FIXEDSLIT', 'NRS_FIXEDSLIT', 'NRS_MSASPEC'
+   - See :ref:`outlier-detection-spec` for algorithm details
+* **Time-Series-Observation (TSO) modes**: 'MIR_LRS-SLITLESS', 'NRC_TSGRISM', 'NIS_SOSS', 'NRS_BRIGHTOBJ', 'NRC_TSIMAGE'
+   - See :ref:`outlier-detection-tso` for algorithm details
+* **IFU Spectroscopic modes**: 'MIR_MRS', 'NRS_IFU'
+   - See :ref:`outlier-detection-ifu` for algorithm details
+* **Coronagraphic Image modes**: 'MIR_LYOT', 'MIR_4QPM', 'NRC_CORON'
+   - See :ref:`outlier-detection-coron` for algorithm details
+
+This step uses the following logic to apply the appropriate algorithm to the
+input data:
+
+#. Interpret inputs (Association, ModelContainer, ModelLibrary, or CubeModel)
+   to identify all input observations to be processed
+
+#. Read in parameters set by user. See :ref:`outlier_detection_step_args` for the full list
+   of parameters.
+
+#. Select outlier detection algorithm based on exposure type in input model ``meta.exposure.type``.
+
+#. Instantiate and run outlier detection class determined for the exposure type
+   using parameter values interpreted from inputs.
+
+#. Update DQ arrays with flags and set SCI, ERR, and variance arrays to NaN at the location
+   of identified outliers.
@@ -1,33 +1,30 @@
 .. _outlier-detection-coron:
 
-Outlier Detection for Coronagraphic Data
-========================================
+Coronagraphic Data
+==================
 
 This module serves as the interface for applying ``outlier_detection`` to coronagraphic
-image observations.
-
-Specifically, this routine performs the following operations:
+image observations. A :py:class:`~jwst.datamodels.CubeModel` serves as the basic format
+for all processing performed by this step. This routine performs the following operations:
 
 #. Extract parameter settings from input model and merge them with any user-provided values.
    See :ref:`outlier detection arguments <outlier_detection_step_args>` for the full list
    of parameters.
 
-#. Convert input data, as needed, to make sure it is in a format that can be processed.
-   A :py:class:`~jwst.datamodels.CubeModel` serves as the basic format for all processing
-   performed by this step.
+#. Do not attempt resampling; data are assumed to be aligned and have an identical WCS.
+   This is true automatically for a CubeModel.
 
-#. Create a median image preserving the spatial dimensions of the cube.
+#. Create a median image over the `groups` (exposures, planes of cube) axis,
+   preserving the spatial (x,y) dimensions of the cube.
 
    * The ``maskpt`` parameter sets the percentage of the weight image values to
-     use, and any pixel with a weight below this value gets flagged as "bad" and
-     ignored when resampled.
+     use, and any pixel with a weight below this value gets flagged as "bad".
 
 #. Perform statistical comparison between median image and original image to identify outliers.
 
    The core detection algorithm uses the following to generate an outlier mask
 
    .. math:: | image\_input - image\_median | > SNR*input\_err
 
-#. Update input data model DQ arrays with mask of detected outliers.
-
-.. automodapi:: jwst.outlier_detection.coron
+#. Update DQ arrays with flags and set SCI, ERR, and variance arrays to NaN at the location
+   of identified outliers.
@@ -1,34 +1,48 @@
 .. _outlier-detection-ifu:
 
-Outlier Detection for IFU Data
+Integral Field Unit (IFU) Data
 ==============================
 
 This module serves as the interface for applying ``outlier_detection`` to IFU
-observations, like those taken with NIRSpec and MIRI.  The code implements the
-basic outlier detection algorithm searching for pixels that are consistent outliers
-in the calibrated images created by the :ref:`calwebb_spec2 <calwebb_spec2>` pipeline.
-After launch it was discovered the
-bad pixels on the MIRI detectors vary with time. The pixels varied from usable to unusable, and at times,
-back to usable  on a time frame that was too short (sometimes as short as 2 days)  to fold into the bad pixel mask applied
-in the :ref:`calwebb_detector1 <calwebb_detector1>` pipeline. 
-At this time it is believed that NIRSpec IFU data also have bad pixels that vary with time, though the time variation is
-still under study.
+observations, like those taken with NIRSpec and MIRI. A :ref:`Stage 3 association <asn-level3-techspecs>`,
+which is loaded into a :py:class:`~jwst.datamodels.ModelContainer` object,
+serves as the basic format for all processing performed by this step.
+
+After launch it was discovered that the bad pixels on the MIRI detectors vary with time.
+The pixels varied from usable to unusable, and at times, back to usable  on a time frame that was too short
+(sometimes as short as 2 days)  to fold into the bad pixel mask applied in the 
+:ref:`calwebb_detector1 <calwebb_detector1>` pipeline. At this time it is believed that NIRSpec IFU data
+also have bad pixels that vary with time, though the time variation is still under study.
+The outlier detection step is designed to flag these pixels as outliers, in addition
+to cosmic ray hits that were not flagged by the jump step.
 
 The basis of the outlier detection flagging for IFU data  is to look for pixels on the detector
 that are regularly discrepant from their neighbors, with a sharper division than could be explained
-by the detector PSF. The algorithm flags pixels that are  outliers when compared to their neighbors for a set of
-input files contained in an association. The neighbor pixel differences are the neighbors in the spatial direction.
-For MIRI data neighbor differences are found to the left and right of every
-science pixel, while for NIRSpec data the neighbor differences are
-determined from pixels above and below every science pixel. The difference between the MIRI MRS and NIRPSpec algorithm for finding the
-spatial pixel differences is due to the opposite dispersion directions between the two instruments. 
-The  pixel differences for each input model 
-in the association are determined and stored in a stack of pixel differences. 
-For each pixel the minimum difference
-through this stack is determined and normalized. The normalization uses a local median of the difference array
-(set by the kernel size). A pixel is flagged as an outlier if  this normalized minimum difference
-is greater than the input threshold percentage.  Pixels that are found to be outliers are flaged in in the DQ array.
-
-
-
-.. automodapi:: jwst.outlier_detection.ifu
+by the detector PSF.
+This routine performs the following operations:
+
+#. Extract parameter settings for the input ModelContainer and merge them with any user-provided values.
+   See :ref:`outlier detection arguments <outlier_detection_step_args>` for the full list of parameters.
+
+#. Loop over cal files, computing nearest-neighbor differences for each pixel
+   in the along-dispersion direction.
+   For MIRI, with the dispersion axis along the y axis, the neighbors that are used to
+   to find the differences are to the left and right of each pixel being examined.
+   For NIRSpec, with the dispersion along the x axis, the neighbors that are used to
+   find the differences are above and below the pixel being examined.
+   The smaller of the two (left, right or up, down) differences is stored as the difference value for each
+   pixel. This avoids artifacts from bright edges.
+
+#. Compare the nearest-neighbor differences across science exposures to find the minimum
+   neighbor difference at each detector pixel.
+
+#. Determine a local spatial median of the minimum difference array using a median filter with a kernel size
+   set by the user according to the ``kernel_size`` parameter.
+
+#. Normalize the minimum difference array by the local median.
+
+#. Select outliers by flagging those normalized minimum values larger than the ``threshold_percent``
+   parameter.
+
+#. Update DQ arrays with flags and set SCI, ERR, and variance arrays to NaN at the location
+   of identified outliers.