Description
The issue
The threshold plugin currently processes multi realization data without any kind of slicing to reduce the amount of data being held in memory. This can lead to enormous memory requirements when processing data from models that produce large numbers of ensemble members (e.g. ECMWF). This is necessary if the realizations are being preserved, but when we are collapsing the realization coordinate, which is common, we could reduce the memory usage considerably.
The work
This work is to re-engineer the threshold plugin to be more memory efficient in these cases, ideally showing consistent memory usage regardless of the number of ensemble members being processed.
Follow on
We may need to undertake similar work for any other improver processing steps that don't use realization slicing and are thus likely to show undesirable memory behaviour when using larger ensembles.
Acceptance criteria:
- the threshold plugin shows more consistent memory usage when being applied to data with differing numbers of realizations when the realization coordinate is being collapsed; this should be demonstrated with statistics.