-
Notifications
You must be signed in to change notification settings - Fork 1
/
4.html
163 lines (163 loc) · 12.9 KB
/
4.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" type="text/css" href="sst_review_style.css">
<title>4 Presentation of Uncertainty</title>
</head>
<body>
<div class="row">
<div class="column side">
<p></p>
</div>
<div class="column middle">
<p><a href="3.7.html">Previous - 3.7 Estimates of Total Uncertainty</a> <a href="index.html">Index</a> <a href="5.html">Next - 5 Minimizing Exposure to Uncertainty</a></p>
<h1>4 Presentation of Uncertainty</h1>
<p>
At present, some groups provide explicit uncertainty estimates based on their analysis techniques
[Kaplan et al., 1998; Smith et al., 2008; Kennedy et al., 2011b, 2011c, Kennedy et al. 2019, Ishii et al., 2005;
Hirahara et al., 2013, Liu et al. 2015, Huang et al. 2016, 2017]. The uncertainty estimates derived
from a particular analysis will tend to misestimate the true uncertainty because they rely on
the analysis method and the assumptions on which it is based being complete and correct.
</p><p>
Comparing uncertainty estimates provided with analyses can be difficult (Kent et al. 2017) because
not all analyses consider the same sources of uncertainties. Consequently, a narrower uncertainty
range does not necessarily imply a better analysis. One way that data set providers could help
users is to provide an inventory of sources of uncertainty that have been considered either
explicitly or implicitly. This would allow users to assess the relative maturity of the
uncertainty analysis and assess when uncertainty estimates are comparable. There is a difficulty
with this approach however, which is that the methods used to process the data may not be
exactly comparable. For example, grid-box sampling uncertainty, which affects a simple gridded
data set such as Kennedy et al. [2019] would not have an exact counterpart in an analysis which
directly interpolates individual observations (e.g. the land component of Rohde et al. 2013).
Both analyses may well address uncertainty that arises due to sparse data coverage, but one could
not easily extract the necessary components of a like-for-like comparison.
</p><p>
There is a further difficulty in supplying and using uncertainty estimates: the traditional means
of displaying uncertainties, the error bar, or error range, does not, on its own, describe or
preserve the covariance structure of the errors. A user wishing to propagate uncertainty
information into and through their own analysis cannot do so correctly without this information.
It is possible to describe it (provided it is not overly complex) using covariance
matrices, which encapsulate not only uncertainties expressed as variances of the expected error
distributions but also the correlations between them.
</p><p>
Unfortunately, the storage requirements for covariance information for all but the
lowest resolution data sets can be prohibitively expensive. EOF-based analyses, like that of
Kaplan et al. [1998], could in principle efficiently store the spatial-error covariances because
only the covariances of the reduced space of principal components need to be kept. For Kaplan et
al. [1998], based on a reduced space of only 80 EOFs, this is a matrix of order 802 elements for
each time step as opposed to 10002 elements for the full-field covariance matrix. The difficulty
with this approach is that not all variability can be resolved by the leading EOFs and excluding
higher-order EOFs will underestimate the full uncertainty. Furthermore, this tells us nothing
about the temporal correlation of the errors. Full space-time error covariance matrices can be
very big indeed.
</p><p>
Another approach to presenting uncertainty information is to use "ensembles". These can take a
number of forms. As a very simple example, one can perturb the best estimate value in a data set by
generating samples of "noise" based on the estimated uncertainties and an assumed probability
distribution such as a Gaussian. Repeating this process provides an ensemble, each member of which
can be propagated through an analysis. The ensemble of results then give an estimate of the
uncertainty in the outcome given the uncertainty in the input. The general idea has been realised
in a number of different ways. This method is often referred to as the Monte-carlo method
and is widely used.
</p><p>
The simple example mentioned just above was applied in a more complex case in Jones and Kennedy [2017].
In that paper they drew samples from spatial covariance matrices and combined these using assumptions
about the temporal correlation of the errors. As a general approach, and even in simple situations,
it can often be simpler than performing a complete "propagation of uncertainty" calculation. It has
the added advantage that it works even in situations where the propagation of uncertainty formulae
breakdown, such as situations where the processing is non-linear or discontinuous, where the errors
are large and the linearity assumtion no longer holds, or where the error distributions themselves
have strange shapes (non-Gaussian, multi modal or whatever) that it would be desirable to preserve
through the uncertainty propagation process.
</p><p>
A problem with this approach though is that typically a measurement can be thought of as the true (albeit
unknowable) value plus an error (again, unknowable). By adding a sample from the estimated
uncertainty distribution, one ends up with a value that is the truth plus the actual error plus an artificial
error. While in some situations this can help to understand how uncertainties propagate, it would
be inappropriate to apply it naively in a study of, say, extreme values as the statistical properties
of the ensemble members are very different from those of the true SSTs.
</p><p>
A second kind of ensemble, often referred to as a parametric ensemble, is created by varying
parameters or analytical choices in an analysis to generate a set of outcomes. Ideally, one would vary
parameters through a plausible range or select from a range of equally defensible choices (e.g. the
choice of 90 or 95% for a significance threshold). Parametric ensembles were used to assess uncertainty
in SST bias estimates by Kennedy et al. [2011c], Kennedy et al. [2019] and in a broader range of
analytic choices by Liu et al. [2015] and Huang et al. [2016a].
This approach allows for the expression of complex error structures and allows users to easily propagate
uncertainty through an analysis. The interpretation of this kind of ensemble can be difficult however as
it is not always possible to vary all the relevant parameters, leading to an incomplete assessment
of uncertainty. Nor can the resulting ensemble be treated as a probability density function (or
approximation thereof). By appropriate parameter choice it may be possible to generate a pdf, however,
some parametric choices are effectively structural (e.g. where distinct modules can perform a given
task in very different ways) and may not fit neatly into such a scheme. A further limitation of the
approach is that the assessed parametric uncertainties are often smaller than the structural uncertainty.
It is worth noting that Kennedy et al. [2019] and
Huang et al. [2016a] generate their ensembles in slightly different ways. Kennedy et al. [2019]
choose continuous parameters randomly for each ensemble members from anywhere within the distribution.
Huang et al. [2016a] choose discrete values even for parameters that can take continuous values. This
means that each parameter can be more evenly sampled, at the expense of not exploring the full range of
possibilities.
</p><p>
A third kind of ensemble, which fixes some of the problems with the first simple kind of ensemble described
above, is where samples are drawn from the posterior distribution of a statistical analysis. For example,
Karspeck et al. [2012] drew samples from the posterior distribution of a statistical analysis of North
Atlantic SSTs. Each
sample provides an SST field that is consistent with the available observations and the estimated
covariance structures of the actual SST field and the observational errors. The resulting samples
have the advantage that each one "looks" like an SST field. Often the best estimate for a statistical
analysis (such as Kriging) will be smooth in areas with few or no observations, but have more
realistic spatial variability in areas where there are many observations. This is unrealistic and
undesirable for many applications such as forcing an atmosphere-only run of a climate model. Samples
solve this problem, at least to the extent that statistical
model describes those structures. Sampling has the added
advantage that it can be combined easily with samples from a parametric ensemble that represents errors
that are hard to treat analytically, such as residual errors associated with pervasive systematic errors.
However, production of samples is
not always computationally efficient. Karspeck et al. [2012] were able to do it for the North
Atlantic region, but the computational costs of extending the analysis unchanged to the rest of
the world could be prohibitive.
</p><p>
Once the decision has been made to present uncertainties as an ensemble, there are a number of ways
that the ensembles can themselves be presented. Mears et al. [2011] provide a best estimate
data set (of Microwave Sounding Unit-derived tropospheric temperatures) with an ensemble of perturbations
to that best estimate. The best estimate is different from
each of the ensemble members. Kennedy et al. [2011c] and Kennedy
et al. [2019] provide an ensemble where each ensemble member is equivalent and there is no single
best estimate. Liu et al. [2015] and Huang et al. [2016a] present an "operational" ensemble member used
as a best estimate, which has parameter choices that are judged to be optimal. The operational member
is one of a larger ensemble. In a sense these three approaches are equivalent - the same information can
be obtained from all three with a small amount of processing - however, they represent perhaps subtly
different ideas about what an ensemble represents and suggest different ways it might be used.
</p><p>
By providing a set of plausible realizations of a data set, it can be relatively easy for
users to assess the sensitivity of their analysis to uncertainties in SST data.
Kent et al. [2017] were able to compare HadSST3 and ERSSTv4 and detect significant differences
between them. By combining ensembles, parametric or otherwise, with estimates from multiple data
sets it is possible to get an idea of the overall uncertainty combining those parts of uncertainty
which can be more easily quantified with structural uncertainty, which is harder to assess
efficiently. For example,
individual ensemble members of HadSST3 were used in Tokinaga et al. [2012], along with other SST
analyses, to show that their results were robust to the estimated bias uncertainties in SSTs.
</p><p>
While ensembles are a practical approach in some respects, they can be impractical for large data sets, such
as those with high spatial or temporal resolution. Current satellite SST data sets have spatial resolutions
measured in kilometres and temporal resolutions of minutes, resolving small scale features and variations
in diurnal cycles. In such cases, generating, storing and retrieving large numbers of ensemble members
can be prohibitively expensive. Another approach [Merchant et al. 2013] is to separate out components
of the uncertainty that correlate
at different scales. Some measurement errors, such as sensor noise, are uncorrelated. Some
uncertainties, for example those related to water vapor in a satellite view, are locally-correlated
at a synoptic scale. Yet others are correlated at all times and places. Grouping uncertainties in
this way, together with information about the how correlations behave can allow users to propagate
uncertainty information more easily. This could be done by estimating effective degrees of freedom
within an area average (as recommended in the Product User Guide for the SST CCI project), by generating
error covariances over small regions relevant to a particular calculation and either sampling from
the distribution or propagating the uncertainties analytically. However, care needs to be taken to
ensure that the correlation scales and their associated shape parameters are realistic e.g. Bellprat et al. [2017].
</p>
<p><a href="3.7.html">Previous - 3.7 Estimates of Total Uncertainty</a> <a href="index.html">Index</a> <a href="5.html">Next - 5 Minimizing Exposure to Uncertainty</a></p>
<br><br><br><br>
</div>
</div>
</body>
</html>