Estimating Uncertainty
I have just endured a BRUTAL several months of
revision to a paper on a LBL-type whale tracking system. A portion
of the revision involved how one estimates the uncertainty in the
resulting whale track. It hurts my head just to think about, but I
wanted to make some notes for future reference.
- My first recommendation is to first read the Guide to
Uncertainty In Measurement and understand the difference between
Type A errors (those you can measure with statistics) and Type B
errors (those you can only estimate) and their recommendations
regarding how to estimate them. If you do this before you even
begin a measurement project it will help you think about what you
need to measure later on. (The GUM is a NIST/ISO document that
isn’t free so you won’t usually find it floating around the web
willy-nilly.)
- When ever possible, try to make a series of ground-truth
measurements (i.e. try to position something for which you know the
position already). Using real measurements to characterize your
uncertainty is far far easier than trying to estimate it by other
means.
- When you have to measure the uncertainty in various parts of a
measurement and then propagate those uncertainties through a
subsequent calculation (using the law of propagation of variance),
it is worth while to list every measurement and how you will assess
it’s uncertainty. For example, in our whale tracking, how do you
know the GPS uncertainty, the detection algorithm uncertainty, the
sound speed and ray-tracking uncertainty, the uncertainty
associated with false detections or multipath, etc. Understanding
this up front may help you design measurements to address them
before you get to writing your paper.
- When attempting to assess the uncertainty in some measurement,
don’t be tempted to look at an O-scope, and roughly characterize
the uncertainty by the maximum variation you see. It might be more
conservative, but it is too hard to describe later. Make actual
measurements (preferably with a data acquisition system) save the
results, calculate a real statistic (RMS, std deviation, etc.)
- Finally, if it were all possible, it would be worth while to
get buy-in from all involved on the strategies for assessing each
of these uncertainties early on in the process.
There’s more to say and more to think about, but this is enough
for now.