Posted by James Youngman on June 29, 1998 at 08:29:38:
I have an application in which a least-squares fit
needs to be done online (i.e. without storing the
whole input vector). I have somple code to do this.
It turns out that I also need to perform
extreme-outlier rejection and I don't know how to
do that (online).
Details: Input data is linear, gradient may be + or
- but the sign is known in advance. N is between 10
and 500,000. The noise is normal, and the signal to
noise ratio varies between roughly 1e-4 and 1e0.
Where the noise ratio approaches 1e0 N approaches
500,000.
As well as the normally distributed noise, there are
rogue measurements which are the only ones that we
must reject. They're wild. We we measure S.D. of the
successive y deltas having rejected these outliers
by eyeball, the measurements to be rejected lie at
Z values in excess of (say) 50.