Learning a Constant from Data
Estimators, detectors, and the floor no estimator beats
Who knows what
Why the error splits in two
Insert the estimator’s own mean and expand, exactly as for the best constant guess: \[\E{(\widehat\Phi-\phi)^2} =\E{(\widehat\Phi-\E{\widehat\Phi})^2} +2\,\underbrace{\E{\widehat\Phi-\E{\widehat\Phi}}}_{=0}\,(\E{\widehat\Phi}-\phi) +(\E{\widehat\Phi}-\phi)^2 .\]
Variance: how much the estimate wobbles from sample to sample. Bias: where it wobbles around. A dartboard with a tight cluster off-center, and a wide cluster on-center.
Noise you can see without the truth
Two panels of twenty economists each forecast whether the economy grows next year (1) or shrinks (0).
Split: half are wrong whatever happens, and we know it today. Unanimous: error \(0\) or \(1\), and only the outcome will tell.
Variance (noise) is measured from the estimates alone; bias needs the truth.
One recipe, three levels
Unbiased is not the same as better
Matching moments
Method of moments With \(p\) unknown parameters, set \(\frac1N\sum_k x_k^n=\E{X^n;\phi}\) for \(n=1,\dots,p\) and solve.
The puzzle (\(p=1\)). The model mean is \(\phi/2\), so \[\widehat\phi_{\text{MoM}}=\frac2N\sum_kx_k=20.96,\] the twice the average guess: unbiased, consistent, one moment.
A Gaussian (\(p=2\)). Matching \(\E{X}\) and \(\E{X^2}\) gives \[\widehat m=\bar x,\qquad \widehat\sigma^2=\frac1N\sum_k(x_k-\bar x)^2,\] the divide-by-\(N\) estimator of the last slide.
Suspiciously wasteful: it reads only the average, while a sample near the edge says far more about where the edge is. The rest of the week explains how much it throws away.
Two moments do not pick a law
The fits differ only in the higher moments, which twelve points estimate very badly. Matching moments answers the questions you asked of it, and no others.
The simplest inference
- Radar. An aircraft in this range cell, or receiver noise alone?
- Alarm. Someone at the door, or a sensor drifting?
- Communication. Was a \(0\) sent, or a \(1\)?
The parameter takes only two values, \(H_0\) and \(H_1\), and the answer must be one of them, with no option to hedge.
With two answers there is no estimator distribution to track: the whole problem is which data lead to which answer.
From a test statistic to regions
Threshold a test statistic: \(\Theta(\mathbf x)\gtrless t\), so that \(\mathcal X_1=\set{\mathbf x:\Theta(\mathbf x)>t}\).
A statistic collapses \(N\)-dimensional data onto one axis, so the region becomes an interval. The question is which statistic.
One threshold, one point on the ROC
The diagonal is a coin flip. Better separation pushes the curve to the top-left corner.
From data to ROC: the intruder alarm
A door sensor reports a score every second. Threshold it, collect the two conditional histograms, sweep the threshold.
Sweeping the threshold
One threshold is one operating point. The curve is the whole family of rules of this form, and choosing a detector means choosing a point on it.
A noisier sensor breaks the simple rule
When an intruder changes the spread of the score and not only its level, a threshold on \(x\) is the wrong shape of rule. What should be thresholded?
The noisier sensor, resolved
The region \(\set{\Lambda>t}\) is two tails, not an interval: an extreme score either way is what the wide alternative predicts. Its ROC lies above the simple threshold’s everywhere.
In Gaussian noise, one number decides everything
A known level in Gaussian noise \(X=\pm A+N\), \(N\sim\Gauss(0,\sigma^2)\). Then \(\ln\Lambda(x)=2Ax/\sigma^2\) is linear in \(x\), so the test is a threshold on \(x\) after all, and \(P_F=Q\!\paren{\tfrac{\gamma+A}{\sigma}}\), \(P_D=Q\!\paren{\tfrac{\gamma-A}{\sigma}}\).
The ROC depends only on \(d=2A/\sigma\), the separation in noise units. That is why detection performance is always quoted as a signal-to-noise ratio.
The reveal: count the alternations
Count the steps where the outcome changes. For a fair, independent coin these are \(N-1\) iid Bernoulli\((\tfrac12)\) indicators, so the same test applies with \(N-1\) in place of \(N\): \[Z_A=\frac{\widehat P_A-\tfrac12}{1/(2\sqrt{N-1})} .\]
People break up runs because runs look unrandom, and alternate at about \(0.6\): \(Z_A\approx2\) at \(N=100\). Same formula, a different statistic: real randomness has longer runs than intuition expects.
The density, read backwards
Hold the data fixed and let the parameter vary: \(f(\mathbf x;\phi)\) becomes the likelihood of \(\phi\). \[\widehat\phi_{\text{ML}}=\argmax_\phi\ \ln f(\mathbf x;\phi) =\argmax_\phi\ \sum_{k=1}^N\ln f_X(x_k;\phi).\]
Detection compared two members of the family. Maximum likelihood compares all of them: \(\widehat\phi_{\text{ML}}\) is the candidate that wins every pairwise likelihood-ratio test at threshold \(t=1\).
Setting the derivative to zero
The Gaussian mean With \(\sigma^2\) known, \(\ln f=-\frac{1}{2\sigma^2}\sum_k(x_k-\phi)^2+\text{const}\), so \[\frac{\partial}{\partial\phi}\ln f=\frac{1}{\sigma^2}\sum_k(x_k-\phi)=0 \quad\Longrightarrow\quad \widehat\phi_{\text{ML}}=\frac1N\sum_k x_k .\]
The sample mean was maximum likelihood all along, for Gaussian data. The logarithm moved no maximum: it turned the product into a sum, one term per observation.
The likelihood sharpens with data
More data barely moves the peak; it makes it sharper. A sharp peak means the data tell nearby values of \(\phi\) apart. The curvature at the peak is the subject of the last part.
The three sampling distributions
The puzzle by maximum likelihood
Each time stamp has density \(1/\phi\) on \([0,\phi]\), and \(0\) beyond it. So \[f(\mathbf x;\phi)=\begin{cases}\phi^{-N}, & \phi\ge\max_kx_k,\\ 0, & \text{otherwise.}\end{cases}\]
No derivative is zero anywhere: the maximum sits on the edge of the admissible set. Any smaller \(\phi\) rules out a time stamp we saw; any larger one spreads the same five points more thinly.
Scaling up the maximum
\(N\) points drop into \([0,\phi]\) and cut it into \(N+1\) gaps. By symmetry the gaps have the same expected length, \(\phi/(N+1)\), including the one between the largest value and the edge.
So the maximum falls short by one gap on average, \(\E{\max_kX_k}=\frac{N}{N+1}\,\phi\), and the fix adds the average gap seen in the data: \[\widehat\phi=\frac{N+1}{N}\max_kx_k=\max_kx_k+\frac{\max_kx_k}{N}=19.8+3.96=23.76 .\]
The puzzle, answered three ways
The maximum can never exceed the truth, so ML is biased low, yet it has the smallest spread. Scaling by \((N+1)/N\) removes the bias for a little extra spread. Both leave the moment estimator far behind.
The floor, and who stands on it
For Gaussian data \(I_N=N/\sigma^2\): no unbiased estimator beats \(\sigma^2/N\), and the sample mean sits exactly on it. The median pays about 25% in spread.
The floor for a variance
For Gaussian data the information about \(\sigma^2\) is \(I=N/(2\sigma^4)\), so every unbiased estimator of the variance has \(\Var{\widehat\Sigma^2}\ge 2\sigma^4/N\).
The unbiased estimator never reaches the bound: it is a lower limit, not a promise. The ML curve sits below it, which is no contradiction, since the bound speaks only about unbiased estimators.
ML against the average
The average cannot tell the three laws apart, because it never looks at the density. ML does: it is twice as good for Laplacian data, and for uniform data its error falls like \(1/N^2\) instead of \(1/N\).