Unbiased estimator

An estimator is unbiased when the mean of its sampling distribution is exactly the parameter, $E(\hat{\theta}) = \theta$, and when that holds for every possible value of the parameter rather than for one convenient case. The sample mean qualifies, $E(\bar{x}) = \mu$ (x-bar, mu), and so does the sample proportion, $E(\hat{p}) = p$ (p-hat). Unbiasedness pins down where the sampling distribution sits and says nothing about how wide it is.

Take the four-value population 1, 2, 3, 6, which has $\mu = 3$, $\sigma^2 = 3.5$ and $\sigma = 1.8708$. Write out all 16 samples of size 2 drawn with replacement and average each statistic across them. The sample mean averages to 3, exactly $\mu$. The sample variance with the $n-1$ divisor averages to 3.5, exactly $\sigma^2$. The sample standard deviation averages to $\sqrt{2} = 1.4142$, which is 75.6 percent of $\sigma$.

So this sentence is false: "the $n-1$ divisor makes $s$ an unbiased estimator of $\sigma$." What $n-1$ buys is $E(s^2) = \sigma^2$. Taking a square root is nonlinear, and the mean of a square root is not the square root of a mean, so the correction does not survive the step. $s$ comes out too small on average, at every sample size, for any population in which it varies at all.

The shortfall shrinks with $n$ without reaching zero: in the same population, all 1,024 samples of size 5 give an average $s$ of 1.7673, or 94.5 percent of $\sigma$. Inference for means uses $s$ in place of $\sigma$ regardless, because at ordinary sample sizes the gap is small and the exact correction depends on the population's shape.

Unbiasedness is also not the whole of a good estimator. Topic 3.1, Estimators, weighs it against variability: an estimator centered perfectly but scattered widely can be worse than one that leans slightly and lands close every time.