MAP estimation: maximum likelihood plus a prior

The problem, from first principles

07-maximum-likelihood-estimation's MLE has a real weakness: with very little data, it trusts that small sample completely, 3 coin flips landing heads twice gives an MLE of 66.7% heads, even though you might have good reason to believe, before seeing any flips, that most coins are close to fair. MLE has no way to incorporate that prior belief, it only ever looks at the data in front of it.

MAP (Maximum A Posteriori) estimation fixes exactly this: it combines the likelihood (how well a parameter explains the data, MLE's whole criterion) with a prior (05-bayes-theorem's prior belief, before seeing the data), and finds the parameter value that maximizes their product, the posterior. With a very informative prior and little data, MAP leans on the prior; with lots of data, MAP converges to the same answer MLE would give, the data eventually overwhelms any reasonable prior.

From theory to code

Theory expresses the (unnormalized) log-posterior as the log-likelihood plus the log-prior, and, for the specific case of a Normal likelihood with a Normal prior on the mean, derives a closed-form MAP estimate: a precision-weighted average of the sample mean and the prior mean.

Implement negative_log_posterior_normal(mean_candidate, x, data_std, prior_mean, prior_std) (reusing 07-maximum-likelihood-estimation's NLL and 06-likelihood-vs-probability's normal_pdf), then map_estimate_normal_mean(x, data_std, prior_mean, prior_std), the closed form.

Constraints

• data_std is assumed known and fixed, only the mean is being estimated.
• map_estimate_normal_mean is closed-form, no search.
• With a very large prior_std (an uninformative prior), the MAP estimate should approach the plain sample mean (MLE).
• With a very small prior_std (an extremely confident prior), the MAP estimate should approach prior_mean itself.

Hints

Open one at a time. Each gives away a little more than the last.