Maximum Likelihood Estimation

TL;DR

Choose parameter values that make the observed data most probable.
Compare likelihoods of candidate parameter values or models and select the highest.
Commonly used to estimate parameters such as a coin’s success rate or the mean of a normal distribution.

Definition

Maximum likelihood estimation (MLE) is a statistical technique used to estimate the values of parameters in a given model. It is based on the idea that the observed data is most likely to have occurred under the model with the highest probability.

Explanation

MLE evaluates the likelihood of the observed data under different parameter values (or models) and selects the parameter values that maximize that likelihood. For candidate parameter values or competing models, compute the probability (likelihood) of the observed data given each candidate, then choose the candidate with the largest likelihood.

Examples

Example 1: Estimating the success rate of a coin

Suppose we have a coin that we believe to be fair. We toss the coin 10 times and obtain the sequence HTHTHTHTHT.

The probability of obtaining this sequence under the assumption that the coin is fair is 1/1024, because each toss has probability 1/2 and there are 10 independent tosses: $\left(\tfrac{1}{2}\right)^{10} = 1/1024.$

Consider two models:

Model 1: The coin is fair (probability of heads is 0.5). The likelihood of the sequence is $\left(\tfrac{1}{2}\right)^{10} = 1/1024.$
Model 2: The coin is biased (probability of heads is not 0.5). The likelihood of the sequence under this model is $p^{8}(1-p)^{2}.$

Since the likelihood of Model 1 is higher than the likelihood of Model 2, we conclude that the coin is likely to be fair.

Example 2: Estimating the mean of a normal distribution

Suppose a population is normally distributed with unknown mean and variance. We sample 10 observations and obtain: 2, 3, 4, 5, 6, 7, 8, 9, 10, 11.

The probability of obtaining this sample under the assumption of a normal distribution with mean 5.5 and variance 2.25 is 0.027.