Lindleys Paradox

TL;DR

A source of conflicting conclusions between Bayesian and frequentist analyses when sample sizes are small.
Frequentist methods emphasize observed results; Bayesian methods weigh those results against prior probabilities.
Small samples can make it difficult for Bayesian analyses to overcome prior beliefs and for frequentist conclusions to be reliable.

Definition

Lindley’s paradox is a statistical concept that describes the tension between Bayesian and frequentist interpretations of probability. This tension arises when a statistical hypothesis is tested using a small sample size, leading to conflicting conclusions about the likelihood of the hypothesis being true.

Explanation

Lindley’s paradox highlights how the same small-sample data can be interpreted differently depending on the statistical paradigm:

Under a frequentist interpretation, inference focuses on the observed data and may conclude that a hypothesis is supported when the proportion or test statistic from the sample favors that hypothesis.
Under a Bayesian interpretation, inference combines the observed data with a prior probability for the hypothesis; with a small sample size, a prior (especially if the prior probability is low or high) can dominate and lead to a different conclusion than the frequentist result.

The paradox therefore emphasizes the role of sample size and prior beliefs: small samples can make it hard for Bayesian updating to override prior probabilities, while frequentist conclusions based solely on observed proportions may not account for prior information.

Examples

Clinical trial example

Imagine a researcher tests whether a medication is effective using a clinical trial with a sample of 10 patients, half of whom receive the medication and half of whom receive a placebo. The results of the trial show that 6 out of the 5 patients who received the medication experienced a significant improvement in their condition, while only 2 out of the 5 patients who received the placebo experienced a similar improvement.

From a frequentist perspective, the greater proportion of improvement among treated patients appears to support the medication’s effectiveness.
From a Bayesian perspective, the small sample size and a likely low prior probability for effectiveness make it difficult to draw a definitive conclusion that the medication is effective.

Coin-toss example

Consider testing whether a coin is fair by tossing it 10 times. If the coin lands heads 5 times and tails 5 times, one may conclude the coin is fair. If it lands heads 8 times and tails 2 times, one may conclude the coin is biased toward heads.

A frequentist interpretation would focus on the observed proportions and may conclude bias when heads appear 8 times.
A Bayesian interpretation, assuming a high prior probability that the coin is fair, may consider the small sample insufficient to overturn that prior and therefore not find compelling evidence of bias.

Notes or pitfalls

Small sample sizes are central to the paradox: they increase the chance that Bayesian and frequentist conclusions conflict.
The paradox underscores the need to consider study limitations and the potential for bias in results when interpreting small-sample hypothesis tests.

Bayesian interpretation of probability
Frequentist interpretation of probability