Law of Large Numbers (LLN) Explained

Introduction

The Law of Large Numbers (LLN) is the anchor of statistics. It states a simple but powerful truth: as you collect more data, the sample average converges to the true expected value.

The Core Promise

{\overset{ˉ}{X}}_{n}\overset{n\rightarrow\infty}{\rightarrow}\mu

Sample mean approaches population mean as sample size grows

Without LLN, machine learning would be impossible. We assume that training loss approximates true generalization error. LLN is the mathematical license for that assumption.

The Casino Intuition

Why Casinos Always Win

Bet on Red (Roulette)

Win: 18/38 = 47.4%

Lose: 20/38 = 52.6%

Expected Value

E[X] = (1)(0.474) + (-1)(0.526)

= -$0.052 per bet

1 game:Player might win. Luck matters.

10 games:Still volatile. Streaks happen.

1,000,000 games:Average profit per game = exactly $0.052.

The casino is not gambling. It is running a business based on LLN.

Mathematical Statement

Let X_{1},X_{2},\ldots,X_{n} be i.i.d. random variables with mean \mu The sample mean is:

{\overset{ˉ}{X}}_{n} = \frac{1}{n}\sum_{i = 1}^{n}X_{i}

Sample mean = sum of observations divided by count

Law of Large Numbers

As n approaches infinity, the sample mean converges to the true mean:

\left. {\overset{ˉ}{X}}_{n}\rightarrow\mu\quad\text{as}\quad n\rightarrow\infty \right.

Interactive: Watch Convergence

Try different distributions and sample sizes. Notice how the running average stabilizes around the true mean as n grows. Small n = noisy. Large n = stable.

coin

die

exponential

Samples0 / 2000

True Mean0.50

Sample Mean0.000

True Mean0.50

Sample Mean0.000

Notice how the "swings" (variance) are huge at the start (small N), but the line inevitably tightens around the True Mean as N grows.

Why It Works: Variance Reduction

The intuition comes from looking at the variance of the sample mean.

Start with variance of sample mean:

Var({\overset{ˉ}{X}}_{n}) = Var\left( \frac{1}{n}\sum_{i = 1}^{n}X_{i} \right)

For independent variables:

= \frac{1}{n^{2}}\sum_{i = 1}^{n}Var(X_{i}) = \frac{1}{n^{2}} \cdot n\sigma^{2}=n21​∑i=1n​Var(Xi​)=n21​⋅nσ2

Result:

Var({\overset{ˉ}{X}}_{n}) = \frac{\sigma^{2}}{n}

The Key Insight

As \left. n\rightarrow\infty \right. variance \left. \rightarrow 0 \right.→0. A random variable with zero variance is a constant. Therefore, the sample mean becomes the constant \mu

Weak vs Strong LLN

There are two versions with different mathematical guarantees.

Weak LLN

Convergence in Probability

{\lim}_{n\rightarrow\infty}P(\mid{\overset{ˉ}{X}}_{n} - \mu\mid > \epsilon) = 0

For any margin \epsilon probability of being far from \mu goes to zero.

Strong LLN

Almost Sure Convergence

P\left( {\lim}_{n\rightarrow\infty}{\overset{ˉ}{X}}_{n} = \mu \right) = 1

The sample average converges with probability 1.

For most ML applications, the distinction does not matter. Both guarantee convergence.

LLN vs CLT

These are often confused. They describe different aspects of the same process.

See Central Limit Theorem for the distribution story.

The Gambler's Fallacy

The Mistake

"I got 10 heads in a row. LLN says it balances to 50%, so tails is 'due' next."

Why It's Wrong

The coin has no memory. LLN works by dilution, not compensation.

Example

After 10 heads:10H, 0T = 100% heads

Flip 1000 more (fair):~510H, ~500T

New ratio:510/1010 = 50.5%

The streak did not disappear. It just became statistically insignificant.

Interactive: Gambler's Fallacy Demo

Start with a streak of heads, then flip more. Watch how the ratio approaches 50% through dilution, not correction.

Initial "Unlucky" Streak

10

Starting with 10 Heads in a row (100%)

Add Fair Flips

0 / 500

Add Flips

Dilution Tank

100.0%

Heads %

50% Target

Total Stats

Heads10

Tails0

Absolute Diff10

Observed: The percentage drops towards 50%, but the absolute number of Heads is still much higher than Tails (diff: 10).

The universe didn't generate extra tails to "fix" the streak. It just buried the streak under a mountain of new, normal data. That is Dilution.

ML Applications

Monte Carlo Methods

Replace intractable integrals with sample averages:

\int f(x)p(x)dx \approx \frac{1}{N}\sum_{i = 1}^{N}f(x_{i})

Used in MCMC, Reinforcement Learning (value estimation), Bayesian inference.

Empirical Risk Minimization

Training loss approximates true generalization loss:

\frac{1}{n}\sum_{i = 1}^{n}L(f(x_{i}),y_{i}) \approx E\lbrack L(f(X),Y)\rbrack

The entire justification for training on finite datasets.

Stochastic Gradient Descent

Mini-batch gradient is an unbiased estimate of full gradient. Over many steps, the noise averages out. SGD converges because of LLN.

AlphaGo & MCTS

Cannot compute exact game tree values. Instead, play thousands of random games from a position. Average outcome converges to true value of the position.