Random Variables & Expectation: Complete Guide

Introduction

In algebra, a variable x stands for a fixed number. In probability, a Random Variable represents a value determined by chance. It could be any value within a range, each with some probability.

Random Variable: A function that maps outcomes of a random process to numbers.

Expectation: The theoretical average if we repeated the process infinitely.

In ML, everything is a random variable: input data X target labels Y and even model parameters \theta (in Bayesian learning). Understanding expectation is how we define what "good" means for a model.

What is a Random Variable?

Formal Definition

\left. X:\Omega\rightarrow\mathbb{R} \right.

A function that assigns a real number to every possible outcome in the sample space \Omega

Example: Two Coin Flips

Sample Space:

{HH, HT, TH, TT}

X = number of Heads:

X(HH) = 2

X(HT) = X(TH) = 1

X(TT) = 0

The random variable X converts outcomes (like "HT") into numbers (like 1) that we can do math with.

Discrete vs Continuous

Discrete

Countable values: dice rolls, coin flips, number of emails.

Described by PMF:

P(X = x)

Each value has a specific probability. Sum of all probabilities = 1.

Continuous

Any value in a range: height, time, temperature.

Described by PDF:

P(a < X < b) = \int_{a}^{b}f(x)dx

P(X = exact value) = 0. Only ranges have nonzero probability.

For details on specific distributions, see Probability Distributions.

Expected Value (Mean)

The Expectation E[X] is the theoretical weighted average. Think of it as the "center of mass" of the distribution.

It answers: "If we repeated this experiment infinitely, what would the average value be?"

Discrete

E\lbrack X\rbrack = \sum_{x}x \cdot P(X = x)

Sum of (value x probability)

Continuous

E\lbrack X\rbrack = \int_{- \infty}^{\infty}x \cdot f(x)dx

Integral of (value x density)

Example: Rigged Die

A die where rolling 6 is 50% likely, and 1-5 are 10% each.

E[X] = (1)(0.1) + (2)(0.1) + (3)(0.1) + (4)(0.1) + (5)(0.1) + (6)(0.5)

E[X] = 0.1 + 0.2 + 0.3 + 0.4 + 0.5 + 3.0 = 4.5

The expected value is 4.5 (you can never roll 4.5!). The high probability on 6 pulls the average upward.

Interactive: Expected Value

See how E[X] acts as the "balance point" of the distribution. Try different probability distributions and watch where the fulcrum lands.

uniform

skewed low

skewed high

bimodal

Randomize

10%

E[X] = 5.50

Click bars to increase weight (+), Right-click to decrease (-)

Fulcrum finds the center of mass

Variance & Standard Deviation

Expectation tells us the center. Variance tells us the spread (how uncertain we are).

Variance: Expected squared deviation from mean

Var(X) = E\lbrack(X - \mu)^{2}\rbrack

Computational formula:

Var(X) = E\lbrack X^{2}\rbrack - (E\lbrack X\rbrack)^{2}

Variance

Units are squared (e.g., dollars squared). Hard to interpret directly.

Standard Deviation

\sigma = \sqrt{Var(X)} Same units as X More interpretable.

Interactive: Variance Comparison

Compare distributions with the same mean but different variances. Higher variance = wider spread = more uncertainty.

Low (σ²=0.5)

σ = 0.71

Medium (σ²=1.5)

σ = 1.22

High (σ²=4)

σ = 2.00

Low Variance

Tall & Narrow

High Certainty

Medium Variance

Balanced Shape

Moderate Certainty

High Variance

Short & Wide

Low Certainty

Key Properties

Linearity of Expectation

For any random variables X and Y (even if dependent!):

E\lbrack aX + bY\rbrack = aE\lbrack X\rbrack + bE\lbrack Y\rbrack

This is incredibly powerful. It lets us break complex problems into simple pieces.

Variance of Sum (Independent Variables)

If X and Y are independent:

Var(X + Y) = Var(X) + Var(Y)

Variances add for independent variables. This is why errors accumulate.

Scaling Properties

E\lbrack aX\rbrack = aE\lbrack X\rbrack

Var(aX) = a^{2}Var(X)

Multiplying by a constant scales variance by a squared!

ML Applications

Expected Risk (Loss Minimization)

We train models to minimize the expected loss on unseen data:

\theta^{\ast} = \arg{\min}_{\theta}E_{(x,y)}\lbrack L(f(x;\theta),y)\rbrack

Since we cannot compute the true expectation, we approximate with the empirical average over training data.

SGD Justification

By linearity of expectation, the expected gradient of a mini-batch equals the expected gradient of the full dataset. This is why Stochastic Gradient Descent works!

Reinforcement Learning

The Value Function is the expected future return:

V(s) = E\left\lbrack \sum_{t = 0}^{\infty}\gamma^{t}r_{t} \mid s_{0} = s \right\rbrack

The agent maximizes expected cumulative reward.

Variance in Model Performance

High variance in predictions means the model is sensitive to training data (overfitting). The bias-variance tradeoff is fundamentally about E\lbrack\text{error}\rbrack vs Var(\text{error})