Probability Distributions: Complete Guide
Introduction
A Random Variable is a variable whose value depends on the outcome of a random event. A Probability Distribution describes the likelihood of each possible value.
Core Question: If I sample from this process, what values am I likely to see, and how often?
Discrete
Countable outcomes: coin flips, dice rolls, number of emails.
Function: PMF (Probability Mass Function)
Rule: \sum P(x) = 1
Continuous
Infinite outcomes in a range: height, time, temperature.
Function: PDF (Probability Density Function)
Rule: \int f(x)dx = 1
PMF, PDF, and CDF
These three functions fully describe any probability distribution.
PMF: Probability Mass Function (Discrete)
Gives the probability of exactly each value: P(X = k)
Example: For a fair die, \text{PMF}(3) = 1/6
PDF: Probability Density Function (Continuous)
Gives probability density, not probability itself. P(X = x) = 0 for any single point!
The probability of a range is the area under the curve: P(a < X < b) = \int_{a}^{b}f(x)dx
CDF: Cumulative Distribution Function (Both)
Gives the probability of being at or below a value: F(x) = P(X \leq x)
Always starts at 0, ends at 1, and never decreases.
Discrete (PMF)
Continuous (PDF)
PDF f(x)
Curve shows density. Any single point has P=0! Only areas have probability.
Integrate / Sum
CDF F(x) = P(X ≤ x)
CDF always starts at 0, ends at 1, and never decreases. (Smooth curve for continuous)
Adjust x valuex = 0.00
F(x) = P(X \leq x) = \int_{- \infty}^{x}f(t)dt CDF height equals the shaded area in the PDF/PMF above it.
Interactive: Distribution Explorer
Select different distributions and adjust their parameters to build intuition for how they behave.
Normal
Binomial
Poisson
Exponential
Beta
Probability Density Function
f(x) = \frac{1}{\sqrt{2\pi 1.0^{2}}}e^{- \frac{(x - 0.0)^{2}}{2(1.0)^{2}}}
Click sample to generate data
Sample Data
Parameters
Mean (μ)0.0
Std Dev (σ)1.0
Theoretical Moments
Expected (Mean)0.00
Variance1.00
Discrete Distributions
Bernoulli
Single trial
One trial with two outcomes: Success (1) with probability p, Failure (0) with probability 1-p.
Mean: p
Variance: p(1 - p)
ML: Logistic Regression output. Binary Cross-Entropy is derived from Bernoulli likelihood.
Binomial
n trials
Number of successes k in n independent Bernoulli trials.
P(k) = \left( \frac{n}{k} \right)p^{k}(1 - p)^{n - k}
Mean: np
Variance: np(1 - p)
ML: A/B testing (how many users convert out of 1000?).
Poisson
Rare events
Number of events in a fixed interval, given average rate lambda.
P(k) = \frac{\lambda^{k}e^{- \lambda}}{k!}
Mean: \lambda
Variance: \lambda
ML: Website traffic spikes, call center volume, Poisson Regression.
Continuous Distributions
Normal (Gaussian)
The King
The bell curve. Defined by mean \mu and variance \sigma^{2} Due to the Central Limit Theorem, sums of random variables converge to this.
f(x) = \frac{1}{\sigma\sqrt{2\pi}}e^{- \frac{(x - \mu)^{2}}{2\sigma^{2}}}
ML: Weight initialization (Xavier/He), L2 regularization = Gaussian prior, VAEs.
Exponential
Waiting time
Models time between Poisson events. Has the memoryless property: P(wait 5 more min) is independent of how long you have waited.
f(x) = \lambda e^{- \lambda x},\quad x \geq 0
Mean: 1/\lambda
Variance: 1/\lambda^{2}
ML: Survival analysis, time-to-failure predictions.
Log-Normal
Heavy tail
If ln(X) is Normal, then X is Log-Normal. Models data that is positive with a heavy right tail.
ML: House prices, incomes, stock prices. This is why we log-transform targets in regression!
Bayesian Prior Distributions
These distributions describe probabilities of probabilities. They are used as priors in Bayesian inference.
Beta Distribution
Defined on [0, 1]. Represents uncertainty about a probability (like a coin's bias).
"I think the coin lands heads 60% of the time, but I'm not sure."
Conjugate prior for Binomial likelihood.
Dirichlet Distribution
Multivariate Beta. Distribution over probability vectors that sum to 1.
Used in Latent Dirichlet Allocation (LDA) for topic modeling.
For more on Bayesian inference, see Bayesian vs. Frequentist.
The Distribution Family Tree
Distributions are not isolated. They are connected through limiting relationships.
Bernoulli \rightarrow→ Binomial: Sum n Bernoullis and you get Binomial(n, p).
Binomial \rightarrow→ Normal: As \left. n\rightarrow\infty \right. Binomial looks Gaussian (CLT).
Binomial \rightarrow→ Poisson: As \left. n\rightarrow\infty \right. and \left. p\rightarrow 0 \right. (with np = \lambda constant), Binomial becomes Poisson.
Exponential \rightarrow→ Gamma: Sum k Exponentials and you get \text{Gamma}(k,\lambda)
Normal{}^{2} \rightarrow→ Chi-Square: Sum of squared standard normals gives Chi-Square.
ML Applications: Loss Functions
Deep learning loss functions are negative log-likelihoods of specific distributions. Choosing a loss function implicitly assumes a distribution on your target variable.
MSE Loss \leftrightarrow↔︎ Gaussian
Minimizing MSE assumes the target y follows a Gaussian distribution around the prediction.
\text{MSE} = - \log P(y\mid\hat{y},\sigma) + \text{const}
BCE Loss \leftrightarrow↔︎ Bernoulli
Binary Cross-Entropy assumes the target follows a Bernoulli distribution.
\text{BCE} = - \lbrack y\log(p) + (1 - y)\log(1 - p)\rbrack
Categorical CE \leftrightarrow↔︎ Multinomial
Categorical Cross-Entropy assumes the target follows a Multinomial (Categorical) distribution.
Regularization \leftrightarrow↔︎ Prior
L2 regularization = Gaussian prior on weights. L1 regularization = Laplace prior on weights.
Key insight: Understanding probability distributions helps you choose the right loss function for your problem. Predicting counts? Consider Poisson loss. Predicting positive values with heavy tails? Consider log-transforming the target (Log-Normal assumption).