Orthogonality & Gram-Schmidt Process

Introduction

"Orthogonal" comes from Greek: orthos (right, correct) + gonia (angle). In everyday language, it means "perpendicular" or "at a right angle." In mathematics and machine learning, orthogonality represents a deeper concept: complete independence.

Two orthogonal vectors share absolutely no common direction. In the context of data, if two features are orthogonal, knowing one tells you nothing about the other. They provide completely unique, non-redundant information. This is why orthogonality is sometimes called the "Holy Grail" of feature engineering.

Why Orthogonality Matters in ML

• Feature Independence: Orthogonal features maximize information content per feature.
• Numerical Stability: Orthogonal matrices are perfectly conditioned (condition number = 1).
• Gradient Flow: Orthogonal weight matrices preserve gradient magnitudes in deep networks.
• Efficient Computation: Inverting orthogonal matrices is trivial (just transpose).

Orthogonality is the mathematical engine behind algorithms like PCA, SVD, QR decomposition, and orthogonal weight initialization in neural networks.

Orthogonal Vectors

Two vectors \mathbf{u} and \mathbf{v} in \mathbb{R}^{n} are orthogonal (written \mathbf{u}\bot\mathbf{v} if their inner product (dot product) is zero:

\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^{T}\mathbf{v} = \sum_{i = 1}^{n}u_{i}v_{i} = 0

This definition extends to any finite dimension n, not just 2D or 3D.

Example: Verifying Orthogonality

Let \mathbf{u} = \lbrack 1,2,3\rbrack and \mathbf{v} = \lbrack 1,1, - 1\rbrack Are they orthogonal?

u · v = (1)(1) + (2)(1) + (3)(-1)

u · v = 1 + 2 - 3 = 0

Yes, u and v are orthogonal!

The Zero Vector

The zero vector 0 is orthogonal to every vector (including itself), since 0 \cdot \mathbf{v} = 0 for any v. However, when we talk about orthogonal sets or bases, we typically exclude the zero vector.

Orthogonal Sets

A set of vectors \{\mathbf{v}_{1},\mathbf{v}_{2},\ldots,\mathbf{v}_{k}\}{v1​,v2​,…,vk​} is an orthogonal set if every pair is orthogonal:

\mathbf{v}_{i} \cdot \mathbf{v}_{j} = 0\quad\text{for~all~}i \neq j all i=j

Key Theorem: An orthogonal set of non-zero vectors is always linearly independent. This makes orthogonal vectors extremely useful as basis vectors.

Geometric Intuition

The dot product has a beautiful geometric interpretation that explains why orthogonal vectors have dot product zero:

\mathbf{u} \cdot \mathbf{v} = \mid\mid\mathbf{u}\mid\mid \cdot \mid\mid\mathbf{v}\mid\mid \cdot \cos(\theta)

When \theta = 90{^\circ} \cos(90{^\circ}) = 0 so the dot product is zero.

Projection Interpretation

The dot product measures "how much of u lies in the direction of v."

\text{proj}_{\mathbf{v}}(\mathbf{u}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\mid\mid\mathbf{v}\mid\mid^{2}}\mathbf{v}

If dot = 0, projection = 0. The vectors share no common direction.

Pythagorean Theorem

For orthogonal vectors, the Pythagorean theorem holds:

\mid\mid\mathbf{u} + \mathbf{v}\mid\mid^{2} = \mid\mid\mathbf{u}\mid\mid^{2} + \mid\mid\mathbf{v}\mid\mid^{2}

This is because the cross-term 2(\mathbf{u} \cdot \mathbf{v}) vanishes.

Interactive: Projection & Orthogonality

Adjust the angle of vector u to see its projection onto v. When the projection vanishes, the vectors are orthogonal.

Vector Projection

Decomposing u into components parallel and perpendicular to v.

Angle45°

0° (Parallel)90° (Perp)180°

Decomposition

Projection (\text{proj}_{v}u)(projv​u)70.7

Parallel

Rejection (u - \text{proj})(u−proj)70.7

Perp

Dot Product: 7071 When the angle is 90° (or 270°), the dot product is zero, and the projection vanishes.

Orthonormal Bases

An orthonormal set is an orthogonal set where every vector also has unit length (norm = 1). This is the "gold standard" for coordinate systems.

1. Orthogonal

\mathbf{e}_{i} \cdot \mathbf{e}_{j} = 0

(if i ≠ j)

2. Normalized

\mid\mid\mathbf{e}_{i}\mid\mid = 1

(unit length)

Combined using Kronecker delta:

\mathbf{e}_{i} \cdot \mathbf{e}_{j} = \delta_{ij}

Why Orthonormal Bases are Powerful

In an orthonormal basis, finding coordinates is trivial. You don't need to solve a system of equations; you just take dot products!

\mathbf{x} = \sum_{i = 1}^{n}(\mathbf{x} \cdot \mathbf{e}_{i})\mathbf{e}_{i}

The coordinate for \mathbf{e}_{i} is just \mathbf{x} \cdot \mathbf{e}_{i} This is why Fourier series and Wavelet transforms (which use orthonormal bases) are computationally feasible.

Orthogonal Matrices

An orthogonal matrix is a square matrix Q whose columns form an orthonormal set. (Confusingly named; should be "orthonormal matrix").

Defining Property

Q^{T}Q = QQ^{T} = I

Which implies: Q⁻¹ = Q^T

Examples

Rotation Matrix

\begin{bmatrix} {\cos\theta} & {- \sin\theta} \\ {\sin\theta} & {\cos\theta} \end{bmatrix}[cosθsinθ​−sinθcosθ​]

Reflection Matrix

\begin{bmatrix} 1 & 0 \\ 0 & {- 1} \end{bmatrix}[10​0−1​]

Permutation Matrix

\begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix}[01​10​]

Identity Matrix

\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}[10​01​]

Properties & Proofs

1. Isometry (Length Preserving)

Multiplying by Q does not change length.

\mid\mid Q\mathbf{x}\mid\mid = \mid\mid\mathbf{x}\mid\mid

Proof: \mid\mid Q\mathbf{x}\mid\mid^{2} = (Q\mathbf{x})^{T}(Q\mathbf{x}) = \mathbf{x}^{T}Q^{T}Q\mathbf{x} = \mathbf{x}^{T}I\mathbf{x} = \mid\mid\mathbf{x}\mid\mid^{2}

2. Angle Preserving

Dot products (and therefore angles) are preserved.

(Q\mathbf{x}) \cdot (Q\mathbf{y}) = \mathbf{x} \cdot \mathbf{y}(Qx)⋅(Qy)=x⋅y

Proof: (Q\mathbf{x})^{T}(Q\mathbf{y}) = \mathbf{x}^{T}Q^{T}Q\mathbf{y} = \mathbf{x}^{T}I\mathbf{y} = \mathbf{x}^{T}\mathbf{y}(Qx)T(Qy)=xTQTQy=xTIy=xTy

3. Determinant

Determinant is always ±1.

\det(Q) = \pm 1

Proof: \det(Q^{T}Q) = \det(I) = 1 and \det(Q^{T}) = \det(Q) so \det(Q)^{2} = 1

4. Eigenvalues

All eigenvalues lie on the complex unit circle.

\mid\lambda\mid = 1

Proof: If Q\mathbf{v} = \lambda\mathbf{v} then \mid\mid\mathbf{v}\mid\mid = \mid\mid Q\mathbf{v}\mid\mid = \mid\lambda\mid\,\mid\mid\mathbf{v}\mid\mid so \mid\lambda\mid = 1

Computational Advantage

Inverting a general matrix is O(n³). Inverting an orthogonal matrix is O(n²) (just transpose!). Also, condition number = 1 means perfect numerical stability.

Gram-Schmidt Process

The Gram-Schmidt process transforms any linearly independent vectors into an orthonormal basis for the same space. It works by iteratively subtracting the projection onto previous vectors.

The Algorithm

Step 1: Normalize first vector

\mathbf{e}_{1} = \frac{\mathbf{v}_{1}}{\mid\mid\mathbf{v}_{1}\mid\mid}

Step 2: Subtract projection on e1

\mathbf{u}_{2} = \mathbf{v}_{2} - (\mathbf{v}_{2} \cdot \mathbf{e}_{1})\mathbf{e}_{1} \mathbf{e}_{2} = \frac{\mathbf{u}_{2}}{\mid\mid\mathbf{u}_{2}\mid\mid}

Step k: Subtract all previous projections

\mathbf{u}_{k} = \mathbf{v}_{k} - \sum_{j = 1}^{k - 1}(\mathbf{v}_{k} \cdot \mathbf{e}_{j})\mathbf{e}_{j} \mathbf{e}_{k} = \frac{\mathbf{u}_{k}}{\mid\mid\mathbf{u}_{k}\mid\mid}

Interactive: Gram-Schmidt

Watch step-by-step how orthogonalization happens.

1

2

3

4

5

1. Define Vectors

Start with two linearly independent vectors.

Initial Vectors

v₁ Angle15°

v₂ Angle60°

Current Operation

Configure vectors...

Back

Next Step

QR Decomposition

Matrix form of Gram-Schmidt: A = QR.

A = QR

• Q: Orthogonal matrix (the e vectors).
• R: Upper triangular matrix (the dot products).

Used for solving least squares (Rx = Q^{T}b and finding eigenvalues.

ML Applications

Orthogonal Weight Initialization

Initializing RNN/LSTM weights as orthogonal matrices prevents vanishing/exploding gradients because \mid\lambda\mid = 1 preserving signal magnitude over time.

PCA & Decorrelation

PCA finds orthogonal directions of maximum variance. This "whitens" or decorrelates data, making downstream learning easier for models.

Orthogonal Regularization

Adding a loss term \mid\mid W^{T}W - I\mid\mid^{2} encourages weights to remain orthogonal during training, improving stability in GANs.