Chapter 6 — A Unifying View: Hilbert Spaces
Companion material for Chapter 6. Shows how MMSE estimation, the Wiener filter, and linear least squares are all instances of orthogonal projection in a Hilbert space.
§ 6.1 Inner Product Spaces
Inner Product as Similarity
The correlation coefficient \rho is the cosine of the angle between two zero-mean random variables in L^2(\Omega). This demo lets you adjust \rho and see the geometry change in real time.
Inner product visualizer in \mathbb{R}^2: move two vectors and see \langle \mathbf{a}, \mathbf{b} \rangle = \|\mathbf{a}\|\|\mathbf{b}\|\cos\theta update.
Hilbert Spaces in This Course
Table: three settings — \mathbb{R}^n (vectors), L^2[0,T] (deterministic signals), L^2(\Omega) (random variables). Columns: inner product, norm, projection formula.
3Blue1Brown — Abstract vector spaces: extending linear algebra to function spaces and general vector space axioms. Directly motivates the Hilbert space framing of this chapter.
Probability Hilbert Space L^2(\Omega)
The autocorrelation function is the inner product in L^2(\Omega). This demo makes the connection between the inner product geometry and the computed ACF values tangible.
§ 6.2 Signal Representations
Basis Expansion
3Blue1Brown — Fourier series as a basis expansion: sine and cosine functions form an orthonormal basis in L^2[0,T]. The projection formula c_n = \langle f, e_n \rangle is the same formula used in this chapter.
Fourier series builder: add basis functions one by one and watch the partial sums converge to a target waveform.
Examples of Orthonormal Bases
Gram–Schmidt orthogonalization: start from a non-orthogonal set of functions, construct an orthonormal basis, and verify orthogonality numerically.
§ 6.3 Projections and the Gramian
Projection onto Orthonormal Vectors
3D animation: project a vector onto a 2D subspace. Show \hat{x} = \sum_i \langle x, e_i \rangle e_i and the error perpendicular to the subspace.
Gramian in Each Setting
Side-by-side Gramian matrices: (1) deterministic vectors [\langle \phi_i, \phi_j \rangle], (2) random variables [\mathbb{E}[\phi_i \phi_j]] = correlation matrix. The math is identical.
MMSE as Projection in L^2(\Omega)
For a bivariate Gaussian, compute the MMSE estimate analytically and verify that the error is orthogonal to the observation X.
§ 6.4 Unifying View
Where the Course Lives
Summary diagram: MMSE, Wiener filter, and Linear Least Squares — all as orthogonal projections, each labeled with its specific Hilbert space setting and projection formula.
Unified Projection Formula
Verify that the projection formula \hat{x} = \Phi(\Phi^H\Phi)^{-1}\Phi^H x gives the same answer in all three settings (vectors, signals, random variables) on a small worked example.
3Blue1Brown — Essence of Linear Algebra (Dot Products and Projections)
3Blue1Brown — Dot products as projections, and the duality between vectors and linear functionals. The geometric core of the unifying view.