Inner Product Spaces

6A. Inner Products and Norms

1) Dot Product

Dot Product

For $x,y\in\mathbb{R}^n$,

$$x\cdot y = x_1y_1+\cdots+x_ny_n.$$

Meaning: The dot product combines two real vectors into a scalar.

2) Inner Product

Inner Product

An inner product on $V$ assigns a scalar

$$\langle u,v\rangle\in\mathbb{F}$$

to every pair $u,v\in V$ and satisfies the following properties.

Positivity

$$\langle v,v\rangle\ge0.$$

Meaning: The inner product of a vector with itself is nonnegative.

Definiteness

$$\langle v,v\rangle=0 \iff v=0.$$

Meaning: Only the zero vector has zero inner product with itself.

Additivity in the First Slot

$$\langle u+v,w\rangle = \langle u,w\rangle+\langle v,w\rangle.$$

Meaning: The inner product is additive in its first argument.

Homogeneity in the First Slot

$$\langle \lambda u,v\rangle = \lambda\langle u,v\rangle.$$

Meaning: Scalars can be pulled out of the first argument.

Conjugate Symmetry

$$\langle u,v\rangle = \overline{\langle v,u\rangle}.$$

Meaning: Reversing the vectors conjugates the inner product.

3) Euclidean Inner Product

Euclidean Inner Product

For $w,z\in\mathbb{F}^n$,

$$\langle w,z\rangle = w_1\overline{z_1} +\cdots+ w_n\overline{z_n}.$$

Meaning: This is the standard inner product on $\mathbb{F}^n$.

4) Inner Product Space

Inner Product Space

An inner product space is a vector space together with an inner product.

Meaning: An inner product adds geometric ideas such as length and orthogonality to a vector space.

5) Basic Properties of an Inner Product

Linearity in the First Slot

For fixed $v$,

$$u\mapsto\langle u,v\rangle$$

is linear.

Meaning: The first argument behaves linearly.

Conjugate Homogeneity in the Second Slot

$$\langle u,\lambda v\rangle = \overline{\lambda}\langle u,v\rangle.$$

Meaning: Scalars pulled from the second argument are conjugated.

6A. Norms and Orthogonality

1) Norm

Norm

For $v\in V$,

$$\|v\| = \sqrt{\langle v,v\rangle}.$$

Meaning: The norm measures the length of a vector.

Basic Properties

$$\|v\|=0 \iff v=0,$$

and

$$\|\lambda v\| = |\lambda|\|v\|.$$

Meaning: Only the zero vector has zero length, and scaling a vector scales its length by the absolute value of the scalar.

2) Orthogonal Vectors

Orthogonal

Vectors $u,v\in V$ are orthogonal if

$$\langle u,v\rangle=0.$$

Meaning: Orthogonality generalizes the idea of perpendicular vectors.

3) Pythagorean Theorem

Pythagorean Theorem

If

$$\langle u,v\rangle=0,$$

then

$$\|u+v\|^2 = \|u\|^2+\|v\|^2.$$

Meaning: The squared lengths of orthogonal components add.

4) Orthogonal Decomposition with One Vector

Orthogonal Decomposition

If $v\neq0$, then

$$u = \frac{\langle u,v\rangle}{\|v\|^2}v + \left( u- \frac{\langle u,v\rangle}{\|v\|^2}v \right),$$

where

$$u- \frac{\langle u,v\rangle}{\|v\|^2}v$$

is orthogonal to $v$.

Meaning: A vector can be separated into a component parallel to $v$ and a component orthogonal to $v$.

5) Cauchy–Schwarz Inequality

Cauchy–Schwarz Inequality

For all $u,v\in V$,

$$|\langle u,v\rangle| \le \|u\|\|v\|.$$

Equality holds if and only if one vector is a scalar multiple of the other.

Meaning: The magnitude of an inner product cannot exceed the product of the vector lengths.

6) Triangle Inequality

Triangle Inequality

$$\|u+v\| \le \|u\|+\|v\|.$$

Meaning: The length of a sum cannot exceed the sum of the lengths.

7) Parallelogram Equality

Parallelogram Equality

$$\|u+v\|^2+\|u-v\|^2 = 2\left(\|u\|^2+\|v\|^2\right).$$

Meaning: The norm induced by an inner product satisfies the parallelogram law.

6B. Orthonormal Bases

1) Orthonormal List

Orthonormal

A list

$$e_1,\ldots,e_m$$

is orthonormal if

$$\langle e_j,e_k\rangle = \begin{cases} 1, & j=k,\\ 0, & j\neq k. \end{cases}$$

Meaning: Orthonormal vectors have length $1$ and are mutually orthogonal.

2) Norm of an Orthonormal Linear Combination

Orthonormal Combination

If $e_1,\ldots,e_m$ is orthonormal, then

$$\left\| a_1e_1+\cdots+a_me_m \right\|^2 = |a_1|^2+\cdots+|a_m|^2.$$

Meaning: With orthonormal vectors, the squared norm is simply the sum of the squared coefficients.

3) Linear Independence

Orthonormal Lists Are Linearly Independent

Every orthonormal list is linearly independent.

Meaning: Orthogonal unit vectors cannot contain redundant directions.

4) Bessel's Inequality

Bessel's Inequality

If $e_1,\ldots,e_m$ is orthonormal, then

$$|\langle v,e_1\rangle|^2 +\cdots+ |\langle v,e_m\rangle|^2 \le \|v\|^2.$$

Meaning: The energy contained in orthonormal components cannot exceed the total squared length of the vector.

5) Orthonormal Basis

Orthonormal Basis

An orthonormal basis is a basis that is also an orthonormal list.

Meaning: An orthonormal basis provides independent perpendicular unit directions spanning the entire space.

6) Coordinates in an Orthonormal Basis

Orthonormal Basis Expansion

If

$$e_1,\ldots,e_n$$

is an orthonormal basis, then

$$v = \langle v,e_1\rangle e_1 +\cdots+ \langle v,e_n\rangle e_n.$$

Meaning: Coordinates in an orthonormal basis are obtained directly using inner products.

7) Parseval's Identity

Parseval's Identity

If $e_1,\ldots,e_n$ is an orthonormal basis, then

$$\|v\|^2 = |\langle v,e_1\rangle|^2 +\cdots+ |\langle v,e_n\rangle|^2.$$

Meaning: The squared norm equals the sum of the squared orthonormal coordinates.

8) Inner Product from Orthonormal Coordinates

Inner Product Formula

If $e_1,\ldots,e_n$ is an orthonormal basis, then

$$\langle u,v\rangle = \sum_{k=1}^{n} \langle u,e_k\rangle \overline{\langle v,e_k\rangle}.$$

Meaning: The inner product can be computed entirely from orthonormal coordinates.

9) Gram–Schmidt Procedure

Gram–Schmidt Procedure

Given linearly independent vectors

$$v_1,\ldots,v_m,$$

define

$$f_1=v_1$$

and

$$f_k = v_k - \sum_{j=1}^{k-1} \frac{\langle v_k,f_j\rangle}{\|f_j\|^2}f_j.$$

Then define

$$e_k = \frac{f_k}{\|f_k\|}.$$

The resulting list

$$e_1,\ldots,e_m$$

is orthonormal.

Meaning: Gram–Schmidt removes previous vector components and then normalizes the result.

Span Preservation

For every $k$,

$$\operatorname{span}(v_1,\ldots,v_k) = \operatorname{span}(e_1,\ldots,e_k).$$

Meaning: Gram–Schmidt changes the basis vectors without changing the subspace they span.

10) Existence of Orthonormal Bases

Existence of an Orthonormal Basis

Every finite-dimensional inner product space has an orthonormal basis.

Meaning: Any finite-dimensional inner product space can be described using perpendicular unit basis vectors.

Extension of an Orthonormal List

Every orthonormal list in a finite-dimensional inner product space can be extended to an orthonormal basis.

Meaning: Existing orthonormal vectors can always be completed to a full orthonormal basis.

11) Schur's Theorem

Schur's Theorem

Every operator on a finite-dimensional complex inner product space has an upper-triangular matrix with respect to some orthonormal basis.

Meaning: Every complex operator can be upper triangularized without losing orthonormality of the basis.

12) Riesz Representation Theorem

Riesz Representation Theorem

If $V$ is finite-dimensional and

$$\varphi:V\to\mathbb{F}$$

is a linear functional, then there exists a unique $v\in V$ such that

$$\varphi(u) = \langle u,v\rangle$$

for every $u\in V$.

Meaning: Every linear functional on a finite-dimensional inner product space can be represented by an inner product with one unique vector.

6C. Orthogonal Complements and Minimization Problems

1) Orthogonal Complement

Orthogonal Complement

For $U\subseteq V$,

$$U^\perp = \{v\in V \mid \langle u,v\rangle=0 \text{ for every }u\in U\}.$$

Meaning: $U^\perp$ contains every vector orthogonal to all vectors in $U$.

2) Basic Properties of Orthogonal Complements

Orthogonal Complement Is a Subspace

$$U^\perp$$

is a subspace of $V$.

Meaning: Vectors orthogonal to $U$ form their own vector space.

Zero and Whole Space

$$\{0\}^\perp=V$$

and

$$V^\perp=\{0\}.$$

Meaning: Every vector is orthogonal to zero, while only zero is orthogonal to every vector.

Reverse Inclusion

If

$$G\subseteq H,$$

then

$$H^\perp\subseteq G^\perp.$$

Meaning: The larger the original set, the smaller its orthogonal complement.

3) Orthogonal Direct Sum

Orthogonal Decomposition

If $U$ is finite-dimensional, then

$$V = U\oplus U^\perp.$$

Meaning: Every vector can be uniquely decomposed into a component in $U$ and a component orthogonal to $U$.

4) Dimension of the Orthogonal Complement

Dimension Formula

If $V$ is finite-dimensional, then

$$\dim U^\perp = \dim V-\dim U.$$

Meaning: The dimensions of a subspace and its orthogonal complement add to the dimension of the whole space.

5) Double Orthogonal Complement

Double Orthogonal Complement

If $U$ is finite-dimensional, then

$$(U^\perp)^\perp = U.$$

Meaning: Taking the orthogonal complement twice returns the original finite-dimensional subspace.

6) Orthogonal Projection

Orthogonal Projection

If

$$V=U\oplus U^\perp$$

and

$$v=u+w, \qquad u\in U, \quad w\in U^\perp,$$

then the orthogonal projection onto $U$ is defined by

$$P_Uv=u.$$

Meaning: Orthogonal projection keeps the component inside $U$ and removes the orthogonal component.

7) Projection onto a One-Dimensional Subspace

Projection onto $\operatorname{span}(u)$

If

$$U=\operatorname{span}(u), \qquad u\neq0,$$

then

$$P_Uv = \frac{\langle v,u\rangle}{\|u\|^2}u.$$

Meaning: This formula extracts the component of $v$ parallel to $u$.

8) Properties of Orthogonal Projection

Range

$$\operatorname{range}P_U=U.$$

Meaning: Projection outputs vectors in $U$.

Null Space

$$\operatorname{null}P_U=U^\perp.$$

Meaning: Vectors orthogonal to $U$ are projected to zero.

Idempotence

$$P_U^2=P_U.$$

Meaning: Projecting twice gives the same result as projecting once.

Orthogonal Error

$$v-P_Uv\in U^\perp.$$

Meaning: The difference between a vector and its projection is orthogonal to the target subspace.

Projection Formula

If $e_1,\ldots,e_m$ is an orthonormal basis of $U$, then

$$P_Uv = \langle v,e_1\rangle e_1 +\cdots+ \langle v,e_m\rangle e_m.$$

Meaning: Projection onto a subspace is the sum of the components along an orthonormal basis of that subspace.

9) Closest Point in a Subspace

Best Approximation Theorem

If $u\in U$, then

$$\|v-P_Uv\| \le \|v-u\|.$$

Equality holds only when

$$u=P_Uv.$$

Meaning: The orthogonal projection is the unique point in $U$ closest to $v$.

6C. Pseudoinverse

1) Restricted Invertible Map

Restriction to $(\operatorname{null}T)^\perp$

If $V$ is finite-dimensional and

$$T\in\mathcal{L}(V,W),$$

then

$$T|_{(\operatorname{null}T)^\perp} : (\operatorname{null}T)^\perp \to \operatorname{range}T$$

is invertible.

Meaning: Removing the null-space directions makes $T$ one-to-one onto its range.

2) Pseudoinverse

Pseudoinverse

The pseudoinverse of $T$, denoted by

$$T^\dagger,$$

is defined by

$$T^\dagger w = \left( T|_{(\operatorname{null}T)^\perp} \right)^{-1} P_{\operatorname{range}T}w.$$

Meaning: The pseudoinverse first projects onto the reachable outputs and then reverses $T$ on the part where it is invertible.

3) Inverse as a Special Case

Invertible Case

If $T$ is invertible, then

$$T^\dagger=T^{-1}.$$

Meaning: The pseudoinverse generalizes the ordinary inverse.

4) Projection Properties

Projection onto the Range

$$TT^\dagger = P_{\operatorname{range}T}.$$

Meaning: Applying $T^\dagger$ and then $T$ projects onto the range of $T$.

Projection onto $(\operatorname{null}T)^\perp$

$$T^\dagger T = P_{(\operatorname{null}T)^\perp}.$$

Meaning: Applying $T$ and then $T^\dagger$ removes the null-space component.

5) Best Approximate Solution

Least-Squares Property

For every $x\in V$ and $b\in W$,

$$\|T(T^\dagger b)-b\| \le \|Tx-b\|.$$

Meaning: $T^\dagger b$ produces an output of $T$ that is as close to $b$ as possible.

6) Minimum-Norm Solution

Minimum-Norm Property

Among all vectors that give the best approximation,

$$T^\dagger b$$

has the smallest norm.

Meaning: The pseudoinverse chooses the shortest solution among all equally good solutions.