Orthogonal Complement

Resources

Axler, Sheldon. 2020. Measure, Integration & Real Analysis. Vol. 282. Graduate Texts in Mathematics. Cham: Springer International Publishing. https://doi.org/10.1007/978-3-030-33143-6.

Main Idea

Within Linear Algebra and Functional Analysis, given some subset $W$ of an Inner Product space $V$ , the orthogonal complement is the set of all vectors $W^{⊥}$ that are orthogonal to every vector in $W,$ as per some inner product on the parent space $V$ . More formally, the orthogonal complement to $W$ is

W^{⊥} = {h \in V : ⟨ h, g ⟩ = 0, \forall g \in U} .

The orthogonal decomposition writes a vector as a sum of a vector in $W$ and a vector in the space orthogonal to $W$ , $W^{⊥}$ .

Properties

$W^{⊥}$ is a closed subspace of $V$ .
$W^{⊥} \cap W \subset {0}$ (if $W = \emptyset$ , then the overlap is also $\emptyset$ , so at most, the spaces share $0$ .)
... see Axler

Linear Algebra Case

Let $W \subset R^{n}$ , then

W^{⊥} := {v \in R^{n} : ⟨ u, v ⟩ = 0 \forall u \in W} .

We can show that if $W$ has an orthogonal basis ${a_{1}, a_{2}, \dots, a_{p}}$ , then it is equivalent for elements of $W^{⊥}$ to be orthogonal to only these basis vectors, as opposed to everything in $W .$ This can be shown by linearity of the inner product and the decomposition of any vector into a linear combination of its basis vectors.

A matrix $A$ defines a subspace through it's Range / Column Space: $W = range (A) = col (A)$ . More formally,

W = {A x : x \in R^{n}} .

For $u \in R^{n}$ , we wish to find its orthogonal decomposition as $u = u^{W} + u^{W^{⊥}}$ . By using an orthogonal basis, we can find the $u^{W}$ term,

u^{W} = \frac{⟨ a_{1}, u ⟩}{⟨ a_{1}, a_{1} ⟩} a_{1} + \frac{⟨ a_{2}, u ⟩}{⟨ a_{2}, a_{2} ⟩} a_{2} + \dots + \frac{⟨ a_{p}, u ⟩}{⟨ a_{p}, a_{p} ⟩} a_{p} .

This is a sum of projections along the basis vectors of the subspace of $W$ . Then, $u^{W^{⊥}} = u - u^{W}$ .

Next, we wish to find $W^{⊥}$ . First, begin with its definition, and plug in $u = A x$ , which must be true for some $x$ , as that is how $u \in W$ is originally defined:

W^{⊥} = {v \in R^{n} : ⟨ A x, v ⟩ = 0 \forall x \in R^{n}} = {v \in R^{n} : x^{T} A^{T} v = 0 \forall x \in R^{n}} .

Because this second condition holds for all $x$ , we can "divide it out", giving

W^{⊥} = {v \in R^{n} : A^{T} v = 0} .

Thus, we have shown that the orthogonal complement of the range of a matrix is the Nullspace / Kernel of the transpose of the matrix!

Note, $\dim W \neq \dim W^{⊥}$ , and $W, W^{⊥} \subset R^{n}$ . There are further properties about $(W^{⊥})^{⊥}$ and the (set) addition of these spaces.

If we instead view $A$ as an operator, $A : R^{n} \to R^{n}$ , then there exists (multiple?) $\tilde{A} : R^{n} \to R^{n}$ such that the range of $A$ is orthogonal to the range of $\tilde{A}$ (over the domain $R^{n}$ ). More generally, the operators do not even need to share the same domain, just the same codomain. Here, $\tilde{A}$ is the matrix associated with the nullspace of $A^{T}$ (i.e. proper dimension and such that $A^{T} \tilde{A} = 0$ ).

For ease of discussion, let the columns of $A$ , $a_{i}$ , be orthonormal. Then, repeating the formula from above:

\begin{aligned} u^{W} & = ⟨ a_{1}, u ⟩ a_{1} + ⟨ a_{2}, u ⟩ a_{2} + \dots + ⟨ a_{p}, u ⟩ a_{p}, \\ = a_{1} a_{1}^{T} u + a_{2} a_{2}^{T} u + \dots + a_{p} a_{p}^{T} u, \\ = (a_{1} a_{1}^{T} + a_{2} a_{2}^{T} + \dots + a_{p} a_{p}^{T}) u, \\ = A A^{T} u . \end{aligned}

Then,

u^{W^{⊥}} = (I - A A^{T}) u .

From this, we can see that the operators that map from a vector to the subspace and it's orthogonal complement are $A A^{T}$ and $I - A A^{T}$ , respectively.