Angles between flats

Angles between subspaces

Now let ${\displaystyle F}$ and ${\displaystyle G}$ be subspaces of the ${\displaystyle n}$-dimensional inner product space over the real or complex numbers. Geometrically, ${\displaystyle F}$ and ${\displaystyle G}$ are flats, so Jordan's definition of mutual angles applies. When for any canonical coordinate ${\displaystyle \xi }$ the symbol ${\displaystyle \hat{\xi }}$ denotes the unit vector of the ${\displaystyle \xi }$ axis, the vectors ${\displaystyle {\hat{y}}_1,\dots ,{\hat{y}}_{\sigma },}$ ${\displaystyle {\hat{w}}_1,\dots ,{\hat{w}}_{\alpha },}$ ${\displaystyle {\hat{z}}_1,\dots ,{\hat{z}}_{\tau }}$ form an orthonormal basis for ${\displaystyle F}$ and the vectors ${\displaystyle {\hat{y}}_1,\dots ,{\hat{y}}_{\sigma },}$ ${\displaystyle {\hat{w}}_1^{\prime },\dots ,{\hat{w}}_{\alpha }^{\prime },}$ ${\displaystyle {\hat{u}}_1,\dots ,{\hat{u}}_{\upsilon }}$ form an orthonormal basis for ${\displaystyle G}$, where

${\displaystyle {\hat{w}}_i^{\prime }={\hat{w}}_i\cos {\theta }_i+{\hat{v}}_i\sin {\theta }_i,i=1,\dots ,\alpha .}$

Being related to canonical coordinates, these basic vectors may be called canonical.

When ${\displaystyle a_i,i=1,\dots ,k}$ denote the canonical basic vectors for ${\displaystyle F}$ and ${\displaystyle b_i,i=1,\dots ,l}$ the canonical basic vectors for ${\displaystyle G}$ then the inner product ${\displaystyle ⟨a_i,b_j⟩}$ vanishes for any pair of ${\displaystyle i}$ and ${\displaystyle j}$ except the following ones.

${\displaystyle \begin{array}{cccc}& ⟨{\hat{y}}_i,{\hat{y}}_i⟩=1,& & i=1,\dots ,\sigma ,\\ & ⟨{\hat{w}}_i,{\hat{w}}_i^{\prime }⟩=\cos {\theta }_i,& & i=1,\dots ,\alpha .\end{array}}$

With the above ordering of the basic vectors, the matrix of the inner products ${\displaystyle ⟨a_i,b_j⟩}$ is thus diagonal. In other words, if ${\displaystyle (a_i^{\prime },i=1,\dots ,k)}$ and ${\displaystyle (b_i^{\prime },i=1,\dots ,\ell )}$ are arbitrary orthonormal bases in ${\displaystyle F}$ and ${\displaystyle G}$ then the real, orthogonal or unitary transformations from the basis ${\displaystyle (a_i^{\prime })}$ to the basis ${\displaystyle (a_i)}$ and from the basis ${\displaystyle (b_i^{\prime })}$ to the basis ${\displaystyle (b_i)}$ realize a singular value decomposition of the matrix of inner products ${\displaystyle ⟨a_i^{\prime },b_j^{\prime }⟩}$. The diagonal matrix elements ${\displaystyle ⟨a_i,b_i⟩}$ are the singular values of the latter matrix. By the uniqueness of the singular value decomposition, the vectors ${\displaystyle {\hat{y}}_i}$ are then unique up to a real, orthogonal or unitary transformation among them, and the vectors ${\displaystyle {\hat{w}}_i}$ and ${\displaystyle {\hat{w}}_i^{\prime }}$ (and hence ${\displaystyle {\hat{v}}_i}$) are unique up to equal real, orthogonal or unitary transformations applied simultaneously to the sets of the vectors ${\displaystyle {\hat{w}}_i}$ associated with a common value of ${\displaystyle {\theta }_i}$ and to the corresponding sets of vectors ${\displaystyle {\hat{w}}_i^{\prime }}$ (and hence to the corresponding sets of ${\displaystyle {\hat{v}}_i}$).

A singular value ${\displaystyle 1}$ can be interpreted as ${\displaystyle \cos 0}$ corresponding to the angles ${\displaystyle 0}$ introduced above and associated with ${\displaystyle F\cap G}$ and a singular value ${\displaystyle 0}$ can be interpreted as ${\displaystyle \cos \pi /2}$ corresponding to right angles between the orthogonal spaces ${\displaystyle F\cap G^{\mathrm{\perp }}}$ and ${\displaystyle F^{\mathrm{\perp }}\cap G}$, where superscript ${\displaystyle \mathrm{\perp }}$ denotes the orthogonal complement.

Variational characterization

The variational characterization of singular values and vectors implies as a special case a variational characterization of the angles between subspaces and their associated canonical vectors. This characterization includes the angles ${\displaystyle 0}$ and ${\displaystyle \pi /2}$ introduced above and orders the angles by increasing value. It can be given the form of the below alternative definition. In this context, it is customary to talk of principal angles and vectors.

Definition

Let ${\displaystyle V}$ be an inner product space. Given two subspaces ${\displaystyle \mathcal{U},\mathcal{W}}$ with ${\displaystyle \dim (\mathcal{U})=k\le \dim (\mathcal{W}):=\ell }$, there exists then a sequence of ${\displaystyle k}$ angles ${\displaystyle 0\le {\theta }_1\le {\theta }_2\le \cdots \le {\theta }_k\le \pi /2}$ called the principal angles, the first one defined as

${\displaystyle {\theta }_1:=min\left\{\arccos \left(\frac{|⟨u,w⟩|}{\Vert u\Vert \Vert w\Vert })\right|u\in \mathcal{U},w\in \mathcal{W}\right\}=\mathrm{\angle }(u_1,w_1),}$

where ${\displaystyle ⟨\cdot ,\cdot ⟩}$ is the inner product and ${\displaystyle \Vert \cdot \Vert }$ the induced norm. The vectors ${\displaystyle u_1}$ and ${\displaystyle w_1}$ are the corresponding principal vectors.

The other principal angles and vectors are then defined recursively via

${\displaystyle {\theta }_i:=min\left\{\arccos \left(\frac{|⟨u,w⟩|}{\Vert u\Vert \Vert w\Vert }\right)|u\in \mathcal{U},w\in \mathcal{W},u\perp u_j,w\perp w_j\mathrm{\forall }j\in \{1,\dots ,i-1\}\right\}.}$

This means that the principal angles ${\displaystyle ({\theta }_1,\dots ,{\theta }_k)}$ form a set of minimized angles between the two subspaces, and the principal vectors in each subspace are orthogonal to each other.

Examples

Geometric example

Geometrically, subspaces are flats (points, lines, planes etc.) that include the origin, thus any two subspaces intersect at least in the origin. Two two-dimensional subspaces ${\displaystyle \mathcal{U}}$ and ${\displaystyle \mathcal{W}}$ generate a set of two angles. In a three-dimensional Euclidean space, the subspaces ${\displaystyle \mathcal{U}}$ and ${\displaystyle \mathcal{W}}$ are either identical, or their intersection forms a line. In the former case, both ${\displaystyle {\theta }_1={\theta }_2=0}$. In the latter case, only ${\displaystyle {\theta }_1=0}$, where vectors ${\displaystyle u_1}$ and ${\displaystyle w_1}$ are on the line of the intersection ${\displaystyle \mathcal{U}\cap \mathcal{W}}$ and have the same direction. The angle ${\displaystyle {\theta }_2>0}$ will be the angle between the subspaces ${\displaystyle \mathcal{U}}$ and ${\displaystyle \mathcal{W}}$ in the orthogonal complement to ${\displaystyle \mathcal{U}\cap \mathcal{W}}$. Imagining the angle between two planes in 3D, one intuitively thinks of the largest angle, ${\displaystyle {\theta }_2>0}$.

Algebraic example

In 4-dimensional real coordinate space R⁴, let the two-dimensional subspace ${\displaystyle \mathcal{U}}$ be spanned by ${\displaystyle u_1=(1,0,0,0)}$ and ${\displaystyle u_2=(0,1,0,0)}$, and let the two-dimensional subspace ${\displaystyle \mathcal{W}}$ be spanned by ${\displaystyle w_1=(1,0,0,a)/\sqrt{1+a^2}}$ and ${\displaystyle w_2=(0,1,b,0)/\sqrt{1+b^2}}$ with some real ${\displaystyle a}$ and ${\displaystyle b}$ such that ${\displaystyle |a|<|b|}$. Then ${\displaystyle u_1}$ and ${\displaystyle w_1}$ are, in fact, the pair of principal vectors corresponding to the angle ${\displaystyle {\theta }_1}$ with ${\displaystyle \cos ({\theta }_1)=1/\sqrt{1+a^2}}$, and ${\displaystyle u_2}$ and ${\displaystyle w_2}$ are the principal vectors corresponding to the angle ${\displaystyle {\theta }_2}$ with ${\displaystyle \cos ({\theta }_2)=1/\sqrt{1+b^2}.}$

To construct a pair of subspaces with any given set of ${\displaystyle k}$ angles ${\displaystyle {\theta }_1,\dots ,{\theta }_k}$ in a ${\displaystyle 2k}$ (or larger) dimensional Euclidean space, take a subspace ${\displaystyle \mathcal{U}}$ with an orthonormal basis ${\displaystyle (e_1,\dots ,e_k)}$ and complete it to an orthonormal basis ${\displaystyle (e_1,\dots ,e_n)}$ of the Euclidean space, where ${\displaystyle n\ge 2k}$. Then, an orthonormal basis of the other subspace ${\displaystyle \mathcal{W}}$ is, e.g.,

${\displaystyle (\cos ({\theta }_1)e_1+\sin ({\theta }_1)e_{k+1},\dots ,\cos ({\theta }_k)e_k+\sin ({\theta }_k)e_{2k}).}$

Basic properties

• If the largest angle is zero, one subspace is a subset of the other.
• If the largest angle is ${\displaystyle \pi /2}$, there is at least one vector in one subspace perpendicular to the other subspace.
• If the smallest angle is zero, the subspaces intersect at least in a line.
• If the smallest angle is ${\displaystyle \pi /2}$, the subspaces are orthogonal.
• The number of angles equal to zero is the dimension of the space where the two subspaces intersect.

Advanced properties

• Non-trivial (different from ${\displaystyle 0}$ and ${\displaystyle \pi /2}$) angles between two subspaces are the same as the non-trivial angles between their orthogonal complements.
• Non-trivial angles between the subspaces ${\displaystyle \mathcal{U}}$ and ${\displaystyle \mathcal{W}}$ and the corresponding non-trivial angles between the subspaces ${\displaystyle \mathcal{U}}$ and ${\displaystyle {\mathcal{W}}^{\perp }}$ sum up to ${\displaystyle \pi /2}$.
• The angles between subspaces satisfy the triangle inequality in terms of majorization and thus can be used to define a distance on the set of all subspaces turning the set into a metric space.
• The sine of the angles between subspaces satisfy the triangle inequality in terms of majorization and thus can be used to define a distance on the set of all subspaces turning the set into a metric space. For example, the sine of the largest angle is known as a gap between subspaces.

Extensions

The notion of the angles and some of the variational properties can be naturally extended to arbitrary inner products and subspaces with infinite dimensions.

Computation

Historically, the principal angles and vectors first appear in the context of canonical correlation and were originally computed using SVD of corresponding covariance matrices. However, as first noticed in, the canonical correlation is related to the cosine of the principal angles, which is ill-conditioned for small angles, leading to very inaccurate computation of highly correlated principal vectors in finite precision computer arithmetic. The sine-based algorithm fixes this issue, but creates a new problem of very inaccurate computation of highly uncorrelated principal vectors, since the sine function is ill-conditioned for angles close to π/2. To produce accurate principal vectors in computer arithmetic for the full range of the principal angles, the combined technique first compute all principal angles and vectors using the classical cosine-based approach, and then recomputes the principal angles smaller than π/4 and the corresponding principal vectors using the sine-based approach. The combined technique is implemented in open-source libraries Octave and SciPy and contributed and to MATLAB.