Math Core

Many of the matrices that show up in practice are symmetric: covariance matrices in statistics, Hessians in calculus, stiffness matrices in engineering. Symmetric matrices are the best-behaved matrices there are. Their eigenvalues are always real, and they can always be diagonalized using an orthonormal basis of eigenvectors, so the messy $P^{-1}$ in $A = PDP^{-1}$ becomes a simple transpose.

Symmetric matrices

Definition

Symmetric matrix

A square matrix $A$ is symmetric if $A^T = A$ . Equivalently, $a_{ij} = a_{ji}$ for every $i$ and $j$ : the entries are mirror images across the main diagonal.

For example,

\begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}, \qquad \begin{bmatrix} 2 & -4 & 0 \\ -4 & 7 & 5 \\ 0 & 5 & -1 \end{bmatrix}

are symmetric, while $\begin{bmatrix} 1 & 2 \\ 3 & 1 \end{bmatrix}$ is not, and a non-square matrix never is. The diagonal entries can be anything; only the off-diagonal pairs must match.

A handy source of symmetric matrices: for any matrix $B$ (any size), $B^TB$ is symmetric, because $(B^TB)^T = B^T(B^T)^T = B^TB$ . You will use this fact in the lesson on the singular value decomposition.

Eigenvectors of a symmetric matrix are orthogonal

Recall from the last unit that eigenvectors for different eigenvalues are always linearly independent. For a symmetric matrix, something much stronger is true.

Orthogonal eigenvectors

If $A$ is symmetric, then any two eigenvectors from different eigenspaces are orthogonal.

The proof is short and worth knowing. Suppose $A\mathbf{v}_1 = \lambda_1\mathbf{v}_1$ and $A\mathbf{v}_2 = \lambda_2\mathbf{v}_2$ with $\lambda_1 \ne \lambda_2$ . Using $\mathbf{x}\cdot\mathbf{y} = \mathbf{x}^T\mathbf{y}$ and $A^T = A$ ,

\lambda_1(\mathbf{v}_1 \cdot \mathbf{v}_2) = (A\mathbf{v}_1)^T\mathbf{v}_2 = \mathbf{v}_1^TA^T\mathbf{v}_2 = \mathbf{v}_1^T(A\mathbf{v}_2) = \lambda_2(\mathbf{v}_1 \cdot \mathbf{v}_2).

So $(\lambda_1 - \lambda_2)(\mathbf{v}_1 \cdot \mathbf{v}_2) = 0$ . Since $\lambda_1 - \lambda_2 \ne 0$ , the dot product $\mathbf{v}_1 \cdot \mathbf{v}_2$ must be $0$ .

Orthogonal diagonalization

If you normalize orthogonal eigenvectors, they become orthonormal, and a square matrix $P$ with orthonormal columns is an orthogonal matrix: it satisfies $P^TP = I$ , so $P^{-1} = P^T$ . That leads to the following definition.

Definition

Orthogonally diagonalizable

A square matrix $A$ is orthogonally diagonalizable if there are an orthogonal matrix $P$ and a diagonal matrix $D$ with

A = PDP^T = PDP^{-1}.

If $A = PDP^T$ , then $A^T = (PDP^T)^T = (P^T)^TD^TP^T = PDP^T = A$ , since a diagonal matrix equals its own transpose. So only symmetric matrices can be orthogonally diagonalizable. The remarkable fact is that the converse is also true.

The spectral theorem for symmetric matrices

An $n \times n$ matrix $A$ is orthogonally diagonalizable if and only if it is symmetric. In that case:

$A$ has $n$ real eigenvalues, counted with multiplicity.
The dimension of each eigenspace equals the multiplicity of its eigenvalue.
Eigenspaces for different eigenvalues are orthogonal.
$A = PDP^T$ , where the columns of $P$ are an orthonormal basis of $\mathbb{R}^n$ made of eigenvectors of $A$ .

The set of eigenvalues of a matrix is called its spectrum, which is where the name comes from. Part 2 is the big surprise: a symmetric matrix never has a deficient eigenspace, so it is always diagonalizable, even with repeated eigenvalues.

The recipe

Find the eigenvalues.
For each eigenvalue, find a basis of its eigenspace.
Make each basis orthonormal. If an eigenspace is one-dimensional, just normalize. If it has dimension 2 or more, run Gram-Schmidt on its basis, then normalize.
Put the orthonormal eigenvectors into the columns of $P$ and the matching eigenvalues on the diagonal of $D$ .

You never need Gram-Schmidt between different eigenspaces: they are automatically orthogonal.

Worked example: A 2 × 2 matrix

Orthogonally diagonalize $A = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix}$ .

The characteristic polynomial is $(3 - \lambda)^2 - 1 = \lambda^2 - 6\lambda + 8 = (\lambda - 4)(\lambda - 2)$ .

For $\lambda = 4$ : $A - 4I = \begin{bmatrix} -1 & 1 \\ 1 & -1 \end{bmatrix}$ , so $x_1 = x_2$ and $\mathbf{v}_1 = (1, 1)$ .

For $\lambda = 2$ : $A - 2I = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}$ , so $x_1 = -x_2$ and $\mathbf{v}_2 = (1, -1)$ .

As promised, $\mathbf{v}_1 \cdot \mathbf{v}_2 = 1 - 1 = 0$ . Both have length $\sqrt{2}$ , so

P = \begin{bmatrix} 1/\sqrt{2} & 1/\sqrt{2} \\ 1/\sqrt{2} & -1/\sqrt{2} \end{bmatrix}, \qquad D = \begin{bmatrix} 4 & 0 \\ 0 & 2 \end{bmatrix}, \qquad A = PDP^T.

Worked example: A repeated eigenvalue

Orthogonally diagonalize $A = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix}$ .

Every row sums to $4$ , so $A(1, 1, 1) = (4, 4, 4)$ and $\lambda = 4$ is an eigenvalue with eigenvector $(1, 1, 1)$ . Next, $A - I = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 1 & 1 \\ 1 & 1 & 1 \end{bmatrix}$ has rank $1$ , so $\lambda = 1$ is an eigenvalue with a two-dimensional eigenspace: the plane $x_1 + x_2 + x_3 = 0$ . (Check with the trace: $4 + 1 + 1 = 6 = 2 + 2 + 2$ .)

A basis for the plane is $\mathbf{w}_1 = (-1, 1, 0)$ , $\mathbf{w}_2 = (-1, 0, 1)$ , but these are not orthogonal ( $\mathbf{w}_1 \cdot \mathbf{w}_2 = 1$ ). Apply Gram-Schmidt:

\mathbf{w}_2 - \frac{\mathbf{w}_2 \cdot \mathbf{w}_1}{\mathbf{w}_1 \cdot \mathbf{w}_1}\mathbf{w}_1 = (-1, 0, 1) - \tfrac{1}{2}(-1, 1, 0) = \left(-\tfrac{1}{2}, -\tfrac{1}{2}, 1\right).

Scale this to $(-1, -1, 2)$ . Now normalize all three eigenvectors:

P = \begin{bmatrix} 1/\sqrt{3} & -1/\sqrt{2} & -1/\sqrt{6} \\ 1/\sqrt{3} & 1/\sqrt{2} & -1/\sqrt{6} \\ 1/\sqrt{3} & 0 & 2/\sqrt{6} \end{bmatrix}, \qquad D = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}.

Notice that $(1, 1, 1)$ was already orthogonal to both $(-1, 1, 0)$ and $(-1, -1, 2)$ , exactly as the theorem guarantees.

Common mistake

Inside an eigenspace of dimension 2 or more, a basis you find by row reduction is usually not orthogonal. If you just normalize those vectors and put them in $P$ , then $P$ is not an orthogonal matrix and $P^T \ne P^{-1}$ . Always run Gram-Schmidt within each multi-dimensional eigenspace.

The spectral decomposition

Write $P = \begin{bmatrix} \mathbf{u}_1 & \cdots & \mathbf{u}_n \end{bmatrix}$ . Multiplying out $PDP^T$ column-by-row gives

A = \lambda_1\mathbf{u}_1\mathbf{u}_1^T + \lambda_2\mathbf{u}_2\mathbf{u}_2^T + \cdots + \lambda_n\mathbf{u}_n\mathbf{u}_n^T.

This is the spectral decomposition of $A$ . Each $\mathbf{u}_j\mathbf{u}_j^T$ is an $n \times n$ matrix of rank $1$ , and it is exactly the matrix of orthogonal projection onto the line spanned by $\mathbf{u}_j$ . So a symmetric matrix acts by splitting a vector into perpendicular pieces along its eigenvectors and scaling each piece by its eigenvalue.

Worked example: Building a matrix from its spectrum

Let $A = \begin{bmatrix} 7 & 2 \\ 2 & 4 \end{bmatrix}$ . Find the spectral decomposition of $A$ .

The characteristic polynomial is $\lambda^2 - 11\lambda + 24 = (\lambda - 8)(\lambda - 3)$ .

For $\lambda = 8$ : $A - 8I = \begin{bmatrix} -1 & 2 \\ 2 & -4 \end{bmatrix}$ gives $x_1 = 2x_2$ , so $\mathbf{u}_1 = \frac{1}{\sqrt{5}}(2, 1)$ .

For $\lambda = 3$ : the eigenvector must be orthogonal to $(2, 1)$ , so $\mathbf{u}_2 = \frac{1}{\sqrt{5}}(1, -2)$ .

\mathbf{u}_1\mathbf{u}_1^T = \frac{1}{5}\begin{bmatrix} 4 & 2 \\ 2 & 1 \end{bmatrix}, \qquad \mathbf{u}_2\mathbf{u}_2^T = \frac{1}{5}\begin{bmatrix} 1 & -2 \\ -2 & 4 \end{bmatrix}.

Check: $8\mathbf{u}_1\mathbf{u}_1^T + 3\mathbf{u}_2\mathbf{u}_2^T = \frac{1}{5}\begin{bmatrix} 32 + 3 & 16 - 6 \\ 16 - 6 & 8 + 12 \end{bmatrix} = \frac{1}{5}\begin{bmatrix} 35 & 10 \\ 10 & 20 \end{bmatrix} = A$ .

Tip

For a $2 \times 2$ symmetric matrix, once you have one eigenvector $(a, b)$ , the other eigenvector is $(b, -a)$ (or $(-b, a)$ ). No second row reduction is needed.

Practice

Practice 1

Which matrix is guaranteed to be orthogonally diagonalizable?

Practice 2

Find the eigenvalues of $A = \begin{bmatrix} 5 & 2 \\ 2 & 2 \end{bmatrix}$ .

Separate answers with commas, e.g. 2, -5

Practice 3

The matrix $P = \begin{bmatrix} 3/5 & -4/5 \\ 4/5 & 3/5 \end{bmatrix}$ is orthogonal. Find the $(1, 2)$ entry of $P^{-1}$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 4

Let $A = \begin{bmatrix} 1 & 0 & 2 \\ 0 & 3 & 0 \\ 2 & 0 & 1 \end{bmatrix}$ . Its eigenvalues are $3$ and $-1$ . What is the dimension of the eigenspace for $\lambda = 3$ ?

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 5

A symmetric $2 \times 2$ matrix $A$ has eigenvalues $3$ and $-2$ , and $(1, 2)$ is an eigenvector for $\lambda = 3$ . Find the $(1, 1)$ entry of $A$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 6

Let $\mathbf{u}_1 = \frac{1}{\sqrt{2}}(1, 1)$ and $\mathbf{u}_2 = \frac{1}{\sqrt{2}}(1, -1)$ , and let $A = 6\mathbf{u}_1\mathbf{u}_1^T - 2\mathbf{u}_2\mathbf{u}_2^T$ . Find the $(1, 2)$ entry of $A$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 7

Find the eigenvalues of $A = \begin{bmatrix} 2 & -1 & -1 \\ -1 & 2 & -1 \\ -1 & -1 & 2 \end{bmatrix}$ , then describe an orthonormal eigenbasis.

Separate answers with commas, e.g. 2, -5

Practice 8

Let $A$ be an $n \times n$ symmetric matrix. Which statement is false?