Math Core

Expressions like $5x_1^2 - 4x_1x_2 + 5x_2^2$ appear whenever you measure energy, variance, or curvature. The cross term $-4x_1x_2$ makes them hard to read: is the expression always positive? What shape is its level curve? Writing the expression with a symmetric matrix and applying the spectral theorem removes the cross terms and answers these questions at once.

Quadratic forms and their matrices

Definition

Quadratic form

A quadratic form on $\mathbb{R}^n$ is a function of the form

Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x},

where $A$ is an $n \times n$ symmetric matrix, called the matrix of the quadratic form.

Multiply it out for $n = 2$ with $A = \begin{bmatrix} a & b \\ b & c \end{bmatrix}$ :

\mathbf{x}^TA\mathbf{x} = \begin{bmatrix} x_1 & x_2 \end{bmatrix}\begin{bmatrix} ax_1 + bx_2 \\ bx_1 + cx_2 \end{bmatrix} = ax_1^2 + 2bx_1x_2 + cx_2^2.

So every term has degree exactly $2$ . The diagonal entries of $A$ are the coefficients of the squares, and each cross-term coefficient is split in half between the two symmetric off-diagonal positions. The same rule holds in any dimension: the coefficient of $x_ix_j$ (with $i \ne j$ ) is $2a_{ij}$ .

Worked example: Finding the matrix

Find the matrix of $Q(\mathbf{x}) = x_1^2 + 2x_2^2 - 3x_3^2 + 4x_1x_2 - 6x_2x_3$ , then compute $Q(1, -1, 2)$ .

The squares give the diagonal $1, 2, -3$ . The $x_1x_2$ coefficient $4$ splits into $2$ and $2$ ; the $x_2x_3$ coefficient $-6$ splits into $-3$ and $-3$ ; there is no $x_1x_3$ term, so those entries are $0$ :

A = \begin{bmatrix} 1 & 2 & 0 \\ 2 & 2 & -3 \\ 0 & -3 & -3 \end{bmatrix}.

Directly: $Q(1, -1, 2) = 1 + 2 - 12 - 4 + 12 = -1$ . With the matrix: $A(1, -1, 2) = (-1, -6, -3)$ , and $(1, -1, 2) \cdot (-1, -6, -3) = -1 + 6 - 6 = -1$ .

Common mistake

The most common mistake is putting the whole cross-term coefficient in one off-diagonal spot, or putting it in both. For $6x_1x_2$ , the entries $a_{12}$ and $a_{21}$ are each $3$ , not $6$ .

Removing the cross terms

Because $A$ is symmetric, the spectral theorem gives $A = PDP^T$ with $P$ orthogonal. Make the change of variable

\mathbf{x} = P\mathbf{y}, \qquad \text{equivalently} \qquad \mathbf{y} = P^T\mathbf{x}.

Then

\mathbf{x}^TA\mathbf{x} = (P\mathbf{y})^TA(P\mathbf{y}) = \mathbf{y}^T(P^TAP)\mathbf{y} = \mathbf{y}^TD\mathbf{y} = \lambda_1y_1^2 + \lambda_2y_2^2 + \cdots + \lambda_ny_n^2.

The principal axes theorem

If $A$ is symmetric, the orthogonal change of variable $\mathbf{x} = P\mathbf{y}$ , where the columns of $P$ are an orthonormal eigenbasis of $A$ , turns $\mathbf{x}^TA\mathbf{x}$ into

\lambda_1y_1^2 + \cdots + \lambda_ny_n^2,

a quadratic form with no cross terms. The columns of $P$ are called the principal axes of the form.

Since $P$ is orthogonal, the change of variable is just a rotation (possibly with a reflection) of the coordinate axes. Lengths and angles don't change; you are simply looking at the same form along better axes.

Worked example: A tilted ellipse

Remove the cross term from $Q(\mathbf{x}) = 5x_1^2 - 4x_1x_2 + 5x_2^2$ and describe the curve $Q(\mathbf{x}) = 21$ .

The matrix is $A = \begin{bmatrix} 5 & -2 \\ -2 & 5 \end{bmatrix}$ , with characteristic polynomial $\lambda^2 - 10\lambda + 21 = (\lambda - 3)(\lambda - 7)$ .

For $\lambda = 3$ : $A - 3I = \begin{bmatrix} 2 & -2 \\ -2 & 2 \end{bmatrix}$ , so $\mathbf{u}_1 = \frac{1}{\sqrt{2}}(1, 1)$ . For $\lambda = 7$ : $\mathbf{u}_2 = \frac{1}{\sqrt{2}}(1, -1)$ .

With $\mathbf{x} = P\mathbf{y}$ and $P = \begin{bmatrix} \mathbf{u}_1 & \mathbf{u}_2 \end{bmatrix}$ , the form becomes $3y_1^2 + 7y_2^2$ . The curve $3y_1^2 + 7y_2^2 = 21$ is

\frac{y_1^2}{7} + \frac{y_2^2}{3} = 1,

an ellipse with semi-axis $\sqrt{7} \approx 2.65$ along the line $y = x$ (direction $\mathbf{u}_1$ ) and semi-axis $\sqrt{3} \approx 1.73$ along the line $y = -x$ (direction $\mathbf{u}_2$ ).

The ellipse 5x² − 4xy + 5y² = 21 with its principal axesOpen in grapher →

Notice that the smaller eigenvalue goes with the longer axis: along $\mathbf{u}_1$ the form grows slowly, so you have to travel farther to reach the level $21$ .

Classifying quadratic forms

The diagonal form $\lambda_1y_1^2 + \cdots + \lambda_ny_n^2$ makes the sign of $Q$ obvious.

Definition

Definiteness

A quadratic form $Q$ (and its symmetric matrix $A$ ) is

positive definite if $Q(\mathbf{x}) > 0$ for every $\mathbf{x} \ne \mathbf{0}$ ; this happens exactly when all eigenvalues of $A$ are positive;
negative definite if $Q(\mathbf{x}) < 0$ for every $\mathbf{x} \ne \mathbf{0}$ ; exactly when all eigenvalues are negative;
indefinite if $Q$ takes both positive and negative values; exactly when $A$ has both a positive and a negative eigenvalue.

If all eigenvalues are $\ge 0$ (some equal to $0$ ) the form is positive semidefinite, and if all are $\le 0$ it is negative semidefinite.

Worked example: An indefinite form

Classify $Q(\mathbf{x}) = x_1^2 + 6x_1x_2 + x_2^2$ .

The matrix $\begin{bmatrix} 1 & 3 \\ 3 & 1 \end{bmatrix}$ has characteristic polynomial $\lambda^2 - 2\lambda - 8 = (\lambda - 4)(\lambda + 2)$ . One eigenvalue is positive and one is negative, so $Q$ is indefinite. You can confirm with two inputs: $Q(1, 1) = 8 > 0$ but $Q(1, -1) = -4 < 0$ . The level curves of $Q$ are hyperbolas, not ellipses.

Tip

For a $2 \times 2$ matrix $\begin{bmatrix} a & b \\ b & c \end{bmatrix}$ you don't need the eigenvalues. Since $\det A = \lambda_1\lambda_2$ and $\operatorname{tr}A = \lambda_1 + \lambda_2$ : if $\det A < 0$ the form is indefinite; if $\det A > 0$ and $a > 0$ it is positive definite; if $\det A > 0$ and $a < 0$ it is negative definite.

Maximizing a quadratic form on the unit sphere

A classic question: over all unit vectors $\mathbf{x}$ , how large and how small can $Q(\mathbf{x}) = \mathbf{x}^TA\mathbf{x}$ be? Change variables to $\mathbf{x} = P\mathbf{y}$ . Since $P$ is orthogonal, $\|\mathbf{y}\| = \|\mathbf{x}\| = 1$ . If the eigenvalues are ordered $\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_n$ , then

\lambda_1y_1^2 + \cdots + \lambda_ny_n^2 \le \lambda_1(y_1^2 + \cdots + y_n^2) = \lambda_1,

with equality when $\mathbf{y} = (1, 0, \ldots, 0)$ , that is, when $\mathbf{x} = \mathbf{u}_1$ . The same argument works for the minimum.

Constrained extremes

Let $A$ be symmetric with largest eigenvalue $\lambda_{\max}$ and smallest eigenvalue $\lambda_{\min}$ . Subject to $\|\mathbf{x}\| = 1$ ,

the maximum of $\mathbf{x}^TA\mathbf{x}$ is $\lambda_{\max}$ , reached at a unit eigenvector for $\lambda_{\max}$ ;
the minimum of $\mathbf{x}^TA\mathbf{x}$ is $\lambda_{\min}$ , reached at a unit eigenvector for $\lambda_{\min}$ .

Worked example: Extremes in three variables

Find the maximum and minimum of $Q(\mathbf{x}) = 4x_1^2 + 4x_2^2 + x_3^2 + 2x_1x_2$ subject to $x_1^2 + x_2^2 + x_3^2 = 1$ , and where they occur.

The matrix is $A = \begin{bmatrix} 4 & 1 & 0 \\ 1 & 4 & 0 \\ 0 & 0 & 1 \end{bmatrix}$ . The upper-left block $\begin{bmatrix} 4 & 1 \\ 1 & 4 \end{bmatrix}$ has eigenvalues $5$ (eigenvector $(1, 1)$ ) and $3$ (eigenvector $(1, -1)$ ), and the last diagonal entry contributes the eigenvalue $1$ with eigenvector $(0, 0, 1)$ .

So the maximum is $5$ , at $\mathbf{x} = \pm\frac{1}{\sqrt{2}}(1, 1, 0)$ , and the minimum is $1$ , at $\mathbf{x} = \pm(0, 0, 1)$ . Check the maximum: $Q\left(\tfrac{1}{\sqrt{2}}, \tfrac{1}{\sqrt{2}}, 0\right) = 2 + 2 + 0 + 1 = 5$ .

Practice

Practice 1

Let $A$ be the matrix of the quadratic form $Q(\mathbf{x}) = 2x_1^2 + 6x_1x_2 - x_2^2$ . Find the $(1, 2)$ entry of $A$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 2

Let $A = \begin{bmatrix} 2 & -1 \\ -1 & 3 \end{bmatrix}$ and $\mathbf{x} = (2, 1)$ . Compute $\mathbf{x}^TA\mathbf{x}$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 3

Classify $Q(\mathbf{x}) = 3x_1^2 + 4x_1x_2 + 3x_2^2$ .

Practice 4

An orthogonal change of variable $\mathbf{x} = P\mathbf{y}$ turns $Q(\mathbf{x}) = x_1^2 - 8x_1x_2 - 5x_2^2$ into $\lambda_1y_1^2 + \lambda_2y_2^2$ . Find $\lambda_1$ and $\lambda_2$ .

Separate answers with commas, e.g. 2, -5

Practice 5

Find the maximum value of $Q(\mathbf{x}) = 7x_1^2 + 4x_1x_2 + 4x_2^2$ subject to $x_1^2 + x_2^2 = 1$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 6

For which values of $k$ is $Q(\mathbf{x}) = x_1^2 + 2kx_1x_2 + 4x_2^2$ positive definite?

Enter an inequality, e.g. x >= 4, -2 < x <= 3, or x < 1 or x > 5

Practice 7

Find the minimum value of $Q(\mathbf{x}) = 2x_1^2 + 2x_2^2 + 5x_3^2 - 2x_1x_2$ subject to $\|\mathbf{x}\| = 1$ .

Enter a number. Fractions like 3/4 and sqrt(2) are OK.

Practice 8

The curve $6x^2 + 4xy + 3y^2 = 14$ is an ellipse. Find the length of its semi-major axis (half the longest diameter).

Enter a number. Fractions like 3/4 and sqrt(2) are OK.