Math Core

So far, linear algebra has been about combining vectors: spans, bases and linear maps. None of that needed length or angle. But many real questions do: how far is a data point from a model, which vector in a subspace is closest to a given one, when do two directions meet at a right angle? This unit adds geometry to $\mathbb{R}^n$ , and it all starts with one simple product.

The inner product

Definition

Inner product (dot product)

For vectors $\mathbf{u}$ and $\mathbf{v}$ in $\mathbb{R}^n$ , the inner product is the number

\mathbf{u} \cdot \mathbf{v} = \mathbf{u}^T\mathbf{v} = u_1v_1 + u_2v_2 + \cdots + u_nv_n.

Multiply matching entries and add. The result is a scalar, not a vector. Writing it as $\mathbf{u}^T\mathbf{v}$ (a $1 \times n$ matrix times an $n \times 1$ matrix) is handy, because it lets you use matrix algebra on inner products.

The inner product has four properties you will use constantly. For all $\mathbf{u}, \mathbf{v}, \mathbf{w}$ in $\mathbb{R}^n$ and every scalar $c$ :

$\mathbf{u} \cdot \mathbf{v} = \mathbf{v} \cdot \mathbf{u}$
$(\mathbf{u} + \mathbf{v}) \cdot \mathbf{w} = \mathbf{u} \cdot \mathbf{w} + \mathbf{v} \cdot \mathbf{w}$
$(c\mathbf{u}) \cdot \mathbf{v} = c(\mathbf{u} \cdot \mathbf{v})$
$\mathbf{u} \cdot \mathbf{u} \ge 0$ , and $\mathbf{u} \cdot \mathbf{u} = 0$ only when $\mathbf{u} = \mathbf{0}$

Property 4 holds because $\mathbf{u} \cdot \mathbf{u} = u_1^2 + \cdots + u_n^2$ is a sum of squares.

Length and distance

In $\mathbb{R}^2$ , the Pythagorean theorem says the vector $(a, b)$ has length $\sqrt{a^2 + b^2}$ . That is exactly $\sqrt{\mathbf{v} \cdot \mathbf{v}}$ , and the same formula works in any dimension.

Length and distance

The length (or norm) of $\mathbf{v}$ is $\|\mathbf{v}\| = \sqrt{\mathbf{v} \cdot \mathbf{v}} = \sqrt{v_1^2 + \cdots + v_n^2}$ .

The distance between $\mathbf{u}$ and $\mathbf{v}$ is $\operatorname{dist}(\mathbf{u}, \mathbf{v}) = \|\mathbf{u} - \mathbf{v}\|$ .

Scaling a vector scales its length: $\|c\mathbf{v}\| = |c|\,\|\mathbf{v}\|$ . A vector of length $1$ is a unit vector. Dividing a nonzero vector by its length, $\mathbf{u} = \dfrac{\mathbf{v}}{\|\mathbf{v}\|}$ , gives the unit vector in the same direction. This is called normalizing $\mathbf{v}$ .

Worked example: Inner product, length and a unit vector

Let $\mathbf{u} = (1, -2, 3)$ and $\mathbf{v} = (2, -1, 2, 4)$ . Find $\mathbf{u} \cdot \mathbf{u}$ , $\|\mathbf{u}\|$ , and the unit vector in the direction of $\mathbf{v}$ .

$\mathbf{u} \cdot \mathbf{u} = 1 + 4 + 9 = 14$ , so $\|\mathbf{u}\| = \sqrt{14}$ .

For $\mathbf{v}$ (a vector in $\mathbb{R}^4$ ): $\|\mathbf{v}\| = \sqrt{4 + 1 + 4 + 16} = \sqrt{25} = 5$ . So the unit vector is

\frac{1}{5}\mathbf{v} = \left(\tfrac{2}{5}, -\tfrac{1}{5}, \tfrac{2}{5}, \tfrac{4}{5}\right).

Check: $\tfrac{4 + 1 + 4 + 16}{25} = 1$ .

Worked example: Distance between two points

Find the distance between $\mathbf{u} = (7, 1)$ and $\mathbf{v} = (3, 2)$ .

$\mathbf{u} - \mathbf{v} = (4, -1)$ , so $\operatorname{dist}(\mathbf{u}, \mathbf{v}) = \sqrt{16 + 1} = \sqrt{17}$ .

Orthogonal vectors

When are $\mathbf{u}$ and $\mathbf{v}$ perpendicular? Geometrically, exactly when the distance from $\mathbf{u}$ to $\mathbf{v}$ equals the distance from $\mathbf{u}$ to $-\mathbf{v}$ . Expand both squared distances:

\|\mathbf{u} - \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2 - 2\,\mathbf{u} \cdot \mathbf{v}, \qquad \|\mathbf{u} + \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2 + 2\,\mathbf{u} \cdot \mathbf{v}.

These are equal exactly when $\mathbf{u} \cdot \mathbf{v} = 0$ . That motivates the definition.

Orthogonality

Vectors $\mathbf{u}$ and $\mathbf{v}$ are orthogonal if $\mathbf{u} \cdot \mathbf{v} = 0$ .

Pythagorean theorem: $\mathbf{u}$ and $\mathbf{v}$ are orthogonal if and only if $\|\mathbf{u} + \mathbf{v}\|^2 = \|\mathbf{u}\|^2 + \|\mathbf{v}\|^2$ .

The zero vector is orthogonal to every vector, since $\mathbf{0} \cdot \mathbf{v} = 0$ .

u · v = 2(−1) + 1(2) = 0, so u and v meet at a right angle.Open in grapher →

Angles

More generally, the law of cosines in the plane spanned by $\mathbf{u}$ and $\mathbf{v}$ gives

\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta,

where $\theta$ is the angle between the vectors. So $\cos\theta = \dfrac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\|\,\|\mathbf{v}\|}$ . A positive inner product means an acute angle, a negative one means an obtuse angle, and zero means a right angle. In statistics, this cosine (for centered data) is the correlation coefficient.

Orthogonal complements

Definition

Orthogonal complement

If $W$ is a subspace of $\mathbb{R}^n$ , its orthogonal complement $W^\perp$ ("W perp") is the set of all vectors orthogonal to every vector in $W$ .

Three facts make $W^\perp$ easy to work with:

A vector is in $W^\perp$ if and only if it is orthogonal to every vector in a spanning set of $W$ . (By linearity, orthogonal to the spanning vectors means orthogonal to all their combinations.)
$W^\perp$ is a subspace of $\mathbb{R}^n$ , and $\dim W + \dim W^\perp = n$ .
For any $m \times n$ matrix $A$ : $(\operatorname{Row} A)^\perp = \operatorname{Nul} A$ and $(\operatorname{Col} A)^\perp = \operatorname{Nul} A^T$ .

The last fact is just matrix multiplication read row by row: $A\mathbf{x} = \mathbf{0}$ says each row of $A$ has inner product $0$ with $\mathbf{x}$ . So to find $W^\perp$ , put a spanning set of $W$ into the rows of a matrix and find its null space.

Worked example: Finding an orthogonal complement

Let $W = \operatorname{Span}\{(1, 2, -1), (0, 1, 1)\}$ in $\mathbb{R}^3$ . Find a basis for $W^\perp$ .

Solve $A\mathbf{x} = \mathbf{0}$ with $A = \begin{bmatrix} 1 & 2 & -1 \\ 0 & 1 & 1 \end{bmatrix}$ . Row 2 gives $x_2 = -x_3$ . Row 1 gives $x_1 = -2x_2 + x_3 = 2x_3 + x_3 = 3x_3$ . With $x_3$ free,

\mathbf{x} = x_3\begin{bmatrix} 3 \\ -1 \\ 1 \end{bmatrix}, \quad \text{so } \{(3, -1, 1)\} \text{ is a basis for } W^\perp.

Check: $(1, 2, -1) \cdot (3, -1, 1) = 3 - 2 - 1 = 0$ and $(0, 1, 1) \cdot (3, -1, 1) = -1 + 1 = 0$ . Geometrically, $W$ is a plane through the origin and $W^\perp$ is its normal line. Note $\dim W + \dim W^\perp = 2 + 1 = 3$ .

Common mistake

To be in $W^\perp$ , a vector must be orthogonal to every vector in the spanning set, not just one of them. Put all the spanning vectors into the rows of one matrix, so the null space handles every condition at once. And put them in the rows, not the columns: $\operatorname{Nul} A$ is orthogonal to the rows of $A$ .

Tip

Always check a claimed orthogonal vector by taking inner products. It takes seconds and catches most arithmetic slips.