{
 "cells": [
  {
   "id": "l15-orthogonalitygramschmidtandprojection-001",
   "cell_type": "markdown",
   "source": "# Lesson 15 - Orthogonality, Gram--Schmidt, and Projection\n\n**Learning goals.** By the end of this lecture, students will be able to:\n\n- interpret dot products as both weighted sums and geometric measures of alignment,\n- determine when vectors are orthogonal and explain why orthogonality is useful computationally,\n- project one vector onto another vector or onto a subspace,\n- construct an orthonormal basis using the Gram--Schmidt process,\n- connect projections and orthonormal bases to least-squares approximation and model reduction ideas.\n\n**Key terms.** orthogonality, projection, orthonormal basis, Gram--Schmidt process, least squares.\n\n## From operations to geometry\n\nLesson 11 introduced the dot product as a weighted sum,\n\n$$\n    \\mathbf{u}^{\\mathsf{T}}\\mathbf{v}=\\sum_{i=1}^n u_i v_i.\n$$\n\nThat interpretation is still important, but the dot product also carries geometric information. It tells us how much two vectors point in the same direction.\n\nFor nonzero vectors $\\mathbf{u}$ and $\\mathbf{v}$, the angle $\\theta$ between them satisfies\n\n$$\n    \\mathbf{u}^{\\mathsf{T}}\\mathbf{v}=\\left\\lVert \\mathbf{u} \\right\\rVert_2\\left\\lVert \\mathbf{v} \\right\\rVert_2\\cos\\theta.\n$$\n\nIf the dot product is positive, the vectors point partly in the same direction. If it is negative, they point partly in opposite directions. If it is zero, the vectors are orthogonal.\n\n## Orthogonality\n\nTwo vectors are orthogonal if\n\n$$\n    \\mathbf{u}^{\\mathsf{T}}\\mathbf{v}=0.\n$$\n\nIn two or three dimensions, orthogonality corresponds to a right angle. In higher dimensions, the picture is harder to draw, but the algebraic condition is the same.\n\nOrthogonality is useful because independent contributions do not interfere. If $\\mathbf{u}\\perp\\mathbf{v}$, then\n\n$$\n    \\left\\lVert \\mathbf{u}+\\mathbf{v} \\right\\rVert_2^2=\\left\\lVert \\mathbf{u} \\right\\rVert_2^2+\\left\\lVert \\mathbf{v} \\right\\rVert_2^2.\n$$\n\nThis is the Pythagorean theorem in vector form.\n\nA set of vectors $\\mathbf{q}_1,\\ldots,\\mathbf{q}_k$ is orthonormal if\n\n$$\n    \\mathbf{q}_i^{\\mathsf{T}}\\mathbf{q}_j =\n    \\begin{cases}\n    1, & i=j,\\\\\n    0, & i\\neq j.\n    \\end{cases}\n$$\n\nIf we collect these vectors as columns of a matrix\n\n$$\n    \\mathbf{Q}=\\begin{bmatrix}\\mathbf{q}_1&\\mathbf{q}_2&\\cdots&\\mathbf{q}_k\\end{bmatrix},\n$$\n\nthen orthonormality is written compactly as\n\n$$\n    \\mathbf{Q}^{\\mathsf{T}}\\mathbf{Q}=\\mathbf{I}.\n$$\n\n## Projection onto a line\n\nSuppose $\\mathbf{q}$ is a unit vector, meaning $\\left\\lVert \\mathbf{q} \\right\\rVert_2=1$. The projection of $\\mathbf{x}$ onto the line spanned by $\\mathbf{q}$ is\n\n$$\n    \\widehat{\\mathbf{x}}=(\\mathbf{q}^{\\mathsf{T}}\\mathbf{x})\\mathbf{q}.\n$$\n\nThe scalar $\\mathbf{q}^{\\mathsf{T}}\\mathbf{x}$ is the component of $\\mathbf{x}$ in the $\\mathbf{q}$ direction. Multiplying by $\\mathbf{q}$ turns that component back into a vector.\n\nIf the direction vector $\\mathbf{v}$ is not normalized, the projection is\n\n$$\n    \\widehat{\\mathbf{x}}=\\frac{\\mathbf{v}^{\\mathsf{T}}\\mathbf{x}}{\\mathbf{v}^{\\mathsf{T}}\\mathbf{v}}\\mathbf{v}.\n$$\n\nThe error $\\mathbf{x}-\\widehat{\\mathbf{x}}$ is orthogonal to the line:\n\n$$\n    \\mathbf{v}^{\\mathsf{T}}\\left(\\mathbf{x}-\\widehat{\\mathbf{x}}\\right)=0.\n$$\n\nThis orthogonality condition is the key to least-squares problems.\n\nThe geometry is the important part. Projection splits one vector into a part the line can explain and a leftover part that points away from the line:\n\n![L15_OrthogonalityGramSchmidtAndProjection figure 1](generated_figures/L15_OrthogonalityGramSchmidtAndProjection_figure_01.png)\n\n**Figure.** Projection onto a line decomposes $\\mathbf{x}$ into a component along the line and an orthogonal residual.\n\nThe formula matters because it computes the picture; the picture matters because it tells us what the formula is trying to do.\n\n**Example.**\n\nLet\n\n$$\n    \\mathbf{x}=\\begin{bmatrix}2\\\\1\\end{bmatrix},\n    \\qquad\n    \\mathbf{v}=\\begin{bmatrix}1\\\\1\\end{bmatrix}.\n$$\n\nThen\n\n$$\n    \\widehat{\\mathbf{x}}\n    =\\frac{\\mathbf{v}^{\\mathsf{T}}\\mathbf{x}}{\\mathbf{v}^{\\mathsf{T}}\\mathbf{v}}\\mathbf{v}\n    =\\frac{3}{2}\\begin{bmatrix}1\\\\1\\end{bmatrix}\n    =\\begin{bmatrix}3/2\\\\3/2\\end{bmatrix}.\n$$\n\nThe error is\n\n$$\n    \\mathbf{x}-\\widehat{\\mathbf{x}}=\\begin{bmatrix}1/2\\\\-1/2\\end{bmatrix},\n$$\n\nand this error is orthogonal to $\\mathbf{v}$.\n\n## Projection onto a subspace\n\nA subspace is a flat set of vectors that is closed under linear combinations. A line through the origin is a one-dimensional subspace; a plane through the origin is a two-dimensional subspace. In higher dimensions the picture is harder to draw, but the rule is the same.\n\nLet the columns of $\\mathbf{Q}\\in\\mathbb{R}^{n\\times k}$ be an orthonormal basis for a subspace $\\mathcal{S}$. The orthogonal projection of $\\mathbf{x}$ onto $\\mathcal{S}$ is\n\n$$\n    \\widehat{\\mathbf{x}}=\\mathbf{Q}\\mathbf{Q}^{\\mathsf{T}}\\mathbf{x}.\n$$\n\nThe vector $\\mathbf{Q}^{\\mathsf{T}}\\mathbf{x}$ contains the coordinates of the projection in the orthonormal basis. Its entries are dot products:\n\n$$\n    \\mathbf{Q}^{\\mathsf{T}}\\mathbf{x}\n    =\n    \\begin{bmatrix}\n    \\mathbf{q}_1^{\\mathsf{T}}\\mathbf{x}\\\\\n    \\mathbf{q}_2^{\\mathsf{T}}\\mathbf{x}\\\\\n    \\vdots\\\\\n    \\mathbf{q}_k^{\\mathsf{T}}\\mathbf{x}\n    \\end{bmatrix}.\n$$\n\nMultiplying by $\\mathbf{Q}$ converts those coordinates back to the original vector space:\n\n$$\n    \\widehat{\\mathbf{x}}\n    =\\mathbf{Q}\\left(\\mathbf{Q}^{\\mathsf{T}}\\mathbf{x}\\right)\n    =\\sum_{i=1}^k \\left(\\mathbf{q}_i^{\\mathsf{T}}\\mathbf{x}\\right)\\mathbf{q}_i.\n$$\n\n![L15_OrthogonalityGramSchmidtAndProjection figure 2](generated_figures/L15_OrthogonalityGramSchmidtAndProjection_figure_02.png)\n\n**Figure.** For orthonormal columns, $\\mathbf{Q}^{\\mathsf{T}}$ measures coordinates by dot products and $\\mathbf{Q}$ reconstructs the projected vector from those coordinates.\n\nThe projection is the best approximation to $\\mathbf{x}$ from the subspace:\n\n$$\n    \\widehat{\\mathbf{x}}=\\arg\\min_{\\mathbf{y}\\in\\mathcal{S}} \\left\\lVert \\mathbf{x}-\\mathbf{y} \\right\\rVert_2.\n$$\n\nThe error is orthogonal to the whole subspace:\n\n$$\n    \\mathbf{Q}^{\\mathsf{T}}\\left(\\mathbf{x}-\\widehat{\\mathbf{x}}\\right)=\\mathbf{0}.\n$$\n\nThis orthogonal-residual condition is the key idea to remember: projection is not only a formula, but the closest available approximation together with an error that points outside the chosen subspace.\n\nIf the basis columns in $\\mathbf{V}$ are linearly independent but not orthonormal, then the projection matrix is\n\n$$\n    \\mathbf{V}\\left(\\mathbf{V}^{\\mathsf{T}}\\mathbf{V}\\right)^{-1}\\mathbf{V}^{\\mathsf{T}}.\n$$\n\nThis formula is useful for interpretation, but it is usually less pleasant computationally than working with an orthonormal basis. If the columns are nearly dependent, $\\mathbf{V}^{\\mathsf{T}}\\mathbf{V}$ can be poorly conditioned, and the formula can magnify roundoff error.\n\n**Example.**\n\n**A subspace projection with an orthonormal basis.** Let\n\n$$\n    \\mathbf{q}_1=\\frac{1}{\\sqrt{3}}\n    \\begin{bmatrix}\n    1\\\\1\\\\1\n    \\end{bmatrix},\n    \\qquad\n    \\mathbf{q}_2=\\frac{1}{\\sqrt{2}}\n    \\begin{bmatrix}\n    -1\\\\0\\\\1\n    \\end{bmatrix}.\n$$\n\nThese vectors are orthonormal. If $\\mathbf{x}=[2,0,1]^T$, then the coordinates of the projection are\n\n$$\n    \\mathbf{q}_1^{\\mathsf{T}}\\mathbf{x}=\\sqrt{3},\n    \\qquad\n    \\mathbf{q}_2^{\\mathsf{T}}\\mathbf{x}=-\\frac{1}{\\sqrt{2}}.\n$$\n\nThe projected vector is\n\n$$\n    \\widehat{\\mathbf{x}}\n    =(\\mathbf{q}_1^{\\mathsf{T}}\\mathbf{x})\\mathbf{q}_1\n    +(\\mathbf{q}_2^{\\mathsf{T}}\\mathbf{x})\\mathbf{q}_2\n    =\n    \\begin{bmatrix}\n    3/2\\\\\n    1\\\\\n    1/2\n    \\end{bmatrix}.\n$$\n\nThe residual $\\mathbf{x}-\\widehat{\\mathbf{x}}=[1/2,-1,1/2]^T$ is orthogonal to both $\\mathbf{q}_1$ and $\\mathbf{q}_2$.\n\n## Least squares as projection\n\nAn overdetermined system\n\n$$\n    \\mathbf{A}\\mathbf{x}\\approx\\mathbf{b}\n$$\n\nusually has more equations than unknowns. We cannot, in general, make $\\mathbf{A}\\mathbf{x}$ equal $\\mathbf{b}$. Instead, least squares chooses $\\mathbf{x}$ to minimize the residual norm:\n\n$$\n    \\mathbf{x}_{\\text{ls}}=\\arg\\min_{\\mathbf{x}}\\left\\lVert \\mathbf{A}\\mathbf{x}-\\mathbf{b} \\right\\rVert_2.\n$$\n\nGeometrically, $\\mathbf{A}\\mathbf{x}_{\\text{ls}}$ is the projection of $\\mathbf{b}$ onto the column space of $\\mathbf{A}$. Therefore, the residual\n\n$$\n    \\mathbf{r}=\\mathbf{b}-\\mathbf{A}\\mathbf{x}_{\\text{ls}}\n$$\n\nis orthogonal to every column of $\\mathbf{A}$. This gives the normal equations\n\n$$\n    \\mathbf{A}^{\\mathsf{T}}\\mathbf{r}=\\mathbf{0},\n$$\n\nor\n\n$$\n    \\mathbf{A}^{\\mathsf{T}}\\mathbf{A}\\mathbf{x}_{\\text{ls}}=\\mathbf{A}^{\\mathsf{T}}\\mathbf{b}.\n$$\n\nThe normal equations are a compact way to express the projection condition. In practical numerical work, QR- or SVD-based algorithms are often preferred because they use orthonormal bases more directly and handle rank or conditioning issues more reliably.\n\n## Gram--Schmidt\n\nGram--Schmidt turns a linearly independent set of vectors into an orthonormal set spanning the same subspace. Suppose we start with $\\mathbf{v}_1,\\ldots,\\mathbf{v}_k$. The first basis vector is\n\n$$\n    \\mathbf{q}_1=\\frac{\\mathbf{v}_1}{\\left\\lVert \\mathbf{v}_1 \\right\\rVert_2}.\n$$\n\nTo build $\\mathbf{q}_2$, remove the component of $\\mathbf{v}_2$ in the $\\mathbf{q}_1$ direction:\n\n$$\n    \\mathbf{w}_2=\\mathbf{v}_2-(\\mathbf{q}_1^{\\mathsf{T}}\\mathbf{v}_2)\\mathbf{q}_1,\n$$\n\nthen normalize:\n\n$$\n    \\mathbf{q}_2=\\frac{\\mathbf{w}_2}{\\left\\lVert \\mathbf{w}_2 \\right\\rVert_2}.\n$$\n\nThe first nontrivial step is the same projection picture used in reverse. The candidate vector $\\mathbf{v}_2$ contains a part already explained by $\\mathbf{q}_1$. Gram--Schmidt subtracts that old part and keeps the new orthogonal direction.\n\n![L15_OrthogonalityGramSchmidtAndProjection figure 3](generated_figures/L15_OrthogonalityGramSchmidtAndProjection_figure_03.png)\n\n**Figure.** Gram--Schmidt removes the component already lying in the old direction, leaving a new vector orthogonal to that direction.\n\nFor the $j$th vector, subtract all previously constructed directions:\n\n$$\n    \\mathbf{w}_j=\\mathbf{v}_j-\\sum_{i=1}^{j-1}(\\mathbf{q}_i^{\\mathsf{T}}\\mathbf{v}_j)\\mathbf{q}_i,\n$$\n\n$$\n    \\mathbf{q}_j=\\frac{\\mathbf{w}_j}{\\left\\lVert \\mathbf{w}_j \\right\\rVert_2}.\n$$\n\n**Algorithm.**\n\n**Algorithm: Modified Gram--Schmidt**\n\n1. Start with linearly independent columns $\\mathbf{v}_1,\\ldots,\\mathbf{v}_k$.\n1. For $j=1,2,\\ldots,k$, set $\\mathbf{w}=\\mathbf{v}_j$.\n1. For each previous basis vector $\\mathbf{q}_i$, subtract $(\\mathbf{q}_i^{\\mathsf{T}}\\mathbf{w})\\mathbf{q}_i$ from $\\mathbf{w}$.\n1. Normalize the remaining vector: $\\mathbf{q}_j=\\mathbf{w}/\\left\\lVert \\mathbf{w} \\right\\rVert_2$.\n\n**Listing.** Modified Gram--Schmidt in Python.\n",
   "metadata": {}
  },
  {
   "id": "l15-orthogonalitygramschmidtandprojection-002",
   "cell_type": "code",
   "metadata": {
    "execution": {
     "iopub.status.busy": "2026-06-30T13:24:19.681817Z",
     "iopub.execute_input": "2026-06-30T13:24:19.681923Z",
     "iopub.status.idle": "2026-06-30T13:24:19.746145Z",
     "shell.execute_reply": "2026-06-30T13:24:19.745785Z"
    }
   },
   "execution_count": 1,
   "source": "import numpy as np\n\ndef modified_gram_schmidt(V):\n    V = V.astype(float)\n    n, k = V.shape\n    Q = np.zeros((n, k))\n    R = np.zeros((k, k))\n    for j in range(k):\n        w = V[:, j].copy()\n        for i in range(j):\n            R[i, j] = Q[:, i] @ w\n            w = w - R[i, j] * Q[:, i]\n        R[j, j] = np.linalg.norm(w)\n        if R[j, j] == 0.0:\n            raise ValueError('dependent input vectors')\n        Q[:, j] = w / R[j, j]\n    return Q, R\n",
   "outputs": []
  },
  {
   "id": "l15-orthogonalitygramschmidtandprojection-003",
   "cell_type": "markdown",
   "source": "The output satisfies approximately\n\n$$\n    \\mathbf{Q}^{\\mathsf{T}}\\mathbf{Q}=\\mathbf{I},\n$$\n\nand\n\n$$\n    \\mathbf{V}=\\mathbf{Q}\\mathbf{R}.\n$$\n\nThis is a QR factorization.\n\n**Example.**\n\n**Discrete polynomial-like basis vectors.** On the grid $i=0,1,2,3$, start with the sampled monomials\n\n$$\n    \\mathbf{v}_0=\n    \\begin{bmatrix}\n    1\\\\1\\\\1\\\\1\n    \\end{bmatrix},\n    \\qquad\n    \\mathbf{v}_1=\n    \\begin{bmatrix}\n    0\\\\1\\\\2\\\\3\n    \\end{bmatrix},\n    \\qquad\n    \\mathbf{v}_2=\n    \\begin{bmatrix}\n    0\\\\1\\\\4\\\\9\n    \\end{bmatrix}.\n$$\n\nThe first vector is the constant shape:\n\n$$\n    \\mathbf{q}_0=\\frac{1}{2}\n    \\begin{bmatrix}\n    1\\\\1\\\\1\\\\1\n    \\end{bmatrix}.\n$$\n\nThe component of $\\mathbf{v}_1$ along $\\mathbf{q}_0$ is\n\n$$\n    \\mathbf{q}_0^{\\mathsf{T}}\\mathbf{v}_1=3,\n$$\n\nso the centered linear part is\n\n$$\n    \\mathbf{w}_1=\\mathbf{v}_1-3\\mathbf{q}_0=\n    \\begin{bmatrix}\n    -3/2\\\\\n    -1/2\\\\\n    1/2\\\\\n    3/2\n    \\end{bmatrix}.\n$$\n\nAfter normalization,\n\n$$\n    \\mathbf{q}_1=\\frac{1}{\\sqrt{5}}\n    \\begin{bmatrix}\n    -3/2\\\\\n    -1/2\\\\\n    1/2\\\\\n    3/2\n    \\end{bmatrix}.\n$$\n\nFor the quadratic sample, remove the constant and linear parts:\n\n$$\n    \\mathbf{w}_2\n    =\n    \\mathbf{v}_2\n    -(\\mathbf{q}_0^{\\mathsf{T}}\\mathbf{v}_2)\\mathbf{q}_0\n    -(\\mathbf{q}_1^{\\mathsf{T}}\\mathbf{v}_2)\\mathbf{q}_1\n    =\n    \\begin{bmatrix}\n    1\\\\-1\\\\-1\\\\1\n    \\end{bmatrix},\n$$\n\nso\n\n$$\n    \\mathbf{q}_2=\\frac{1}{2}\n    \\begin{bmatrix}\n    1\\\\-1\\\\-1\\\\1\n    \\end{bmatrix}.\n$$\n\n![L15_OrthogonalityGramSchmidtAndProjection figure 4](generated_figures/L15_OrthogonalityGramSchmidtAndProjection_figure_04.png)\n\n**Figure.** A discrete constant-linear-quadratic orthonormal basis on four sample points. The scale is less important than the shapes and their orthogonality.\n\nThis is a discrete analogue of building orthogonal polynomials: begin with simple powers of the grid index, then remove what has already been represented. A sampled response can then be reconstructed from basis coefficients. For example, if\n\n$$\n    \\mathbf{x}=\n    \\begin{bmatrix}\n    2\\\\2\\\\3\\\\5\n    \\end{bmatrix},\n$$\n\nthen\n\n$$\n    \\mathbf{q}_0^{\\mathsf{T}}\\mathbf{x}=6,\n    \\qquad\n    \\mathbf{q}_1^{\\mathsf{T}}\\mathbf{x}=\\sqrt{5},\n    \\qquad\n    \\mathbf{q}_2^{\\mathsf{T}}\\mathbf{x}=1,\n$$\n\nand the reconstruction is\n\n$$\n    \\mathbf{x}\n    =\n    6\\mathbf{q}_0+\\sqrt{5}\\mathbf{q}_1+\\mathbf{q}_2.\n$$\n\nThe three coefficients say how much constant, linear, and quadratic shape is present in this four-point signal.\n\n## Why orthonormal bases are computationally useful\n\nIf $\\mathbf{Q}$ has orthonormal columns, then projecting onto its column space is simple:\n\n$$\n    \\widehat{\\mathbf{x}}=\\mathbf{Q}\\mathbf{Q}^{\\mathsf{T}}\\mathbf{x}.\n$$\n\nCoordinates are computed by dot products:\n\n$$\n    \\mathbf{c}=\\mathbf{Q}^{\\mathsf{T}}\\mathbf{x}.\n$$\n\nLengths are preserved under square orthogonal transformations:\n\n$$\n    \\left\\lVert \\mathbf{Q}\\mathbf{z} \\right\\rVert_2=\\left\\lVert \\mathbf{z} \\right\\rVert_2\n$$\n\nwhen $\\mathbf{Q}^{\\mathsf{T}}\\mathbf{Q}=\\mathbf{I}$ and $\\mathbf{Q}$ is square. These properties make orthonormal bases stable and convenient in least squares, model reduction, and iterative methods.\n\n**Example.**\n\n**A Galerkin system from a projection condition.** Suppose we want an approximate solution of $\\mathbf{A}\\mathbf{x}=\\mathbf{b}$ in the two-dimensional subspace spanned by the orthonormal columns of $\\mathbf{Q}$. We write\n\n$$\n    \\mathbf{x}\\approx \\mathbf{Q}\\mathbf{y}.\n$$\n\nThe residual is\n\n$$\n    \\mathbf{r}=\\mathbf{b}-\\mathbf{A}\\mathbf{Q}\\mathbf{y}.\n$$\n\nA Galerkin condition asks that this residual be orthogonal to the same subspace:\n\n$$\n    \\mathbf{Q}^{\\mathsf{T}}\\mathbf{r}=\\mathbf{0}.\n$$\n\nSubstituting the residual gives the smaller system\n\n$$\n    \\mathbf{Q}^{\\mathsf{T}}\\mathbf{A}\\mathbf{Q}\\mathbf{y}\n    =\n    \\mathbf{Q}^{\\mathsf{T}}\\mathbf{b}.\n$$\n\nThis system did not come out of thin air: it is the orthogonality condition applied to the residual. Krylov and finite-element methods use this same projection logic with more carefully chosen subspaces.\n\n## Connection to Krylov ideas\n\nLesson 14 ended with repeated matrix-vector products. There, the iteration index $k$ was a sweep or generation counter: Richardson, Jacobi, and Gauss--Seidel repeatedly converted a residual into a new population estimate, mostly keeping only the current iterate. Krylov methods keep more of that generated history. Starting from a residual $\\mathbf{r}_0$, one can build the sequence\n\n$$\n    \\mathbf{r}_0,\\quad \\mathbf{A}\\mathbf{r}_0,\\quad \\mathbf{A}^2\\mathbf{r}_0,\\quad \\ldots\n$$\n\nThe span of the first $k$ vectors is a Krylov subspace:\n\n$$\n    \\mathcal{K}_k(\\mathbf{A},\\mathbf{r}_0)=\\operatorname{span}\\{\\mathbf{r}_0,\\mathbf{A}\\mathbf{r}_0,\\ldots,\\mathbf{A}^{k-1}\\mathbf{r}_0\\}.\n$$\n\nThe vectors in this sequence can become nearly dependent, so orthonormalization is essential in many Krylov methods. Gram--Schmidt applied to this sequence is the core idea behind Arnoldi iteration, GMRES, and related algorithms.\n\nWe will not develop GMRES fully here. The important message for this course is that the same ingredients keep appearing:\n\n$$\n    \\text{matrix-vector products} + \\text{orthogonality} + \\text{projection}.\n$$\n\n## Connection to eigenvectors\n\nOrthogonality also appears in eigenvalue problems. For real symmetric matrices, eigenvectors associated with distinct eigenvalues are orthogonal. This is one reason symmetric eigenvalue problems are especially well behaved.\n\nPower iteration in Lesson 16 repeatedly multiplies by $\\mathbf{A}$ and normalizes. More advanced eigenvalue algorithms keep several directions at once, orthogonalize them, and project the matrix onto a lower-dimensional subspace. The geometry in this chapter is therefore not separate from eigenvalue computation; it is one of the foundations.\n\n## Exercises\n\n1. Let $\\mathbf{u}=[1,2,-1]^T$ and $\\mathbf{v}=[2,-1,0]^T$. Compute $\\mathbf{u}\\cdot\\mathbf{v}$. Are the vectors orthogonal?\n1. Project $\\mathbf{x}=[2,1]^T$ onto the line spanned by $\\mathbf{v}=[1,1]^T$. What is the projection error, and how can you check that it is orthogonal to $\\mathbf{v}$?\n1. Apply the first two steps of Gram--Schmidt to $\\mathbf{v}_1=[1,1,0]^T$ and $\\mathbf{v}_2=[1,0,1]^T$. Give $\\mathbf{q}_1$ and $\\mathbf{q}_2$.\n1. Explain why orthonormal columns make projection and coordinate calculations easier.\n\n**Answers.**\n\n1. The dot product is $1(2)+2(-1)+(-1)(0)=0$, so the vectors are orthogonal.\n1. The projection coefficient is $3/2$, so\n$$\n        \\widehat{\\mathbf{x}}=\n        \\frac{3}{2}\n        \\begin{bmatrix}\n        1\\\\\n        1\n        \\end{bmatrix}\n        =\n        \\begin{bmatrix}\n        3/2\\\\\n        3/2\n        \\end{bmatrix}.\n$$\n\nThe error is $[1/2,-1/2]^T$, and its dot product with $[1,1]^T$ is zero.\n\n1. The first vector is\n$$\n        \\mathbf{q}_1=\\frac{1}{\\sqrt{2}}\n        \\begin{bmatrix}\n        1\\\\\n        1\\\\\n        0\n        \\end{bmatrix}.\n$$\n\nSubtracting the part of $\\mathbf{v}_2$ along $\\mathbf{q}_1$ gives\n\n$$\n        \\mathbf{w}_2=\n        \\begin{bmatrix}\n        1/2\\\\\n        -1/2\\\\\n        1\n        \\end{bmatrix},\n        \\qquad\n        \\mathbf{q}_2=\\frac{1}{\\sqrt{6}}\n        \\begin{bmatrix}\n        1\\\\\n        -1\\\\\n        2\n        \\end{bmatrix}.\n$$\n\n1. If $\\mathbf{Q}$ has orthonormal columns, coordinates are $\\mathbf{Q}^T\\mathbf{x}$, and the projection is $\\mathbf{Q}\\mathbf{Q}^T\\mathbf{x}$. No normal-equation solve is needed.\n\n## End-of-lecture summary\n\n**Takeaway.**\n\n- The dot product measures both weighted sums and geometric alignment.\n- Orthogonal vectors have zero dot product and independent length contributions.\n- Projection gives the closest vector in a line or subspace, and the projection error is orthogonal to that subspace.\n- Least squares is a projection problem onto a column space.\n- Gram--Schmidt builds an orthonormal basis from independent vectors.\n- Orthonormal bases support stable projections, least-squares solves, Krylov methods, and eigenvalue algorithms.\n",
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.13.13",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}