Bright, Edelman & Johnson, Matrix Calculus (for Machine Learning and Beyond)

Lecture notes for MIT 18.S096 / 18.063, the IAP short course taught by alan-edelman and steven-g-johnson in January 2023, written up by paige-bright with both lecturers (arxiv:2501.14787v1, 7 Jan 2025, 101pp, CC BY-NC-SA 4.0, held in raw/). Filed by arXiv under math.HO with cross-lists to cs.LG, math.NA and stat.ML — a mathematics-education paper that three other fields claim. Shared via Telegram, ingested 2026-08-22.

16 hours of lectures, and the closing chapter says that was not enough: "if you came into this class thinking that taking derivatives is easy and you already learned everything there is to know about it in first-year calculus, hopefully we've convinced you that it is an enormously rich subject that is impossible to exhaust in a single course."

The curriculum claim, which is the reason to hold it

The course opens by locating itself in a gap in MIT's own catalog. After 18.01 (single-variable) and 18.02 (vector calculus), "it seems as though this sequence of material is being cut off arbitrarily":

Scalar → Vector → Matrices → Higher-Order Arrays?

"After all, this is how the sequence is portrayed in many computer programming languages, including Julia! Why should calculus stop with vectors?" The justification offered is that linear algebra "has gradually taken over a much larger part of today's tools" across machine learning, statistics and engineering, so the calculus has to follow it up the type hierarchy. And the generalisation is not routine — the hook is one line: d/dA A² ≠ 2A for a non-scalar matrix A.

This is the corpus's third distinct finding about where the canon stops, and the first where the authors propose to move the edge rather than describe it. See matrix-calculus and the synthesis.

The organising idea: a derivative is a linear operator

Chapter 2 rebuilds the derivative so it will survive the generalisation. In one variable, δf = f(x+δx) − f(x) = f′(x)δx + o(δx); the notes then define, for arbitrary inputs,

df = f(x + dx) − f(x) = f′(x)[dx]

and say explicitly why the square brackets matter: "we avoid dividing by dx, because soon we will allow x (and hence dx) to be something other than a number — if dx is a vector, we won't be able to divide by it!" So f′ is a map taking a point to a linear operator, and that operator eats a direction. "Don't write dx f′(x), which will generally be nonsense!" Making this precise needs a norm, which is why Banach spaces appear in Chapter 5 — introduced to justify what o(δx) means, not as a topic.

One line settles a tension this spoke tracks. Remark 6: "Here, δx is not an infinitesimal but rather a small number." The notes use differential notation throughout and disclaim the infinitesimal reading in a single sentence — a text that reaches for d while refusing what keisler-elementary-calculus builds on, and the cleanest instance yet of the corpus's ε-δ default being stated rather than assumed.

What it covers

Jacobians of matrix functions, worked through the 2×2 matrix-square example and its 4×4 Jacobian, then Kronecker products — the key identity, the Jacobian in Kronecker notation, and the computational cost of using it.
Finite differences — accuracy, order of accuracy, roundoff, and the question the chapter opens with: "Why compute derivatives approximately instead of exactly?"
Derivatives in general vector spaces — matrix dot product and norm, Banach spaces.
Root-finding, optimization, and adjoint differentiation — Newton's method, nonlinear and engineering/physical optimization, then reverse-mode "adjoint" differentiation and worked adjoint examples.
Derivative of the determinant and inverse, with the characteristic polynomial and the logarithmic derivative as applications.
Forward and reverse-mode automatic differentiation — via dual numbers (with the Babylonian square root as the example), why naive symbolic differentiation fails, AD on computational graphs, the forward-vs-reverse cost comparison, and forward-over-reverse for second derivatives.
Differentiating ODE solutions — forward and reverse/adjoint sensitivity analysis.
Calculus of variations — functionals, inner products of functions, arc-length minimisation, the Euler–Lagrange equations.
Derivatives of random functions — stochastic programs, stochastic differentials and the reparameterization trick, and handling discrete randomness.
Second derivatives as bilinear maps, generalized quadratic approximation, Hessians in optimization.
Derivatives of eigenproblems — differentiating on the unit sphere and on orthogonal matrices, then the symmetric eigendecomposition.

The subject-boundary claim

The course description makes an assertion about what AD is: "modern automatic differentiation is more computer science than calculus (it is neither symbolic formulas nor finite differences)." Chapter 8 is built to demonstrate it — dual numbers and computational graphs, with symbolic differentiation shown failing on its own terms. That is a mathematics course arguing that one of its own chapters belongs to another discipline, which no other text here does.

The computational tool assumed

Julia, with "numerous example numerical computations" and installation instructions for students; alan-edelman runs the Julia lab at MIT CSAIL. Course material lives at github.com/mitmath/matrixcalc, and the January 2022 run is on mit-opencourseware. Chapter 14 names the escape hatches in both ecosystems — custom frule/rrule via ChainRules in Julia, jax.custom_jvp/jax.custom_vjp in Python. So the tool assumption is strong but the notes are explicit about the equivalent in the other language.

What it leaves out, by name

Chapter 14, "Where We Go From Here", lists the omissions rather than implying completeness: writing custom Jacobian–vector and vector–Jacobian products when AD "hits something it cannot handle"; complex derivatives, where |z|, Re(z) and Im(z) break ordinary differentiability and Wirtinger (CR) calculus is the usual answer; and the wider literature of matrix-factorization derivatives, "some of them quite tricky to derive." Only carroll-geometric-function-theory, with its §1.4 "What Is Not in This Book", does the same thing here.

matrix-calculus · linear-algebra · vector-calculus · keisler-elementary-calculus · carroll-geometric-function-theory · alan-edelman · steven-g-johnson · paige-bright · massachusetts-institute-of-technology · mit-opencourseware · arxiv · mathematics