Matrix calculus

Differential calculus on functions whose inputs and outputs are matrices, operators, functions or random variables rather than numbers or vectors — the derivative of a matrix inverse, of a factorization, of an ODE solution, of a functional. One source: edelman-johnson-matrix-calculus.

The move that makes it work

Drop the quotient. First-year calculus defines f′(x) as a limit of Δf/Δx, and that definition dies as soon as the input stops being something you can divide by. The notes define instead

df = f(x + dx) − f(x) = f′(x)[dx],

so that f′(x) is a linear operator and dx is the direction it acts on. The derivative at a point is no longer a number or a matrix of numbers; it is the linear map that best approximates the change. Everything else follows from being able to say that in any vector space with a norm — which is why Banach spaces show up as machinery rather than as a topic.

The generalisation is not a formality. d/dA A² ≠ 2A for a non-scalar matrix A, because A and dA need not commute: the correct answer is the operator dA ↦ A·dA + dA·A. That single inequality is the subject's advertisement.

What the layer contains

• Jacobians of matrix functions, and Kronecker products as the notation that turns an operator back into a matrix when you need one — at a cost the source is careful to state.
• Adjoint / reverse-mode differentiation, better known as backpropagation: computing the gradient of one scalar output with respect to very many inputs in roughly the cost of evaluating the function once, instead of once per input. The efficiency argument, not the chain rule, is why this exists.
• Automatic differentiation — forward mode via dual numbers, reverse mode over computational graphs, and forward-over-reverse for second derivatives. Distinct from both symbolic differentiation and finite differences.
• Second derivatives as bilinear maps, with the Hessian as the matrix you get when the inputs are vectors.
• Derivatives of determinants and inverses, of eigenproblems (on the sphere, on orthogonal matrices, of the symmetric eigendecomposition), of ODE solutions (forward and adjoint sensitivity analysis), of functionals (the calculus of variations and Euler–Lagrange), and of random functions (the reparameterization trick).

Where it sits in the spoke

Directly above vector-calculus and directly on top of linear-algebra — the source's own framing is that MIT's 18.01 → 18.02 sequence "is being cut off arbitrarily" and that the next term of Scalar → Vector → Matrices → Higher-Order Arrays is missing from the standard curriculum. Whether that gap is real or an artifact of one institution's catalog is not something one source can settle, but the argument is made explicitly and with a reason: linear algebra has spread into machine learning, statistics and engineering, so its calculus has to follow.

It is also the spoke's first subject that argues part of itself belongs to another discipline: "modern automatic differentiation is more computer science than calculus." The AD chapter is where this page touches ../machine-learning-wiki — backpropagation is the same object seen from the training side there, and the mechanism lives here.

Related

edelman-johnson-matrix-calculus · vector-calculus · linear-algebra · mathematics