Browser inference
Running model inference client-side inside a web browser — the browser sub-regime of the on-device / edge end of llm-inference. It’s the same execution discipline as native local inference (llama-cpp, local-llm-stack) but constrained to what a web page can reach: no CUDA/Metal, so the runtime dispatches through browser APIs instead.
The browser’s hardware-access stack
Where a native engine talks to the GPU directly, browser inference goes through a portable layer:
- WebAssembly (WASM) — the CPU execution substrate (e.g. XNNPACK kernels).
- WebGPU — the modern GPU-compute API that replaced WebGL for ML; the browser’s answer to CUDA/Metal (Google’s ML Drift kernels target it in litertjs).
- WebNN — an emerging API exposing the device NPU, experimental in Chrome/Edge.
Why it’s its own regime
The binding constraints differ from both datacenter and native-edge inference. The win is privacy +
zero server cost + low latency (nothing leaves the device); the costs are the browser sandbox
(portable APIs, capability gaps across browsers, download size of model + WASM) rather than raw VRAM. So
the spoke’s “same pipeline at both ends of the hardware spectrum” thesis gains a third point:
datacenter GPU → native on-device → in-browser — each defined by how it reaches the silicon.
litertjs (Google) is the founding instance; the regime is model-agnostic (vision/audio/embeddings
as much as LLMs), a seam to ../webperf-wiki (client-side performance) and ../web-browsers-wiki
(WebGPU/WebNN as browser capabilities).
Related
litertjs · llm-inference · local-llm-stack · llama-cpp · quantization