LiteRT.js
LiteRT.js is Google’s “JavaScript binding of LiteRT for running AI directly inside the web browser” — the flagship engine of the browser-inference regime, extending the spoke’s on-device edge stack from native apps onto the web page. LiteRT is the runtime formerly known as TensorFlow Lite (TFLite); LiteRT.js brings it to the browser and succeeds TensorFlow.js (claimed up to ~3× faster).
Execution mechanism
Client-side, no server round-trip. A WASM runtime loads a .tflite model and dispatches to hardware
backends through browser APIs:
- CPU — XNNPACK (multi-threaded), via WebAssembly.
- GPU — ML Drift, Google’s kernels over WebGPU.
- NPU — the emerging WebNN API (experimental in Chrome/Edge).
Results can stay GPU-resident (avoid read-back) for real-time pipelines. Google’s claims: ~3× over other web runtimes, and 5–60× GPU/NPU-over-CPU on demanding workloads. The API is the familiar inference loop — load WASM → compile model → run → read outputs.
Scope note (why it broadens the spoke)
LiteRT.js is general-ML, not LLM-only: the demos are vision-heavy (YOLO detection, depth estimation, image upscaling) plus audio and embedding models. It matters here because the subject is inference execution mechanics — the same “make a trained model run efficiently on the hardware you have” discipline as vllm/llama-cpp, now on browser hardware. LLMs do run on this stack (its sibling MediaPipe ships on-device LLM inference), but LiteRT.js itself is model-agnostic — so it extends the spoke into the browser sub-regime of the edge, with the caveat that the regime is broader than LLMs.
Tier / caveats
T2 — first-party Google Developers Blog: authoritative for the mechanism (backends, .tflite, API),
but the 3× / 5–60× figures are vendor-reported with no neutral benchmark — the spoke’s standing
which-lever-bought-what question applies here too.
Related
browser-inference · local-llm-stack · llama-cpp · quantization · llm-inference