Local LLM stack (platforms vs. engines)

The tooling people assemble to run an LLM on their own machine — the on-device / single-user regime the synthesis names against the datacenter regime. The llms-local-list catalog draws the load-bearing distinction inside it: platforms sit on top of engines.

• Engines are the runtimes that actually execute the model — load weights, manage the kv-cache, run token-sampling, and emit tokens. The canonical local engine is llama-cpp (C/C++, GGUF quantization, the de-facto core of most consumer tools); the list groups it with Ollama, vLLM, SGLang, and Apple's MLX. Note that vllm and SGLang are datacenter engines that also show up here — people run them locally on a single box — so the engine list spans both regimes.
• Platforms are the turnkey apps wrapped around an engine: LM Studio, Jan, and LocalAI. They bundle model download, a chat UI, and a local API endpoint so a user never touches the engine directly. This is the layer the spoke had only implied — the llama-cpp page notes Ollama and LM Studio are built on llama.cpp, but the platform tier itself wasn't named until now.

The platform/engine line does not hold everywhere (added 2026-09-04). omlx is a turnkey app — a signed DMG, a SwiftUI menubar, a welcome wizard, a web dashboard in eight languages — and it is also a serving engine with a paged kv-cache, continuous-batching, LRU model eviction and an OpenAI/Anthropic API. It does not sit on top of an engine the way LM Studio sits on llama.cpp; it wraps MLX/mlx-lm directly and supplies the scheduling and caching itself. So the two-row map above is a description of the llama.cpp lineage rather than a law of the category, and the Apple Silicon corner is where it comes apart first: MLX ships a generation loop but no server, which leaves the server-shaped hole that oMLX fills.

A distinct on-device surface sits beside these native engines: in-browser inference, where the engine runs inside a web page through WASM/WebGPU/WebNN rather than as a native binary. litertjs (Google) is the flagship — same "run a model on the user's own device" goal as llama.cpp/Ollama, but reaching the hardware through browser APIs instead of CUDA/Metal.

Above the runtime sit the UI (Open WebUI, Lobe Chat, SillyTavern, …) and builder layers (agent frameworks, MCP, RAG, evaluation) — real parts of the local-LLM experience but out of this spoke's scope: they belong to agentic-tooling-wiki and llm-providers-wiki. What lands here is the execution stack: platform → engine → the prompt-to-token pipeline the rest of the wiki documents.

Why it matters to the wiki

It grounds the edge / on-device regime with the tools that actually fill it. The synthesis argued that inference runs at two ends of the hardware spectrum and that quantization is the edge lever; this page supplies the players at that end — the platform/engine separation that explains why "running a model locally" usually means picking a turnkey app (LM Studio) without ever choosing an engine, while a power user reaches past it to llama-cpp or vllm directly. It's a map, not a measurement — sourced from a T4 curated list (llms-local-list), so it names the players without benchmarking them.

Related

llama-cpp · vllm · quantization · llm-inference · llms-local-list · omlx · mlx