Local LLM stack (platforms vs. engines)

The tooling people assemble to run an LLM on their own machine — the on-device / single-user regime the synthesis names against the datacenter regime. The llms-local-list catalog draws the load-bearing distinction inside it: platforms sit on top of engines.

• Engines are the runtimes that actually execute the model — load weights, manage the kv-cache, run token-sampling, and emit tokens. The canonical local engine is llama-cpp (C/C++, GGUF quantization, the de-facto core of most consumer tools); the list groups it with Ollama, vLLM, SGLang, and Apple’s MLX. Note that vllm and SGLang are datacenter engines that also show up here — people run them locally on a single box — so the engine list spans both regimes.
• Platforms are the turnkey apps wrapped around an engine: LM Studio, Jan, and LocalAI. They bundle model download, a chat UI, and a local API endpoint so a user never touches the engine directly. This is the layer the spoke had only implied — the llama-cpp page notes Ollama and LM Studio are built on llama.cpp, but the platform tier itself wasn’t named until now.

A distinct on-device surface sits beside these native engines: in-browser inference, where the engine runs inside a web page through WASM/WebGPU/WebNN rather than as a native binary. litertjs (Google) is the flagship — same “run a model on the user’s own device” goal as llama.cpp/Ollama, but reaching the hardware through browser APIs instead of CUDA/Metal.

Above the runtime sit the UI (Open WebUI, Lobe Chat, SillyTavern, …) and builder layers (agent frameworks, MCP, RAG, evaluation) — real parts of the local-LLM experience but out of this spoke’s scope: they belong to agentic-tooling-wiki and llm-providers-wiki. What lands here is the execution stack: platform → engine → the prompt-to-token pipeline the rest of the wiki documents.

Why it matters to the wiki

It grounds the edge / on-device regime with the tools that actually fill it. The synthesis argued that inference runs at two ends of the hardware spectrum and that quantization is the edge lever; this page supplies the players at that end — the platform/engine separation that explains why “running a model locally” usually means picking a turnkey app (LM Studio) without ever choosing an engine, while a power user reaches past it to llama-cpp or vllm directly. It’s a map, not a measurement — sourced from a T4 curated list (llms-local-list), so it names the players without benchmarking them.

Related

llama-cpp · vllm · quantization · llm-inference · llms-local-list