oMLX

An Apache-2.0 inference server that runs language models on a Mac and is driven from the menu bar. Its own line: "LLM inference, optimized for your Mac — continuous batching and tiered KV caching, managed directly from your menu bar." Python 3.11–3.13 over Apple's MLX and mlx-lm, macOS 15.0+ (Sequoia), Apple Silicon only (M1–M5). Repository metadata read 2026-09-04: 21,412 stars, 1,827 forks, 1,272 open issues, 2,549 commits, created 2026-02-13, last push 2026-09-03, homepage omlx.ai.

This is the spoke's first source in which the whole datacenter serving stack is assembled for one laptop. local-llm-stack separates turnkey platforms from engines and puts MLX in the engine row; oMLX sits between them — a server with continuous-batching, a paged kv-cache and multi-model scheduling, wrapped in a SwiftUI menubar app and a web dashboard.

Tier note. T1 for what the project is — architecture, flags, licence, supported model families, repository and release metadata read from the GitHub API. The performance numbers below are the author's own, on hardware the author chose, and the project's benchmark page (omlx.ai/benchmarks) has not been read here.

The cache is the point

Block-based cache management with prefix sharing and copy-on-write, described in the README as "inspired by vLLM" — so PagedAttention's scheme ported onto unified memory. What is new to this corpus is that it has two tiers and one of them is a disk:

Hot tier (RAM) — frequently used blocks, write-back.
Cold tier (SSD) — when the hot tier fills, blocks are written out as safetensors. A later request with a matching prefix restores them from disk instead of recomputing prefill, including after the server has been restarted.

Every KV-cache lever this wiki holds so far lives inside one process's lifetime: kv-cache as the reuse that makes decode cheap, prefill-decode-kv-cache as the thing prefill produces, kv-cache-isolation as the reason a shared cache is a leak surface. A prefix cache that survives omlx stop is a different claim — it makes prefill a cost you can pay once ever, not once per boot, and it is the one thing here the datacenter stacks in this spoke do not describe doing.

The stated motive is coding agents: the author wanted "past context stays cached and reusable across requests… making local LLMs practical for real coding work with tools like Claude Code."

The 30× lever that is off by default

The README's sharpest number is a warning about its own build. Native Metal kernels for GLM-5.2, MiniMax M3 and Qwen3.5 are not built by a plain pip install -e .; the affected families then "silently fall back to much slower generic paths." For GLM-5.2 the fused DSA prefill is "roughly 30x faster with the kernels (measured 845 vs ~29 tok/s on an M3 Ultra)", and the fallback also uses more memory (issue #2137).

Two things about that. The gap is enormous and it is silent — no error, no warning at runtime, just a server an order of magnitude slower than the same server on the same machine. And the reason is a toolchain absence rather than a modelling choice: building the kernels needs the full Metal toolchain, which Command Line Tools alone do not ship (xcrun: error: unable to find utility "metal"), so the DMG and a --with-custom-kernel Homebrew build get them and the obvious source install does not. The project ships a one-liner to check (python -c "from omlx.custom_kernels import native_kernel_status; print(native_kernel_status())"), which is an admission of how easy it is to be on the slow path without knowing.

What else it serves

Continuous batching through mlx-lm's BatchGenerator; max concurrent requests configurable, default 8.
Multi-model serving in one process: an EnginePool with LRU eviction, per-model TTL, model pinning and manual load/unload, plus a ProcessMemoryEnforcer whose default ceiling is system RAM minus 8 GB.
Beyond text: vision-language models, OCR (DeepSeek-OCR, DOTS-OCR, GLM-OCR), embeddings (BERT, BGE-M3, ModernBERT) and rerankers (ModernBERT, XLM-RoBERTa) run on the same batching and cache stack.
API surface: OpenAI and Anthropic shapes — /v1/chat/completions, /v1/completions, /v1/messages, /v1/embeddings, /v1/rerank, /v1/models — with streaming usage stats and Anthropic adaptive thinking.
Tool calling across eight parser families, taken from mlx-lm: JSON <tool_call> (Llama, Qwen, DeepSeek), XML <function=…> (Qwen3.5), <start_function_call> (Gemma), <arg_key> (GLM 4.7/5), namespaced MiniMax, [TOOL_CALLS] (Mistral), Kimi K2, Longcat.
Profiles — named bundles of per-model settings exposed as their own model IDs (qwen3-8b:thinking) that serve on the same engine with settings overlaid per request, "no extra memory, no reload."
Experimental multi-Mac inference — one model split across unequal-memory Macs using MLX pipeline ranks over Ring or Thunderbolt RDMA/JACCL, with byte-aware shard planning and measured compute/link rebalancing. Source builds only.

There is also a Claude Code accommodation that is really a statement about context accounting: oMLX scales the reported token counts so a small-context model triggers the client's auto-compact at the right moment, and holds the SSE connection open so a long prefill does not read-timeout. The client's context management assumes the server's numbers mean what a frontier API means by them.

What the numbers do and don't say

The v0.6.4 release (2026-08-29) splits its own audience: the macOS 26–27 DMG has 22,343 downloads against 1,982 for the macOS 15 Sequoia build — eleven to one toward the newer OS — while the three Python wheels together take about a thousand. Whatever this project is, it is overwhelmingly consumed as a signed desktop app by people on current macOS, not as a pip install by people who read the architecture section.

The other dated number is 1,272 open issues on a repository not yet seven months old, against 2,549 commits. That is a snapshot, not a verdict — a fast-moving project with 21k stars collects reports faster than a solo maintainer closes them — but it is the number to re-read rather than the star count.

Gaps

~~No independent benchmark of any of this.~~ Corrected 2026-09-23: mlx-chronos, a third-party benchmark, has seven oMLX runs. On one M2 with one model, oMLX decoded fastest (31–32 tok/s against mlx-lm's 26–28 and Ollama's 23), but its cached first token was no faster than cold (0.42 s against 0.41 s) while mlx-lm and Ollama cut theirs by half or more. The test prompts are one sentence long, so the long-context SSD-cache claim below is still unmeasured. (Earlier note, re-checked 2026-09-15: a comparative study of MLX, MLC-LLM, Ollama and llama.cpp exists — arXiv 2511.05502 — and community runs put MLX at ~3× llama.cpp decode on an M4 Pro until long contexts; none of it runs oMLX itself, so the sentence stands for the server and not for its runtime.) The 845-vs-29 figure is one model on one machine with no methodology published here; the tiered cache's claimed benefit — prefill you never pay twice — has no hit-rate or latency measurement attached in the README at all, which is the same shape as the corpus's standing complaint about self-reported serving numbers. The project's own benchmark page is unread, and a cold-tier hit rate on a real coding session is the measurement that would settle whether the SSD tier is a lever or a feature.

kv-cache · continuous-batching · local-llm-stack · mlx · paged-attention-paper · vllm · llama-cpp · prefill-decode-kv-cache · quantization