mlx-Chronos

An open benchmark and community leaderboard for local inference engines on Apple Silicon: mlx-lm, oMLX, vllm-mlx, Rapid-MLX and Ollama's MLX backend (igurss/mlx-chronos, Apache-2.0, 23★, created May 2026). Found on 2026-09-23 while re-checking the absence claim on omlx; results read from the repository's submitted JSON files.

T2: a third party measuring engines it did not build, with the protocol published and each run saved as a file anyone can re-check. Weak points, stated below, limit what it can show.

Method

Each run records hardware (chip, memory, thermal state, power source), engine version and model, then measures cold time to first token, cached time to first token (the same prompt sent again), request and decode tokens per second separately, and peak RAM, with mean, standard deviation, minimum and maximum. Decode speed is reported apart from total speed, which is the split openmetal-tokens-per-second says a fair measurement needs.

What the submitted runs show (21 files, June–July 2026)

Same machine (Apple M2, 8 GB) and same model (Qwen3.5-4B, NVFP4), run on 2026-06-12:

oMLX decodes fastest, by about 20% over mlx-lm. It is also the only engine whose cached first token is no faster than cold, and that holds in its other five runs too (M2 Max and M5 Pro, five different model builds). mlx-lm roughly halves the second request, and Ollama cuts it by 70–80%.

Limits

• Very short prompts. The cached test repeats one sentence. oMLX's cache is designed for long coding-agent contexts kept across restarts, and this test cannot show that benefit or its absence. What it does show is that oMLX gets no short-prompt reuse where the other two do.
• Few runs from few people. Seven oMLX files, two mlx-lm, ten Ollama, on three kinds of Mac.
• The leaderboard is new and its author is asking for feedback on the method.

Related

omlx · mlx · llama-cpp · tokens-per-second · openmetal-tokens-per-second · kv-cache · synthesis