Lev

An open-source decision engine written by dmitri-sotnikov in Jolt, built as a local, offline alternative to TypeSafe's hosted Jev (system-one-models-and-jev). It exposes the same wire API — unstructured state plus a set of typed questions in, typed answers with calibrated probabilities out — but runs entirely on local hardware with open weights, with no subscription and no data leaving the machine. Read from its launch post, introducing-lev; the repository itself was not fetched during this ingest.

Two-tier inference, not one

Where a single-pass classifier like laya or Jev must answer every question the same way, Lev routes:

• Fast tier — one of three laya checkpoints (ModernBERT-large English, ModernBERT-large typed-decisions, mmBERT-base multilingual), picked by content and language. ~95 ms per short question on a laptop CPU.
• Slow tier — any GGUF chat model through a statically-linked llama.cpp, default Qwen3.5-4B (Qwen). Candidate answers are scored by token log-probability rather than generated as free text, which skips the autoregressive "thinking" pass entirely and still reaches 95.1% on the launch post's own adversarial set — against 61.1% for the encoder alone.

A confidence gate, thresholded separately per question type (choice/score/noul — see typed-decision), decides which tier answers a given question. On the post's 144-case adversarial set, gating at 0.5 escalated 126 of 144 cases and lifted accuracy from 61.1% to 92.4% at roughly double the per-case latency (0.117 s → 0.270 s) — paid only on the cases that need it, since an easier 120-case mix never escalated at all. A noul (yes/no) question's confidence is max(p, 1-p), which structurally can't fall below 0.5, so a single global threshold silently never escalates any binary question — the reason the gate is per-type rather than global.

Calibration and debiasing as operator tools, not fixed weights

Three features the post frames as things an operator runs on their own traffic, not properties baked into the checkpoints:

• Refit — fits a temperature per question-type/option-count bucket by minimizing negative log-likelihood on labeled traffic, without changing any answer. Demonstrated cutting laya's published mean calibration error 0.466 → 0.081, and Lev's own 20-way-choice error 0.57 → 0.10.
• Debias mode — re-asks a wide-choice question once per rotation of the option order and averages the probabilities, against a measured ~14% option-order flip rate on the encoder tier (~13% on Jev, by the same set).
• Constraints decoder — a post-pass linking related typed questions so the answer set stays mutually consistent (a phishing-flagged email can't also be marked "needs reply").

Packaging

Ships as a single statically-linked executable (jolt binary) with llama.cpp built in — copy it next to the model weights, no interpreter or container. The build uses Jolt's native FFI to reach ICU (tokenization), BLAS/Accelerate, and llama.cpp without the JVM.

What it does not do

The decision space must be declared up front — no open-ended generation, no drafting or summarizing. The English encoder checkpoint's context caps at 512 tokens, with silent end-truncation past that. Calibration refits need re-running as traffic drifts. Against Jev's larger context and built-in escalation to hosted frontier models, Lev trades reach for locality and transparency.

Related

introducing-lev · dmitri-sotnikov · laya · llama-cpp · model-routing · quantization · jolt-clojure · local-llm-stack · typed-decision · system-one-models-and-jev · typesafe-ai · qwen