Meaning-Typed Programming (MTP)

Dantanarayana, Kang, Sivasothynathan, Clarke, Li, Kashmira, Flautner, Tang, and Mars — "MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programming," PACMPL, OOPSLA2, Article 314 (October 2025). Peer-reviewed. The paper behind byllm and jaseci's by operator, and the corpus's rare instance of an agent-tooling claim backed by a controlled user study and a 13-benchmark suite rather than a star count.

The idea

Prompt engineering makes developers hand-write the text sent to an LLM and hand-parse what comes back. MTP's bet: source code already carries the semantic information a prompt needs — names, types, structure — so a compiler can extract it and build the prompt automatically. Three parts:

• The by operator — replace a function body with by gpt4() and the call becomes an LLM invocation. def calc_age(birth: date, today: date) -> int by gpt4() is the paper's own example.
• MT-IR — a meaning-based intermediate representation, built at compile time, that preserves variable names, function signatures, and types as the semantic payload for the prompt.
• MT-Runtime — binds runtime values into MT-IR at each by call site, generates the context-aware prompt, and handles type-safe parsing of the response.

Evaluation against DSPy and LMQL

Three-way comparison across a 13-problem benchmark suite (math, translation, essay review, RPG level generation, a Wikipedia ReAct agent, and more):

• Code reduction. MTP needs on average 4.0× fewer lines modified than LMQL and 5.3× fewer than DSPy to add an LLM feature to existing code (worst case 1.3×, best case 10.7× vs DSPy on the "Odd Word Out" benchmark).
• User study (N=20, blind to the frameworks beforehand). Developers finished tasks 3.2× faster with 45% fewer lines of code than with DSPy or LMQL. On the hardest task (RPG level generator), LMQL's success rate was zero — participants could not hand-craft a working prompt in 90 minutes. MTP scored highest on ease-of-use, documentation clarity, and overall satisfaction across all five surveyed criteria.
• Accuracy. Average correctness across the benchmark suite (GPT-4o, 100 trials/benchmark): 90.85% for LMQL, 98.23% for DSPy, 98.92% for MTP — MTP wins 9 of 13 benchmarks against DSPy despite doing no manual prompt engineering at all. On GSM8K across ten LLM generations, MTP tracks DSPy closely on current-generation GPT models and pulls ahead as models get better, while DSPy's accuracy plateaus — the paper's read: less prompt engineering matters less as models get smarter, not more.
• Cost and tokens. MTP uses fewer tokens than DSPy on every benchmark tested, translating to 3.5×–4×+ cost and runtime savings on individual benchmarks (math_problem, odd_word_out) at GPT-4o pricing.
• Robustness. Renaming 25–50% of identifiers to meaningless strings left correctness at 98%; it degrades gracefully from there — 70% correctness at 75% renaming, 20% at 100% (including function signatures). Semantically consistent but misleading renames (Level→Boss) hurt more than random abbreviation (60% correctness), which the paper reads as evidence MTP is extracting real meaning, not just presence-of-identifier.

What this adds to the spoke's evidence gap

synthesis.md has an open, repeated finding that this corner's most-adopted packages (superpowers, ruflo, cc-thinking-skills's subjects) ship with no measurement at all. MTP is the counter-case: a pre-registered-style comparison against two named prior systems, a blind user study, and a released benchmark suite. It doesn't resolve the gap elsewhere in the corpus, but it is direct evidence that evaluating an LLM-integration abstraction is tractable when someone chooses to do it.

Caveats

• The user study population is 20 undergraduates from one NLP class, not professional developers.
• GPTScore (an LLM-as-judge) grades most of the open-ended benchmarks; only three of thirteen use exact-match ground truth.
• DSPy's compiled/optimized mode (extra training examples, template search) is not the version MTP is compared against on the code-reduction and cost figures — only on the GSM8K accuracy chart, where MTP without training still beats compiled DSPy on all LLMs except GPT-4.

Related

byllm · jaseci · agentskills-spec · loop-engineering