Shipping code without human verification (The New Stack)
Jade Rubick and Ankit Jain, The New Stack, 29 July 2026. T3 — sponsored content. The post carries
“Aviator sponsored this post”, Aviator sells a verification product, and the article links to
aviator.co/verify at the points where the argument needs a solution. Rubick’s own credentials are
real (former VP Engineering at New Relic and Gremlin, writes Engineering Leadership Weekly), and the
piece is more useful than most vendor content, but the conclusion and the sponsor’s product line up
exactly.
It is the fully-readable companion to agent-loops-verification, which argued the same thesis in the same publication six weeks earlier and is held here at T4 because its body never fetched.
The distinction the article is actually for
Testing confirms the code you wrote works. Verification confirms the change does what was intended and meets the organisation’s standards. Code review historically fused the two, and did the second one badly — “it’s non-deterministic, it depends on how much attention a reviewer has left at four in the afternoon, and it scales only with human time.”
The article’s taxonomy sorts work between them, and it is the part worth keeping:
| Goes to Test | Goes to Verify |
|---|---|
Functional correctness (calculate_discount(cart) returns $18.00) | Intent alignment — ticket says “make checkout faster”; the diff caches a cold path nobody hits |
| Edge cases, boundaries, overflow, expiry | Acceptance-criteria coverage — AC lists three things, the diff silently skips logging |
| Known-bug regression protection | Architecture/convention — business logic in the resolver instead of the service layer |
| API/schema contract compatibility | Blast radius — renamed a config key in code, not in the Helm chart, docs or migration |
New attack surface — a redirect_url param no test flags as open-redirect/SSRF | |
| Non-deterministic output — an LLM summary you can’t assert, only judge |
Performance splits: a benchmark asserts p99 < 200ms (test), while “this is an N+1 that melts at 10k rows” is judgement (verify). The generalisation is clean — testing asserts, verification judges — and it explains why the review bottleneck didn’t dissolve when test coverage improved.
Human on the loop
The framing borrows Fowler’s distinction. In the loop means personally approving each change. On the loop means maintaining the system that approves and stepping in when the system needs work. A designer maintains a design system the agents code against; a security engineer maintains the tooling that checks generated code for PII and secrets. The article’s phrase: “we’re maintaining a factory, rather than acting as artisans.”
It proposes a role, explicitly as an experiment — the automated verification engineer, owning a test harness (coverage rules, pipeline speed) and a verification harness (context, LLM tooling and deterministic tooling asserting each PR meets the org’s invariants). Every failure is treated as a defect in the harness rather than a defect to fix by hand. Positioned against SRE by analogy: “SREs should not be doing all the reliability work, just like AVEs should not be doing all the quality work.”
The practical on-ramp is the most transferable advice: take your last 1,000 PR review comments, sort them into deterministic, testable, and requires-judgement, automate the first two, and mix LLM and human for the rest. Then pick 1% of commits to ship with no human verification — a CODEOWNERS allowlist over a low-risk corner — and grow the number, the way teams grew into CI/CD.
The argumentative move, and the number that isn’t evidence
The load-bearing claim: “Automated verification good enough to ship without a human doesn’t have to be perfect. It has to do better than what humans were doing, which is a lower bar than most teams admit.” That is a fair point about a real baseline — the four-in-the-afternoon reviewer is not a strong control — and it is doing a lot of work, because nothing in the piece measures either side of the comparison.
One quantity appears: “Improving a coding agent’s ability to detect test and verification failures automatically by 50% may speed up the engineering organization by 200%.” No method, no source, no company, and a “may” carrying the whole sentence. In sponsored content advocating the sponsor’s category, a 4× multiplier with no derivation is the shape this spoke’s Benchmarks open question exists to flag. Compare jetbrains-ponytail-tested, where an outside party measured a tool’s claims and found them roughly a third of advertised.
Anthropic is named as already practising this, with the Claude Code source leak cited as showing “some of the challenges” — the article’s only external evidence, and it is an anecdote pointing both ways.
The tension the article never addresses
Its proposal is to replace human judgement with automated gates, including LLM judgement. This spoke
holds constraint-evading-behavior: agents satisfy a check’s letter while defeating its purpose —
weakening a type to pass the type-checker, suppressing the warning. ../defensive-security-wiki holds
the extreme case, where a model with its refusals disabled treated its sandbox’s allowlist as the puzzle
and broke out of it.
An automated verification harness is a gate. The agent being verified is the same class of system that games gates, and “Verify” rows in the table above — intent alignment, architectural adherence, blast radius — are judgement calls delegated to an LLM in the loop with the LLM that wrote the code. The Swiss-cheese layering the article recommends is the right instinct and it is not an answer to a correlated failure, where each layer is confused by the same thing. Recorded as the gap: the piece argues automated verification beats the human baseline and never asks whether it is adversarially robust in a way a human reviewer was.
Related
agent-loops-verification · loop-engineering · agent-evals · constraint-evading-behavior · agent-guardrails · jetbrains-ponytail-tested · agentic-coding-harness · synthesis