Agent guardrails (autonomy boundaries)

The discipline of bounding what an agent may do without a human checkpoint — the safety counterpart to autonomy. Its organizing principle is reversibility / recovery cost: grant high autonomy where a mistake is cheap to undo (refactors, unit tests), require pre-execution approval where the penalty is steep or irreversible (dropped production tables, prod deploys, IAM changes) agents-never-do-alone. Autonomy is graduated by blast radius, not all-or-nothing.

The bright lines

Categories that should require human approval before execution agents-never-do-alone: destructive file ops (rm -rf, git reset --hard), DB writes/migrations (DROP, TRUNCATE, DELETE sans WHERE), cloud infra (terraform apply, kubectl delete, IAM), production deployments (regardless of code quality), auth/security logic (failures surface in incident reports, not unit tests), and secrets/credentials.

Mechanisms

AGENTS.md contract — setup + coding + safety rules + “Definition of Done”; the cross-vendor sibling of the CLAUDE.md / spec-driven-development “constitution”.
blocked_commands.md — an explicit block list; don’t trust the agent to infer limits.
Two-agent review loop — an implementer + an egoless diff-reviewer: the agent-orchestration adversarial-verification pattern repurposed as a safety gate.
Human-in-the-loop checkpoints — the gating itself; in framework terms, LangChain’s HumanInTheLoopMiddleware (agent-middleware) and the supervised-board model of agent-kanban.
Final reports + deploy-timing reserved for humans (the agent lacks live-incident context).

The counterexample — a constitution instead of enforcement

auto-company runs fully unattended and states its limits as prose the agent reads (no gh repo delete, no wrangler delete, no force-push to main, no deleting ~/.ssh) while conceding that its sandbox boundary “relies on underlying CLI configurations” — Codex danger-full-access or Claude bypassPermissions, operating directly on the host. Its FAQ’s remedy for cycles stalling on permission prompts is to turn the prompts off.

That reverses this page’s principle twice over: the block list is advisory rather than enforced, and the checkpoint layer is removed at the point where autonomy gets uncomfortable. Compare run-claude-code-agents-24-hours, which reaches the same waived-permissions state deliberately but pays for it with containment (sandbox plus recoverable backups) — autonomy bought, not assumed. The two make the enforcement-vs-instruction distinction concrete, and the wiki has no incident data either way yet.

Moving the boundary to the tools (nono)

The mechanisms above bound the agent. nono argues the boundary is in the wrong place: what actually holds dangerous privilege is the tools the agent delegates to — git, gh, curl, kubectl — and a single agent-level policy hands all of them the same secrets and the same network. So it runs each delegated tool in its own child sandbox with separate filesystem grants, network rules, credentials and policy, and injects tokens through a proxy with L7 endpoint filtering rather than exposing them to the agent at all.

Two things follow that this page previously had no answer for. An agent cannot widen a tool’s permissions from inside its session, because that policy was never its to hold — which closes the auto-company failure mode structurally rather than by asking nicely. And the “turn the prompts off when they get annoying” escape hatch stops being available, because the restriction isn’t a prompt. Enforcement over instruction, taken one layer deeper than containment of the agent process.

Policies are composable JSON with profile inheritance and a public registry, which makes a restriction a shareable versioned artifact — the packaging move agent-skills made for capability, applied to its opposite. Still unaudited: a sandbox is judged by escapes, and none of that evidence exists here yet.

Phase-scoped tool availability (statewright)

nono moves the boundary onto the tools; statewright moves it onto the workflow phase. A state machine defines phases (plan → implement → test), and each phase exposes only the tools that belong to it — read-only while planning, edit tools once implementing, test commands only while testing — rejecting any out-of-phase call with a note on how to transition. The novelty for this page is the motive: every mechanism above bounds the agent for safety, and this one bounds it for reliability, on the finding that an over-tooled agent edits during review and deploys before tests pass. Enforcement-not-instruction, generalized from “don’t do harm” to “stay on task” — with the destructive-op blocks still applying inside a permissive phase, so it layers on the blast-radius model rather than replacing it. Its evidence (2/10 → 10/10 on a 5-task subset) is the strongest single data point for structure over capability in the corpus and one of the smallest, discussed under the Benchmarks open question in synthesis.

Where it sits

This names and operationalizes a reliability/governance thread recurring across the wiki: microsoft-scout‘s continuous policy-conformance audit, hermes-agent‘s command-approval / container isolation, and the review/eval discipline behind the agentic-coding-harness (“structure substitutes for capability” includes structure that says no). It’s the explicit counterweight to the autonomy push of durable-agents and self-improving-agents — and a strong candidate to be itself a stack of agent-middleware.

Design-time guardrails, not just runtime

The bright lines above are runtime gates (approve before the destructive act). constraint-evading-behavior adds the design-time face: make invalid states unrepresentable and turn checks into hard failures (-Werror, hooks that reject warning-suppressions) so the agent can’t take the easy way out in the first place. Same discipline — structure that says no — moved earlier, into the types and the build config rather than a pre-execution prompt. It also sharpens the enforcement-not-instruction point: a guardrail the agent can cheese (satisfy vacuously) is instruction, not enforcement.

Spend as a bound (2026-07-28)

The bounds above are about reversibility and recovery cost — how bad is it if the agent is wrong. gemini-managed-agents-hooks adds an axis those miss: budget. max_total_tokens caps input, output and thinking together; on the limit the run “safely pauses”, returns status: "incomplete", keeps its environment, and resumes on a fresh budget via previous_interaction_id.

That’s the bound an autonomous multi-turn loop actually needs. Its characteristic failure isn’t a destructive action, it’s a run that never converges and quietly spends — invisible to a guardrail that only asks whether an action is reversible. Pausing rather than aborting makes it a durability mechanism at the same time: a spend ceiling and a resumable checkpoint are one construct viewed from two sides.

The same source shows a deny path that explains itself: a pre_tool_execution hook returning {"decision": "deny", "reason": "..."} skips the call and feeds the reason into the model’s context. Useful, and double-edged — per constraint-evading-behavior, telling a model exactly why it was blocked also tells it what would pass.

A posture ladder with a floor under it (added 2026-08-03)

qm supplies a governance shape this page hadn’t seen: not a per-run permission prompt and not a constitution in prose, but an org-level posture that narrower scopes may only tighten. Three rungs — Strict (every harness tool call pauses for approval), Auto (a classifier screens provenance-labelled external data and tool results before the model sees them), Dangerous (neither).

The structural move is the floor. A predeclared command policy — approval rules and hard denials for recursive deletes, destructive SQL and similar — applies in every posture, Dangerous included. So review is configurable and effects are not, and a user turning safety down cannot turn that particular class off. Compare auto-company, where the limits are prose in a CLAUDE.md and the FAQ’s remedy for a stalled cycle is to disable the prompts.

Two admissions from the same source belong here, because they bound what this page can claim. Its command policy is “a speed bump against mistakes and injection, not a sandbox boundary” — obfuscation, encoding, or writing then executing a script evades it. And a line that applies to every human-in-the-loop design in this wiki: “An approval means a human accepted the displayed action under the information available at that time, not that the resulting behavior is safe.” Approval gates the decision a human was shown, not the effect that follows.

agents-never-do-alone · agentic-coding-harness · agent-orchestration · agent-middleware · spec-driven-development · constraint-evading-behavior · microsoft-scout · hermes-agent · nono · nolabs-ai · gemini-managed-agents-hooks · qm · agent-scope-isolation