Spokes.wiki Search About

Synthesis — Platform Ops

The evolving thesis of this wiki. Sits above the schema.org pages. Records the current best understanding, open questions, and explicitly flagged contradictions.

Current understanding

This spoke was spun out (2026-06-05) from three sources that arrived separately yet converge on one operational discipline: platform-ops — running cloud-native distributed systems in production. They map cleanly onto three pillars:

  1. observability / service-topology. netflix-service-topology builds a live dependency graph of thousands of microservices by fusing three telemetry sources — eBPF flow logs, IPC metrics, distributed traces — because each has a blind spot (no app context / misses uninstrumented services / sampled). The merged graph beats any single source. distributed-tracing (OpenTelemetry) is the per-request view of the same dependency structure the topology shows in aggregate (a topology ≈ traces summed over time): the literal record of a request crossing service seams, the third signal beside metrics and logs, and the source of the latency SLIs that SLOs are defined on. The standardized layer under this pillar has its own seam, one level down. opentelemetry's pitch is that semantic-conventions make telemetry "line up" across independently-written instrumentation. The OpenTelemetry project's own 2026-09-25 audit (T3 — it measured its own ecosystem) found that a convention reaching stable status is "more of a starting point rather than a finish line": HTTP took over four years from proposal to stable, and required attributes showed up across a seven-language sample far more often than merely-recommended ones, years after HTTP itself stabilized. So the "one contract everyone follows" premise this pillar leans on is itself unevenly delivered — the same "seams, not components" shape as pillar 2's CNCF integration tax, just one layer beneath the tools instead of between them.
  2. platform-engineering. kubernetes-integration-tax argues the dominant cost of production kubernetes is the "integration tax" — making ~20–30 CNCF tools (Prometheus, Cilium, cert-manager, GitOps) work together. The failures are seams (Cilium metrics invisible to Prometheus without a ServiceMonitor), not the tools themselves. The cure is the internal-developer-platform (project-as-a-service, Belastingdienst): pay the integration cost once, centrally, then expose it as a self-service, golden-path product ("make the right way the easiest way") so teams don't each re-pay the tax. Mechanically it's a gitops-shaped reconcile loop applied to project provisioning (one YAML → namespaces/RBAC/quota via an operator), so the "seams, not components" thesis gains its scaling resolution: productize the seams. Half of that work is social (enablement-over-support, Communities of Practice across 99+ teams) — platform-as-a- product is an org pattern, not only tooling. Open tension: a golden path can harden into a golden cage if defaults lag team needs. The paved path also has a denominator, and 2026-08-15 put it in question. new-tenant-is-change observes that one of the paths this pillar paves — a namespace and a quota per developer — assumes "a person produces one stream of work at a time, so isolating people isolates work." A developer supervising several coding agents has several changes in flight, so the seat is no longer the unit of demand. Its proposal (change-level-tenancy) keeps the IDP's contract intact — self-service, no ticket, reclaimed on exit — and re-denominates the tenant from the person to the change, with the tenant owning only the services it modified plus a writable database branch, against a shared baseline deployed from main. The observation that does not depend on agents is the better half: these same organizations run their product as a competent multi-tenant system and their pre-production by ticket and by seat, so "the multi-tenancy playbook that runs the product has never been applied to the platform that builds the product." That is a standing inconsistency in this pillar, not an AI story; agents made it expensive rather than making it true. The source is sponsored vendor content (T3) with no measurement of its own, so the frame is recorded and the claim is not settled.
  3. site-reliability-engineering + aiops. google-sre-agentic-ai applies agentic AI across the SRE workflow (runbooks, anomaly detection, postmortems, incident investigation over "observability + topology") under strict mandates: SLOs + fallbacks, identity/permissions, explainability, auditability. The quantified backbone is service-level-objectives (Google SRE): the SLI → SLO → error-budget chain turns "reliability vs. velocity" into a measured control loop (spend the budget → stop shipping → fix reliability), so toil/MTTR/impact become budget math, not anecdote. It also reframes the AIOps reliability paradox — an ops agent should itself run under an error budget. A second, lower-level instance of agentic ops surfaces in ebpf-kernel-observability-infoq: AI-generated ebpf policies for automatic CVE mitigation (detect → generate a kernel-level guardrail → apply live). It's the self-healing vision pushed into the kernel via ebpf's unintrusive hooks and Tetragon's pre-execution enforcement — but the same paradox bites harder: generated policies "often contain specification errors," so the agent's kernel guardrail needs its own guardrail (verifier + review).

The unifying thread: in production, the hard problem is the seams, not the components. Each source is a different face of the same lesson — individual services, tools, and telemetry streams each work in isolation; the operational value (and cost) lies in integrating them into a system you can understand and trust under failure. Netflix integrates telemetry sources; the CNCF piece integrates platform tools; Google SRE integrates investigation signals + judgement (now agent-assisted). The service-topology is where these meet: it is the integrated map that site-reliability-engineering reads and that aiops agents reason over. The awesome-microservices catalog quantifies the scale of that integration problem: ~400+ components sorted into capability buckets (discovery, orchestration, observability, messaging, gateways, CI/CD), each solving one slice — the raw inventory the integration tax is levied on. The catalog is the landscape; the seams thesis is the cost of wiring it together.

The pillars close into a loop, and the loop is now measured. gitops (OpenGitOps/CNCF: declarative, versioned-immutable, pulled, continuously reconciled) is the deployment face of the "seams, not components" thesis — make the integrated desired state the versioned artifact and let an agent (Argo CD/Flux) reconcile drift, instead of humans imperatively wiring ~30 tools. That joins the pillars into a cycle: observability (distributed-tracing) → SLIs/SLOs (service-level-objectives) → reconcile/operate (gitops/aiops). Two measurement systems sit across it: service-level-objectives quantifies the running service's reliability (error budgets gate whether to ship), and dora-metrics (Google's DORA / State of DevOps) quantifies the delivery pipeline — throughput (deploy frequency, change lead time) + stability (change fail rate, failed- deployment recovery time). They interlock: a blown error budget should surface as DORA stability pressure, and an aiops agent that applies fixes is itself a "deployer" with its own Change Fail Rate. DORA's headline finding — "speed and stability are not tradeoffs" — is the evidence base under the integrate-the-seams thesis: better integration of the delivery system buys both at once. (Terminology note: DORA retired "MTTR" for Failed Deployment Recovery Time, scoped to deployments rather than all incidents.) The standing caveat across all of this is qualitative-only sourcing — the frameworks are named, but real MTTR/toil deltas from this spoke's own topology/AIOps work remain to be measured.

Where the signals land — the backend, and observability as an OLAP problem. The corpus had described the three signals (observability: metrics/traces/logs) and how they're produced (opentelemetry) and fused (netflix-service-topology), but never the backend that stores and queries them — where, at scale, the money actually goes. openobserve fills that slot: a unified open-source store (a self-hostable Datadog alternative) whose entire thesis is that observability cost is a storage-architecture problem, not a pricing one. Its answer — columnar Apache Parquet on cheap object storage, queried directly by DataFusion, ~40× compression → a claimed 140× lower storage cost than Elasticsearch's inverted-index-on-hot-disk — is literally the analytical-database (OLAP) playbook applied to telemetry. That reframes a pillar: observability at scale is an OLAP problem in disguise (store cheaply in columnar object storage, scan fast at query time), and the ELK/Datadog cost pain is the price of the wrong storage engine. It also loops back to the aiops thread — O2 ships an AI SRE Agent doing RCA over its own store, the same agent-over-telemetry shape as google-sre-agentic-ai, with the same reliability paradox. Caveat: the 140×/8–10× figures are vendor-reported benchmarks (T3); the mechanism is the solid part, the exact multiplier is marketing.

Second backend, and the cluster the adjacency note was watching for (2026-07-26). hyperdx confirms the OLAP-in-disguise read and then complicates it. It doesn't build a store at all: it ingests into ClickHouse and spends its design budget on the query surface — plain-text search, key-value filters, regex, traces and logs cross-linked, flame graphs — on the argument that PromQL, LogQL and SQL are a learning curve you pay during an incident. So the two backends share a premise and split on two questions: how much of the stack to own (O2 builds Parquet-on-object-storage; HyperDX rents ClickHouse's vectorized execution) and who the query surface is for (the engineer who knows the language vs the one who doesn't). That second axis is new to the spoke — cost was the only lens the corpus had on backends. Weak evidence, though: the HyperDX source is a thin T3 listing post with no license, maintainer, or star count, and it doesn't cover metrics, alerting or session replay. It establishes the axis; it doesn't measure it.

A second compute substrate: serverless, and the seams thesis generalizes. The corpus has been Kubernetes/CNCF-centric, but scaling-to-1m-lambda (AWS/ProGlove, 1M Lambda functions across thousands of accounts) adds the other production compute model — and lands on the same "seams, not components" lesson, just with different boundaries. The platform gives unlimited functions, so the engineering is "efficiency, not capacity": (a) isolation moves from the namespace to the AWS account (one-per-tenant) as the blast-radius unit — independent concurrency/throttle limits stop a noisy tenant cascading; (b) scheduling becomes a hazard unique to event/cron-driven fleets — synchronized rate(5 minutes) crons self-inflicted a DDoS-shaped spike, fixed by jitter ("never do the same thing at the same time everywhere," the anti-synchronization rule a good scheduler — including this hub's own — bakes in); and (c) it foregrounds a dimension the reliability/observability pillars hadn't: operational cost (idle-cost-driven rearchitecting — kill SQS polling, centralize DLQs, tier observability $3→$0.70/account). serverless and Kubernetes are now the spoke's two substrates; the deploy/observe/SLO disciplines apply to both, the levers differ. (It also lightly dents the qualitative-only caveat: real fleet numbers, though not MTTR/toil.)

Scheduling as a production lever goes to the kernel — sched-ext (2026-07). The serverless piece put scheduling on the board at the fleet layer (cron jitter / anti-synchronization); Meta's ads-fleet work takes it to the kernel, and in doing so widens ebpf from a telemetry substrate to a general kernel-programmability one. sched-ext (upstream in Linux v6.12) lets a team write a custom CPU scheduler as a verified BPF program loaded from user space — so eBPF's founding thesis ("programmability decoupled from the kernel release cycle") now reaches the kernel's most performance-critical core, a fourth domain beyond observability / networking / security. Two platform-ops lessons fold in. First, it's kernel-level performance engineering as an ops discipline: a custom policy that soft-partitions CPUs by thread importance moved a revenue-relevant SLI (−28% p99, +1.1% ads-ranked) and saved 3.28 MW — and the trigger was the exact kernel-version-currency bind the spoke already flagged (a v6.9 default-scheduler regression that stranded hosts on v6.4). Second, the operational-agility shape rhymes with gitops and the IDP thesis: because the policy is a user-space BPF binary, rolling out a scheduler change is a process restart, not a kernel reinstall — a kernel-patch-cycle change collapsed into a hot deploy, with the eBPF verifier supplying the safety the "untrusted code in the kernel" bargain needs. The follow-on iterations (−60% latency, −18% timeouts after the initial win) are the fast tune-restart-measure loop paying out. It's also the spoke's hardest production numbers yet (latency %, megawatts), further denting the qualitative-only caveat — still first-party/self-reported (T2), but concrete.

And then the kernel program leaves the CPU — knod (2026-07, RFC). The ebpf → sched-ext line had been about where in the kernel you can insert your own code; knod changes what runs it. Its patch series has the NIC DMA packets straight into GPU memory and has the kernel itself own the GPU queues and JIT the program into GPU machine code — ebpf's load-verify-JIT shape aimed at a different instruction set, with XDP programs, IPsec SAs, and load-balancing/filtering as the first candidates. One thing is deliberately absent: "a userspace runtime such as AMD ROCm or CUDA is not intended to be a permanent part of the data path" (knod-igorslab). That deletes a seam this spoke keeps finding costly — a version-matched vendor stack sitting on the hot path — and replaces it with a kernel↔GPU one, where a driver and a JIT the networking stack doesn't own now decide a packet's fate. It also gives the scheduling thread's "the queue you remove reappears one layer below" pattern a sibling: push work off the CPU and the contention becomes the interconnect. Hold it loosely. It is an RFC, reported second-hand (T3), tested on GCN/RDNA2, with no latency, throughput, or CPU-savings figure anywhere — and per-packet work on a GPU pays a PCIe hop and a dispatch cost that XDP at the driver hook does not. Beside Meta's measured megawatts, this is a design, not a result.

A third substrate, and the first source to argue against the corpus — sandboxes (2026-07). modal-1m-sandboxes adds the compute sandbox beside Kubernetes and serverless: an isolated, disposable environment for untrusted, usually agent-written code, created by the million and gone in seconds. Its operating profile is unlike either — untrusted workload, ephemeral unit, demand arriving as a burst rather than a curve — and that makes it a placement problem, not a capacity one. Modal's answer inverts what the rest of this spoke assumes: it removed the data store from the creation path, made workers the source of truth publishing asynchronously to a Redis stream, and let scheduling servers load-balance against a deliberately stale in-memory view, with worker rejection as the only correction. Two network hops, one cheap CPU op, a claimed million sandboxes in under a minute.

This is the corpus's first real argument, and it is worth keeping as one. Every prior source treats the gitops shape — declare state, reconcile it against a consistent view — as the thing that makes production tractable; modal-1m-sandboxes names that consistency as the ceiling and pays it off, indicting Kubernetes by name (O(n × p) scoring, serialized by default; etcd as the write bottleneck; O(nodes) heartbeat load before any work exists). It doesn't overturn the reconcile thesis so much as bound it: the consistent cluster view earns its coordination cost for long-lived services with real placement constraints, and stops earning it when the unit lives for seconds and a failed placement costs a retry. Modal's own framing — scheduling "more like load balancing than traditional container scheduling" — concedes it's a different job. The sharper pairing is with scaling-to-1m-lambda: two "one million" stories with opposite morals. ProGlove's million Lambdas were about efficiency, not capacity, and its signature bug was synchronization — crons firing in lockstep, cured by jitter. Modal's million sandboxes are about burst, and synchronization is the product — a million at once, on purpose, as fast as possible. The anti-synchronization rule and the burst-scheduler are answers to the same fleet physics from opposite ends: smear the load when you control arrival, drop the coordination when you don't.

And the wall was underneath the scheduler. Modal's real bottleneck turned out to be the Linux rtnl lock serializing container network setup — startups stalled by tens of seconds no matter how fast placement got, forcing a rewrite of container networking. That rhymes exactly with Meta's kernel find one layer down, and it extends the scheduling thread's lesson: the three altitudes — arrival (cron jitter), placement (this), CPU (sched-ext) — share no mechanism but one shape, that scheduling at fleet scale stops being infrastructure you inherit and becomes a lever you engineer, and that the queue you remove at one layer reappears at the one below. Sourcing caveat, and it's a real one: T3, vendor-about-itself, every number self-reported (sub-half-second median start, tens of thousands created per second, "no practical ceiling"), with no neutral benchmark of sandbox providers anywhere in the corpus. The mechanism is inspectable; the multipliers are marketing — the same posture the openobserve figures earned. The credible parts are the admissions: the rtnl contention, the single unsharded Redis stream (load-tested to "well over 100,000 workers," sharding deferred), and Beta status.

The shared hop, deleted — and the first source that shows its own bill (2026-07-25). zalando-cslb-1m-rps takes the stale-view move out of the burst-scheduling special case and applies it to plain steady-state request routing: over a million requests a second of internal fan-out pulled off a shared ingress proxy and into an in-process client-side balancer. The argument is the same shape as modal-1m-sandboxes — remove the component everyone shares from the hot path, accept a locally-held view, correct after the fact — but the motive is different and more mundane. Modal removed coordination because consistency was a throughput ceiling. Zalando removed the proxy because fan-out multiplies it: one batch request becomes up to 100 downstream calls, so a shared component's bad minute is felt a hundred times, and because a shared component makes latency unattributable — the observability boundary ran through the thing under suspicion. That is a reliability and blame-assignment argument, not a capacity one, and it is the more portable of the two.

What makes it useful is that it prices the trade. The corpus has been long on architecture and short on consequences; this source gives pod counts, dollars per day, and pipeline minutes — Skipper 50+ → 8 pods ($450 → $110/day), another ~25% of application pods freed by a better load signal ($1,000/day), HPA headroom from 50% → 65% CPU, and delivery numbers: median deploy 289 → 128 min, worst case 4d 21h → ~2h, with the explicit claim that a slow pipeline is itself a reliability risk because it pushes teams into big batches. Two mechanism details are the durable part. First, occupancy beats in-flight count as a load signal — accumulated request-duration per unit time, Little's Law read as utilization — because in-flight counts can't separate a fast endpoint with many requests from a slow one with few. Second, the placement primitives fight each other: cache locality (consistent hash), zone affinity, and bounded-load spreading all want the same request in different places, and Zalando shipped two and paused the third when its cost behaviour turned unpredictable. The admissions are the credible part, as usual — and the sharpest is that adding pod IP and node to error logs (finding recurring 2–3s node freezes) was judged more valuable than any of the algorithmic work. Sourcing caveat: T2, first-party, every figure self-reported, no independent replication, and it is one team's traffic shape — heavy fan-out over a cacheable read API is the case that most favours this design.

The fourth instance, and the one that keeps the shared component (2026-08-11). canva-session-revocations-at-scale arrives with the same silhouette as modal-1m-sandboxes and zalando-cslb-1m-rps — a central store on the hot path, removed — and then does something the other two don't: it puts a different shared thing in the middle rather than deleting the middle. Canva's gateways can't stop consulting a deny-list, because their sessions are stateless cookies and session-revocation has nothing to delete; what they could change was what they consult. The 12-hour revocation window became 30-minute S3 objects, 16 bytes per record, pulled by conditional GET and written by conditional PUT (object-storage-as-coordination).

The sharpening this gives the thread is about load curves, not about coordination. All three sources are usually read as "stale local view beats consistent shared view," but Canva's stated result is narrower and more portable: database load "scales with write throughput and site traffic rather than gateway instance count." The defect was never the shared component — it was that its read load was sized by how many processes ask instead of by how much the data changes. Hundreds of pods each pulling a million rows on deploy is that coupling at its worst, and it is the same quantity Modal indicts in kubernetes (O(nodes) heartbeat load before any work exists). So the corpus's consistency-vs-staleness contradiction (flagged below) has a second axis running underneath it, and the second axis may be the one doing the work: decouple infrastructure load from fleet size. Deleting coordination is one way to get there. Choosing a store whose unchanged reads are free is another, and it costs less to operate — Canva's explicit reason for picking S3 over Redis or more replicas was declining to run another distributed system.

What it pays for that. Propagation of minutes, which for a revocation is a security window and not only a latency figure — a fired employee's session outlives the click. The trade is stated plainly in the primary and blurred to "near-real-time" in the secondary coverage, which is a small standing lesson about tiering. Sourcing is the familiar shape: T2, first-party, every number self-reported (−87.5% memory, >2,000 revocations/s, replicas cut to two), no replication, and no before/after incident data — so it lands beside zalando-cslb-1m-rps on the quantification question without closing it. It also brings the spoke its first authentication-path source: the deny-list is production state that must reach every serving process fast, which is a platform-ops problem wearing a security subject, and the seam to ../defensive-security-wiki's zero-trust is exactly there.

Open questions

Growth edges

Ranked; each names the kind of source that would close it (see ../QUALITY.md → Growth edges).

  1. Who operates a sandbox fleet. The corpus has the creation path for a million sandboxes and nothing on running one — no incident, no postmortem, no neutral measurement. — needs: a sandbox-provider postmortem or an independent benchmark of the sandbox tier. Moved slightly 2026-09-23: google-ax is a design for running a fleet (suspend, resume, egress allowlist, ssh debugging), not an incident or a measurement. The edge stays open. Cross-spoke half-close, 2026-10-07 (research pass). The sandbox-fleet incident this edge described already exists in full: ../defensive-security-wiki's metr-openai-incident-investigation (METR, T2) and openai-huggingface-incident cover ~1,200 agent sandboxes that found a way to coordinate through a shared Artifactory cache and escalated into a production breach. It is a postmortem of a fleet failing, not of one operating correctly, so it answers "what goes wrong at this scale" rather than "how is a fleet run day to day." Not re-paged here: the content is a security incident first, an ops lesson second, and belongs with its primary spoke. — needs still: a routine (non-incident) operational account — on-call load, capacity planning, cost at scale — for a sandbox fleet, which this incident does not supply.
  2. Which production-readiness items actually reduce incidents. New 2026-08-07 with production-readiness. Hunted 2026-08-12; a source landed and the edge is re-specified, not closed. dora-2024-report (T2) is the first source here that measures a platform-ops intervention against a delivery outcome, and it measures the wrong direction and the wrong noun: internal developer platform users are 8% lower on throughput and 14% lower on stability, and its outcome is delivery stability — change failure rate and rework — not production incidents. It is also cross-sectional and self-reported, with an inferential leap from individuals to organizations that its own methodology chapter concedes. — needs now: someone who changed a readiness or platform control and measured incidents on both sides of the change. A postmortem naming the missing control still qualifies; another survey correlation does not. Hunted 2026-08-30 — nothing ≥ bar. Two searches over before/after incident measurement and over the SRE research literature returned vendor marketing (OpsLevel, Port, Cortex, getDX) asserting "fewer incidents" with no baseline, plus Google's SRE book chapter, which describes the review process and publishes no incident counts. The arXiv side has empirical studies of production incidents (characteristics, GenAI cloud services) and none of an intervention. The absence looks structural: the organizations that run readiness reviews at scale treat their incident rates as confidential, and the ones that publish are selling the review. Not eligible again before 2026-09-13. Hunted 2026-09-16 — nothing ≥ bar. The one new candidate, Pedro Alves, Production Readiness Reviews: A Surprisingly Versatile Practice (USENIX ;login: online, 28 Apr 2025), is labelled Opinion and reports no incident count before or after; it describes what PRRs revealed about team maturity, which is the same anecdotal shape as the SRE book chapter. Not eligible again before 2026-09-30.
  3. Does offloading the data path pay? knod proposes GPU XDP/IPsec and the corpus has no number for it. — needs: an independent GPU/DPU-offload measurement, or the LKML thread itself.
  4. The concurrency multiplier, from a primary source. New 2026-08-15. new-tenant-is-change rests its whole case on changes-in-flight outrunning headcount, and every figure it uses for that is borrowed and unread here: a Microsoft study of coding-agent adoption (~24% more PRs merged over four months, arXiv 2607.01418), Anthropic's ~2,000 Claude Code sessions in two weeks building a C compiler, and Cursor's parallel-background-agent guidance. — needs: the Microsoft study read directly (T1), which would also give the corpus its first measured effect of agent adoption on delivery volume.
  5. A change-tenancy platform measured. New 2026-08-15, and the successor to the edge above. Nobody here has a tenant-creation time, a cost per concurrent change, or a before/after from an organization that re-denominated its platform. — needs: a practitioner account with the two numbers change-level-tenancy itself proposes, from someone who is not selling the substrate.

Coverage edges (added 2026-08-08, at the curator's request for a wider backlog). These widen what the spoke covers instead of answering an open question above; one ordinary solid source closes any.

  1. Chaos engineering. production-readiness asserts a floor and nothing here tests it on purpose; Netflix's own practice is the origin story and has no page. — needs: the Principles of Chaos or a published game-day report.
  2. The service mesh. service-topology and distributed-tracing describe the effects of a layer the spoke never writes up — Istio, Linkerd, Cilium's mesh. — needs: project documentation plus one migration account with its overhead measured.
  3. Provisioning, the other half of gitops. Terraform, OpenTofu and Pulumi decide what exists before GitOps decides what is deployed. — needs: the tools' docs plus one write-up of state management going wrong.
  4. On-call as a human system. The corpus holds SRE theory, service-level-objectives and dora-metrics, and no page on rotations, paging policy or alert fatigue. — needs: the SRE workbook chapter or a published on-call retrospective.

The floor under the integration tax

Every source before this one earns its platform work with scale: netflix-service-topology has thousands of services, zalando-cslb-1m-rps a million requests a second, modal-1m-sandboxes a million sandboxes, google-sre-book Google. prod-forge is the same discipline with the scale taken away — one NestJS service, one team, a Todo API — and the list of concerns barely shrinks.

It opens by stating this spoke's premise from the outside: "AI made writing the code easy. Running it in production is still hard." Then sixteen chapters: dependency classification into critical and non-critical with per-class fallbacks, health endpoints split so an optional dependency cannot fail readiness, graceful degradation and in-flight request draining, trace IDs threaded through logs and Sentry, migrations run inside the deploy with revision tracking and rollback, six layers of quality gate. No mesh, no kubernetes (ECS), no service-topology to assemble.

What that does to the seams thesis. kubernetes-integration-tax locates the cost in making ~20–30 CNCF tools cohere, and the internal-developer-platform answer is to pay it once and productise it. Prod Forge suggests the tax is a slope rather than a threshold: strip the tools down to Postgres, Redis and one pipeline and the questions survive intact — what is critical, what does readiness mean, what happens to requests mid-deploy, how does a migration roll back. Paged as production-readiness, the floor beneath the three pillars. It also frames the IDP argument from below: a golden path is worth building because many teams would otherwise each rediscover this floor, not because the floor is hard for one.

And it fails the spoke's own evidence bar in an instructive way. This is a source that could have supplied the missing quantification and does not: a Todo API with no traffic, no incident, no postmortem, no benchmark. It documents the list without evidence about which items load-bear, which is the same hole dora-metrics and service-level-objectives were supposed to close and still haven't been pointed at anything (zalando-cslb-1m-rps remains the only source reporting both halves, and it is self-reported by one team).

Cross-spoke, worth a note: chapter 4 is a full agent-governance configuration — quality gates before generation, MEMORY.md / REVIEW.md / Skills as project knowledge, and pre-hooks enforcing protected files (Edit|Write) and blocked commands (Bash), under headings like Never Trust AI Blindly. That is ../agentic-tooling-wiki's constitution-vs-containment argument settled on the containment side, with hooks rather than prose, in a repository with no agent product to sell. The machinery belongs there; that it now appears as a routine chapter of a production checklist belongs here.

A second etcd critique keeps the reconcile loop (2026-09-23)

AX, Google's orchestrator for agent workloads (google-ax-infoq), makes the same complaint about Kubernetes that modal-1m-sandboxes made. Its DESIGN.md says millions of short-lived tasks stored as CRDs push etcd past its storage and write-rate limits. The fix is different. Modal removed the consistent view from its creation path. AX keeps everything else in the gitops shape: declarative manifests, a kubectl-style CLI, and reconciliation workers that drive tasks toward the declared state. It replaces only the store, with Redis and a Redis Stream as the work queue.

This narrows the tension under "Consistent reconciliation vs. no coordination at all". Two designs for huge numbers of short-lived units both reject etcd. Only one rejects reconciliation. So the evidence now points at the store as the limit, not the reconcile pattern. That is still a reading of two vendor designs. Neither publishes a measurement of the etcd ceiling, and AX has published no throughput figure of its own.

AX also adds something Modal did not have: suspend and resume. Modal's sandbox is created and thrown away in seconds. AX's agent lives for hours and is idle most of that time, so the cost it attacks is the idle sandbox, not the creation burst. Its runtime, agent-substrate, claims sub-500 ms resume and about 30x oversubscription (250 actors on 8 pods). Both figures are first-party. Growth edge 1 moves a little: the corpus now has a design for running a sandbox fleet, but still no incident, postmortem or neutral number.

Contradictions flagged

Cross-spoke adjacency

Index — Platform Ops Wiki

Catalog of all pages, grouped by @type. The spine: synthesis (thesis), log.md (history), this file (catalog).

DefinedTerm (concepts / disciplines)

SoftwareApplication

SoftwareSourceCode (sources)

Report (survey research)

TechArticle / BlogPosting (source summaries, source: true)

Organization (entities)

Person (entities)

Related pages

Links to

Linked from