platform-ops-wiki

Synthesis — Platform Ops

The evolving thesis of this wiki. Sits above the schema.org pages. Records the current best understanding, open questions, and explicitly flagged contradictions.

Current understanding

This spoke was spun out (2026-06-05) from three sources that arrived separately yet converge on one operational discipline: platform-ops — running cloud-native distributed systems in production. They map cleanly onto three pillars:

observability / service-topology. netflix-service-topology builds a live dependency graph of thousands of microservices by fusing three telemetry sources — eBPF flow logs, IPC metrics, distributed traces — because each has a blind spot (no app context / misses uninstrumented services / sampled). The merged graph beats any single source. distributed-tracing (OpenTelemetry) is the per-request view of the same dependency structure the topology shows in aggregate (a topology ≈ traces summed over time): the literal record of a request crossing service seams, the third signal beside metrics and logs, and the source of the latency SLIs that SLOs are defined on.
platform-engineering. kubernetes-integration-tax argues the dominant cost of production kubernetes is the “integration tax” — making ~20–30 CNCF tools (Prometheus, Cilium, cert-manager, GitOps) work together. The failures are seams (Cilium metrics invisible to Prometheus without a ServiceMonitor), not the tools themselves. The cure is the internal-developer-platform (project-as-a-service, Belastingdienst): pay the integration cost once, centrally, then expose it as a self-service, golden-path product (“make the right way the easiest way”) so teams don’t each re-pay the tax. Mechanically it’s a gitops-shaped reconcile loop applied to project provisioning (one YAML → namespaces/RBAC/quota via an operator), so the “seams, not components” thesis gains its scaling resolution: productize the seams. Half of that work is social (enablement-over-support, Communities of Practice across 99+ teams) — platform-as-a- product is an org pattern, not only tooling. Open tension: a golden path can harden into a golden cage if defaults lag team needs.
site-reliability-engineering + aiops. google-sre-agentic-ai applies agentic AI across the SRE workflow (runbooks, anomaly detection, postmortems, incident investigation over “observability + topology”) under strict mandates: SLOs + fallbacks, identity/permissions, explainability, auditability. The quantified backbone is service-level-objectives (Google SRE): the SLI → SLO → error-budget chain turns “reliability vs. velocity” into a measured control loop (spend the budget → stop shipping → fix reliability), so toil/MTTR/impact become budget math, not anecdote. It also reframes the AIOps reliability paradox — an ops agent should itself run under an error budget. A second, lower-level instance of agentic ops surfaces in ebpf-kernel-observability-infoq: AI-generated ebpf policies for automatic CVE mitigation (detect → generate a kernel-level guardrail → apply live). It’s the self-healing vision pushed into the kernel via ebpf‘s unintrusive hooks and Tetragon’s pre-execution enforcement — but the same paradox bites harder: generated policies “often contain specification errors,” so the agent’s kernel guardrail needs its own guardrail (verifier + review).

The unifying thread: in production, the hard problem is the seams, not the components. Each source is a different face of the same lesson — individual services, tools, and telemetry streams each work in isolation; the operational value (and cost) lies in integrating them into a system you can understand and trust under failure. Netflix integrates telemetry sources; the CNCF piece integrates platform tools; Google SRE integrates investigation signals + judgement (now agent-assisted). The service-topology is where these meet: it is the integrated map that site-reliability-engineering reads and that aiops agents reason over. The awesome-microservices catalog quantifies the scale of that integration problem: ~400+ components sorted into capability buckets (discovery, orchestration, observability, messaging, gateways, CI/CD), each solving one slice — the raw inventory the integration tax is levied on. The catalog is the landscape; the seams thesis is the cost of wiring it together.

The pillars close into a loop, and the loop is now measured. gitops (OpenGitOps/CNCF: declarative, versioned-immutable, pulled, continuously reconciled) is the deployment face of the “seams, not components” thesis — make the integrated desired state the versioned artifact and let an agent (Argo CD/Flux) reconcile drift, instead of humans imperatively wiring ~30 tools. That joins the pillars into a cycle: observability (distributed-tracing) → SLIs/SLOs (service-level-objectives) → reconcile/operate (gitops/aiops). Two measurement systems sit across it: service-level-objectives quantifies the running service’s reliability (error budgets gate whether to ship), and dora-metrics (Google’s DORA / State of DevOps) quantifies the delivery pipeline — throughput (deploy frequency, change lead time) + stability (change fail rate, failed- deployment recovery time). They interlock: a blown error budget should surface as DORA stability pressure, and an aiops agent that applies fixes is itself a “deployer” with its own Change Fail Rate. DORA’s headline finding — “speed and stability are not tradeoffs” — is the evidence base under the integrate-the-seams thesis: better integration of the delivery system buys both at once. (Terminology note: DORA retired “MTTR” for Failed Deployment Recovery Time, scoped to deployments rather than all incidents.) The standing caveat across all of this is qualitative-only sourcing — the frameworks are named, but real MTTR/toil deltas from this spoke’s own topology/AIOps work remain to be measured.

Where the signals land — the backend, and observability as an OLAP problem. The corpus had described the three signals (observability: metrics/traces/logs) and how they’re produced (opentelemetry) and fused (netflix-service-topology), but never the backend that stores and queries them — where, at scale, the money actually goes. openobserve fills that slot: a unified open-source store (a self-hostable Datadog alternative) whose entire thesis is that observability cost is a storage-architecture problem, not a pricing one. Its answer — columnar Apache Parquet on cheap object storage, queried directly by DataFusion, ~40× compression → a claimed 140× lower storage cost than Elasticsearch’s inverted-index-on-hot-disk — is literally the analytical-database (OLAP) playbook applied to telemetry. That reframes a pillar: observability at scale is an OLAP problem in disguise (store cheaply in columnar object storage, scan fast at query time), and the ELK/Datadog cost pain is the price of the wrong storage engine. It also loops back to the aiops thread — O2 ships an AI SRE Agent doing RCA over its own store, the same agent-over-telemetry shape as google-sre-agentic-ai, with the same reliability paradox. Caveat: the 140×/8–10× figures are vendor-reported benchmarks (T3); the mechanism is the solid part, the exact multiplier is marketing.

Second backend, and the cluster the adjacency note was watching for (2026-07-26). hyperdx confirms the OLAP-in-disguise read and then complicates it. It doesn’t build a store at all: it ingests into ClickHouse and spends its design budget on the query surface — plain-text search, key-value filters, regex, traces and logs cross-linked, flame graphs — on the argument that PromQL, LogQL and SQL are a learning curve you pay during an incident. So the two backends share a premise and split on two questions: how much of the stack to own (O2 builds Parquet-on-object-storage; HyperDX rents ClickHouse’s vectorized execution) and who the query surface is for (the engineer who knows the language vs the one who doesn’t). That second axis is new to the spoke — cost was the only lens the corpus had on backends. Weak evidence, though: the HyperDX source is a thin T3 listing post with no license, maintainer, or star count, and it doesn’t cover metrics, alerting or session replay. It establishes the axis; it doesn’t measure it.

A second compute substrate: serverless, and the seams thesis generalizes. The corpus has been Kubernetes/CNCF-centric, but scaling-to-1m-lambda (AWS/ProGlove, 1M Lambda functions across thousands of accounts) adds the other production compute model — and lands on the same “seams, not components” lesson, just with different boundaries. The platform gives unlimited functions, so the engineering is “efficiency, not capacity”: (a) isolation moves from the namespace to the AWS account (one-per-tenant) as the blast-radius unit — independent concurrency/throttle limits stop a noisy tenant cascading; (b) scheduling becomes a hazard unique to event/cron-driven fleets — synchronized rate(5 minutes) crons self-inflicted a DDoS-shaped spike, fixed by jitter (“never do the same thing at the same time everywhere,” the anti-synchronization rule a good scheduler — including this hub’s own — bakes in); and (c) it foregrounds a dimension the reliability/observability pillars hadn’t: operational cost (idle-cost-driven rearchitecting — kill SQS polling, centralize DLQs, tier observability $3→$0.70/account). serverless and Kubernetes are now the spoke’s two substrates; the deploy/observe/SLO disciplines apply to both, the levers differ. (It also lightly dents the qualitative-only caveat: real fleet numbers, though not MTTR/toil.)

Scheduling as a production lever goes to the kernel — sched-ext (2026-07). The serverless piece put scheduling on the board at the fleet layer (cron jitter / anti-synchronization); Meta’s ads-fleet work takes it to the kernel, and in doing so widens ebpf from a telemetry substrate to a general kernel-programmability one. sched-ext (upstream in Linux v6.12) lets a team write a custom CPU scheduler as a verified BPF program loaded from user space — so eBPF’s founding thesis (“programmability decoupled from the kernel release cycle”) now reaches the kernel’s most performance-critical core, a fourth domain beyond observability / networking / security. Two platform-ops lessons fold in. First, it’s kernel-level performance engineering as an ops discipline: a custom policy that soft-partitions CPUs by thread importance moved a revenue-relevant SLI (−28% p99, +1.1% ads-ranked) and saved 3.28 MW — and the trigger was the exact kernel-version-currency bind the spoke already flagged (a v6.9 default-scheduler regression that stranded hosts on v6.4). Second, the operational-agility shape rhymes with gitops and the IDP thesis: because the policy is a user-space BPF binary, rolling out a scheduler change is a process restart, not a kernel reinstall — a kernel-patch-cycle change collapsed into a hot deploy, with the eBPF verifier supplying the safety the “untrusted code in the kernel” bargain needs. The follow-on iterations (−60% latency, −18% timeouts after the initial win) are the fast tune-restart-measure loop paying out. It’s also the spoke’s hardest production numbers yet (latency %, megawatts), further denting the qualitative-only caveat — still first-party/self-reported (T2), but concrete.

And then the kernel program leaves the CPU — knod (2026-07, RFC). The ebpf → sched-ext line had been about where in the kernel you can insert your own code; knod changes what runs it. Its patch series has the NIC DMA packets straight into GPU memory and has the kernel itself own the GPU queues and JIT the program into GPU machine code — ebpf‘s load-verify-JIT shape aimed at a different instruction set, with XDP programs, IPsec SAs, and load-balancing/filtering as the first candidates. One thing is deliberately absent: “a userspace runtime such as AMD ROCm or CUDA is not intended to be a permanent part of the data path” (knod-igorslab). That deletes a seam this spoke keeps finding costly — a version-matched vendor stack sitting on the hot path — and replaces it with a kernel↔GPU one, where a driver and a JIT the networking stack doesn’t own now decide a packet’s fate. It also gives the scheduling thread’s “the queue you remove reappears one layer below” pattern a sibling: push work off the CPU and the contention becomes the interconnect. Hold it loosely. It is an RFC, reported second-hand (T3), tested on GCN/RDNA2, with no latency, throughput, or CPU-savings figure anywhere — and per-packet work on a GPU pays a PCIe hop and a dispatch cost that XDP at the driver hook does not. Beside Meta’s measured megawatts, this is a design, not a result.

A third substrate, and the first source to argue against the corpus — sandboxes (2026-07). modal-1m-sandboxes adds the compute sandbox beside Kubernetes and serverless: an isolated, disposable environment for untrusted, usually agent-written code, created by the million and gone in seconds. Its operating profile is unlike either — untrusted workload, ephemeral unit, demand arriving as a burst rather than a curve — and that makes it a placement problem, not a capacity one. Modal’s answer inverts what the rest of this spoke assumes: it removed the data store from the creation path, made workers the source of truth publishing asynchronously to a Redis stream, and let scheduling servers load-balance against a deliberately stale in-memory view, with worker rejection as the only correction. Two network hops, one cheap CPU op, a claimed million sandboxes in under a minute.

This is the corpus’s first real argument, and it is worth keeping as one. Every prior source treats the gitops shape — declare state, reconcile it against a consistent view — as the thing that makes production tractable; modal-1m-sandboxes names that consistency as the ceiling and pays it off, indicting Kubernetes by name (O(n × p) scoring, serialized by default; etcd as the write bottleneck; O(nodes) heartbeat load before any work exists). It doesn’t overturn the reconcile thesis so much as bound it: the consistent cluster view earns its coordination cost for long-lived services with real placement constraints, and stops earning it when the unit lives for seconds and a failed placement costs a retry. Modal’s own framing — scheduling “more like load balancing than traditional container scheduling” — concedes it’s a different job. The sharper pairing is with scaling-to-1m-lambda: two “one million” stories with opposite morals. ProGlove’s million Lambdas were about efficiency, not capacity, and its signature bug was synchronization — crons firing in lockstep, cured by jitter. Modal’s million sandboxes are about burst, and synchronization is the product — a million at once, on purpose, as fast as possible. The anti-synchronization rule and the burst-scheduler are answers to the same fleet physics from opposite ends: smear the load when you control arrival, drop the coordination when you don’t.

And the wall was underneath the scheduler. Modal’s real bottleneck turned out to be the Linux rtnl lock serializing container network setup — startups stalled by tens of seconds no matter how fast placement got, forcing a rewrite of container networking. That rhymes exactly with Meta’s kernel find one layer down, and it extends the scheduling thread’s lesson: the three altitudes — arrival (cron jitter), placement (this), CPU (sched-ext) — share no mechanism but one shape, that scheduling at fleet scale stops being infrastructure you inherit and becomes a lever you engineer, and that the queue you remove at one layer reappears at the one below. Sourcing caveat, and it’s a real one: T3, vendor-about-itself, every number self-reported (sub-half-second median start, tens of thousands created per second, “no practical ceiling”), with no neutral benchmark of sandbox providers anywhere in the corpus. The mechanism is inspectable; the multipliers are marketing — the same posture the openobserve figures earned. The credible parts are the admissions: the rtnl contention, the single unsharded Redis stream (load-tested to “well over 100,000 workers,” sharding deferred), and Beta status.

The shared hop, deleted — and the first source that shows its own bill (2026-07-25). zalando-cslb-1m-rps takes the stale-view move out of the burst-scheduling special case and applies it to plain steady-state request routing: over a million requests a second of internal fan-out pulled off a shared ingress proxy and into an in-process client-side balancer. The argument is the same shape as modal-1m-sandboxes — remove the component everyone shares from the hot path, accept a locally-held view, correct after the fact — but the motive is different and more mundane. Modal removed coordination because consistency was a throughput ceiling. Zalando removed the proxy because fan-out multiplies it: one batch request becomes up to 100 downstream calls, so a shared component’s bad minute is felt a hundred times, and because a shared component makes latency unattributable — the observability boundary ran through the thing under suspicion. That is a reliability and blame-assignment argument, not a capacity one, and it is the more portable of the two.

What makes it useful is that it prices the trade. The corpus has been long on architecture and short on consequences; this source gives pod counts, dollars per day, and pipeline minutes — Skipper 50+ → 8 pods (~~$450 → $110/day), another ~25% of application pods freed by a better load signal (~~$1,000/day), HPA headroom from 50% → 65% CPU, and delivery numbers: median deploy 289 → 128 min, worst case 4d 21h → ~2h, with the explicit claim that a slow pipeline is itself a reliability risk because it pushes teams into big batches. Two mechanism details are the durable part. First, occupancy beats in-flight count as a load signal — accumulated request-duration per unit time, Little’s Law read as utilization — because in-flight counts can’t separate a fast endpoint with many requests from a slow one with few. Second, the placement primitives fight each other: cache locality (consistent hash), zone affinity, and bounded-load spreading all want the same request in different places, and Zalando shipped two and paused the third when its cost behaviour turned unpredictable. The admissions are the credible part, as usual — and the sharpest is that adding pod IP and node to error logs (finding recurring 2–3s node freezes) was judged more valuable than any of the algorithmic work. Sourcing caveat: T2, first-party, every figure self-reported, no independent replication, and it is one team’s traffic shape — heavy fan-out over a cacheable read API is the case that most favours this design.

The fourth instance, and the one that keeps the shared component (2026-08-11). canva-session-revocations-at-scale arrives with the same silhouette as modal-1m-sandboxes and zalando-cslb-1m-rps — a central store on the hot path, removed — and then does something the other two don’t: it puts a different shared thing in the middle rather than deleting the middle. Canva’s gateways can’t stop consulting a deny-list, because their sessions are stateless cookies and session-revocation has nothing to delete; what they could change was what they consult. The 12-hour revocation window became 30-minute S3 objects, 16 bytes per record, pulled by conditional GET and written by conditional PUT (object-storage-as-coordination).

The sharpening this gives the thread is about load curves, not about coordination. All three sources are usually read as “stale local view beats consistent shared view,” but Canva’s stated result is narrower and more portable: database load “scales with write throughput and site traffic rather than gateway instance count.” The defect was never the shared component — it was that its read load was sized by how many processes ask instead of by how much the data changes. Hundreds of pods each pulling a million rows on deploy is that coupling at its worst, and it is the same quantity Modal indicts in kubernetes (O(nodes) heartbeat load before any work exists). So the corpus’s consistency-vs-staleness contradiction (flagged below) has a second axis running underneath it, and the second axis may be the one doing the work: decouple infrastructure load from fleet size. Deleting coordination is one way to get there. Choosing a store whose unchanged reads are free is another, and it costs less to operate — Canva’s explicit reason for picking S3 over Redis or more replicas was declining to run another distributed system.

What it pays for that. Propagation of minutes, which for a revocation is a security window and not only a latency figure — a fired employee’s session outlives the click. The trade is stated plainly in the primary and blurred to “near-real-time” in the secondary coverage, which is a small standing lesson about tiering. Sourcing is the familiar shape: T2, first-party, every number self-reported (−87.5% memory, >2,000 revocations/s, replicas cut to two), no replication, and no before/after incident data — so it lands beside zalando-cslb-1m-rps on the quantification question without closing it. It also brings the spoke its first authentication-path source: the deny-list is production state that must reach every serving process fast, which is a platform-ops problem wearing a security subject, and the seam to defensive-security-wiki’s zero-trust is exactly there.

Open questions

Build vs. buy the topology. Netflix built a bespoke system on internal infra (Pekko/Kafka/KV store). What does this look like for teams without Netflix-scale platform engineering? Partly answered (2026-06-09): the off-the-shelf stack is opentelemetry (vendor-neutral instrumentation → any backend) + prometheus (metrics/alerting) over ebpf (low-overhead kernel telemetry, the same substrate Netflix leans on). The remaining gap is the topology-graph assembly Netflix built on top — OTel/Prom give the signals, not the merged dependency graph. Also note ebpf‘s operational cost (CAP_BPF, verifier complexity limits, kernel-version dependence) — the second open question, now sourced.
(new 2026-07-17) Who operates a sandbox fleet? The corpus now has the creation path for a million sandboxes (modal-1m-sandboxes) and nothing on running one: what observability over a million second-lived units looks like (a trace per sandbox is absurd; what replaces it?), whether SLOs on a burst substrate mean anything when the unit outlives nothing, and what happens when isolation fails on untrusted agent code. No incident, no postmortem, no neutral measurement — the vendors publish their architectures, not their outages. Natural next sources: a sandbox-provider postmortem, or any independent benchmark of the sandbox tier (cloud-wiki’s parallel search for a “VPSBenchmarks-for-PaaS” came up empty, so temper expectations).
(new 2026-07-22) Does offloading the data path off the CPU actually pay? knod proposes running XDP/IPsec on a GPU and the corpus has no number for it — not from the reporting (knod-igorslab), and not from the DPU/SmartNIC side either (NVIDIA DOCA is parked in the hub _inbox, and is a vendor telling its own story). The question is whether per-packet work amortizes a PCIe hop and a dispatch cost well enough to beat a CPU running XDP at the driver hook, and at what packet rate the crossover sits. Natural next sources: the LKML thread itself (maintainer pushback is the real signal at RFC stage), or any independent GPU/DPU-offload measurement.
(new 2026-08-07) Which parts of the production-readiness floor actually reduce incidents? prod-forge writes the list down completely and demonstrates it on a Todo API with no incident history (production-readiness). The corpus holds two measurement frameworks and has never applied either to a readiness checklist. Natural next sources: a postmortem naming which control was missing, or anyone reporting change-fail-rate before and after adopting one of these practices.
(new 2026-08-11) How long may a revoked session live? canva-session-revocations-at-scale trades instant session-revocation for minutes of propagation delay and does not say how that number was chosen, whether any compliance regime constrains it, or what the fleet does for the revocations that can’t wait (a compromised admin, a leaked cookie). The corpus has no source treating revocation latency as a security budget the way error budgets treat reliability. Natural next sources: an incident where the delay mattered, a break-glass/forced-refresh design, or the defensive-security-wiki zero-trust literature on continuous verification intervals.
AIOps reliability paradox. Agents drafting postmortems and investigating incidents must themselves be reliable. How is agent error handled in the loop, and does it interact with the “AI doesn’t deliver reliable production software” critique parked in the hub _inbox (ai-productionization cluster)? Open.
eBPF’s operational cost. netflix-service-topology leans on eBPF for coverage; the overhead, security, and kernel-version constraints of eBPF at scale aren’t covered here. Natural next source.
Quantification. Like most founding sets, claims are qualitative (the “integration tax,” the three-source merge). Hard numbers on toil reduction or incident MTTR would strengthen the thesis. Now framed (2026-06-10 + 2026-06-12): two measurement systems — service-level-objectives (reliability of the running service) + dora-metrics (delivery throughput & stability). Remaining: applying them to this spoke’s own claims (real MTTR/toil deltas from the topology/AIOps work), not just naming the frameworks. Best instance so far (2026-07-25): zalando-cslb-1m-rps reports both halves at once — reliability (no LB incidents post-migration, flattened latency spikes) and DORA-shaped delivery deltas (deploy 289 → 128 min) — from one change, plus the dollar cost. Still self-reported, still one team.
(new 2026-07-25) When is owning the load balancer worth it? zalando-cslb-1m-rps makes the case at 1M req/s with ~100× fan-out over a cacheable read API — the shape that most favours CSLB. The corpus has nothing on where the line sits: at what fan-out or request rate an in-process balancer stops paying for its staleness, RBAC, on-call, and per-caller CPU, and nothing at all comparing it against the service-mesh sidecar answer (Envoy/xDS) that Zalando skipped without benchmarking. Natural next sources: a mesh-side account of the same trade, or anyone reporting a CSLB migration that didn’t pay off.

Growth edges

Ranked; each names the kind of source that would close it (see ../QUALITY.md → Growth edges).

Who operates a sandbox fleet. The corpus has the creation path for a million sandboxes and nothing on running one — no incident, no postmortem, no neutral measurement. — needs: a sandbox-provider postmortem or an independent benchmark of the sandbox tier.
Which production-readiness items actually reduce incidents. New 2026-08-07 with production-readiness. Hunted 2026-08-12; a source landed and the edge is re-specified, not closed. dora-2024-report (T2) is the first source here that measures a platform-ops intervention against a delivery outcome, and it measures the wrong direction and the wrong noun: internal developer platform users are 8% lower on throughput and 14% lower on stability, and its outcome is delivery stability — change failure rate and rework — not production incidents. It is also cross-sectional and self-reported, with an inferential leap from individuals to organizations that its own methodology chapter concedes. — needs now: someone who changed a readiness or platform control and measured incidents on both sides of the change. A postmortem naming the missing control still qualifies; another survey correlation does not.
Does offloading the data path pay? knod proposes GPU XDP/IPsec and the corpus has no number for it. — needs: an independent GPU/DPU-offload measurement, or the LKML thread itself.

Coverage edges (added 2026-08-08, at the curator’s request for a wider backlog). These widen what the spoke covers instead of answering an open question above; one ordinary solid source closes any.

Chaos engineering. production-readiness asserts a floor and nothing here tests it on purpose; Netflix’s own practice is the origin story and has no page. — needs: the Principles of Chaos or a published game-day report.
The service mesh. service-topology and distributed-tracing describe the effects of a layer the spoke never writes up — Istio, Linkerd, Cilium’s mesh. — needs: project documentation plus one migration account with its overhead measured.
Provisioning, the other half of gitops. Terraform, OpenTofu and Pulumi decide what exists before GitOps decides what is deployed. — needs: the tools’ docs plus one write-up of state management going wrong.
On-call as a human system. The corpus holds SRE theory, service-level-objectives and dora-metrics, and no page on rotations, paging policy or alert fatigue. — needs: the SRE workbook chapter or a published on-call retrospective.

The floor under the integration tax

Every source before this one earns its platform work with scale: netflix-service-topology has thousands of services, zalando-cslb-1m-rps a million requests a second, modal-1m-sandboxes a million sandboxes, google-sre-book Google. prod-forge is the same discipline with the scale taken away — one NestJS service, one team, a Todo API — and the list of concerns barely shrinks.

It opens by stating this spoke’s premise from the outside: “AI made writing the code easy. Running it in production is still hard.” Then sixteen chapters: dependency classification into critical and non-critical with per-class fallbacks, health endpoints split so an optional dependency cannot fail readiness, graceful degradation and in-flight request draining, trace IDs threaded through logs and Sentry, migrations run inside the deploy with revision tracking and rollback, six layers of quality gate. No mesh, no kubernetes (ECS), no service-topology to assemble.

What that does to the seams thesis. kubernetes-integration-tax locates the cost in making ~20–30 CNCF tools cohere, and the internal-developer-platform answer is to pay it once and productise it. Prod Forge suggests the tax is a slope rather than a threshold: strip the tools down to Postgres, Redis and one pipeline and the questions survive intact — what is critical, what does readiness mean, what happens to requests mid-deploy, how does a migration roll back. Paged as production-readiness, the floor beneath the three pillars. It also frames the IDP argument from below: a golden path is worth building because many teams would otherwise each rediscover this floor, not because the floor is hard for one.

And it fails the spoke’s own evidence bar in an instructive way. This is a source that could have supplied the missing quantification and does not: a Todo API with no traffic, no incident, no postmortem, no benchmark. It documents the list without evidence about which items load-bear, which is the same hole dora-metrics and service-level-objectives were supposed to close and still haven’t been pointed at anything (zalando-cslb-1m-rps remains the only source reporting both halves, and it is self-reported by one team).

Cross-spoke, worth a note: chapter 4 is a full agent-governance configuration — quality gates before generation, MEMORY.md / REVIEW.md / Skills as project knowledge, and pre-hooks enforcing protected files (Edit|Write) and blocked commands (Bash), under headings like Never Trust AI Blindly. That is agentic-tooling-wiki’s constitution-vs-containment argument settled on the containment side, with hooks rather than prose, in a repository with no agent product to sell. The machinery belongs there; that it now appears as a routine chapter of a production checklist belongs here.

Contradictions flagged

The paved road is measured, and it slows delivery down. The spoke’s spine says the hard problem is the seams, and the internal-developer-platform is the answer: pay the kubernetes-integration-tax once, centrally, and hand teams the result. dora-2024-report measures that answer across a survey where 89% of respondents use an IDP and finds the productivity case and the delivery case pointing opposite ways — individuals +8% productive and teams +10% better, against ~8% lower throughput and 14% lower stability (change failure rate and rework both up), with a further 6% throughput cost where the platform is mandatory for the whole lifecycle. The golden-cage caveat this wiki wrote from first principles in June now has a number on it.

Both claims stand. The readings DORA leaves open are not equivalent for this spoke: an IDP adding handoffs is an argument that the integration tax was relocated rather than paid — the seam moved into the platform — while the “teams ship more freely, so instability is experimentation” reading says the metric is measuring confidence, not damage. The first would revise the spoke’s thesis; the second would revise dora-metrics. Nothing here decides between them, and the corpus should not pick the flattering one.

Recorded with the source’s limits attached: cross-sectional, self-reported, delivery-stability rather than incidents, and published by a company that sells platform tooling — which cuts toward trusting a finding this unflattering to platform tooling.
Consistent reconciliation vs. no coordination at all. The corpus’s spine — gitops‘s continuously-reconciled desired state, kubernetes‘s control loop, the IDP’s one-YAML provisioning — assumes a consistent view is what makes production tractable. modal-1m-sandboxes holds the opposite: on the sandbox-creation path a consistent view is the ceiling, so it runs with no data store, asynchronous worker-published state, and a deliberately stale scheduler view, indicting Kubernetes’ O(n × p)/etcd/heartbeat costs by name. Both claims stand. The reconciling read (not yet sourced, so held as a hypothesis) is that they answer different workloads — consistency pays for long-lived services with placement constraints, and stops paying when the unit is ephemeral, untrusted, and arrives in a burst where a rejected placement costs only a retry. Note the self-interest: Modal sells the alternative it is arguing for (T3). Worth revisiting if a neutral source, or a Kubernetes-side rebuttal, lands.
The name says Git; the specification does not (2026-08-12). Re-reading opengitops.dev found that none of the four v1.0.0 principles mentions Git, or any version-control system. Principle 2 asks only for storage that “enforces immutability, versioning and retains a complete version history”; principle 3 says agents pull “from the source.” Object storage with versioning satisfies the letter of both. Meanwhile every implementation this spoke touches — Argo CD under project-as-a-service, Flux — reconciles from a repository.

Not a contradiction between sources so much as one between a standard and its own name, and it is load-bearing here because this spoke keeps invoking gitops as a shape rather than a product: the reconcile-from-declaration loop it shares with kubernetes‘s control loop and with the IDP’s one-YAML provisioning. If the specification is storage-agnostic, that generalization is stronger than the wiki has been claiming, and the “Git” in the name is doing rhetorical rather than technical work. Both readings recorded on gitops; the practical consequence is that a conformance claim should be checked against the four principles rather than against whether a repository is involved.
Two “1M” stories, opposite morals. scaling-to-1m-lambda: never do the same thing at the same time everywhere (synchronization is the bug; jitter is the fix). modal-1m-sandboxes: do a million things at the same time, deliberately, in under a minute (synchronization is the product). Not a true contradiction — one controls arrival, the other cannot — but the pair is the sharpest statement of the fleet-scheduling trade the spoke has, and shouldn’t be flattened into one lesson.

Cross-spoke adjacency

agentic-tooling-wiki owns the agent-building stack (ADK, MCP, Gemini Enterprise Agent Platform). aiops / google-sre-agentic-ai is the application of that stack to operations — the same tools, a different altitude. This split mirrors the hub’s general rule (tools vs. their application). Watch for bridge sources: agents acting over a service-topology. New seam (2026-06-15): agentic-tooling’s [[agent-loops-verification]] argues that as agent loops replace prompts, verifying agent-written cloud-native code becomes a runtime problem — validating behaviour against a real running system (Kubernetes ephemeral environments). That verification substrate is this spoke’s domain: the loop/verification paradigm is agentic-tooling’s, the runtime that makes the feedback truthful is platform-ops’. A candidate cluster if more “verify agents against live infra / CI-CD” sources arrive.
cloud-wiki owns cloud-hosting providers & pricing (where you rent servers). This spoke owns the practice of operating what runs on them. Managed-Kubernetes pricing → cloud-wiki; running kubernetes in production → here. Live seam (2026-07-17): modal is now paged in both senses — cloud-wiki’s synthesis already names Modal in its open question on where agent-written code gets executed (alongside E2B and AWS’s Firecracker-backed [[aws-lambda-microvms]]), which is the offering; modal-1m-sandboxes is the scheduler behind the offering, which is here. modal-1m-sandboxes was routed to this spoke with cloud-wiki as runner-up. If more sandbox-vendor comparisons arrive they belong there; more scheduler/fleet internals belong here.
defensive-security-wiki (new seam, 2026-08-11 via canva-session-revocations-at-scale). session-revocation is a security control and a fleet-state-distribution problem at the same time. The split: the posture — continuous verification, what warrants revoking, zero-trust — is that spoke’s; the delivery mechanism — how a deny-list reaches hundreds of gateways without a stampede, and what staleness that buys — is this one’s. Canva routed here on dominant substance (the whole source is architecture and load curves) with defensive-security as runner-up. If sources on session/token security policy start arriving, they belong there and should cross-link back.
llm-inference-wiki shares a latency/throughput sensibility but at the model- serving layer, not the distributed-systems-ops layer.
analytical-databases-wiki (new seam, 2026-07-11 via openobserve). An observability backend that stores telemetry as columnar Parquet on object storage, queried by DataFusion, is running the OLAP playbook of ClickHouse/DuckDB — the reason the log bill collapses. The split: the operating-telemetry / observability practice is this spoke; the columnar-store engine as subject (DataFusion, ClickHouse, DuckDB) is analytical-databases. O2 routes here (subject = observability platform) with the storage mechanism cross-linked. Cluster strengthened 2026-07-26: hyperdx is the second instance and the more literal one — it runs directly on ClickHouse, which is a paged subject in analytical-databases-wiki, so the seam is now two sources deep. The split holds unchanged (practice here, engine there), but if a third arrives the case for a joint page on telemetry-as-OLAP-workload gets hard to refuse.
Parked adjacency (hub _inbox): nvidia-doca-in-silicon-security (ai-infrastructure) is the hardware/silicon-security layer beneath this one — related but a distinct sub-domain. Fold in only if a hardware-ops cluster forms here.

Index — Platform Ops Wiki

Catalog of all pages, grouped by @type. The spine: synthesis (thesis), log.md (history), this file (catalog).

DefinedTerm (concepts / disciplines)

platform-ops — umbrella: running cloud-native distributed systems in production; the three pillars. · domain
site-reliability-engineering — SRE; reliability/operations as a software problem, SLOs & incident response. · practice
platform-engineering — building/integrating the internal platform teams run on (the “integration tax”). · practice
production-readiness — the floor beneath the three pillars: what a service needs before people can be woken up by it (critical vs non-critical dependencies, liveness vs readiness, degradation and draining, migrations with a rollback, gates before merge). The integration tax as a slope, not a threshold — and documented in this corpus only by checklists, never by outcomes · practice
observability — metrics/traces/logs + topology; fuse multiple signals, none is complete alone. · practice
service-topology — the live service-dependency graph; blast-radius & local-vs-upstream. · mechanism
aiops — AI/agents applied to operations; constrained & accountable, not autonomous. · practice
ebpf — sandboxed in-kernel programs; low-overhead telemetry/networking/security + scheduling substrate · source · mechanism
sched-ext — eBPF-based extensible kernel scheduler framework (Linux v6.12); write a custom CPU scheduler as a verified BPF program, hot-swappable from user space; eBPF’s 4th domain · mechanism
knod — in-kernel network offload device: RFC patch series running XDP / IPsec on an AMD GPU, with the kernel owning the GPU queues and JIT-ing the program (no ROCm/CUDA in the data path); eBPF’s load-verify-JIT shape aimed off the CPU · mechanism
service-level-objectives — SLI/SLO/SLA + error budgets; the quantification backbone gating velocity vs reliability · source · standard
gitops — Git as source of truth; declarative + pulled + continuously reconciled (Argo CD/Flux); the deploy/operate loop · source · practice
distributed-tracing — spans/traces across services; the per-request view of the service topology; third observability signal · source · mechanism
dora-metrics — DORA Four Keys: deploy frequency/lead time (throughput) + change-fail-rate/recovery-time (stability); delivery-performance quantification, complements SLOs · source · standard
internal-developer-platform — IDP / golden paths / platform-as-a-product; the self-service product platform-engineering builds; the cure for the integration tax · mechanism
serverless — the per-invocation compute model (AWS Lambda); the spoke’s second production substrate beside kubernetes; ops shifts from capacity → efficiency/governance · mechanism
compute-sandbox — isolated, disposable environment for untrusted / agent-written code (container or microVM); the spoke’s third substrate: untrusted workload + ephemeral unit + burst arrival → a placement problem, not a capacity one · mechanism
container-scheduling — the placement layer: which worker runs a container. The consistent-view (kubernetes O(n × p) + etcd) vs stale-view (modal async workers) axis; distinguishes the arrival / placement / CPU altitudes · mechanism
client-side-load-balancing — CSLB: the caller routes in-process instead of via a shared proxy. Endpoint watch + consistent hash + occupancy load signal + bounded load + fade-in; the request-level sibling of container-scheduling · mechanism
session-revocation — killing an already-authenticated session when auth is stateless: there is nothing to delete, so every gateway carries a deny-list. Bound it by token lifetime, hold it in memory; what’s left is fleet seeding and propagation delay · mechanism
object-storage-as-coordination — S3 conditional PUT (compare-and-swap → optimistic concurrency, no lock service) + conditional GET (change-only polling → a propagation channel) over immutable chunks; a shared component whose load tracks writes, not readers · mechanism

SoftwareApplication

kubernetes — container orchestration; the production substrate of platform engineering.
opentelemetry — vendor-neutral CNCF telemetry standard (traces/metrics/logs); the off-the-shelf instrumentation layer · source
prometheus — CNCF metrics + alerting toolkit; pull-based time series + PromQL · source
openobserve — open-source unified observability backend (logs/metrics/traces/RUM); Rust + Parquet-on-object-storage + DataFusion; the “Datadog alternative” cost play (claimed 140× cheaper log storage vs Elasticsearch) · source · T3 · medium.com (url-only)
hyperdx — open-source observability UI on ClickHouse: plain-text/regex search + key-value filters instead of PromQL/LogQL/SQL, linked traces↔logs, flame graphs, OTel/Fluentd/Logstash/HTTP ingest; the query-surface answer where openobserve is the storage-cost one · source · T3 · opensourceprojects.dev
modal — AI-workload cloud whose Sandboxes product is paged here as a published scheduler design: no data store on the creation path, workers as source of truth, stale-view load balancing. Vendor-about-itself (T3); the mechanism is the durable part · modal.com
webernetes — ngrok’s browser-based Kubernetes simulator (TS, no real cluster); teaches Pod/Service/Deployment/controller primitives — educational, not operational · source · T1 · github.com

SoftwareSourceCode (sources)

prod-forge — MIT reference implementation (NestJS + Terraform on AWS, 217★) opening with “AI made writing the code easy. Running it in production is still hard.”: sixteen documented chapters from git flow to rollback, on a Todo API. The spoke’s smallest-scale source and its clearest statement of production-readiness; also ships a full agent-governance chapter (protected files, blocked commands, mandatory review) that belongs to agentic-tooling-wiki. No incident, no benchmark, no traffic · source · T1 · github.com/prod-forge

Report (survey research)

dora-2024-report — Accelerate State of DevOps Report 2024 (Google Cloud / DORA, 10th report, 120 pp, read from the PDF): the four keys become five metrics on two factors with rework rate added to stability; the 2024 clusters (Elite 5% change-fail / Medium 10% / High 20% / Low 40% — the ladder is not monotone); and the finding this spoke did not want — 89% use an internal developer platform, and using one means individuals +8% productive, teams +10% better, throughput −8%, stability −14%, mandatory use a further −6%. AI: +25% adoption → org +2.3%, team +1.4%; adding AI-powered experiences → −10% delivery stability · source · T2 · services.google.com

TechArticle / BlogPosting (source summaries, `source: true`)

google-sre-book — Google’s first-party SRE Book (intro): Treynor’s definition, error budgets, the 50% toil cap, change-risk. · source · T1 · sre.google (url-only)
otel-observability-primer — OpenTelemetry’s canonical definition of observability (vs monitoring; traces/metrics/logs; instrumentation). · source · T1 · opentelemetry.io (url-only)
kubernetes-docs-overview — Kubernetes first-party docs: definition, Borg/2014 lineage, the control-loop capabilities, and what it is not (no built-in PaaS/monitoring). · source · T1 · kubernetes.io (url-only)
netflix-service-topology — InfoQ: Netflix maps thousands of microservices in real time (eBPF + IPC + traces). (url-only)
google-sre-agentic-ai — Google Cloud: agentic AI across the SRE workflow. (url-only)
kubernetes-integration-tax — CNCF: the Kubernetes integration tax (Prometheus, Cilium, production reality). (url-only)
project-as-a-service — InfoQ/KubeCon: Belastingdienst’s IDP pattern (one YAML → namespaces/RBAC/quota; golden paths; enablement over support; 99+ teams). (url-only)
ebpf-kernel-observability-infoq — InfoQ podcast (Dan Fineran/Isovalent): eBPF safety (verifier-as-bouncer), observability without instrumentation, Tetragon security + live CVE patching, AI-generated eBPF policies (AIOps seam) · source · T3 · infoq.com (url-only)
awesome-microservices — mfornos: curated catalog of ~400+ microservice-architecture tools/principles (Capabilities = discovery/orchestration/observability/messaging/gateways); the landscape map under the spoke’s seams thesis · source · T3 · github.com (url-only)
scaling-to-1m-lambda — AWS Architecture Blog: ProGlove scales to 1M Lambda functions / thousands of accounts; serverless ops at scale (account-per-tenant isolation, the cron self-DDoS→jitter lesson, cost-driven rearchitecting, StackSets ceilings) · source · T3 · aws.amazon.com (url-only)
modal-1m-sandboxes — Modal: 1M concurrent sandboxes in <1 min (was 50k/customer) by deleting central coordination — no datastore on the creation path, workers→Redis stream, stale-view load balancing, direct RPC; explicit kubernetes O(n × p)/etcd critique; the rtnl kernel-lock wall; the corpus’s first argument against its own reconcile thesis · source · T3 · modal.com (url-only)
knod-igorslab — Igor’s Lab: knod shifts network processing to AMD GPUs — NIC DMAs into GPU memory, kernel JITs the program, ROCm/CUDA kept off the data path; GCN/RDNA2, RFC stage, no benchmarks and no developer named · source · T3 · igorslab.de (url-only)
meta-sched-ext-ads — Engineering at Meta: custom sched-ext CPU scheduler for the 5M-req/s ads fleet (soft-partition CPUs, user-space BPF policy = restart-to-deploy); −28% p99, +1.1% ads-ranked, 3.28 MW saved; kernel-level perf as an ops discipline · source · T2 · engineering.fb.com (url-only)
canva-session-revocations-at-scale — Canva Engineering: hundreds of millions of stateless cookie sessions, with the revocation deny-list moved out of MySQL into 30-minute S3 chunks of a 12-hour window (16-byte sorted records, conditional GET/PUT, ZooKeeper election as an optimization only); −87.5% gateway memory, read replicas → 2, >2,000 revocations/s, DB load decoupled from fleet size; propagation is minutes · source · T2 · canva.dev (url-only)
canva-session-revocation-infoq — InfoQ: the news write-up of the above and the arriving source; adds Adam Urban’s “immutable objects and conditional writes as coordination primitives” reading, and a looser “near-real-time” framing of a minutes-long propagation window · source · T3 · infoq.com (url-only)
zalando-cslb-1m-rps — Zalando Engineering: >1M req/s of internal fan-out moved off the shared Skipper proxy into an in-process CSLB; occupancy over in-flight, EndpointSlice watch over poll, N-ring fade-in; 50→8 proxy pods, −25% app pods, deploys 289→128 min; AZ affinity built then paused · source · T2 · engineering.zalando.com (url-only)

Organization (entities)

zalando — European fashion retailer; publishes its engineering work and maintains the Skipper ingress proxy · entity · Corporation
canva — Australian design platform; operator of the hundreds-of-millions-of-sessions auth fleet behind session-revocation · entity · Corporation
infoq — software-engineering news outlet and QCon organizer; this spoke’s most frequent secondary publisher (4 sources), standing tier T3 · entity · NewsMediaOrganization

Person (entities)

llew-vallis — Canva engineer; author of Session Revocations at Scale · entity