Observability
The ability to understand a system’s internal state from its external outputs — classically metrics, traces, and logs, extended here to the service-topology (the live dependency graph). A pillar of platform-ops and the substrate site-reliability-engineering investigates over.
The canonical definition (and the contrast with monitoring)
Per the otel-observability-primer, observability “lets you understand a system from the outside … without knowing its inner workings” — which is the key contrast with monitoring: monitoring watches predefined signals for known failure modes, while observability supports open-ended questions about the unknown unknowns (“why is this happening?”) after the fact. It rests on telemetry in three signals — traces (a request’s path across services, see distributed-tracing), metrics (aggregated numbers over time), and logs (timestamped, usually not request-scoped) — emitted by instrumentation in the code. The bar OTel sets: enough instrumentation that you never have to ship more code just to debug an issue.
A recurring design lesson from the sources
No single telemetry source is complete; fuse several. netflix-service-topology makes this explicit, merging three sources because each has a blind spot:
| Source | Strength | Blind spot |
|---|---|---|
| eBPF network flow logs | broad, kernel-level coverage | no application context |
| IPC metrics (instrumented) | rich app-level endpoints | misses uninstrumented services |
| Distributed traces | real request paths | sampled → incomplete |
The merged graph beats any single source — a concrete instance of the platform-ops through-line that the value is in integrating signals, not collecting them (cf. the kubernetes-integration-tax, where Cilium metrics are invisible to Prometheus without the right glue).
Where the signals land: the backend and its cost
Instrumentation (opentelemetry) produces the three signals; something has to store and query them, and at scale the storage bill dominates. openobserve is the spoke’s first unified backend — a single store for logs/metrics/traces (plus RUM) whose whole pitch is that the storage layer, not the pricing sheet, is where observability gets expensive. Its answer — columnar Parquet on object storage, queried directly — reframes observability-at-scale as an analytical-database (OLAP) problem: store cheaply, scan fast. prometheus is the narrower, metrics-only precedent (its own time-series store, no logs/traces).
hyperdx is the second backend here and it moves the complaint: not what storage costs, but what the query surface costs. PromQL, LogQL and SQL each have to be learned before the system will answer anything, which is a poor property during an incident — so HyperDX puts plain-text search, key-value filters and regex over ClickHouse instead. Between them the two sources mark out the axis: an observability backend is an OLAP engine plus a decision about how much of the stack to own (O2 builds its store; HyperDX rents ClickHouse’s) and who the query surface is for.
Used by
site-reliability-engineering reads these signals to locate failures; in particular aiops agents (google-sre-agentic-ai) reason over observability + topology to investigate incidents.
Related
otel-observability-primer · opentelemetry · openobserve · hyperdx · prometheus · distributed-tracing · service-topology · ebpf · site-reliability-engineering · platform-ops · netflix-service-topology