Log — Operational Databases Wiki

Append-only history. Entries start with ## [YYYY-MM-DD] <op> | <title>. <op> ∈ {split, ingest, query, lint}.

[2026-07-16] split | operational-databases-wiki created from _inbox (3 sources)

Spun out of the hub _inbox databases-beyond-olap cluster once it reached 3 — turso (embedded OLTP engine, parked 2026-06-23), awesome-db-tools (general DB-tooling catalog, parked 2026-06-24), and planetscale-768-servers (distributed-SQL sharding explainer, the 2026-07-16 trigger). Both prior park notes predicted a 3rd general-DB source would trip the spin-out; the human chose a new spoke over broadening the OLAP-only analytical-databases-wiki (keeping that spoke’s columnar identity intact). Domain: transactional/OLTP + distributed-SQL databases and the tooling around them, across the embedded↔distributed spectrum — the operational sibling to analytical-databases-wiki.

[2026-07-16] ingest | Turso (in-process SQLite-compatible OLTP engine)

Founding source. turso — a Rust rewrite of SQLite, in-process OLTP; MVCC BEGIN CONCURRENT writes, io_uring async I/O, CDC, exact vector search, tantivy FTS, 7-language bindings, built-in MCP server (9 tools). v0.7.0 (Jul 2026), MIT, ~22.9k★, beta but in production (Turso Cloud/Kin AI/Spice.ai). Typed SoftwareApplication + source (T1, primary repo). Anchors the embedded pole of operational-databases; noted the MCP/agent seam (seekdb). Maker-org deferred (thin).

[2026-07-16] ingest | Making 768 servers look like 1 (PlanetScale)

Founding source + spin-out trigger. planetscale-768-servers — sharding a transactional SQL DB so 256 shards × 3 = 768 servers (~4 TB each, ~1 PB, millions QPS) present as one logical DB. Motivates sharding via the single-writer WAL / replicas-duplicate-not-distribute / multi-day-backup bottlenecks; mechanism = proxy router (Vitess/MySQL, Neki/Postgres) + hash shard key + cross-shard aggregation behind an NLB. Typed TechArticle + source (T2, vendor blog). New concept database-sharding; new Org planetscale (maintains Vitess, built Neki). Anchors the distributed pole.

[2026-07-16] ingest | awesome-db-tools (curated DB-tooling catalog)

Founding source. awesome-db-tools (mgramin) — ~200+ tools for working with databases (not engines), 20+ categories (clients, migration, backup, monitoring, HA/sharding, security, SQL utils…), spanning OLTP+OLAP+NoSQL; CC0, ~5.2k★. Typed Collection + source (T3, link catalog / page-candidate source). The tooling member of the spoke; flagged the open question of whether cross-type DB tooling coheres with the OLTP-engine core.

Also created concepts online-transaction-processing (the OLTP-vs-OLAP boundary that defines the spoke) and the operational-databases umbrella. Spine files (index/synthesis/this log) written. avoid-ai-writing self-pass (clean). Site rebuild + verify run at split (required for structural page moves).

[2026-07-29] ingest | Rethinking Data: three-tier stack → client-side event sourcing (InfoQ podcast)

Routed from the hub (Telegram). Five pages: the source summary rethinking-data-client-event-sourcing (PodcastEpisode, T2), two concepts (local-first-architecture, event-sourcing), the artifact livestore, and the person johannes-schickling (Prisma founder, LiveStore maintainer, the episode’s guest).

The substance the spoke didn’t have: this is the first source about where the OLTP store lives relative to the user, and it extends the founding embedded↔distributed axis past turso into the client. It’s also the first material on merge semantics — event log vs CRDT vs query-sync — which the “where’s the distributed-consistency substance?” open question had been waiting for, arriving from the client end rather than the NewSQL end that question anticipated. Both were noted in synthesis rather than quietly closed: the server-side consensus story is still missing.

Recorded as a real tension rather than smoothing it: planetscale-768-servers shards to preserve one globally consistent database, and local-first abandons global consistency on purpose. Both OLTP, opposite convictions about what OLTP protects. Schickling draws his own boundary (financial systems, social graphs = wrong fit), which is what makes it a trade rather than a contradiction.

T2 with a stated caveat: independent publisher, interested guest — the event-sourcing-beats-CRDTs claim comes from the person maintaining an event-sourcing framework, with no benchmark. Added as its own open question, and a CRDT-side source (Automerge/Yjs/Loro) is now the named gap.

Not paged, one source each, per the ENTITIES recursion discipline: the sync engines (ElectricSQL, PowerSync, Zero), the CRDT libraries, Overtone, Prisma-as-org, and the Local-First Conference. Nicholas Schiefer and Geoffrey Litt are credited on livestore but not paged — Litt in particular is a research-wiki-adjacent name (tools-for-thought) and would be worth a node if he recurs. avoid-ai-writing run. Verify deferred per hub policy (content-only, no page moves).

[2026-08-03] ingest | Pagination across shards (Medium, Umesh Kumar Yadav) — PAYWALLED STUB

Routed from the hub (Telegram). New: sharded-pagination-interview-post (T4 stub). Updated: database-sharding (the aggregation bullet now names its expensive cases), synthesis (open question), index. Body not retrieved. WebFetch returned the member-only preamble; a Firecrawl scrape returned the same paywall page. Per ../HUB.md edge handling this is the “Firecrawl also fails” case, so the source is recorded as an honest stub rather than dropped or written from guesswork. Nothing about the solution — keyset vs offset, scatter-gather merge, cursor encoding — is attributed to it. What the preview does establish: the framing question (order table across many shards, user opens page 100,000) and the anti-pattern SELECT * FROM orders ORDER BY create_time DESC LIMIT 1000000, 10, called “one of the slowest ways to retrieve data”. Genre is interview prep, not documentation. Kept because it names a real gap: database-sharding compressed cross-shard aggregation into one bullet and planetscale-768-servers never says what an ORDER BY/LIMIT costs once rows sit on 256 shards. Deep pagination is the sharpest ordinary case — the router can’t know which shard holds rows 1,000,000-1,000,010 without asking every shard for its first 1,000,010. T4 justified: paywalled, unread, single-author interview-prep content, no editor, no cited sources. Resolution path recorded on the page and in synthesis — a friend link, or Vitess docs on ORDER BY/LIMIT handling (planetscale already owns two nodes here). Entity discovery skipped (author unknown beyond a byline; nothing to page on evidence). Verify deferred per hub policy (content-only). avoid-ai-writing run.

[2026-08-03] ingest | Vitess query-serving documentation (v23.0)

Pulled at the human’s instruction (“pull the Vitess docs”) as the readable substitute for the paywalled sharded-pagination-interview-post. New: vitess-query-serving-docs (T1 source summary), cross-shard-queries (DefinedTerm/mechanism), vitess (SoftwareApplication). Updated: database-sharding (the aggregation bullet now points at the mechanism), sharded-pagination-interview-post (records that the substitute landed and what it didn’t supply), synthesis (open question moved from unanswered to narrowed), index. Read: concepts/execution-plans, concepts/vtgate, user-guides/sql/vexplain, reference/features/ sharding, reference/compatibility/mysql-compatibility — v23.0, on 2026-08-03. Docs pages were thin through WebFetch summarisation, so the primary markdown was pulled from raw.githubusercontent (vitessio/website) and read directly. Substance: the planning rule is “push down as much work as possible to the underlying MySQL instances… when this is not possible… collect input from multiple sources and merge the results”. Plans are operator trees evaluated leaf-to-root. A routing operator names keyspace, sharded-ness and vindex; scatter is defined as a sharded keyspace addressed without a vindex, with the explicit caveat that not every multi-shard query counts as scatter. VTGate is the merge point and speaks the MySQL wire protocol “as if it is a MySQL Server”. The instrument the corpus was missing: VEXPLAIN TRACE annotates operators with NoOfCalls, Avg/MedianNumberOfRows and ShardsQueried; VEXPLAIN KEYS reports grouping/join/filter columns as candidate sharding keys without executing. Together they answer “how badly does this query scatter” and “which vindex would stop it”. Honest limit recorded in three places: none of the pages read gives a cost model for deep LIMIT/OFFSET or recommends keyset/cursor pagination. The keyset remedy is the obvious consequence of the mechanism and is deliberately not written as sourced — cross-shard-queries states the problem and says plainly that the fix is uncited here. T1 justified: first-party reference docs for shipped software, version-pinned, describing behaviour VEXPLAIN can confirm against a live cluster. Caution: intent and mechanism, not measurement; VEXPLAIN exists only from v16 and operator names move between releases. Verify deferred per hub policy (content-only). avoid-ai-writing run.

[2026-08-08] ingest | Principles of Distributed Computing (Wattenhofer, ETH Zürich)

Routed from the hub (Telegram). T1 — 321-page graduate course text, freely published by ETH’s DISCO group, theorems with proofs. New pages: podc-lecture-notes (source), consensus, consistency-models, plus entities roger-wattenhofer and eth-zurich.

Closes the spoke’s #1 growth edge, from the other side. The edge asked for engine internals; this is the theory those engines were built against. Three results reframe what the corpus already held:

FLP, stated exactly (Thm 16.14): “There is no deterministic algorithm which always achieves consensus in the asynchronous model, with f > 0.” So a store committing across shards is not solving consensus — it is buying partial synchrony (a timeout) or randomisation. The corpus can now ask which premise a vendor weakened.
Byzantine agreement needs f < n/3, the origin of 3f+1 replica counts; authentication moves the bound.
CAP is a theorem with a model — Fox & Brewer 1999, proven by Gilbert & Lynch 2002 for the asynchronous model, relaxable under partial synchrony. That replaces pick-two-of-three with the real move: weaken consistency by a named amount. Causal consistency is the vocabulary the spoke was missing when rethinking-data-client-event-sourcing arrived arguing merge semantics from an event-sourcing maintainer’s position.

Recorded honestly: fifteen of twenty-two chapters are out of scope here — graph colouring, distributed sorting, MIS, locality lower bounds, social networks, synchronizers, communication complexity, wireless, self-stabilisation, labeling schemes, game theory. The source’s centre of gravity sits outside the spoke holding it, which is a real routing tension and is written on the source page, in synthesis, and in the hub log. If more distributed-theory sources arrive, a distributed-systems-theory spoke is the natural home and this should be revisited rather than defended.

Edge refreshed: the consistency edge is now partly closed — the engine half stands, and Raft (what production systems actually run) is in neither the book nor this wiki. Added as edge 4.

Entities: 2 created.

[2026-08-09] ingest | Postgres and MySQL — the base case the spoke never wrote (via research pass)

Coverage edge 5 closed. Sources are the projects’ own documentation, both T1: PostgreSQL’s Architectural Fundamentals and Introduction to MVCC (postgresql.org/docs/current), and the MySQL 8.4 Reference Manual’s Introduction to InnoDB (dev.mysql.com).

New pages: postgresql, mysql, and multiversion-concurrency-control (the mechanism both describe, which also takes a quarter off edge 6).

What was wrong before. Every page here is defined against single-node OLTP — sharding is what you do when one node stops being enough, local-first is what you do when the round trip to it hurts — and the wiki never said what that node does. MySQL was worse than absent: it existed only as the thing vitess shards, so all of its properties arrived filtered through planetscale‘s product.

Two facts earned their place immediately. Postgres forks a backend process per connection, which is why pooling is not optional and which measures the distance to turso (no server at all) and to vitess (the endpoint is a proxy). And InnoDB’s clustered index puts table data inside the primary key index, making the primary key and the shard key one decision — the same decision cross-shard-queries describes at the fleet level, where vitess‘s vindexes make it again.

Both engines land on MVCC from different implementations — new heap versions plus vacuum against an undo log — and the shared guarantee is the single-node form of what consistency-models asks of distributed systems.

Scope recorded honestly: these two doc pages stop before WAL, checkpointing, crash recovery, replication and the planner. That is edge 5’s successor, and it is where planetscale-768-servers already reports the real bottlenecks.

Entities: 0 created — no vendor org pages; both sources are project documentation, and the Turso-maker deferral note in index.md applies here too.

[2026-08-10] ingest | The Byzantine Generals Problem (Lamport, Shostak & Pease, 1982)

Routed here by the hub (runner-up: research-wiki, which owns tla-plus and the formal-methods lineage). Source: https://lamport.azurewebsites.net/pubs/byz.pdf, T1, ACM TOPLAS Vol. 4 No. 3, July 1982, pp. 382–401, 20 pages, SRI International. The arriving link carried a Facebook fbclid parameter; stripped.

Pages: byzantine-generals-problem (source summary), leslie-lamport (Person, canonical node). consensus updated with a correction; synthesis gained the same under “The limits the mechanisms were built against”; ../research-wiki’s tla-plus cross-links to the new Person node.

The primary source corrects the secondary one. consensus said, from podc-lecture-notes, that “authentication changes the bound.” The paper’s Theorem 2 says algorithm SM(m) solves the problem for any number of generals and any number of traitors, and Section 5 drops the connectivity requirement to the weakest one that keeps the loyal subgraph connected. The bound is not moved, it is gone. Both statements are now on the page — the summary’s and the paper’s — per the record-don’t- overwrite rule, with the primary marked as the authority.

Read locally with pypdf, not summarised from a secondary account. Extracted the full 20 pages to text first, which is where the two admissions came from that no summary of this paper carries: detecting an absent message (assumption A3) needs synchronized clocks, and clock synchronization under faults is “as difficult a problem as the Byzantine Generals Problem itself”; and A4’s unforgeable signature “can never be guaranteed” — only made improbable, and against a malicious operator rather than a random malfunction it becomes a cryptography problem.

The finding that pays rent here. The closing sentence — “The only way to reduce the cost is to make assumptions about the type of failure that may occur” — is what every engine in this spoke does by assuming crash-stop. Raft and Paxos are affordable because a node is assumed to die rather than lie. That is a purchase, and it is the wrong one when a node is corrupted rather than dead. It joins the FLP framing already in consensus: a production store claiming consensus is always asserting something, and the question is what.

Also recorded: Section 6 shows the generals were never a metaphor about malice. The motivating system is majority voting over redundant processors in avionics and missile defence, and the canonical failure is physical — a faulty input unit putting a marginal signal on a wire that some processors read as 0 and others as 1, or two processors reading a clock as it ticks.

Entity: leslie-lamport paged on the recurrence rule (../ENTITIES.md) — named by this paper here and by tla-plus in ../research-wiki. SRI International deferred at one mention.

Does not close the standing gap in consensus: Raft and the engines’ own accounts (Spanner TrueTime, CockroachDB, Vitess failover) are still absent. This is more theory, which was already the well-covered side.

[2026-08-10] ingest | The Part-Time Parliament (Lamport, 1998) — Paxos

Routed here by the hub (runner-up: research-wiki, same as the byzantine paper an hour earlier). Source: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/12/The-Part-Time-Parliament.pdf, T1, ACM TOCS Vol. 16 No. 2, May 1998, pp. 133–169, 33 pages, Digital Equipment Corporation, minor corrections 2000. The arriving link carried an fbclid parameter; stripped. Read locally with pypdf.

Pages: part-time-parliament. consensus gained a “Why majorities, exactly” section and a confirmation on the add-timing branch; leslie-lamport rewritten to hold both papers; synthesis gained “Lamport, twice, an hour apart”.

The pair is the finding, not either paper. byzantine-generals-problem closes by saying the only way to make arbitrary-fault tolerance affordable is to assume something about how components fail. This paper’s §4 takes that discount and states both halves: Paxos “does not tolerate arbitrary, malicious failures, nor does it guarantee bounded-time response. However, consistency is maintained despite the (benign) failure of any number of processes and communication paths.” Crash-stop is a purchase, and it is why the engines in this spoke carry no 3f+1 replica count.

Answered a question the spoke was using without asking. Why majorities? Condition B2 — any two quorums intersect — plus B3, which forces a new ballot to adopt the latest earlier vote its quorum knows about. The shared member carries a decision forward. Majority is the cheapest quorum system, not the principle.

Confirmed rather than inferred. consensus had Paxos filed under “add timing” from Wattenhofer. The paper says it outright: any protocol meeting the progress condition “must involve measuring the passage of time,” footnoted to Fischer, Lynch and Paterson (1985) with the remark that “many centuries were to pass before a rigorous proof of this result was given.”

Recorded honestly: the paper declines to specify leader selection — the section ends with the debate’s outcome unknown and “no record exists of the presidential selection protocol that was ultimately used.” The part production systems find hardest is the part left open.

Paged as one page rather than splitting a paxos Thing page off from the paper; split when a second Paxos source arrives. Entity leslie-lamport updated in place, not duplicated. Digital Equipment Corporation deferred at one mention, as SRI International was.

Routing note. Three theory sources now sit in a spoke defined by operational databases. The trigger the 2026-08-08 note set has been reached; recorded in synthesis as a live question for the hub, not settled here.

[2026-08-10] ingest | The Chubby lock service (Burrows, OSDI 2006)

Routed here by the hub (runner-up: platform-ops-wiki, for the operational half). Source: https://static.googleusercontent.com/media/research.google.com/en//archive/chubby-osdi06.pdf, T1, OSDI 2006, 16 pages, Google. fbclid stripped from the arriving link. Read locally with pypdf.

Pages: chubby-lock-service. consensus‘s “what this corpus still lacks” is half closed; synthesis gained “And then the implementation showed up”.

Third Lamport-adjacent source in two hours, and the one that changes the block’s shape. The two papers this morning were results; this is a system running them, with an outage budget. It is the production witness consensus had been asking for since 2026-08-08.

The master lease is the reusable idea. A master wins a majority plus promises that no rival will be elected for a few seconds, and can then serve reads alone because the promise proves no other master exists. Writes still cost a majority. Cheap linearizable reads bought with a time-bounded promise rather than a quorum — invisible from the algorithm, and the thing this spoke’s engines actually need.

Consensus rented rather than linked. Google had a Paxos client library and shipped a central lock service anyway. The structural reason, not the organisational one, is the good one: a consensus library needs a quorum of the application’s own servers, while a lock service lets a single client make progress safely. The quorum is relocated into a service that specialises in holding one — the same shape as part-time-parliament‘s trade, one level up.

The finding to hold against future engines. 61 outages over 700 cell-days, most under 15s; six data losses in a few dozen cell-years — four database software bugs, two operator errors, none from hardware, and the operator errors occurred during upgrades meant to fix the software bugs. Replication handled what it was built for; everything that was lost was lost to code and people.

And the paper corrects its own premise. Designed as a lock service, most popular use is naming (60% of open files), because consistent caching with explicit invalidation beats a DNS TTL when polling is expensive — 3k processes at a 60s TTL would need 150k lookups/s against a ~50k/s DNS server. Locks are 31 ppm of RPC traffic; KeepAlives are 93%.

Tier honesty: peer-reviewed and therefore T1, but a first-party account of one company’s system with no comparison and no external replication, and the numbers are a 2006 snapshot.

Routing note. This weakens rather than strengthens the distributed-systems-theory split flagged earlier today: Chubby is an operational system, not a theorem, and it makes the consensus block look like a stratum this spoke needed rather than stray theory. Recorded in synthesis; a fifth theory source with no system attached should force the call.

Entity: Mike Burrows deferred at one mention, consistent with SRI International and Digital Equipment Corporation. google linked cross-wiki, not duplicated.

[2026-08-10] ingest | Paxos Made Simple (Lamport, 2001) — and the paxos page splits out

Routed here by the hub. Source: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/12/paxos-simple-Copy.pdf, T1, dated 01 Nov 2001, 14 pages (ACM SIGACT News 32(4)). fbclid stripped. Read locally with pypdf.

Pages: paxos-made-simple (source) and paxos — the algorithm as its own concept page, which part-time-parliament‘s ingest this morning said to create “when a second Paxos source arrives.” It has. consensus, chubby-lock-service, part-time-parliament and leslie-lamport all re-pointed at it; synthesis gained a section.

Not a duplicate, and the difference is the value. The 1998 paper states B1–B3 and proves them. This one derives them — P1 → P2 → P2a → P2b → P2c, each strengthening forced by a counterexample — and claims the algorithm “follows almost unavoidably from the properties we want it to satisfy.” Both are held: the result, and the reasoning that makes it inevitable.

It is also where the field’s vocabulary comes from. Proposers / acceptors / learners, and prepare-promise / accept-accepted. Every implementation speaks this paper’s language rather than the 1998 paper’s.

Two ideas worth carrying past Paxos. When you cannot observe the future, forbid it — the phase-1 promise exists because a proposer cannot know what acceptors will later accept (Chubby’s master lease is the same trick with a clock). And, stated flatly here where 1998 only implies it: a failed leader election costs availability, never correctness. The paper spells out the dueling-proposer livelock in full and then says safety holds “regardless of the success or failure of the election.”

Routing note — this does not trigger the theory split. It is the same algorithm the spoke already holds, told again; it deepens one page instead of widening the theory footprint. Trigger unchanged: a source opening new distributed-systems territory with no system attached.

Cross-spoke lead, not a route. That Lamport had to republish his own paper in plain English before the field could use it — content unchanged, exposition fixed — is evidence for ../research-wiki’s diffusion-and-adoption cluster. Noted on the source page; that spoke owns it if it wants it.