Why CockroachDB refused writes to a healthy 155 KiB row

A worksheet in prod stopped saving. The pod was healthy. 404 MiB of a 2 GiB limit, 655m of 1500m, no restarts. I didn't believe that, so I went and looked at the database too. Three active queries cluster-wide, 12% CPU, all three nodes live. Idle. Nothing was exhausted, nothing had crashed, and the service still couldn't write. The software It's a collaborative editor. Teachers build worksheets, whiteboards and lesson plans, and several people can have the same document open at once. Every document is a CRDT, built on Loro. The browser holds a replica and applies edits to it locally, then pushes them over a WebSocket to a sync server. The server keeps its own copy of each open document in memory, merges whatever arrives into it, and writes the result to CockroachDB v25.x. That last step is the one that matters here. Persisting a document means exporting the entire Loro doc as a snapshot and writing it into a single BYTEA column, on a single row. Not an append-only log of updates, which is the usual way to store a CRDT. The whole document, on every save. The document that stopped saving was 155 KiB. Its range was 1 GiB. A wild goose chase to find the root cause The red herring: the same service had an unrelated CPU problem running that day, readiness probes flapping, the node pegged, hundreds of timeout errors in the logs. I went through all of it. Every bit real, none of it connected to this. Two separate problems on one service on the same day, and the louder one wasn't the one refusing writes. A second false trail: I noticed payload sizes varied a lot from one document to the next and read that as clients sending incremental deltas, which would make the write volume real edits. That was wrong. Varying payload size doesn't imply a delta. A CRDT snapshot of a changing document is a different size every time, the same way two zip files of slightly different inputs come out different sizes. The check that settles it is dividing the payload by the stored snapshot: document type writes per resource payload ÷ snapshot worksheet 995 0.81 lesson plan 57 0.94 whiteboard 5.9 0.87 text document 18 0.82 Near 1.0 means the client sent as many bytes as the entire stored document. Every document type was doing it. Worksheets weren't doing anything different in kind. They were doing it 169 times more often than whiteboards, and that was the whole difference between a wasteful system and a broken one. How to wedge a CockroachDB range The error was in the logs the whole time, buried at a much lower volume than the noise. split failed while applying backpressure to Put [/Table/111/60/"..."/0] on range r725: could not find valid split key Four things had to be true at once for that, each one reasonable on its own. CockroachDB is MVCC (Multiversion Concurrency Control), so a write never overwrites anything. Every write stores a new copy of the row under the same key at a new timestamp, and the previous copies stay exactly where they are. The key in the storage engine isn't the row; it's the row plus a timestamp. That's what lets a transaction read a consistent view of the database without locking the rows it reads. A transaction reading at timestamp T sees the newest committed version at or below T of every key it touches. CockroachDB runs SERIALIZABLE by default and there is more machinery than that behind it, since reads leave marks in the timestamp cache that push later writers, and a read that meets an unresolved intent below its own timestamp has to wait on it. But keeping every committed version around is what the rest is built on top of. It's also what AS OF SYSTEM TIME, follower reads and incremental backups are built on. All three are reads at an older timestamp, and they only work if the data as of that timestamp is still on disk. So old versions can't be dropped at write time. Something has to guarantee they're still there for anyone reading in the past. They get collected later by the MVCC GC queue, once they're older than gc.ttlseconds, which was four hours here. Which means the storage a row occupies isn't its size. It's its size multiplied by how many times you wrote it in the last four hours. The whole row is one key. CockroachDB stores a row as one key per column family, and this table never defined any beyond the default, so every column sits in the same one. One document, one key, however large the snapshot gets. A split has to cut between two keys. Ranges are kept under range_max_bytes by splitting, and a split picks a key and cuts the keyspace there: everything below goes to one range, everything above to the other. If every byte in a range belongs to one key and the copies differ only by timestamp, there's nowhere to put the boundary. They can't be separated anyway, because the range is what serves reads of that key at any timestamp, so all of them have to live together. The client was pushing every 2.2 seconds, whether or not anything had changed. Here's what the range actually looked like: keys 1 versions 6,766 val_bytes 1024.02 MiB live 0.151 MiB One key. Nearly seven thousand copies of it. A gigabyte of stored versions against 155 KiB of actual row. 0.47 writes per second against a 14,400 second GC window predicts 6,768 versions. There were 6,766. The range was holding exactly one GC window of writes, which is where this stopped being a mystery and became arithmetic. I liked that part a lot. Nothing was queued or deferred to get there, which is worth being explicit about. Every one of those writes applied immediately: proposed, replicated, committed, visible to the next read. The range grew because that is what a range does when you write to it. Splitting is not part of the write path. Splitting happens on the split queue. Each store walks its replicas on a timer, reads their size straight off the MVCC stats it already maintains, and queues anything over range_max_bytes. That's deliberately asynchronous, because a split isn't a local operation. It's a distributed transaction that carves the keyspace in two, writes a new range descriptor, and updates the meta ranges that tell the rest of the cluster where keys live. You don't want that on the hot path of a Put. So there's always a gap between "this range is too big" and "this range has been split", and under normal load, the queue closes it in seconds. Backpressure is what stops a range from outrunning the queue when it doesn't. At twice range_max_bytes, the KV layer stops letting writes into a range with a split pending: range_max_bytes 536,870,912 (512 MiB) backpressure at 1,073,741,824 r725 1,073,844,534 It doesn't reject them outright. It holds the batch, waiting for the range to come back under the threshold, and the write fails only when the request runs out of time. That distinction is why the failure surfaced to us as persist timeouts rather than as a clean error, and it's the whole design assumption: the split you're waiting on is going to happen. 100 KiB over the line. And the split was never going to happen. I sampled the version count twice, 25 seconds apart, to be sure writes were genuinely frozen rather than merely slow. 6,766 both times. While wedged, the range produced about 390 log lines every 15 minutes, continuously, because failed persists retried with no backoff. Which leaves GC as the only thing that could end it, and GC runs on a queue too, with the same asynchronous, scored shape as the split queue. Each replica tracks a statistic called gc_bytes_age, the volume of collectable garbage multiplied by how long it's been collectable, and the queue prioritises by that rather than by raw size. When it gets to a range it computes a threshold of now - gc.ttlseconds, drops every version older than that, and advances the range's own GC threshold so that later reads below it are refused rather than served wrong. Two things follow from that shape. GC can never reach anything inside the TTL window, so during a wedge a range can only shed what has already aged past it. And because the queue is scored and periodic rather than continuous, recovery begins when the queue reaches the range, not when the first version becomes collectable. The second of those is the part I can't fully account for. Getting back under the line needed almost nothing, since the range was sitting 100 KiB over a 1024 MiB threshold, and yet writes stayed refused for 75 to 90 minutes every time. Aging alone doesn't explain a gap that size, so what dominates it has to be when the GC queue got round to the range. I never pinned that down more precisely, and the incident was resolved before it mattered enough to. The cycle itself is legible enough without it. Roughly two hours of rewriting to rebuild a gigabyte, then the wedge, then GC clears it and it starts over. Five times across two days, always the same row. And gc.ttlseconds is a floor on retention rather than a target. Retained bytes are write rate times version size times that window, and nothing in the system pushes back on the product. Raft never failed in any of this. No quorum loss, no elections, nothing. But it sits underneath every part of it, and it's the reason the size limit exists at all. A range isn't a storage bucket. It's a Raft group: three replicas by default, one of them holding the lease. Every write to that row was a Raft proposal, which the leaseholder proposed, a quorum accepted, and each replica then applied to its own copy. So those 6,766 versions weren't 6,766 disk writes. They were 6,766 rounds of distributed consensus, each shipping a full 155 KiB snapshot across the network, and the gigabyte existed three times over, once per replica. Size matters to Raft in two more places. A replica that falls far enough behind can't be caught up from the log, because the leader has already truncated the entries it would need, so it gets sent a Raft snapshot instead: the entire range, over the network. Same story when a node is decommissioned and its replicas are rebuilt elsewhere. A 1 GiB range is a 1 GiB transfer, and until it lands that replica isn't contributing to quorum. Keeping ranges small is what keeps rebalancing and recovery cheap enough to happen automatically. The split is a Raft operation too, committing a new range descriptor through this same group. None of that got as far as running. It failed at the first step, choosing the key to cut at. So the rule that trapped us exists to keep Raft groups small enough to move around, and we'd built one that could never be divided. One note if you're coming from Postgres. This isn't a page split. The storage engine is Pebble, an LSM tree, so there are no pages and no fillfactor to tune. Splitting a range is a decision about distribution across a sorted keyspace, not about storage layout. The page-split intuition is the obvious one to reach for and it doesn't transfer. Where the writes came from Two thousand consecutive pushes for the wedged document: payload size min 130,009 median 130,009 max 130,009 distinct clients 1 inter-write gap p50 2.24s Not one byte of variance across any of them. One client, sending the whole document every 2.2 seconds, unchanged. The row's lifetime write counter was at 41,201, which at that cadence is about 25 hours of continuous pushing, and lines up with the first wedge the previous afternoon. Someone left a tab open. On the client, the checkpoint gate asked "did any command run during this dispatch?" instead of "did the document change?". A layout loop that measures rendered block heights and reports them back kept producing command work, so it kept re-exporting and re-sending the entire document. On the server, nothing compared the incoming bytes against what was already stored, so each one landed as a fresh 155 KiB version of an identical document. Five ways out, in the order we considered them 1. Raise range_max_bytes First thing suggested, first thing rejected. It moves the ceiling for every range in the table and does nothing about the accumulation. The ceiling here comes from storing one document per row, not from that number being too small. 2. Lower gc.ttlseconds What we actually did, because it needed no deploy: ALTER TABLE resources CONFIGURE ZONE USING gc.ttlseconds = 600; Retained bytes are write rate times version size times retention window. We couldn't touch the write rate without shipping code, so we took the window from four hours to ten minutes. That's 155 KiB × 0.47/s × 600s, or about 43 MiB, against 1024 MiB before. One range went from 728 MiB to 95 MiB in two minutes. Another went from 579 MiB to 168 MiB. gc_bytes_age on the first fell from 7.2e12 to 3.6e10. That speed deserves a note, because deleting from an LSM frees nothing immediately. Pebble writes deletion markers and the space comes back at compaction, whenever that happens to be. But the size the split queue reads is the MVCC stats, not the disk footprint, and GC updates those the moment it runs. So the range stopped counting as oversized well before it stopped occupying the bytes, which is the only reason a one-line config change unwedged production in minutes. This settles at a steady state rather than counting down to anything. Versions arrive and expire at the same rate, so the pile reaches a size and stays there. I checked system.protected_ts_records first, and it's worth being precise about which direction that check runs in. A protected timestamp pins the GC threshold: while one is held, GC cannot collect anything newer than it, which is how a backup or a changefeed keeps its reads valid for as long as it takes to finish. So a record sitting on this table wouldn't have been the thing at risk. It would have defeated the fix. GC would have refused to drop below it, the range would never have drained, and prod would have stayed wedged with a config change applied and nothing to show for it. The table was empty. What an empty table doesn't tell you is whether anything was reading historically without taking a protected timestamp. Those are the consumers this actually breaks, and after the change an AS OF SYSTEM TIME read further back than ten minutes on that table fails outright. 3. Ship the coalescing persist queue There was already an open PR for one: 750 ms debounce, 5 second ceiling. A 5 second ceiling caps a continuously-edited document at 2,880 versions per GC window, which moves the wedge threshold to roughly 364 KiB. The wedged document was 155 KiB on disk, so 364 KiB sounds like room. It isn't much, and there are two thresholds here rather than one. 364 KiB is where a document wedges. 182 KiB is the earlier one, where its range crosses range_max_bytes and starts attempting splits it can't finish, without being backpressured yet. Another worksheet had already reached 188 KB and was still climbing, which puts it past the first line and heading for the second. So this buys headroom without removing the ceiling. It also doesn't compare content, so identical resends still get written, only less often. I'm not actually sure it would have prevented this one. All of that assumes the 5 second ceiling binds, but at a 2.2 second arrival rate the 750 ms debounce expires between pushes, so each push probably still flushes on its own and the rate doesn't move at all. I haven't tested it. 4. Hash the snapshot, skip the write when it matches The real fix, and the one that shipped. It takes almost all of these writes to zero no matter what any client does, and it holds for whatever version of the client happens to be running, which matters when your clients are browser tabs you can't force to reload. 5. Chunk the snapshot, or move to an append-only update log The only option that removes the single-key property instead of buying room underneath it. Also the largest change by a wide margin, and not something you do on a Wednesday afternoon with prod wedging every few hours. The monitoring None of our alerts could have caught this, and it wasn't a threshold problem. OOMKilled, MemoryHigh, CpuThrottled, CrashLooping, Down. Every one of them stays quiet on a service that is perfectly healthy and simply not permitted to make progress. We had no signal for that shape of failure. CockroachDB was already exporting exactly the right counter. queue_split_process_failure goes up on every failed split attempt, and healthy clusters don't fail splits, so rate(queue_split_process_failure[15m]) > 0 is about as clean a signal as you get. The node holding the lease for r725 was at 2,114. Nothing was scraping it. The rule I then wrote on top of it failed twice over. Prometheus evaluated it and it went active, but Alertmanager dropped the notification on the floor: the default receiver is null behind an allow-list regex on alert names, and a new name that isn't in that regex gets discarded without a trace. So it fired and nobody heard it. Any new rule here needs an entry in that regex or its own route, which is not a thing you find out by testing the expression. Once it was routed, it stayed active for about 25 minutes after the condition cleared. Not because counters only go up, which was my first guess: rate() does return to zero. It's that the 15 minute lookback keeps the rate positive until the window slides past the last failed split, and Alertmanager's resolve delay adds the rest. Shorten the window and you trade that against missing sparse failures. The one I'd hand to someone else is the dedup counter we added afterwards, which counts writes skipped because the content hash matched what was stored. It started out near the entire write volume. That means a fall toward zero is the direction that should worry you, because it says writes have gone back to being genuinely distinct and the ceiling is live again. Every other metric on that dashboard alarms upward. Three CockroachDB quirks worth knowing before you go looking, none of them well documented: crdb_internal.tables keys on table_id, not id. SHOW ZONE CONFIGURATION wants the real database name, and these tables live in defaultdb. And round() errors out if you mix decimal and float8, so cast first.