Contents
Jepsen’s analysis of ParticleDB contains a recovery log that is worth reading line by line. After the test harness flipped two bits in a write-ahead log file, the server restarted and logged a checksum mismatch in a segment tail, said it was “replaying valid prefix of already-sealed corrupt WAL segment”, reported recovered_entries=242 and corruption_offset=25184, and then served clients. One key that had read back 44 elements before the restart now returned [81]. Several other keys lost acknowledged writes. No client saw an error.
The bug is not the corruption. Disks and kernels corrupt data, and that was the point of the fault injection. The bug is the recovery policy: it treated a damaged sealed segment the way a database should treat a torn tail after a crash.
A torn tail and a sealed segment mean different things
After a crash, the last record in the newest WAL segment may be half written. The segment was never sealed, the write was never acknowledged as durable, and truncating at the first bad record loses nothing a client was promised. Stopping at the first checksum failure and discarding the rest is correct there.
A sealed segment is different. It was closed after its contents were flushed, and later segments exist after it. If a checksum fails inside it, the cause is not an interrupted write. It is media corruption, a misdirected write, or a bug. Everything after the bad record, in that segment and in the later ones, may contain commits that were acknowledged. Replaying “the valid prefix” and moving on turns a detectable fault into silent loss of committed transactions.
This is my reading of the log, not Jepsen’s wording, and the report does not publish ParticleDB’s recovery code. The log line itself says the segment was sealed and corrupt, so the server had the information needed to tell the two cases apart and chose to continue.
The vendor says 1.1.19-rc.12 now detects bit flips and refuses to start. Jepsen did not test that build, so treat it as a claim.
The misdirected-write case was worse
For issue #112, the harness shuffled 13 four-kilobyte chunks, about a tenth, of the first .wal file. After restart the server logged “No data found in storage, starting fresh” and answered “table not found” to every query. The WAL file was later zero bytes.
Starting fresh is the right behavior for an empty data directory and the wrong behavior for a directory that contains a WAL which failed validation. Those are two states, and a recovery path that cannot distinguish them will eventually initialize over data someone wanted. The report does not say whether checksums flagged the shuffled chunks, so I can only say the outcome was total loss with a normal-looking startup.
The practical rule is small. Recovery should have exactly three outcomes: clean, truncated at an unsealed tail with the truncation logged, or refused with a nonzero exit. If your engine has a fourth, such as “replayed what it could,” find out what triggers it.
Isolation was a startup flag, not a SQL statement
The second thing that matters for anyone using a Postgres-compatible engine is where isolation is configured. According to the report, ParticleDB ignored or rejected SET TRANSACTION ISOLATION LEVEL. In some cases it accepted other levels and “silently ignored them (despite documentation to the contrary)”. The level was chosen by a --txn-mode flag at server start, and the documentation did not say what each mode guaranteed.
Application code written against Postgres assumes the opposite: the session or transaction asks for SERIALIZABLE and either gets it or gets an error. An engine that accepts the statement and does nothing breaks that contract without any signal. Orkes’s unpublished operator guide, which Jepsen quotes, mapped the modes to claimed models, and Jepsen’s results against those claims were:
| Mode | Claimed model | Observed |
|---|---|---|
occ-v3 | Snapshot Isolation | Lost writes, G-single, internal anomalies |
zero-copy-occ | Snapshot Isolation | Lost writes, G-single |
serializable-ssi | Serializable | Lost writes |
table-2pl | Strong Serializable | Read skew, lost writes, deadlocks |
strict-serializable (rc-18) | Strict serializable | Stale reads, internal anomalies with process kills |
The lost-write finding (#110) was in healthy single-node runs, with no faults, using an append workload and the strictest WAL sync mode. About 31 of roughly 20,000 keys lost acknowledged writes, and on one key the list 1, 2, 3 was replaced by a list starting 4, 5, 6, 7. The report says it happened every few minutes in the default occ mode in rc-22.
Vendor claims against measured behavior
The report is explicit about the gap. The home page said “Full ACID. 159K TPS.” and “formally proven” with TLA+, with no specification provided. The docs disagreed with each other about whether sync or groupsync was more durable. Jepsen also notes that Orkes funded the research, and that after testing Orkes labeled the transaction modes “experimental”, which the documentation had not said during the engagement.
The throughput figure is a vendor number. Nothing in the report measures it, and with lost acknowledged writes it would not be a meaningful comparison anyway.
Jepsen counts 96 issues and says all but three occurred in healthy single-node instances, often with one client and no faults. It did not test replication, since the builds provided did not include it, and it has not tested any 1.1.19 build. Orkes says 1.1.19-rc.13 resolves the remaining issues. That is also unverified.
What to take from it
You do not run ParticleDB, probably. The reusable parts are checks you can apply to any engine that stores its own log:
- Find where recovery distinguishes an unsealed tail from a sealed segment. If it does not, a mid-log checksum failure is probably handled as a tail.
- Find the code path that prints “starting fresh” and confirm it requires an empty data directory.
- If you use a Postgres wire-compatible database, send
SET TRANSACTION ISOLATION LEVEL SERIALIZABLEand check that the engine either honors it or errors. Then read what the docs say about how isolation is actually selected. - Treat a database that lets you inject faults at the file level as a feature. Jepsen’s setup used process kills, LazyFS to drop unsynced writes, file truncation, bit flips and shuffled 4 KB chunks. The same categories are cheap to run against a staging instance of whatever you operate.
Sources
- Jepsen: ParticleDB 1.1.4 to 1.1.18.11.rc-22. All log excerpts, issue numbers, counts and vendor responses above come from this report. I read most of it; the final portion, covering type safety, arithmetic and strings, I did not read.
- The WAL torn-tail versus sealed-segment distinction is my own interpretation and is not stated in the report.