Notes from building a database from scratch — one layer at a time, including the parts that were wrong the first time.
Re-embedding treated as a latency problem: durable state publishes updates to a queue, the hot store ingests at its own pace, and versioned embeddings turn a model change into a cutover rather than a migration.
A durability claim you have not tried to break is a hypothesis. Here's the harness that tries.
A log, a memtable and a stack of files are three things. A read has to make them look like one.
Entries, then an index, then eight bytes at the very end that tell you where the index starts.
The log was appending correctly, replaying correctly, and passing its tests — and would still have lost data on power loss.
BTreeMap gets you 90% of a memtable. The last 10% is a three-state answer that Option can't express.
The design post that started it — LSM-trees, write-ahead logs, Raft consensus and multi-region sharding. 500,000 impressions, and conversations with engineers at Netflix, Airbnb, Google and Amazon.
Four bytes of length, then the key, then four bytes of length, then the value. And one number that means "this was a delete".
One rule for the whole project: no crates, in the database or its tests. Here is what that rule is for.