kill -9, in a loop

The engine claims an acknowledged write is on disk. There is exactly one way to find out, and it is not a unit test.

Every layer so far has been tested the polite way: call the function, assert on what it returned, drop everything cleanly. None of that touches the claim the engine actually makes, which is about what happens when nothing gets to finish.

So: a separate binary that does nothing but write, and a test that kills it without warning.

The writer

src/bin/crash_writer.rs takes a data directory and a value size, then writes keys in a tight loop, forever, printing each key only after the write has been acknowledged. It never exits on its own and never cleans up. It exists to be murdered.

The test

Spawn it, let it get going, then SIGKILL. Not SIGTERM — there must be no handler, no unwinding, no destructor, no final flush. The process stops mid-syscall, exactly the way it would if someone tripped over the cable.

Then reopen the directory and check the invariant: every key the writer printed must still be readable. Not most. Every one. Anything it printed, the engine had already said was safe.

What is explicitly not checked is the last key or two that were in flight when the kill landed. Those were never acknowledged, so losing them is correct behaviour. A test that demanded them would be testing a promise the database never made.

What it catches

This is the shape of test that would have caught a torn record — a crash halfway through writing a length prefix, leaving four bytes that claim a key longer than the rest of the file. Replay has to treat a short read as end-of-log and stop, not panic. Doing it in a loop, at different moments, is how you land the kill in the interesting places.

It is also the test that the flush()-versus-fsync() bug would have walked straight past, and it is worth being honest about why: SIGKILL does not clear the page cache. The kernel still has the bytes. To actually test power loss you need the machine to lose power, or a filesystem that will lie to you on demand. This harness tests process death, which is the weaker claim — it is just the strongest one you can check from inside a test suite.