Recovering a SEV-1 Postgres disaster from a volume Kubernetes had orphaned
A botched rolling restart re-bootstrapped a fresh, empty database over two of three replicas. The only intact copy was a PVC the cluster had orphaned when it scaled down — and we should never have been that lucky.
initdb is supposed to run once, at the beginning of a database’s life. During a rolling restart it ran again — and started writing a fresh, empty database over two of three replicas. By the time the graphs looked wrong, most of the cluster believed the empty copy was the truth.
Why this is the frightening kind of incident
There was no error. Every dashboard was green; the pods were healthy; the pipeline had passed. The system was confidently serving the wrong data. Those are the incidents that end careers — not the loud ones, the quiet ones where the machine is certain and wrong.
What I owned
Finding the one surviving copy — a PersistentVolumeClaim the cluster had orphaned rather than deleted when an old replica scaled away — and getting the data back from it before the healthy-looking emptiness became permanent. Then the unglamorous follow-through: the persistentVolumeClaimRetentionPolicy and statefulset guardrails that mean the next person doesn’t survive on luck.
The honest part
We recovered because of a default we hadn’t chosen on purpose, not because of a plan. I say that plainly in the write-up, because a recovery you can’t reproduce isn’t a capability — it’s a warning. The value a client buys here isn’t heroics on the night; it’s the boring settings that make the heroics unnecessary next time.