Schrödinger's Backup - It's Only Good Until You Try to Restore It

Every DBA has a backup strategy. Far fewer have a tested restore strategy — and the gap between the two only becomes visible at the worst possible moment. Until you actually attempt a restore, your backup exists in a kind of superposition: it might be perfectly fine, or it might be silently corrupt, incomplete, or built on assumptions about cluster state that no longer hold. You won’t know which until you open the box. This talk is about closing that gap for Percona XtraDB Cluster deployments. We’ll look at why “the backup job completed successfully” tells you almost nothing about recoverability, and walk through the PXC-specific failure modes that only surface during an actual restore: quorum loss, node discovery and ProxySQL routing issues after topology changes, and the operational surprises that show up when a cluster comes back up in a state nobody rehearsed for.

We’ll also look at a real-world case that made this uncomfortably concrete: the March 2026 drone strikes on AWS’s ME-CENTRAL-1 and ME-SOUTH-1 regions, which took multiple availability zones offline at once — a failure mode most DR plans quietly assume away. It’s a useful reminder that “highly available” infrastructure can fail in ways that have nothing to do with disks or power supplies, and that redundancy across zones is not the same thing as a tested recovery path.

We’ll cover:

• Why a successful backup job and a successful restore are different claims entirely
• PXC-specific pitfalls when restoring under quorum loss or multi-node failure
• Building a real backup verification and restore-testing process, not just cron jobs and hope
• Operational lessons from PXC incidents where cluster topology changed unexpectedly
• A practical checklist for testing your own backups before you’re forced to find out the hard way