When Your Benchmarks Argue Against You

Every engineer has been asked to prove a platform change was worth it, using numbers nobody fully trusts.

I’m a product manager with no engineering background, and I set out to prove our new MySQL and PostgreSQL platform beat the one we already run. The benchmarks argued back. So we dug in: did we have a real problem, a broken benchmark, or just a bad assumption? Lucky for us; all three! This is a lesson in what casual testing hides, why the ugly result is usually the valuable one, and how a benchmark that looked grim at first became the evidence we needed. We came out with answers rather than targets.

This isn’t a benchmarking tutorial. It’s about making sure your numbers survive the next customer pitch or executive review.