When 3,951x Is Not a Speedup: Measurement Integrity in Database Benchmarking

A reported 3,951× speedup can be mathematically correct yet technically misleading. In one real example, the figure compared storage rows scanned rather than query latency—two fundamentally different measurements that became conflated into a dramatic performance claim.

Every database engineer has encountered benchmark claims that promise dramatic performance improvements. This session examines five recurring benchmark reporting errors that can make otherwise convincing results misleading:

• Metric conflation – reporting one measurement as evidence of another.
• Trivial workload bias – benchmarking workloads that are too small or simple to represent production behavior.
• Topology substitution – presenting single-node measurements as evidence for distributed systems.
• Scale substitution – extrapolating small-scale experiments to production workloads.
• Stale measurements – publishing benchmark results that no longer correspond to the software version being evaluated.

Using concrete examples drawn from PostgreSQL performance engineering, we’ll explore how these errors arise, why they influence engineering decisions, and how they can be detected before benchmark results are published or relied upon.

For each reporting error, the session introduces a corresponding verification rule, providing a practical framework for producing and evaluating benchmark reports that are reproducible, defensible, and representative of real-world database performance.

Whether you publish benchmarks, review them, or rely on them when selecting technologies, you’ll leave with a practical methodology for deciding whether an impressive speedup is genuinely meaningful—or merely a measurement artifact.