A War Story of Survival: Taming Memory Spikes for Percona Server for MySQL on Kubernetes with Swap

For years, the conventional wisdom in the Kubernetes world has been: disable swap. This stance prioritizes performance predictability, but it creates a brutal environment for stateful workloads like MySQL, especially under unexpected memory spikes. This is a real-world story of a production Kubernetes cluster hosting MySQL . We faced crippling, unpredictable Out-of-Memory (OOM) kills during specific, high-load operations, even with meticulously tuned memory requests and limits. The issue was the inherent memory overheads of the database, and the occasional, unavoidable burst in query memory usage. Our solution was a radical (and now increasingly viable) shift: leveraging the Kubelet’s beta/stable support for swap memory, specifically using the LimitedSwap behavior. This talk will detail our journey: The Crisis: How constant OOM kills threaten stability, despite “correct” configuration. The Investigation: Deep-diving into the memory profile of a Percona Operator-managed PS pod, accounting for the mysqld process, and other essential overhead. The Solution: Implementing Kubelet swap configuration (LimitedSwap on cgroup v2-enabled nodes) and the necessary host-level kernel tuning (e.g., vm.swappiness). The Victory: Demonstrating the concrete, measurable reduction in OOM kills and improved system resilience without catastrophic performance degradation, turning a database from brittle to Burstable Quality of Service (QoS)-resilient.

Swap is reclamation equality, not emergency memory. Without swap, anonymous pages (buffer pool, GR XCom/message cache, certification info, connection buffers) are locked in RAM. Under pressure the kernel can only reclaim file pages, so it thrashes on page cache and code segments instead. Disabling swap doesn’t avoid disk I/O under contention; it just moves it somewhere worse.

Attendees will learn how to safely and effectively enable swap for their own stateful workloads on Kubernetes, providing a critical buffer against memory pressure and significantly improving the operational stability of database-as-a-service deployments.