8% Utilized, 100% Billed: Fixing Kubernetes Overprovisioning Without Breaking Production
Developers had learned, reasonably, that an OOM-killed pod at 2am is a worse outcome than an inflated cloud bill - so requests crept upward every incident, and nobody ever walked them back down.
What we were solving
Mid-market SaaS platform team running multiple production Kubernetes clusters across staging and prod, cloud spend rising steadily with no corresponding growth in real traffic, no dedicated platform-cost function.
- Resource requests had been set defensively after past incidents and never revisited - actual utilization sat far below what was reserved and billed.
- Staging and test environments were provisioned like production and then forgotten, quietly running 24/7 at production-grade sizing.
- Engineers had no incentive to right-size their own workloads - the cost signal never reached the team that controlled the request values.
What we built
- Instrumented real per-pod utilization against reserved requests across every namespace to find the actual gap, not an estimate.
- Rolled out vertical rightsizing recommendations with a canary-first rollout - one namespace at a time, watched against real incident and latency data before wider rollout.
- Put a live cost-per-namespace view in front of the teams that owned the workloads, so the request values they set were now visibly tied to a number they cared about.
Architecture notes for your engineers
- Rightsizing changes were staged behind the same canary/rollback tooling used for application deploys, so a bad recommendation could be reverted exactly like a bad code change.
- Non-production environments were moved to scheduled scale-to-zero outside working hours as a separate, lower-risk first win.
Key results
- Cluster spend reduced by over a third within one quarter
- CPU utilization more than tripled with zero production incidents from rightsizing
- Non-production environments no longer run at full size around the clock
- Cost-per-namespace visibility now sits with the teams that actually control it
What it was built on
Representative tools and patterns — exact vendors vary per client environment.
Cost & utilization
Rollout safety
Platform
What we'd tell the next team
- Overprovisioning is a trust problem before it's a technical one - engineers pad requests because the cost of underprovisioning (an incident) is more visible to them than the cost of overprovisioning (a bigger bill).
- Canary the rightsizing the same way you'd canary a deploy - the fastest way to lose the team's trust is a rightsizing recommendation that causes the next incident.
- Put the cost number in front of the people who set the request values, or the fix reverts to defensive padding the next time something breaks.
Questions this engagement anticipated
Why do Kubernetes clusters end up so overprovisioned?
Engineers set resource requests defensively after incidents and rarely revisit them, since the cost of underprovisioning - a pod getting OOM-killed - is far more visible to them than the cost of overprovisioning a larger cloud bill. The incentive only points one direction unless someone corrects it.
How do you rightsize without causing new incidents?
By rolling changes out canary-first, one namespace at a time, watched against real latency and error data with automated rollback - the same discipline used for application deploys, not a single cluster-wide sweep.
This is one of several case studies on cloud posture, cost & devsecops at scale.
See the rest of the cluster →Compare your situation to this case.
Bring your constraints - environment, timeline, and budget. We scope before we quote.