July 2024 to now
Delhi
DevOps Engineer at CogniAI
I'm the only infrastructure engineer. Everything between a merged pull request and a running service is mine: the clouds, the clusters, the pipelines and the bill.
PlatformCloud and cost
What I did
- Moved production from AWS to GCP. EKS to GKE and RDS to Cloud SQL, service by service, with the old side running until the new one was proven. The company now depends less on one vendor and has less to run long term.
- Cut infrastructure cost by 30%. Used Kubecost to find idle capacity across EKS clusters, EC2 instances and VPCs, then right-sized or removed it.
- Put every environment in Terraform. Dev, staging and production come from the same modules, so they stop drifting apart and nobody provisions anything by hand.
- Built GitOps delivery. GitHub Actions builds, ArgoCD deploys, and blue-green releases mean a bad version rolls back in one step.
- Set up observability. OpenTelemetry traces, SigNoz dashboards and Slack alert bots, so the team sees a problem before a customer reports it.
Case study: one GKE cluster
The cluster ran on three large nodes at 11 to 14% CPU. Pods reserved 16.1 cores and used about 3.2, and about 25 of them, the ingress included, had no resource requests at all. Kubecost put a third of the node spend down to idle capacity.
The goal was to cut cost as far as possible while keeping room to autoscale at peak, with no production outage during the move.
- Set requests for about 30 apps from 11 days of real usage: CPU at two to three times the average, memory at the peak plus 15%.
- Added a disruption budget to every workload, going from 3 to 63, so draining a node could never take down every copy of a service.
- Replaced the fixed pool with autoscaling pools: a steady on-demand pool, plus a Spot pool that falls back to on-demand when Spot runs out.
- Kept customer-facing apps off Spot. A Spot reclaim gives about 30 seconds of warning, and those apps take three minutes to become ready.
- Moved the apps over one group at a time, drained the old nodes against the budgets, and deleted the old pool.
| Measure | Before | After |
|---|---|---|
| Monthly bill | $922 | $630 |
| CPU reserved | 16.1 cores | 7.6 cores |
| vCPUs paid for | 24 | 14 |
| Node CPU use | 11 to 14% | 24 to 38% |
| Room to scale | 3 to 6 nodes, one pool | 3 to 12 steady, 0 to 9 Spot |
| Downtime during the move | none |
Measured with kubectl, Kubecost and the GCP bill.
The usage data also turned up problems that had nothing to do with cost. I fixed them before shrinking anything:
- A customer-facing API was running at 199% of its CPU target and stuck at its replica limit. It now sits at about 30%.
- The ingress couldn't autoscale at all, because it had no requests to measure against. It scales normally now.
- An ML inference service was peaking at 1730 MiB against an 1800 MiB limit. It got headroom and hasn't been killed for memory since.
- A forgotten service had been crash-looping for 40 hours. Nothing used it, so I deleted it.
- Kubecost itself cost about $51 a month to run, and Managed Prometheus was adding about $118 in network charges. Managed Prometheus is now off, and OpenCost is replacing Kubecost.
Tools
AWS, GCP, GKE, EKS, Cloud SQL, RDS, Terraform, Kubernetes, Helm, ArgoCD, GitHub Actions, OpenTelemetry, SigNoz, Kubecost, Traefik, NGINX