Beyond Raw RPS: Building Goodput-Driven Custom Metrics Exporters for Kubernetes HPA
Standard Kubernetes Horizontal Pod Autoscalers rely on blunt resource metrics like CPU or raw request throughput, which fail to capture service quality. When complex or inference-heavy workloads degrade, scaling on raw throughput actually exacerbates latencies by provisioning nodes for failing requests.
At a Glance
The architecture flow from application goodput metrics through the Prometheus adapter to the Kubernetes HPA for intelligent scaling.
flowchart LR
App[Application Logic]:::core --> Metrics[Prometheus Exporter]:::core
Metrics --> Adapter[Prometheus Adapter]:::core
Adapter --> HPA[Kubernetes HPA]:::core
HPA --> K8s[Kubernetes Scheduler]:::core
K8s --> Pods[New Pod Replicas]:::core
Client[Traffic Source]:::external --> App
classDef external fill:#dbeafe,stroke:#2563eb,stroke-width:1px,color:#1e3a8a
classDef core fill:#dcfce7,stroke:#16a34a,stroke-width:1px,color:#14532d
The Failure of Resource-Based Scaling
I remember the first time our cluster folded under a traffic spike. We were scaling on CPU utilization, so as the system hit its limits and requests started backing up, CPU spiked across all nodes. The Horizontal Pod Autoscaler (HPA) saw that spike and responded by spinning up more replicas. Unfortunately, those new pods entered an already saturated environment, competing for the same constrained resources. Instead of fixing the backlog, we just added more participants to the failure, turning a manageable latency issue into a total system collapse.
The lesson here is that resource utilization—CPU, memory, or even raw request-per-second (RPS) counts—is often a lagging indicator of health. When your application is failing or latencies are blowing past your SLA, raw throughput metrics don’t distinguish between a successful transaction and a 500 error. You end up scaling to support “work,” even when that work is just failing faster.
Designing for Goodput
To fix this, we need to shift from measuring throughput to measuring “goodput”—the rate of successful, SLA-compliant transactions. If your service performs inference, a request that hits a timeout or returns an error isn’t a success; it’s a wasted cycle.
The architecture for this involves a custom Prometheus exporter that tracks the application’s internal state rather than just the container’s hardware metrics. You need to expose a metric that increments only when a request meets your success criteria. A simple implementation looks like this:
1
2
3
4
5
6
7
8
9
10
11
12
13
// Example: Tracking successful requests in a Go-based service
var successfulRequests = prometheus.NewCounter(prometheus.CounterOpts{
Name: "http_requests_completed_total",
Help: "Total number of requests completed within SLA",
})
func handler(w http.ResponseWriter, r *http.Request) {
start := time.Now()
// ... process request ...
if time.Since(start) < 200*time.Millisecond {
successfulRequests.Inc()
}
}
By exposing this via a /metrics endpoint, you provide the HPA with a meaningful signal. You are no longer scaling because the CPU is hot; you are scaling because your “goodput” is approaching the capacity limit of your current replica count.
Implementing the HPA Logic
Once your application exports these counters, you need to bridge the gap to the Kubernetes HPA. You can use the Prometheus Adapter to translate that custom http_requests_completed_total metric into a form the HPA can consume.
In your HPA manifest, you define a Pods or Object metric that targets your custom counter. Instead of a CPU target of 80%, you set a target based on the rate of successful requests per pod:
1
2
3
4
5
6
7
8
metrics:
- type: Pods
pods:
metric:
name: http_requests_completed_total
target:
type: AverageValue
averageValue: 50 # Scale when avg successful requests/pod exceeds 50/sec
This configuration forces the cluster to maintain the desired success rate. If the rate of successful transactions per pod drops, the HPA will scale out to compensate. If the system is failing and goodput is low, the HPA won’t incorrectly assume it needs to scale up, preventing the “death spiral” I encountered earlier.
When to Skip the Complexity
Custom exporters require maintenance. You are now responsible for the lifecycle of that metric, including unit testing your success thresholds and managing the Prometheus Adapter configuration. If your service is a simple CRUD app with predictable latency, standard CPU-based scaling is usually sufficient and far easier to debug.
This approach is specifically for high-stakes environments—like LLM inference or complex data processing pipelines—where a failed request is expensive and resource contention is a constant threat. In those cases, the engineering effort required to build a goodput-aware infrastructure is worth the stability you gain.