Beyond Raw RPS: Building Goodput-Driven Custom Metrics Exporters for Kubernetes HPA
Standard Kubernetes Horizontal Pod Autoscalers rely on blunt resource metrics like CPU or raw request throughput, which fail to capture service quality. When complex or inference-heavy workloads degrade, scaling on raw throughput actually exacerbates latencies by provisioning nodes for failing requests.