Autoscaling Beyond CPU: Custom Metrics, Queue Depth, and Predictive Scaling

I’ve written before about what adding more instances doesn’t actually fix — this is the narrower question of what should trigger adding them in the first place. CPU-based autoscaling is the default in every tutorial, and it’s a reasonable default for exactly one kind of workload: CPU-bound compute where request cost scales predictably with processor time. Most real services aren’t that, and scaling them on CPU alone produces the specific, recognizable failure of infrastructure that “looks fine” on every dashboard right up until it isn’t.

Why CPU Is the Wrong Signal for a Lot of Real Services

An API that spends most of its time waiting — on a database query, a downstream HTTP call, a lock — can be completely saturated from a user’s perspective (every connection slot full, requests queuing, response times climbing) while its CPU utilization sits comfortably at 20%, because waiting isn’t CPU work. A Horizontal Pod Autoscaler watching CPU alone sees a healthy-looking number and does nothing, while actual users are experiencing a service that’s effectively down. This isn’t a rare edge case — it’s the normal shape of most I/O-bound web services, which is most web services.

Custom Metrics: Scaling on What’s Actually Predictive for This Service

Kubernetes’ custom metrics API lets an HPA scale on any metric your monitoring stack already exposes — Prometheus, in the common setup, via the prometheus-adapter bridging Prometheus queries into the format the HPA controller understands:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_in_flight
        target:
          type: AverageValue
          averageValue: "50"   # scale to keep ~50 in-flight requests per pod

This scales on requests actually in flight per pod — a direct measure of “how loaded does this specific pod feel right now” — rather than a proxy metric (CPU) that only correlates with load for certain kinds of work. The right metric is genuinely service-specific: request concurrency for a typical API, queue depth for a worker consuming from a message queue, active WebSocket connections for a real-time service. There’s no universal second metric to replace CPU with — the right one is whatever variable actually predicts this particular service falling over.

Queue Depth: The Case Where CPU Isn’t Even Wrong, It’s Irrelevant

For a worker pool consuming from SQS, RabbitMQ, or Kafka, CPU utilization of the workers themselves says almost nothing about whether more workers are needed — a worker can be pegged at 100% CPU doing genuinely necessary processing with a completely empty queue, or sitting at 5% CPU while messages back up by the thousands because each one waits on a slow downstream call. The metric that actually matters is upstream of the workers entirely:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  scaleTargetRef:
    name: worker
  minReplicaCount: 1
  maxReplicaCount: 50
  triggers:
    - type: aws-sqs-queue
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/.../my-queue
        queueLength: "5"   # target ~5 messages per replica

KEDA is worth naming specifically here because it solves the other half of this problem that a bare HPA can’t: scaling from zero. A queue-driven worker with genuinely no work to do doesn’t need one idle replica burning resources waiting for the next message — KEDA can scale a deployment to zero replicas entirely when the queue’s empty, and back up the moment a message arrives, which a standard HPA (whose minimum is always at least one running replica) structurally can’t do.

Predictive Scaling: Reacting to Load You Can See Coming

Reactive autoscaling — watch a metric, add replicas once it crosses a threshold — has an inherent lag: metrics collection, HPA evaluation interval, then new pod scheduling and startup time, commonly adding up to 1-3 minutes before new capacity actually starts serving traffic. For load with a genuinely predictable pattern — a daily traffic spike at a known hour, a recurring batch job — that lag is avoidable rather than something to just tolerate, by scaling ahead of the pattern instead of in reaction to it:

# scheduled pre-scale, e.g. via a CronJob patching the HPA's minReplicas
0 8 * * 1-5   kubectl patch hpa api-hpa -p '{"spec":{"minReplicas":10}}'
0 20 * * 1-5  kubectl patch hpa api-hpa -p '{"spec":{"minReplicas":2}}'

This is a coarse, low-tech version of what cloud providers sell as managed “predictive scaling” (AWS’s does this from historical CloudWatch patterns automatically) — the underlying idea is the same either way: known, recurring load doesn’t need to wait for a metric to cross a threshold before capacity starts scaling, because the pattern is already known in advance. It’s not a replacement for reactive scaling, which still needs to handle the load the schedule didn’t predict — it’s a floor that removes the startup lag for the load you already know is coming.

The Trap: Scaling on a Metric That’s Itself a Symptom, Not a Cause

Response time (p95 latency) looks like an obviously good scaling signal — it’s literally what users experience — but it’s a lagging indicator that’s already evidence of a problem in progress, not an early warning of one approaching. By the time p95 latency has climbed enough to trigger a scale-up, users have already had a bad experience for however long it took to get there, and the new replicas still need their own startup time on top of that delay. Request concurrency or queue depth are leading indicators of the same underlying pressure — they climb before latency visibly degrades, which is the entire point of scaling on them instead: the goal is to add capacity before the user-facing symptom appears, not in response to it.

The Checklist

• CPU is the right signal for CPU-bound compute and the wrong one for most I/O-bound services — check which kind you actually have before defaulting to it.
• Scale workers on queue depth, not worker CPU — the queue is upstream of the workers and reflects actual backlog, which CPU utilization doesn’t.
• Use KEDA specifically when scale-to-zero matters — a bare HPA structurally can’t go below one replica.
• Pre-scale ahead of genuinely predictable load patterns rather than accepting reactive scaling’s inherent 1-3 minute lag for load you already know is coming.
• Prefer leading indicators (concurrency, queue depth) over lagging ones (p95 latency) — a lagging metric only fires after users have already felt the problem.