Autoscaling on custom metrics

By default an application scales on CPU and memory. Every cluster also runs Prometheus and the Prometheus adapter, so an application can scale on its own metrics instead: requests per second, queue length, job latency and so on.

Both settings below go in the application's deployment template in CI/CD (Devtron).

1. Let Prometheus scrape the metric

Expose the metric from your application in Prometheus format, then add a monitor to the deployment template's rawYaml. The release: prometheus-stack label is what makes the cluster's Prometheus pick it up.

Scrape every pod directly with a PodMonitor:

rawYaml:
  - apiVersion: monitoring.coreos.com/v1
    kind: PodMonitor
    metadata:
      name: demo-app-monitor
      labels:
        release: prometheus-stack
    spec:
      selector:
        matchLabels:
          app: demo
      podMetricsEndpoints:
        - port: app
          path: /metrics

Or scrape through the application's Service with a ServiceMonitor:

rawYaml:
  - apiVersion: monitoring.coreos.com/v1
    kind: ServiceMonitor
    metadata:
      name: web-app-monitor
      labels:
        release: prometheus-stack
    spec:
      namespaceSelector:
        matchNames:
          - staging
      selector:
        matchLabels:
          app: web-app
      endpoints:
        - targetPort: 5000
          path: /metrics
          interval: 15s

Check that the metric arrives in Grafana (the address is on the cluster's Overview tab) before you scale on it.

2. Scale on the metric

In the template's autoscaling section, turn off the CPU and memory targets and add the metric under extraMetrics.

A per-pod metric. Add replicas when the pods average more than 1,000 packets per second:

autoscaling:
  enabled: true
  MinReplicas: 1
  MaxReplicas: 10
  TargetCPUUtilizationPercentage: 0
  TargetMemoryUtilizationPercentage: 0
  extraMetrics:
    - type: Pods
      pods:
        metric:
          name: packets_per_second
        target:
          type: AverageValue
          averageValue: 1k

A metric of another object. Scale background workers on the job latency reported by the web Service, adding replicas while it is above 600 seconds:

autoscaling:
  enabled: true
  MinReplicas: 1
  MaxReplicas: 5
  TargetCPUUtilizationPercentage: 0
  TargetMemoryUtilizationPercentage: 0
  extraMetrics:
    - type: Object
      object:
        metric:
          name: worker_queue_latency
        describedObject:
          apiVersion: v1
          kind: Service
          name: web-app-service
        target:
          type: Value
          value: "600"

Deploy the template. Kubernetes then adds and removes pods to keep the metric at the target. When the new pods do not fit on the existing nodes, the cluster autoscaler adds nodes, up to each node group's maximum.

Tips

• Keep MaxReplicas within what your node groups' maximum sizes can hold.
• A counter that only goes up (a name ending in _total) is exposed to the autoscaler as a per-second rate, under the same name without _total.
• If the pod count never changes, check that the metric has namespace and pod labels in Prometheus. The adapter needs them to match the metric to your pods.