Karpenter for Kubernetes Autoscaling — Node Provisioning, Spot Instances, and Cost Optimization

Nodes on Demand, Not by Guesswork

The classic Kubernetes scaling story has two halves. The Horizontal Pod Autoscaler adds pods; the Cluster Autoscaler adds nodes. The catch is that the Cluster Autoscaler thinks in terms of node groups — pools of identical instances you sized in advance. When a pod cannot schedule, it grows a group by one, whatever instance type that group happens to contain.

That works, but it leaves money and time on the table. You pre-declare instance types, you run several groups to cover different shapes of workload, and you wait for the group to react. Bin-packing across a fleet of fixed pools is somebody else's job.

Karpenter takes a different route. It watches for unschedulable pods, reads their exact resource requests and constraints, and provisions a right-sized node — often a single instance type it chose from hundreds — directly through the EC2 API. No node group in the middle. When the node is no longer needed, Karpenter removes it. It is a continuation of the cost-optimization work most platform teams are already doing, but automated at the node layer.

How It Differs from Cluster Autoscaler

The mental shift is from groups of nodes to constraints on nodes. You stop enumerating instance types and instead describe the boundaries — architectures, families, zones, capacity types — within which Karpenter is free to choose.

• Groupless. No Auto Scaling Groups or EKS managed node groups to size. Karpenter launches instances directly.
• Flexible. One NodePool can span dozens of instance types; Karpenter bin-packs pending pods and picks the cheapest instance that fits.
• Fast. It reacts to pending pods and calls the EC2 fleet API directly, rather than nudging an ASG desired-count and waiting.
• Consolidating. It actively repacks workloads onto fewer or cheaper nodes when utilization drops.

You still keep a small managed node group for Karpenter itself (and other cluster-critical add-ons) so the controller has somewhere stable to run. Everything else can move to Karpenter-managed capacity.

Installing Karpenter on EKS

Karpenter ships as a Helm chart on the public Amazon ECR registry. Before installing you need the supporting AWS plumbing: an IAM role for the controller (via IRSA or EKS Pod Identity), an instance role for the nodes it launches, and an SQS interruption queue with EventBridge rules. The getting-started guide provisions all of this with CloudFormation and eksctl; for repeatable environments, wire the same resources into your Terraform modules.

export KARPENTER_NAMESPACE="kube-system"
export KARPENTER_VERSION="1.13.0"
export CLUSTER_NAME="prod-eks"

# Public ECR is an unauthenticated pull — log out of any stored creds.
helm registry logout public.ecr.aws || true

helm upgrade --install karpenter \
  oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace "${KARPENTER_NAMESPACE}" --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.interruptionQueue=${CLUSTER_NAME}" \
  --set controller.resources.requests.cpu=1 \
  --set controller.resources.requests.memory=1Gi \
  --set controller.resources.limits.cpu=1 \
  --set controller.resources.limits.memory=1Gi \
  --wait

Karpenter runs in kube-system by convention. If your account has never used EC2 Spot, create the service-linked role once so instance launches do not fail with ServiceLinkedRoleCreationNotPermitted:

aws iam create-service-linked-role \
  --aws-service-name spot.amazonaws.com || true

The NodePool: Your Provisioning Constraints

A NodePool (karpenter.sh/v1) defines the shape of the nodes Karpenter is allowed to create. The requirements block uses well-known labels to bound instance selection; the disruption block controls when nodes get removed. It references an EC2NodeClass through nodeClassRef.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        team: platform
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      # Absolute max node lifetime — forces regular AMI/security refresh.
      expireAfter: 720h
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["4"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
  limits:
    cpu: "1000"
    memory: 1000Gi
  weight: 10

A few of these fields carry real operational weight. limits is your circuit breaker — Karpenter stops provisioning once the pool's aggregate CPU or memory hits the cap, which protects you from a runaway workload spinning up a fortune in instances. weight orders multiple NodePools when several could satisfy a pod (higher wins). expireAfter caps node age so nodes are recycled onto fresh AMIs on a schedule.

The EC2NodeClass: AWS-Specific Node Shape

Where the NodePool is cloud-neutral, the EC2NodeClass (karpenter.k8s.aws/v1) holds the AWS specifics: which AMI family to boot, which subnets and security groups to attach, the node IAM role, tags, and disk layout. Subnets and security groups are discovered by tag or ID rather than hard-coded, so the same manifest survives infrastructure changes.

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023
  amiSelectorTerms:
    - alias: al2023@latest
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  tags:
    team: platform
    managed-by: karpenter
  blockDeviceMappings:
    - deviceName: /dev/xvda
      ebs:
        volumeSize: 100Gi
        volumeType: gp3
        encrypted: true
        deleteOnTermination: true

The alias form of amiSelectorTerms pins an AMI family and version stream (here, the latest Amazon Linux 2023). When AWS publishes a new AMI in that stream, Karpenter detects the change as drift and gracefully rolls nodes onto it — more on that below. Valid families include AL2, AL2023, Bottlerocket, Windows2019/2022/2025, and Custom.

Spot Instances the Right Way

Spot capacity is where Karpenter earns its keep. The karpenter.sh/capacity-type requirement accepts spot, on-demand, and reserved. When more than one is allowed, Karpenter prioritizes reserved first, then spot, then on-demand — so listing all three gives you the cheapest available capacity with automatic fallback.

For fault-tolerant, stateless work — batch jobs, CI runners, and the kind of Spark executors that survive node loss — a spot-first NodePool is the default. Keep a separate on-demand pool, selected by a higher weight or a node selector, for stateful or latency-critical services that should not be reclaimed.

# Spot-only pool for interruptible batch workloads.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: spot-batch
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
      taints:
        - key: workload-type
          value: batch
          effect: NoSchedule
  limits:
    cpu: "2000"
  weight: 1

Karpenter uses a price-capacity-optimized allocation strategy for Spot: it weighs both price and the depth of available capacity, which lowers the chance of interruption compared with a pure lowest-price strategy. Give it a broad instance selection and it has more pools to fall back on.

Consolidation: Shrinking the Bill

Provisioning is only half the value; consolidation is the other. Karpenter continuously looks for ways to run the same pods on fewer or cheaper nodes, evaluating three mechanisms in order:

• Empty node consolidation — nodes with no workload pods are deleted, in parallel.
• Multi-node consolidation — two or more nodes are removed together, possibly launching a single cheaper replacement.
• Single-node consolidation — one node is replaced by a cheaper instance that still fits its pods.

consolidationPolicy: WhenEmptyOrUnderutilized enables all three; WhenEmpty limits it to genuinely empty nodes, which is the conservative choice for workloads sensitive to rescheduling. consolidateAfter is the cooldown Karpenter waits after a pod schedules or leaves before treating a node as a consolidation candidate — raise it to reduce churn on bursty clusters.

Drift: Keeping Nodes in Spec

Drift is how Karpenter keeps running nodes aligned with their declared configuration. It stores a hash of the NodePool and EC2NodeClass template on each NodeClaim; when the hash no longer matches — you changed the AMI alias, a security group, or block device settings — the node is marked Drifted and replaced gracefully.

The most useful case is AMIs. Point amiSelectorTerms at a version stream and a newly published AMI triggers drift, so security patches roll through the fleet without a manual node cycle. Behavioral fields — spec.weight, spec.limits, and the spec.disruption block — are explicitly excluded from drift, since changing them should not force node replacement.

Interruption Handling

AWS can reclaim a Spot instance with a two-minute warning. Karpenter listens for that warning — plus scheduled change events, instance stop/terminate events, and status-check failures — through the SQS interruption queue you configured at install time via settings.interruptionQueue. EventBridge forwards the events; Karpenter taints, drains, and terminates the affected node while launching a replacement in parallel.

Two minutes is enough for graceful shutdown of most stateless services, but not for everything. Design workloads to checkpoint or drain quickly, and use a PodDisruptionBudget so a wave of Spot reclamations cannot take down every replica at once. For a broader view of failure signals, pair this with your SLO and alerting strategy so interruption spikes surface before they become incidents.

Disruption Budgets: Rate-Limiting Change

Consolidation, drift, and emptiness are voluntary disruptions, and you can rate-limit them with spec.disruption.budgets. The default is a single budget of nodes: 10%. You can express budgets as a percentage or a fixed count, scope them to specific reasons (Drifted, Underutilized, Empty), and gate them to a cron schedule with a duration window.

  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      # Never voluntarily disrupt more than 10% at once.
      - nodes: "10%"
      # Freeze all voluntary disruption during business hours (UTC).
      - nodes: "0"
        schedule: "0 9 * * mon-fri"
        duration: 8h
        reasons: ["Underutilized", "Drifted"]

When several budgets apply, the most restrictive wins. One caveat worth internalizing: budgets only bound voluntary disruption. Forceful actions — expiration via expireAfter, Spot interruption, and node auto-repair — are not rate-limited by budgets, because they are reacting to something that is going to happen regardless.

Watching What Karpenter Does

Karpenter exposes Prometheus metrics and emits Kubernetes events on every provisioning and disruption decision. When capacity behaves unexpectedly, the events on a NodeClaim usually tell you why a node was created or removed.

# Why did Karpenter make a decision? Read the events.
kubectl get nodeclaims
kubectl describe nodeclaim <name>

# Controller logs — provisioning and disruption reasoning.
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter -f

Scrape the controller's metrics endpoint and alert on nodes stuck pending, sustained Spot interruption rates, and NodePools hitting their limits — the last one silently starves scheduling once reached.

Production Checklist

• Keep the controller off Karpenter capacity. Run Karpenter on a small managed node group so it can always reschedule itself.
• Set limits on every NodePool. Aggregate CPU/memory caps are your defense against a runaway spend.
• Widen instance selection. More instance types means better bin-packing and lower Spot interruption risk.
• Separate spot and on-demand pools. Route stateful and latency-critical work to on-demand via weight or node selectors.
• Add PodDisruptionBudgets. They govern how consolidation and interruption evict your pods.
• Use amiSelectorTerms aliases. Let drift roll security-patched AMIs through the fleet automatically.
• Set expireAfter. Cap node age so nodes never drift far from a fresh, patched image.
• Keep IMDSv2 blocking on. The v1 default hardens node metadata access — leave it unless a workload truly needs it.
• Tune consolidateAfter for bursty clusters. A longer cooldown trades a little cost for far less node churn.
• Read the upgrade notes. v1 was a breaking change from v1beta1; always review the release changelog before bumping versions.

Karpenter changes how you think about cluster capacity: you describe constraints, and it finds the cheapest compliant node in real time, then continuously repacks the cluster to keep it tight. Combined with Spot, consolidation, and disruption budgets, it turns node cost from a quarterly right-sizing chore into a control loop that runs every minute.