# Karpenter Migration Playbook & Lessons Learned

**Date:** 2026-04-15
**Account:** 594765360358 | **Region:** ca-central-1
**First cluster migrated:** dev-mgmt

## Lessons Learned

### 1. Karpenter Cannot Provision Itself
**Problem:** Draining the only node evicts Karpenter too — then nothing can launch a replacement.
**Solution:** Keep the ASG with min=1 as a system node. Karpenter runs on the ASG node and manages additional workload nodes. Never drain the ASG node completely.

### 2. No Default VPC Breaks Karpenter v1.3.0
**Problem:** Karpenter's dry-run `RunInstances` validation fails with `VPCIdNotSpecified` in accounts without a default VPC.
**Solution:** Upgrade to Karpenter v1.11.1+ which handles this correctly. Do NOT create a default VPC as a workaround.

### 3. Downsize the ASG, Don't Delete It
**Problem:** Deleting the ASG removes the system node that runs Karpenter, CoreDNS, and kube-system pods.
**Solution:** Update the ASG launch template to a smaller instance type (e.g., t3.medium) and keep min=1. Use instance refresh to cycle the node to the new type.

### 4. Use Instance Refresh, Not Manual Termination
**Problem:** Manually terminating ASG instances and adjusting desired count is error-prone.
**Solution:** Update the LT/overrides first, then `start-instance-refresh --preferences '{"MinHealthyPercentage": 0}'` to replace in-place.

### 5. t3.medium Is Too Small for System Node
**Problem:** t3.medium only supports ~6 pods (3 ENIs × 2 secondary IPs). ArgoCD + Linkerd + system pods exceed this.
**Root cause:** AWS VPC CNI assigns one IP per pod from the ENI. t3.medium has 3 ENIs with limited IPs = max ~6 pods.
**Solution:** Use t3.large (3 ENIs × 12 IPs = 35 pods) minimum for system node. Updated LT to t3.large (v8).

### 6. Karpenter IAM Needs ListInstanceProfiles
**Problem:** v1.11.1 added instance profile garbage collection that requires `iam:ListInstanceProfiles`.
**Solution:** Include it in the controller policy from the start.

### 7. IMDS Hop Limit Must Be 2
**Problem:** Karpenter defaults to `httpPutResponseHopLimit: 1`. Pods in containers can't reach IMDS, causing `no EC2 IMDS role found` errors when using AWS SDK, SOPS, or helm-secrets.
**Solution:** Set `metadataOptions.httpPutResponseHopLimit: 2` in EC2NodeClass:
```yaml
spec:
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 2
    httpTokens: required
```

### 8. CI/CD Runner Access — EKS Access Entries + KMS
**Problem:** CCI runners on Karpenter nodes use `KarpenterNodeRole` instead of the old ASG node role. Deploys to other clusters fail because:
1. The Karpenter role isn't in the target cluster's EKS access entries
2. The Karpenter role isn't in the KMS key policy for SOPS decryption

**Solution — EKS access entries:** Add the Karpenter node role to every cluster the runners deploy to:
```bash
aws eks create-access-entry --cluster-name <target-cluster> \
  --principal-arn arn:aws:iam::594765360358:role/KarpenterNodeRole-<mgmt-cluster>
aws eks associate-access-policy --cluster-name <target-cluster> \
  --principal-arn arn:aws:iam::594765360358:role/KarpenterNodeRole-<mgmt-cluster> \
  --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy \
  --access-scope type=cluster
```

**Solution — KMS key policy:** Add the Karpenter node role to the SOPS KMS key:
```bash
# Get current policy, add role to "Allow use of the key" and "Allow attachment" statements
aws kms get-key-policy --key-id <key-id> --policy-name default | jq -r '.Policy' | jq '
  .Statement |= map(
    if .Sid == "Allow use of the key" then
      .Principal.AWS += ["arn:aws:iam::594765360358:role/KarpenterNodeRole-<cluster>"]
    elif .Sid == "Allow attachment of persistent resources" then
      .Principal.AWS += ["arn:aws:iam::594765360358:role/KarpenterNodeRole-<cluster>"]
    else . end
  )' > /tmp/kms-policy-updated.json
aws kms put-key-policy --key-id <key-id> --policy-name default --policy "$(cat /tmp/kms-policy-updated.json)"
```

**Solution — IAM policies:** Attach the same policies as the old node role (especially `tbi-clusterautoscaler` for `kms:*`):
```bash
aws iam attach-role-policy --role-name KarpenterNodeRole-<cluster> \
  --policy-arn arn:aws:iam::594765360358:policy/tbi-clusterautoscaler
```

### 9. Karpenter Node Role Needs Same Permissions as Old Node Role
**Problem:** The old ASG node role had extra policies (KMS, SQS, DynamoDB, Kinesis, CloudWatch) that workloads depend on.
**Solution:** Compare policies on old vs new role and attach what's missing:
```bash
# List old role policies
aws iam list-attached-role-policies --role-name <old-node-role>
# Attach missing ones to Karpenter role
aws iam attach-role-policy --role-name KarpenterNodeRole-<cluster> --policy-arn <policy-arn>
```

## Repeatable Playbook for dev/uat Clusters

### Prerequisites
- Helm 3.x installed
- kubectl access to the cluster
- AWS CLI with appropriate profile

### Step 1: Create IAM Roles

```bash
CLUSTER="<cluster-name>"
ACCOUNT="594765360358"
R="ca-central-1"
OIDC_ID="<from eks describe-cluster>"

# Controller Role (IRSA)
# - Trust: OIDC provider → system:serviceaccount:kube-system:karpenter
# - Policy: EC2 full describe/run/terminate, IAM instance profiles, EKS describe, SSM, pricing

# Node Role (EC2)
# - Trust: ec2.amazonaws.com
# - Managed policies: EKSWorkerNodePolicy, EKS_CNI_Policy, ECR ReadOnly, SSM ManagedInstanceCore

# Update aws-auth ConfigMap with new node role
```

### Step 2: Install Karpenter

```bash
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --kube-context $CLUSTER \
  --namespace kube-system \
  --version "1.11.1" \
  --set "settings.clusterName=${CLUSTER}" \
  --set "settings.clusterEndpoint=<endpoint>" \
  --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:aws:iam::${ACCOUNT}:role/KarpenterControllerRole-${CLUSTER}" \
  --set "replicas=1" \
  --wait --timeout 5m
```

### Step 3: Create NodePool + EC2NodeClass

```yaml
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  role: KarpenterNodeRole-<cluster>
  amiSelectorTerms:
    - alias: al2023@latest
  subnetSelectorTerms:
    - tags:
        Name: "TBI-development-vpc-private-eks-*"
  securityGroupSelectorTerms:
    - id: <cluster-sg>
    - id: <node-sg>
  metadataOptions:
    httpEndpoint: enabled
    httpProtocolIPv6: disabled
    httpPutResponseHopLimit: 2
    httpTokens: required
  tags:
    karpenter.sh/cluster: <cluster>
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        managed-by: karpenter
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["m5", "m6i", "m7i", "t3", "t3a"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["medium", "large", "xlarge", "2xlarge"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
  limits:
    cpu: "100"    # adjust per cluster
    memory: "200Gi"
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 60s
```

### Step 4: Downsize ASG (Don't Delete)

```bash
# Update overrides to smaller types
aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name <asg-name> \
  --mixed-instances-policy '{
    "LaunchTemplate": {
      "LaunchTemplateSpecification": {"LaunchTemplateId": "<lt-id>", "Version": "$Latest"},
      "Overrides": [
        {"InstanceType": "t3.large"},
        {"InstanceType": "t3a.large"}
      ]
    }
  }'

# Update LT base instance type
aws ec2 create-launch-template-version \
  --launch-template-id <lt-id> \
  --source-version '$Latest' \
  --launch-template-data '{"InstanceType": "t3.large"}'

# Set new version as default
aws ec2 modify-launch-template --launch-template-id <lt-id> --default-version <new-ver>

# Instance refresh to cycle nodes
aws autoscaling start-instance-refresh \
  --auto-scaling-group-name <asg-name> \
  --preferences '{"MinHealthyPercentage": 0}'
```

### Step 5: Verify

```bash
kubectl get nodes -o wide
kubectl get nodeclaims
kubectl get nodepool
kubectl get ec2nodeclass
kubectl get pods --all-namespaces --field-selector=status.phase=Pending
```

## Cluster-Specific Notes

### dev cluster (9 × m5.xlarge)
- Heavy workload: 28 namespaces, ~220 pods
- Memory-bound (~36% avg, peaks to 63%)
- Use t3.large for ASG system node (not medium)
- NodePool limits: cpu=100, memory=200Gi
- Allow xlarge and 2xlarge in NodePool for bin-packing

### uat cluster (7 × m5.xlarge)
- Moderate workload: ~120 pods
- Very low CPU (4%), moderate memory (35%)
- Use t3.large for ASG system node
- NodePool limits: cpu=60, memory=150Gi
- Consider Spot instances for uat (add "spot" to capacity-type)

## Reference

- Karpenter docs: https://karpenter.sh/docs/
- IAM policy reference: `/tmp/karpenter-controller-policy-v3.json`
- aws-auth backup: `/tmp/aws-auth-backup.yaml`
- Related docs: `docs/EKS-ACC-GX-HIS-CLUSTER-REFERENCE.md`