# Dev Account Cost Optimization Tracker

**Account:** 594765360358 | **Region:** ca-central-1 | **Profile:** tbi-dev
**Started:** 2026-04-14 | **Baseline:** ~$10,647/mo (March 2026)
**Target:** Cut $5,000/mo

## Completed

### Day 1 (2026-04-14)

| # | Action | Savings (with HST) | Status |
|---|---|---|---|
| 1 | ECR lifecycle policy - keep 5 images per repo (336 repos, ~54K images removed) | ~$1,695/mo | ✅ Done |
| 2 | EBS snapshots - deregistered 60 unused AMIs + deleted 112 snapshots (12,888 GB) | $783/mo | ✅ Done |
| 3 | EBS volumes - deleted 7 unused volumes (681 GB) | $80/mo | ✅ Done |
| 4 | EKS dev-win cluster deleted + node terminated | $196/mo | ✅ Done |
| 5 | 6 empty ASGs + 6 launch templates removed | — | ✅ Done |

### Day 2 (2026-04-15)

| # | Action | Savings (with HST) | Status |
|---|---|---|---|
| 6 | EKS tbi-acc-gx cluster deleted (2 × m5.large nodes + control plane) | ~$305/mo | ✅ Done |
| 7 | EKS tbi-acc-his cluster deleted (1 × m5.xlarge node + control plane) | ~$292/mo | ✅ Done |
| 8 | 7 NLBs deleted (gx, his, webclinq02, new-dev + 2 internal + 1 external) | ~$110/mo | ✅ Done |
| 9 | 3 ALBs deleted (tbi-websvc-internal-1, tbi-websites-internal-1, c60-tbi-websvc-qa-staging-int) | ~$50/mo | ✅ Done |
| 10 | Stopped ttsop-prod RDS (db.t3.small, 0 connections) | ~$33/mo | ✅ Done |
| 11 | Terminated 21 stopped EC2 instances + EBS volumes (~2TB freed) | ~$246/mo | ✅ Done |
| 12 | Karpenter on dev-mgmt: m5.2xlarge → t3.xlarge (ASG) + Karpenter | ~$214/mo | ✅ Done |

### Day 3 (2026-04-16)

| # | Action | Savings (with HST) | Status |
|---|---|---|---|
| 13 | dev cluster: 9 × m5.xlarge → 2 × m5.2xlarge (ASG) + Karpenter | ~$565/mo | ✅ Done |
| 14 | Fixed ECR lifecycle image issues (2 batches, requirements.yaml updated) | — | ✅ Done |
| 15 | Fixed tbi-acc-travelbrandsagent (OTEL, HOTEL_API_ROUTE, cms-node-connector) | — | ✅ Done |

### Day 4 (2026-04-17)

| # | Action | Savings (with HST) | Status |
|---|---|---|---|
| 16 | Removed redundant datalogger (Filebeat/Logstash) from dev + uat | ~$0 (resources freed for consolidation) | ✅ Done |
| 17 | Copied tbi-logger + tbi-cms-node-connector images to dev ECR | — | ✅ Done |

| **Total completed** | | **~$4,590/mo** | |

## Remaining Tasks

### Quick Wins (low effort)

| # | Action | Est. Savings (with HST) | Notes |
|---|---|---|---|
| 11 | Remove DAX clusters in us-east-1 | ~$202/mo | dax.t3.medium ($119) + dax.t2.medium ($60). Need us-east-1 access — DAX API blocked on tbi-dev profile |
| 12 | ~~Stop `ttsop-prod` RDS~~ | ~~$33/mo~~ | ✅ Done (Day 2) |
| 13 | ~~DocumentDB - stop 1 of 2~~ | — | Both actively used (13-18 connections). Leave as-is |
| 14 | Review FSx ($107/mo) | ~$121/mo | Evaluate if needed in dev |
| 15 | Cost Optimization Hub EBS volumes (62 recommended) | ~$365/mo | Run: `aws cost-optimization-hub list-recommendations --filter '{"resourceTypes":["EbsVolume"]}'` |

### Medium Effort (2-3 weeks)

| # | Action | Est. Savings (with HST) | Notes |
|---|---|---|---|
| 11 | OpenSearch consolidation (4 → 2 domains) | ~$339/mo | Merge tbi-gg-dev-logs + tbi-gg-uat-logs. Downsize c7g.xlarge |
| 12 | ElastiCache downsize | ~$170/mo | `tbi-secure-redtag` is r6g.large memcached — oversized for dev |
| 13 | NAT Gateway consolidation (3 → 1) | ~$170/mo | Currently $186/mo in hours + $54 data transfer |
| 14 | ALB consolidation (11 → 3-4) | ~$260/mo | 7 NLBs + 4 ALBs. Audit which are actually needed |
| 15 | CloudWatch log retention + trim metrics | ~$90/mo | Reduce retention periods, remove unused custom metrics |
| 16 | RDS Graviton migrations | ~$12/mo | `tbi-gigi-contract-mgmt-rds` and `disney` → Graviton |
| 17 | RDS Aurora downsize | ~$113/mo | `tbi-redtag-dev-aurora-mysql-3-instance-1` r6g.large → r6g.medium (NOT t4g — CPU spikes to 92%). 50-65 avg connections, peaks 117 |

### Large Effort (1 month)

| # | Action | Est. Savings (with HST) | Notes |
|---|---|---|---|
| 18 | EKS consolidation with Karpenter (6 → 2 clusters, 80% node reduction) | ~$3,730/mo | dev-mgmt DONE. dev and uat next |
| 19 | Prod-shared AMI migration (34 AMIs) | ~$250/mo | See `docs/SHARED-AMIS-PROD-REFERENCE.md`. Create images in prod, then deregister from dev |

## Karpenter Migration Status

### dev-mgmt ✅ Complete (stabilizing)
- **Karpenter v1.11.1** installed via Helm OCI
- **IAM**: KarpenterControllerRole-dev-mgmt (IRSA) + KarpenterNodeRole-dev-mgmt (EC2)
- **NodePool**: default — t3/t3a/m5/m6i/m7i, medium-xlarge, on-demand, 16 CPU / 64Gi limits
- **EC2NodeClass**: default — AL2023, private-eks subnets, cluster SG + node SG
- **ASG**: downsized LT from m5.2xlarge → t3.medium (v7), overrides: t3/t3a medium+large
- **Current nodes**: t3.medium (ASG/system) + t3a.xlarge (Karpenter/workloads)
- **Cost**: $362/mo → $173/mo (52% savings)
- **Note**: Linkerd pods in CrashLoopBackOff — may need identity cert bootstrap fix

### dev — Next
- 9 × m5.xlarge ($1,629/mo) → target 3-4 right-sized nodes (~$500/mo)
- Same Karpenter setup pattern as dev-mgmt

### uat — Next
- 7 × m5.xlarge ($1,267/mo) → target 2-3 right-sized nodes (~$350/mo)
- Same Karpenter setup pattern as dev-mgmt

### Savings Projection

| Category | Savings (with HST) |
|---|---|
| Completed | $2,754/mo |
| Quick wins (#6-10) | ~$794/mo |
| Medium effort (#11-17) | ~$1,267/mo |
| Large effort (#18-19) | ~$3,980/mo |
| **Grand total potential** | **~$8,795/mo** |
| **Projected new bill** | **~$1,850/mo** (down from $10,647) |

## Key Resources

- ECR lifecycle script: `/tmp/ecr-lifecycle.sh`
- AMI cleanup script: `/tmp/ami-snapshot-cleanup-safe.sh`
- Snapshot cleanup script: `/tmp/ebs-snapshot-cleanup.sh`
- Prod shared AMIs reference: `docs/SHARED-AMIS-PROD-REFERENCE.md`

## Infrastructure Inventory (as of 2026-04-14)

### EKS Clusters (5 remaining)
- `dev` — 9 × m5.xlarge nodes
- `dev-mgmt` — 1 × m5.2xlarge node
- `tbi-acc-gx` — 2 × m5.large nodes
- `tbi-acc-his` — 1 × m5.xlarge node
- `uat` — 7 × m5.xlarge nodes

### RDS Instances
- `disney` — db.t3.small, postgres
- `tbi-access-dev-aurora-mysql-3` — db.t4g.medium, aurora-mysql
- `tbi-copolo-rds` — db.t4g.medium, postgres, 150GB
- `tbi-documentdb-1` — db.t3.medium, docdb
- `tbi-documentdb-2` — db.t3.medium, docdb
- `tbi-gigi-contract-mgmt-rds` — db.t3.medium, postgres, 50GB
- `tbi-redtag-dev-aurora-mysql-3-instance-1` — db.r6g.large, aurora-mysql
- `ttsop-prod` — db.t3.small, postgres, 200GB

### OpenSearch Domains
- `tbi-gg-dev-logs` — r7g.large.search, 200GB
- `tbi-gg-uat-logs` — r7g.large.search, 200GB
- `opensearch-tbidevelopment` — c7g.xlarge.search, 500GB
- `tbi-dev-os-redtag` — r7g.large.search, 100GB

### ElastiCache
- `demo-0001-001` — cache.t4g.micro, valkey
- `tbi-access-valkey-001` — cache.t3.medium, valkey
- `tbi-copolo-valkey-001` — cache.t4g.medium, valkey
- `tbi-gg-valkey-001` — cache.m5.large, valkey
- `tbi-redtag-redis-001` — cache.t3.medium, redis
- `tbi-secure-redtag` — cache.r6g.large, memcached

### NAT Gateways (3)
- nat-05f334da58766d046
- nat-02e8b6e8abd08f804
- nat-0a2b82bad1c811e14

### Load Balancers (11)
- 7 NLBs (network)
- 4 ALBs (application): mgt-apps, tbi-websvc-internal-1, tbi-websites-internal-1, c60-tbi-websvc-qa-staging-int