# Post-Migration Issues & Fixes Log

**Date:** 2026-04-16
**Account:** 594765360358 | **Region:** ca-central-1

## Summary

After cost optimization work (ECR lifecycle policy, EKS cluster deletions, Karpenter migration), several cascading issues were identified and resolved.

---

## Issue 1: ECR Lifecycle Policy Deleted Active Images

**Impact:** 22+ pods in ImagePullBackOff across dev cluster
**Cause:** ECR lifecycle policy (keep 5 latest) deleted older image tags still referenced by running deployments via Helm releases.

**Services Affected (batch 1 - fixed 2026-04-14):**
- tbi-gg-contract-management, tbi-gg-sabre-flight-api, tbi-his-traveljapan-com
- tbi-his-traveljapan-dynamic-path, tbi-itravel-dynamic-path, tbi-rlv-dynamic
- tbi-sunquest-dynamic-path, tbi-itravel2000-cron, tbi-itravel2000-cron-2, tbi-itravel2000-cron-3

**Services Affected (batch 2 - fixed 2026-04-16):**
- tbi-gg-amadeus-flight-api, tbi-gg-amadeus-ndc-connector, tbi-gg-configuration-api
- tbi-gg-flight-core-api, tbi-gg-notification-api, tbi-pl-booking-path
- tbi-rlv-attraction-frontend, tbi-rlv-frontend-service, tbi-rlv-giftcard-service
- tbi-sunquest-ca, tbi-acc-mybff, tbi-corporate-booking-path, tbi-mybff-booking-path

**Fix:** Updated `tbi-environments/dev/requirements.yaml` with latest available ECR tags. Committed in two batches.

**Lesson:** Before applying ECR lifecycle policies, cross-reference running deployment image tags with ECR. Consider keeping 10+ images instead of 5, or use tag-based rules (keep all `develop-*` tags newer than 30 days).

---

## Issue 2: Cross-Account ECR Images Can't Pull on New Nodes

**Impact:** `tbi-cms-node-connector` and `tbi-logger` (datalogger) pods fail with ImagePullBackOff
**Cause:** These pods use images from the prod ECR (`489176460030.dkr.ecr.ca-central-1.amazonaws.com`). Previously worked because images were cached on old nodes. After node cycling (Karpenter/ASG refresh), new nodes can't pull from prod ECR.

**Services Affected:**
- `tbi-cms-node-connector` — image `489176460030.dkr.ecr.ca-central-1.amazonaws.com/tbi-cms-node-connector:2.2.0`
- `tbi-logger` (datalogger DaemonSet) — image `489176460030.dkr.ecr.ca-central-1.amazonaws.com/tbi-logger:1.0.3`

**Fix (tbi-cms-node-connector):** Switched to local dev ECR copy:
```
594765360358.dkr.ecr.ca-central-1.amazonaws.com/tbi-cms-node-connector:2.1.6-uat-7
```

**Fix (tbi-logger):** Pending — needs either cross-account ECR pull permissions or image copy to dev ECR.

**Long-term fix:** Set up ECR replication from prod to dev, or rebuild these images in the dev CI pipeline.

---

## Issue 3: tbi-acc-travelbrandsagent /login Returns 404

**Impact:** `https://access-dev.travelbrandsagent.com/login` returns 404
**Root causes (3 issues combined):**

### 3a: OpenTelemetry PHP Extension Crashes Laravel Bootstrap
- **Symptom:** `OpenTelemetry Exception` in `Application.php:1131`, every request
- **Cause:** Dev secrets had OTEL enabled; UAT did not
- **Dev secrets (broken):**
  ```
  OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.monitor.tbidevelopment.ca/v1/traces
  OTEL_ENABLED=true
  OTEL_SERVICE_NAME=tbi-acc-travelbrandsagent
  ```
- **UAT secrets (working):** No OTEL vars at all
- **Fix:** Removed all OTEL vars from dev secrets. Note: the endpoint was also hardcoded in the app — the PHP extension auto-initializes regardless of env vars. Dev team should investigate the OpenTelemetry PHP extension compatibility.

### 3b: HOTEL_API_ROUTE Pointed to Deleted Cluster
- **Cause:** Dev secrets pointed to `https://tbi-hotel-service.gimmonix.tbidevelopment.ca` which was on `tbi-acc-gx` cluster (deleted during cost optimization)
- **Fix:** Updated to `http://tbi-acc-hotel-service.access.svc.cluster.local` (matching UAT pattern)

### 3c: tbi-cms-node-connector Down (502)
- **Cause:** Pod crashed on startup due to `EAI_AGAIN` DNS error for `redis-access.dev.tbidevelopment.ca`. Nodemon doesn't auto-restart after DNS failures. Then pod moved to new node and hit cross-account ECR pull issue (Issue 2).
- **Fix:** Switched to local ECR image (see Issue 2 fix)

**Secrets changes committed:** `tbi-environments/dev/secrets/secrets.tbi-acc-travelbrandsagent.yaml`
- Removed OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_ENABLED, OTEL_SERVICE_NAME
- Updated HOTEL_API_ROUTE to cluster-local

---

## Issue 4: Karpenter Node Kubelet TLS Errors

**Impact:** Can't exec, port-forward, or get logs from pods on certain Karpenter nodes
**Symptom:** `remote error: tls: internal error` when accessing kubelet API
**Affected node:** `ip-100-65-43-234.ca-central-1.compute.internal`
**Cause:** Intermittent kubelet certificate issue on newly launched Karpenter nodes. Usually resolves on its own or after node replacement.
**Workaround:** Delete pod to reschedule on a healthy node.

---

## Issue 5: Helm Reverts kubectl set image Changes

**Impact:** `kubectl set image` fixes are temporary — Helm reverts them on next sync/deploy
**Cause:** Helm stores the desired state. Next `helm upgrade` overwrites manual changes.
**Fix:** Must update `tbi-environments/dev/requirements.yaml` (for image tags) or secrets files, then redeploy via Helm.

---

## Services with Pre-existing Issues (Not Caused by Migration)

| Service | Issue | Status |
|---|---|---|
| tbi-logger (datalogger) | Cross-account ECR pull from 489176460030 | Needs fix |
| tbi-rlv-affiliate-booking-path | CrashLoopBackOff — app error | Dev team |
| tbi-redtag-mobile-hotel-cronone-service | CrashLoopBackOff — app error | Dev team |
| debug-pod (redtag namespace) | CrashLoopBackOff for 73 days | Should be deleted |
| Linkerd (dev-mgmt) | CrashLoopBackOff — identity cert | Pre-existing |

---

## Preventive Measures for Future

1. **ECR lifecycle policy:** Consider tag-based rules instead of count-based. Keep all `develop-*` tags newer than 30 days.
2. **Cross-account images:** Audit all deployments using prod ECR images. Set up ECR replication or rebuild in dev CI.
3. **Secrets audit after cluster deletion:** When deleting clusters (e.g., tbi-acc-gx, tbi-acc-his), search all secrets for URLs referencing those clusters.
4. **Helm-managed deployments:** Never use `kubectl set image` for permanent fixes. Always update via requirements.yaml + Helm upgrade.
5. **Node cycling:** Before cycling nodes, ensure all images are pullable (not just cached). Run the ECR image verification script.

## Verification Script

Check all running images exist in ECR:
```bash
kubectl --context tbi-dev get pods --all-namespaces -o jsonpath='{range .items[?(@.status.phase=="Running")]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | \
  grep "594765360358" | sort -u | while read IMG; do
  REPO=$(echo "$IMG" | sed 's|594765360358.dkr.ecr.ca-central-1.amazonaws.com/||' | cut -d: -f1)
  TAG=$(echo "$IMG" | cut -d: -f2)
  EXISTS=$(aws ecr describe-images --profile tbi-dev --region ca-central-1 \
    --repository-name "$REPO" --image-ids imageTag="$TAG" \
    --query 'imageDetails[0].imagePushedAt' --output text 2>/dev/null || echo "MISSING")
  [ "$EXISTS" = "None" ] && EXISTS="MISSING"
  [ "$EXISTS" = "MISSING" ] && echo "❌ $REPO:$TAG"
done
```