# Shared Docker Image Migration Guide

**For services that share a Docker image across multiple deployments**

> **Use Case:** Primary service builds Docker image, secondary service(s) reuse the same image
> **Example:** `tbi-package-travelbrands` (builds image) → `tbi-pl-dynamic` (reuses image)
> **Last Updated:** 2025-12-04

---

## Table of Contents

1. [Overview](#overview)
2. [Architecture Pattern](#architecture-pattern)
3. [Migration Checklist](#migration-checklist)
4. [Common Issues and Solutions](#common-issues-and-solutions)
5. [Testing and Verification](#testing-and-verification)

---

## Overview

### When to Use This Pattern

Use the shared image pattern when:
- Multiple services run the same codebase with different configurations
- Services share application code but deploy to different namespaces
- You want to avoid building the same Docker image multiple times

**Example:**
- `tbi-package-travelbrands` deploys to `access` namespace (main brand)
- `tbi-pl-dynamic` deploys to `privatelabel` namespace (white-label partners)
- Both use the same Docker image from one build process

### Key Differences from Standard Migration

| Standard Pattern | Shared Image Pattern |
|------------------|---------------------|
| Each service builds its own image | Primary service builds, others reuse |
| Independent pipelines | Coordinated pipeline with auto-trigger |
| One values file per service | Primary updates values, secondary reads |
| Simple deployment | Image tag synchronization required |

---

## Architecture Pattern

### Repository Structure

```
📦 tbi-package-travelbrands (Primary - builds image)
├── src/                    # Application code
├── package.json            # Dependencies
├── composer.json           # PHP dependencies
└── VERSION                 # Base version number

📦 tbi-pl-dynamic (Secondary - reuses image)
├── src/                    # Same codebase (or minimal config overrides)
└── .archived/              # Archived Docker/CI configs

📦 tbi-devops-environments (CI/CD configs)
├── dev/
│   ├── tbi-package-travelbrands/
│   │   ├── .circleci/config.yml       # Build + Deploy pipeline
│   │   ├── docker/Dockerfile           # Image build definition
│   │   └── scripts/deploy-services.sh  # Deployment script
│   └── tbi-pl-dynamic/
│       ├── .circleci/config.yml       # Deploy-only pipeline
│       └── scripts/deploy-services.sh  # Deployment script
├── qa/   # Same structure
└── uat/  # Same structure

📦 tbi-environments (Values & Secrets)
├── dev/
│   ├── values/
│   │   ├── tbi-package-travelbrands.yaml   # Contains image tag
│   │   └── tbi-pl-dynamic.yaml             # Uses same image tag
│   ├── requirements.yaml                    # Helm chart versions
│   └── secrets/
│       ├── secrets.tbi-package-travelbrands.yaml
│       └── secrets.tbi-pl-dynamic.yaml
└── qa/, uat/  # Same structure
```

### Deployment Flow

```
Developer pushes to tbi-package-travelbrands/develop
            ↓
CircleCI: Lint & Test (package.json, composer.json)
            ↓
Build Docker image with tag: {VERSION}-{BRANCH}-{BUILD_NUM}
            ↓
Push image to ECR: tbi-package-travelbrands:1.0.0-develop-123
            ↓
Deploy to Kubernetes (access namespace)
            ↓
Update dev/values/tbi-package-travelbrands.yaml
Update dev/requirements.yaml
            ↓
Send SNS success notification
            ↓
Trigger tbi-pl-dynamic deployment (empty commit)
            ↓
═══════════════════════════════════════════════
            ↓
tbi-pl-dynamic pipeline starts
            ↓
Read image tag from dev/values/tbi-package-travelbrands.yaml
            ↓
Verify image exists in ECR (with 12-minute retry)
            ↓
Deploy to Kubernetes (privatelabel namespace)
            ↓
Update dev/values/tbi-pl-dynamic.yaml
Update dev/requirements.yaml
            ↓
Send SNS success notification
```

---

## Migration Checklist

### Prerequisites

- [ ] Both service repositories exist
- [ ] Application code is identical or compatible
- [ ] Different namespaces identified (e.g., access vs privatelabel)
- [ ] CircleCI projects created for both services
- [ ] AWS ECR repository created for primary service

---

## Phase 1: Primary Service (Image Builder)

### Service: `tbi-package-travelbrands` (or your primary service)

### Step 1: Create Directory Structure

```bash
cd /path/to/tbi-devops-environments

# Create for each environment
for env in dev qa uat; do
  mkdir -p "$env/tbi-package-travelbrands"/{.circleci,.github/workflows,docker,scripts}
done
```

- [ ] Directories created for all environments

### Step 2: Stage Dockerfile

```bash
cd /path/to/tbi-package-travelbrands
cp Dockerfile ../tbi-devops-environments/dev/tbi-package-travelbrands/docker/
```

**Important Dockerfile Requirements:**
- Must support BuildKit for SSH agent forwarding (if using private repos)
- Should accept `AWS_ACCOUNT_ID` and `AWS_REGION` build args
- Example:
  ```dockerfile
  # syntax=docker/dockerfile:1.4
  FROM php:8.0-apache

  # Install dependencies
  RUN apt-get update && apt-get install -y git ssh

  # Setup SSH for private repos
  RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

  WORKDIR /var/www/html

  # Copy application code
  COPY . .

  # Install composer dependencies
  RUN --mount=type=ssh composer install --no-interaction --prefer-dist

  # Install npm dependencies
  RUN npm install --force
  RUN npm run build
  ```

- [ ] Dockerfile copied and verified

### Step 3: Create CircleCI Config

Copy from template or create new:

**Key sections required:**

1. **lint-and-test job:**
   ```yaml
   lint-and-test:
     docker:
       - image: cimg/php:8.0-node
     resource_class: small
     steps:
       - checkout
       - add_ssh_keys:
           fingerprints:
             - "YOUR_DEPLOY_KEY_FINGERPRINT"
       - restore_cache  # Composer cache
       - restore_cache  # npm cache
       - run: composer install
       - run: npm install --force
       - run: npm run build
       - run: Generate VERSION tag
       - persist_to_workspace:
           paths:
             - VERSION
   ```

2. **build-and-push-images job:**
   ```yaml
   build-and-push-images:
     docker:
       - image: cimg/aws:2023.09
     steps:
       - checkout
       - attach_workspace
       - setup_remote_docker:
           docker_layer_caching: true
       - run: Stage Dockerfile from tbi-devops-environments
       - run: Generate image tags
       - run: Login to AWS ECR
       - run: Build and push Docker image
   ```

3. **deploy-to-kubernetes job:**
   ```yaml
   deploy-to-kubernetes:
     executor: aws-cli/default
     resource_class: tbi-local-runner/loadtest
     steps:
       - checkout
       - attach_workspace
       - run: Stage deployment scripts
       - run: Setup kubectl and helm-secrets
       - run: Package Helm charts and deploy
       - run: Send deployment success notification
       - run: Send deployment failure notification (when: on_fail)
       - run: Trigger tbi-pl-dynamic deployment
   ```

**Critical Configuration Points:**
- [ ] Branch filters match environment (develop → dev, qa → qa, uat → uat)
- [ ] Image tag format: `${BASE_VERSION}-${CIRCLE_BRANCH}-${CIRCLE_BUILD_NUM}`
- [ ] ECR repository name matches service name
- [ ] Immutable tags enabled (`--image-tag-mutability IMMUTABLE`)
- [ ] SNS notifications configured
- [ ] Auto-trigger step for secondary service included

### Step 4: Create Deployment Script

Create `scripts/deploy-services.sh`:

**Required functions:**

1. **setup_environment():** Set environment variables
   ```bash
   ENVIRONMENT="${DEPLOY_ENV:-dev}"  # Use context variable
   SERVICE_NAME="tbi-package-travelbrands"
   NAMESPACE="access"
   CHART_VERSION="${IMAGE_TAG}"
   ```

2. **clone_repos():** Clone tbi-environments and tbi-devops-common

3. **package_helm_chart():** Package and push to S3

4. **update_values_file():** Update image URL in values file
   ```bash
   IMAGE_URL="${ECR_REGISTRY}/${SERVICE_NAME}:${IMAGE_TAG}"
   yq eval ".image = \"${IMAGE_URL}\"" -i ${VALUES_FILE}
   ```

5. **update_requirements():** Update Helm chart version tracking
   ```bash
   yq eval "(.dependencies[] | select(.name == \"${SERVICE_NAME}\").version) = \"${CHART_VERSION}\"" \
     -i ${REQUIREMENTS_FILE}
   ```

6. **push_values_update():** Commit BOTH values and requirements files
   ```bash
   git add ${ENVIRONMENT}/values/${SERVICE_NAME}.yaml
   git add ${ENVIRONMENT}/requirements.yaml  # ← CRITICAL!
   git commit -m "Update ${SERVICE_NAME} to ${CHART_VERSION}"
   git push origin master
   ```

7. **deploy_with_helm():** Deploy using helm secrets

8. **verify_deployment():** Wait for rollout completion

**Common Mistake to Avoid:**
- ❌ Forgetting to call `update_requirements()` function
- ❌ Not committing `requirements.yaml` alongside values file
- ✅ Always commit both files together for ArgoCD tracking

- [ ] Deployment script created with all functions
- [ ] requirements.yaml update included
- [ ] Script tested locally

### Step 5: Configure GitHub Workflow (Secrets Management)

Copy workflow file:
```bash
cp ../other-service/.github/workflows/manage-secrets.yml \
   tbi-devops-environments/dev/tbi-package-travelbrands/.github/workflows/
```

Update service name in workflow file:
```yaml
service:
  type: choice
  options:
    - tbi-package-travelbrands
```

**Also add to master branch of service repo:**
```bash
cd /path/to/tbi-package-travelbrands
mkdir -p .github/workflows
cp ../tbi-devops-environments/dev/tbi-package-travelbrands/.github/workflows/manage-secrets.yml \
   .github/workflows/
git checkout master
git add .github/workflows/manage-secrets.yml
git commit -m "Add secrets management workflow"
git push origin master
```

**Why master branch?**
- GitHub Actions only discovers workflows from the default branch
- Workflow can still run against any environment/branch

- [ ] Workflow added to tbi-devops-environments
- [ ] Workflow added to master branch of service repo
- [ ] Service name updated

### Step 6: Setup Secrets in tbi-environments

```bash
cd /path/to/tbi-environments/dev

# Create secrets file
cat > secrets/secrets.tbi-package-travelbrands.yaml << 'EOF'
configs:
    dotenv: |-
        APP_ENV=development
        DATABASE_HOST=your-db-host
        DATABASE_PASSWORD=your-db-password
        REDIS_HOST=your-redis-host
        AWS_REGION=ca-central-1
        # Add all required environment variables
EOF

# Encrypt
sops -e -i secrets/secrets.tbi-package-travelbrands.yaml

# Commit
git add secrets/secrets.tbi-package-travelbrands.yaml
git commit -m "Add secrets for tbi-package-travelbrands (dev)"
git push origin master
```

Repeat for qa, uat environments.

- [ ] Secrets created and encrypted for all environments
- [ ] Secrets committed to tbi-environments

### Step 7: Configure CircleCI

1. Go to CircleCI → Project Settings → Advanced
2. Set **Config source:** `tbi-devops-environments`
3. Set **Config filepath:** `dev/tbi-package-travelbrands/.circleci/config.yml`
4. Set **Checkout source:** `tbi-package-travelbrands` (your app repo)

**Add CircleCI contexts:**
- `devops-mgt` (all environments)
- `devops-mgt-qa` (qa only, contains `DEPLOY_ENV=qa`)
- `devops-mgt-uat` (uat only, contains `DEPLOY_ENV=uat`)

**Context variables required:**
```
DEPLOY_ENV=dev|qa|uat  (environment-specific context)
AWS_ACCOUNT_ID=...
AWS_REGION=ca-central-1
GITHUB_TOKEN=...
```

- [ ] CircleCI config source configured
- [ ] Contexts assigned to workflows
- [ ] DEPLOY_ENV set for each environment

### Step 8: Add Dependency Fix (if Node.js + React)

If using react-select or emotion-based libraries, add explicit dependencies:

```json
{
  "dependencies": {
    "@emotion/react": "^11.10.4",
    "@emotion/cache": "^11.10.3",
    "react-select": "^4.3.1"
  }
}
```

Regenerate package-lock.json:
```bash
npm install --force --ignore-scripts
git add package.json package-lock.json
git commit -m "Add explicit emotion dependencies"
git push origin develop
```

**Why?**
- `npm install --force` bypasses peer dependency checks
- react-select requires @emotion packages but won't install them automatically
- Explicit dependencies prevent runtime errors: `registerStyles is not a function`

- [ ] Emotion dependencies added (if applicable)
- [ ] package-lock.json regenerated

### Step 9: Test Primary Service Deployment

```bash
# Push to trigger build
cd /path/to/tbi-package-travelbrands
git checkout develop
# ... make a small change ...
git commit -m "Test centralized CI/CD pipeline"
git push origin develop
```

**Monitor CircleCI:**
1. Watch lint-and-test job
2. Watch build-and-push-images job
3. Watch deploy-to-kubernetes job
4. Verify SNS notification received
5. Check tbi-pl-dynamic pipeline was triggered

**Verify image in ECR:**
```bash
aws ecr describe-images \
  --repository-name tbi-package-travelbrands \
  --region ca-central-1
```

**Verify deployment:**
```bash
kubectl get pods -n access -l app=tbi-package-travelbrands
kubectl logs -n access deployment/tbi-package-travelbrands --tail=50
```

- [ ] CircleCI pipeline completes successfully
- [ ] Image pushed to ECR
- [ ] Deployed to Kubernetes
- [ ] requirements.yaml updated in tbi-environments
- [ ] SNS notification received
- [ ] No errors in pod logs

---

## Phase 2: Secondary Service (Image Consumer)

### Service: `tbi-pl-dynamic` (or your secondary service)

### Step 10: Create Directory Structure

```bash
cd /path/to/tbi-devops-environments

for env in dev qa uat; do
  mkdir -p "$env/tbi-pl-dynamic"/{.circleci,.github/workflows,scripts}
done
```

**Note:** No `docker/` directory needed - reuses primary's image

- [ ] Directories created

### Step 11: Archive Old Configs in Service Repo

```bash
cd /path/to/tbi-pl-dynamic

mkdir -p .archived
mv Dockerfile .archived/Dockerfile.archived
mv .circleci .archived/.circleci.archived  # If exists

git add .archived
git commit -m "Archive old Docker and CI configs - now in tbi-devops-environments"
git push origin develop
```

- [ ] Old configs archived

### Step 12: Create Deploy-Only CircleCI Config

**Key differences from primary service:**

1. **No build job** - only deploy
2. **Read image tag from tbi-environments**
3. **Verify image exists in ECR**
4. **Different namespace**

```yaml
jobs:
  deploy-to-kubernetes:
    executor: aws-cli/default
    steps:
      - checkout
      - run:
          name: Get latest image tag from tbi-environments
          command: |
            # Clone tbi-environments
            git clone ${GITHUB_REPO} /tmp/tbi-environments
            cd /tmp/tbi-environments

            # Map branch to environment
            case ${CIRCLE_BRANCH} in
              develop) envFolder="dev" ;;
              qa) envFolder="qa" ;;
              uat) envFolder="uat" ;;
              *) envFolder="dev" ;;
            esac

            # Read image tag from PRIMARY service values file
            VALUES_FILE="${envFolder}/values/tbi-package-travelbrands.yaml"
            IMAGE_URL=$(grep "^image:" $VALUES_FILE | awk '{print $2}')
            IMAGE_TAG=$(echo $IMAGE_URL | awk -F: '{print $2}')

            # Save IMAGE_TAG
            echo "export IMAGE_TAG=${IMAGE_TAG}" >> $BASH_ENV
            echo "${IMAGE_TAG}" > IMAGE_TAG

      - persist_to_workspace:
          paths:
            - IMAGE_TAG

      - run:
          name: Verify image exists in ECR
          command: |
            IMAGE_TAG=$(cat IMAGE_TAG | tr -d '[:space:]')

            # Check immediately (should exist)
            if aws ecr describe-images \
              --repository-name tbi-package-travelbrands \
              --image-ids imageTag=${IMAGE_TAG} > /dev/null 2>&1; then
              echo "✓ Image found"
              exit 0
            fi

            # If not found, wait and poll (race condition handling)
            INITIAL_WAIT=120  # 2 minutes
            sleep $INITIAL_WAIT

            # Poll with retries (10 more minutes)
            MAX_ATTEMPTS=60
            SLEEP_INTERVAL=10

            for attempt in $(seq 1 $MAX_ATTEMPTS); do
              if aws ecr describe-images \
                --repository-name tbi-package-travelbrands \
                --image-ids imageTag=${IMAGE_TAG} > /dev/null 2>&1; then
                echo "✓ Image found after ${attempt} attempts"
                exit 0
              fi
              sleep $SLEEP_INTERVAL
            done

            echo "ERROR: Image not found after 12 minutes"
            exit 1

      - run: Stage deployment scripts
      - run: Setup kubectl and helm-secrets
      - run:
          name: Package Helm charts and deploy
          command: |
            # Verify DEPLOY_ENV from context
            if [ -z "${DEPLOY_ENV}" ]; then
              echo "ERROR: DEPLOY_ENV not set"
              exit 1
            fi

            if [ "${DEPLOY_ENV}" != "uat" ]; then  # Adjust per environment
              echo "WARNING: DEPLOY_ENV mismatch"
              exit 1
            fi

            chmod +x scripts/deploy-services.sh
            bash scripts/deploy-services.sh

      - run: Send deployment success notification
      - run: Send deployment failure notification (when: on_fail)
```

**Critical Points:**
- [ ] Reads image tag from **primary service** values file
- [ ] Uses same ECR repository name as primary
- [ ] Has 12-minute retry logic for image availability
- [ ] Verifies DEPLOY_ENV matches expected environment
- [ ] Deploys to different namespace (privatelabel vs access)

### Step 13: Create Deployment Script

Similar to primary service script, but:

```bash
# Different service name and namespace
SERVICE_NAME="tbi-pl-dynamic"
NAMESPACE="privatelabel"

# Image URL points to PRIMARY service's ECR repo
IMAGE_URL="${ECR_REGISTRY}/tbi-package-travelbrands:${IMAGE_TAG}"
```

**Important:**
- Uses IMAGE_TAG from workspace (set by CircleCI)
- Points to tbi-package-travelbrands ECR repository
- Updates tbi-pl-dynamic values file with that image URL
- Still updates requirements.yaml

- [ ] Deployment script created
- [ ] Points to correct ECR repository
- [ ] Uses correct namespace
- [ ] Updates requirements.yaml

### Step 14: Configure GitHub Workflow

Same as primary service:

```bash
# Add to tbi-devops-environments
cp workflow-template .../tbi-devops-environments/dev/tbi-pl-dynamic/.github/workflows/

# Add to master branch of service repo
cd /path/to/tbi-pl-dynamic
git checkout master
mkdir -p .github/workflows
cp workflow-file .github/workflows/manage-secrets.yml
git add .github/workflows/
git commit -m "Add secrets management workflow"
git push origin master
```

- [ ] Workflow added to both locations

### Step 15: Setup Secrets

```bash
cd /path/to/tbi-environments/dev

cat > secrets/secrets.tbi-pl-dynamic.yaml << 'EOF'
configs:
    dotenv: |-
        APP_ENV=development
        # Add environment-specific variables for privatelabel
        # May share some with tbi-package-travelbrands or be different
EOF

sops -e -i secrets/secrets.tbi-pl-dynamic.yaml
git add secrets/secrets.tbi-pl-dynamic.yaml
git commit -m "Add secrets for tbi-pl-dynamic (dev)"
git push origin master
```

- [ ] Secrets created for all environments

### Step 16: Configure CircleCI

1. Go to CircleCI → tbi-pl-dynamic project
2. Set **Config source:** `tbi-devops-environments`
3. Set **Config filepath:** `dev/tbi-pl-dynamic/.circleci/config.yml`
4. Set **Checkout source:** `tbi-pl-dynamic`

**Workflow context:**
```yaml
workflows:
  deploy-tbi-pl-dynamic:
    jobs:
      - deploy-to-kubernetes:
          context:
            - devops-mgt
            - devops-mgt-uat  # For UAT branch
          filters:
            branches:
              only:
                - develop  # For dev
                - uat      # For UAT
```

- [ ] CircleCI configured
- [ ] Correct contexts assigned

### Step 17: Test Secondary Service Deployment

**Manual trigger (empty commit):**
```bash
cd /path/to/tbi-pl-dynamic
git checkout develop
git commit --allow-empty -m "Test centralized CI/CD for tbi-pl-dynamic"
git push origin develop
```

**Automatic trigger:**
- Should be triggered automatically after tbi-package-travelbrands deployment

**Monitor CircleCI:**
1. Watch image tag retrieval
2. Watch ECR verification (may take up to 12 minutes if race condition)
3. Watch deployment
4. Verify SNS notification

**Verify deployment:**
```bash
kubectl get pods -n privatelabel -l app=tbi-pl-dynamic
kubectl logs -n privatelabel deployment/tbi-pl-dynamic --tail=50
```

**Verify values files:**
```bash
cd /path/to/tbi-environments

# Check primary service values
cat dev/values/tbi-package-travelbrands.yaml | grep image:

# Check secondary service values (should match image tag)
cat dev/values/tbi-pl-dynamic.yaml | grep image:

# Check requirements.yaml updated for both
cat dev/requirements.yaml | grep -A2 "name: tbi-package-travelbrands"
cat dev/requirements.yaml | grep -A2 "name: tbi-pl-dynamic"
```

- [ ] Secondary service deploys successfully
- [ ] Image tag matches primary service
- [ ] requirements.yaml updated for both services
- [ ] SNS notification received
- [ ] Pods running in correct namespace

---

## Phase 3: Repeat for Other Environments

### Step 18: Replicate to QA Environment

```bash
cd /path/to/tbi-devops-environments

# Copy dev configs to qa
cp -r dev/tbi-package-travelbrands/ qa/
cp -r dev/tbi-pl-dynamic/ qa/

# Update CircleCI configs
# Change branch filters: develop → qa
# Change contexts: devops-mgt-uat → devops-mgt-qa (if applicable)

git add qa/
git commit -m "Add QA configs for tbi-package-travelbrands and tbi-pl-dynamic"
git push origin master
```

**Update CircleCI workflow filters:**
```yaml
filters:
  branches:
    only:
      - qa
```

**Add secrets:**
```bash
cd /path/to/tbi-environments/qa
# Create and encrypt secrets for both services
```

- [ ] QA configs created
- [ ] Branch filters updated
- [ ] Secrets configured

### Step 19: Replicate to UAT Environment

```bash
cp -r dev/tbi-package-travelbrands/ uat/
cp -r dev/tbi-pl-dynamic/ uat/

# Update branch filters: develop → uat
# Update contexts: add devops-mgt-uat
```

**Important for UAT:**
- Must use `devops-mgt-uat` context with `DEPLOY_ENV=uat`
- Add DEPLOY_ENV verification in deploy step
- Update SNS notification subject with environment

**Example DEPLOY_ENV verification:**
```yaml
- run:
    name: Package Helm charts and deploy
    command: |
      if [ -z "${DEPLOY_ENV}" ]; then
        echo "ERROR: DEPLOY_ENV not set"
        exit 1
      fi

      if [ "${DEPLOY_ENV}" != "uat" ]; then
        echo "WARNING: DEPLOY_ENV=${DEPLOY_ENV} but expected 'uat'"
        exit 1
      fi

      bash scripts/deploy-services.sh
```

- [ ] UAT configs created
- [ ] DEPLOY_ENV verification added
- [ ] Secrets configured
- [ ] Contexts properly stacked (devops-mgt + devops-mgt-uat)

---

## Common Issues and Solutions

### Issue 1: requirements.yaml Not Updating

**Symptom:**
- values file updated but requirements.yaml shows old version
- ArgoCD doesn't detect changes

**Cause:**
- `update_requirements()` function not called in deployment script
- requirements.yaml not committed alongside values file

**Solution:**
```bash
# Edit deploy-services.sh
# Add step to call update_requirements():

echo "=== Step 6: Update requirements.yaml ==="
update_requirements

# Ensure push_values_update commits both files:
git add ${ENVIRONMENT}/values/${SERVICE_NAME}.yaml
git add ${ENVIRONMENT}/requirements.yaml  # ← Add this
```

✅ **Verify fix:**
```bash
cd /path/to/tbi-environments
git log --oneline -n 5 dev/requirements.yaml
# Should see recent commits
```

### Issue 2: DEPLOY_ENV Defaulting to Wrong Environment

**Symptom:**
- UAT pipeline deploying to dev/ folder
- Scripts using wrong secrets path

**Cause:**
- Script defaults: `ENVIRONMENT="${DEPLOY_ENV:-dev}"`
- DEPLOY_ENV not set in CircleCI context

**Solution:**
1. **Add DEPLOY_ENV to CircleCI context:**
   - Go to CircleCI → Organization Settings → Contexts
   - Create/edit `devops-mgt-uat` context
   - Add variable: `DEPLOY_ENV=uat`

2. **Add verification in CircleCI config:**
   ```yaml
   - run:
       command: |
         if [ -z "${DEPLOY_ENV}" ]; then
           echo "ERROR: DEPLOY_ENV not set in CircleCI context!"
           exit 1
         fi
   ```

✅ **Verify fix:**
```bash
# Check CircleCI build logs
# Should see: "DEPLOY_ENV is set to: uat"
```

### Issue 3: Image Not Found in ECR

**Symptom:**
```
ERROR: Image not found in ECR after 12 minutes
```

**Causes:**
1. Primary service build failed
2. Image tag mismatch
3. Race condition (secondary triggered before primary finished)

**Solution:**

**A. Check primary service build:**
```bash
# In CircleCI, verify tbi-package-travelbrands build succeeded
# Check ECR directly:
aws ecr describe-images \
  --repository-name tbi-package-travelbrands \
  --region ca-central-1 \
  | grep imageTag
```

**B. Verify image tag consistency:**
```bash
# In tbi-environments, check values file:
cat dev/values/tbi-package-travelbrands.yaml | grep image:

# Should match ECR tags:
# image: 594765360358.dkr.ecr.ca-central-1.amazonaws.com/tbi-package-travelbrands:1.0.0-develop-123
```

**C. Increase wait time (if race condition):**
```yaml
# In tbi-pl-dynamic config, increase INITIAL_WAIT:
INITIAL_WAIT=180  # 3 minutes instead of 2
```

✅ **Prevention:**
- Primary service auto-triggers secondary after successful deployment
- 12-minute retry should handle most race conditions

### Issue 4: Emotion Dependency Runtime Error

**Symptom:**
```
TypeError: (0, _emotion_utils__WEBPACK_IMPORTED_MODULE_5__.registerStyles) is not a function
```

**Cause:**
- react-select requires @emotion packages as peer dependencies
- `npm install --force` bypasses peer dependency resolution
- Missing explicit dependencies in package.json

**Solution:**
```bash
cd /path/to/tbi-package-travelbrands

# Add to package.json:
"dependencies": {
  "@emotion/react": "^11.10.4",
  "@emotion/cache": "^11.10.3"
}

# Regenerate lock file
npm install --force --ignore-scripts

git add package.json package-lock.json
git commit -m "Add explicit emotion dependencies"
git push origin develop
```

✅ **Verify fix:**
- Build should complete without errors
- No runtime errors in browser console

### Issue 5: GitHub Actions Workflow Not Visible

**Symptom:**
- Go to repo → Actions tab
- See "Get started with GitHub Actions" page
- Workflow not listed in sidebar

**Cause:**
- GitHub Actions only discovers workflows from default branch (usually master)
- Workflow only exists on develop/qa/uat branch

**Solution:**
```bash
cd /path/to/service-repo

# Copy workflow to master branch
git checkout master
git pull origin master

mkdir -p .github/workflows
git checkout develop -- .github/workflows/manage-secrets.yml

git add .github/workflows/
git commit -m "Add secrets workflow to master for GitHub Actions discovery"
git push origin master
```

✅ **Verify fix:**
- Go to GitHub repo → Actions tab
- "Manage Secrets" workflow now visible in left sidebar
- Can run workflow against any environment

### Issue 6: package-lock.json Missing Error in CircleCI

**Symptom:**
```
error computing cache key: template: cacheKey:1:10: executing "cacheKey" at <checksum "package-lock.json">:
error calling checksum: open package-lock.json: no such file or directory
```

**Cause:**
- CircleCI cache key uses `checksum "package-lock.json"`
- File deleted or not in repository

**Solution:**
```bash
# DON'T delete package-lock.json
# Instead, regenerate it:

cd /path/to/service-repo
npm install --force --ignore-scripts

git add package-lock.json
git commit -m "Regenerate package-lock.json"
git push origin develop
```

**Why `--ignore-scripts`?**
- Skips postinstall scripts (like node-sass build)
- Prevents build failures on incompatible Node versions
- Still generates valid package-lock.json for CircleCI cache

✅ **Verify fix:**
- CircleCI cache restoration succeeds
- Build uses cached dependencies

### Issue 7: SNS Notifications Not Received

**Symptom:**
- Deployments succeed but no email notifications

**Causes:**
1. SNS subscription not confirmed
2. Wrong SNS topic ARN
3. Email address not subscribed

**Solution:**

**A. Check SNS topic:**
```bash
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices \
  --region ca-central-1
```

**B. Subscribe email:**
```bash
aws sns subscribe \
  --topic-arn arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices \
  --protocol email \
  --notification-endpoint your-email@example.com \
  --region ca-central-1

# Check email and confirm subscription
```

**C. Verify notification step in CircleCI:**
```yaml
- run:
    name: Send deployment success notification
    command: |
      aws sns publish \
        --region ca-central-1 \
        --topic-arn "arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices" \
        --subject "[${DEPLOY_ENV^^}] ✅ Service Deployed" \
        --message file:///tmp/sns-message.txt
```

✅ **Verify fix:**
- Run deployment
- Check email for notification
- Should receive within 1-2 minutes

---

## Testing and Verification

### Pre-Deployment Checklist

**Before pushing to trigger pipeline:**

- [ ] All CircleCI configs committed to tbi-devops-environments
- [ ] All deployment scripts include requirements.yaml update
- [ ] Secrets configured for all environments
- [ ] CircleCI project settings configured
- [ ] Contexts assigned to workflows
- [ ] DEPLOY_ENV set in environment-specific contexts
- [ ] GitHub workflows added to master branch
- [ ] Dependencies resolved (emotion packages if applicable)

### Post-Deployment Verification

**After successful deployment:**

1. **Check CircleCI Build:**
   ```bash
   # All jobs should be green
   # Check each step's logs for errors
   ```
   - [ ] lint-and-test passed
   - [ ] build-and-push-images passed
   - [ ] deploy-to-kubernetes passed
   - [ ] SNS notification sent

2. **Verify Image in ECR:**
   ```bash
   aws ecr describe-images \
     --repository-name tbi-package-travelbrands \
     --region ca-central-1 \
     --query 'imageDetails[0].imageTags' \
     --output table
   ```
   - [ ] Image exists with correct tag
   - [ ] Tag format: `{VERSION}-{BRANCH}-{BUILD_NUM}`

3. **Check Kubernetes Pods:**
   ```bash
   # Primary service
   kubectl get pods -n access -l app=tbi-package-travelbrands
   kubectl logs -n access deployment/tbi-package-travelbrands --tail=50

   # Secondary service
   kubectl get pods -n privatelabel -l app=tbi-pl-dynamic
   kubectl logs -n privatelabel deployment/tbi-pl-dynamic --tail=50
   ```
   - [ ] Pods running (not CrashLoopBackOff)
   - [ ] No errors in logs
   - [ ] Environment variables loaded correctly

4. **Verify Values Files:**
   ```bash
   cd /path/to/tbi-environments

   # Check primary service
   cat dev/values/tbi-package-travelbrands.yaml | grep image:

   # Check secondary service (should match)
   cat dev/values/tbi-pl-dynamic.yaml | grep image:

   # Both should have same image tag
   ```
   - [ ] Image tags match between services
   - [ ] Image URL points to correct ECR repository

5. **Verify requirements.yaml:**
   ```bash
   cat dev/requirements.yaml | grep -A3 "name: tbi-package-travelbrands"
   cat dev/requirements.yaml | grep -A3 "name: tbi-pl-dynamic"
   ```
   - [ ] Chart versions updated for both services
   - [ ] Versions match image tags

6. **Check Git History:**
   ```bash
   cd /path/to/tbi-environments
   git log --oneline -n 5 dev/values/
   git log --oneline -n 5 dev/requirements.yaml
   ```
   - [ ] Recent commits from CircleCI bot
   - [ ] Commit messages indicate version updates
   - [ ] Both values and requirements updated together

7. **Test Application:**
   ```bash
   # Test primary service endpoint
   curl https://tbi-package-travelbrands.dev.tbidevelopment.ca/health

   # Test secondary service endpoint
   curl https://tbi-pl-dynamic.dev.tbidevelopment.ca/health
   ```
   - [ ] Both services respond correctly
   - [ ] No 5xx errors
   - [ ] Application functionality works

8. **Monitor for 15 minutes:**
   ```bash
   kubectl logs -n access deployment/tbi-package-travelbrands -f
   kubectl logs -n privatelabel deployment/tbi-pl-dynamic -f
   ```
   - [ ] No repeated errors
   - [ ] No crash loops
   - [ ] Memory/CPU stable

### Rollback Procedure

If deployment fails:

1. **Check CircleCI logs for error details**

2. **Rollback using Helm:**
   ```bash
   # List releases
   helm list -n access
   helm list -n privatelabel

   # Rollback primary service
   helm rollback tbi-package-travelbrands -n access

   # Rollback secondary service
   helm rollback tbi-pl-dynamic -n privatelabel
   ```

3. **Revert Git commits:**
   ```bash
   cd /path/to/tbi-environments
   git log --oneline -n 5
   git revert <commit-hash>
   git push origin master
   ```

4. **Investigate and fix issue before retrying**

---

## Reference Implementation

**Use these services as templates:**

- **Primary Service:** `tbi-package-travelbrands`
  - Path: `tbi-devops-environments/dev/tbi-package-travelbrands/`
  - Complete example with build + deploy

- **Secondary Service:** `tbi-pl-dynamic`
  - Path: `tbi-devops-environments/dev/tbi-pl-dynamic/`
  - Complete example with deploy-only

**Key Files to Reference:**

```
tbi-devops-environments/
├── dev/tbi-package-travelbrands/
│   ├── .circleci/config.yml          # Full pipeline
│   ├── docker/Dockerfile              # Image build
│   └── scripts/deploy-services.sh     # Deployment logic
└── dev/tbi-pl-dynamic/
    ├── .circleci/config.yml          # Deploy-only
    └── scripts/deploy-services.sh     # Image tag sync

tbi-environments/
└── dev/
    ├── values/
    │   ├── tbi-package-travelbrands.yaml  # Primary values
    │   └── tbi-pl-dynamic.yaml            # Secondary values
    └── requirements.yaml                   # Chart versions
```

---

## Summary

### Key Patterns Implemented

✅ **Shared Docker Image**
- Primary service builds once
- Secondary service reuses the same image
- Reduces build time and ensures consistency

✅ **Image Tag Synchronization**
- Secondary reads tag from primary's values file
- 12-minute retry logic handles race conditions
- Both services always use matching versions

✅ **Auto-Triggering**
- Primary deployment triggers secondary via empty commit
- Coordinated rollout across namespaces
- Ensures both services stay in sync

✅ **requirements.yaml Tracking**
- Both values file AND requirements.yaml updated
- ArgoCD and Helm chart version tracking
- Git history shows complete audit trail

✅ **Environment Isolation**
- DEPLOY_ENV context variable controls environment
- Prevents cross-environment deployments
- Clear separation between dev/qa/uat

✅ **SNS Notifications**
- Success and failure notifications
- Environment-tagged subjects
- Complete deployment details

---

## Additional Resources

- **MIGRATION-CHECKLIST.md:** General migration guide
- **DEVELOPER-GUIDE.md:** Developer workflows
- **QUICK-REFERENCE.md:** Common commands
- **SECRETS-PIPELINE-SETUP.md:** Secrets management details

---

**Questions?** Contact DevOps team or open an issue in `tbi-devops-environments` repository.