# Deployment Script Guide for TBI Services

**Comprehensive guide for Claude agents and DevOps engineers**

> **Purpose:** Document deployment script patterns, common fixes, and troubleshooting for TBI microservices
> **Last Updated:** 2026-01-12
> **Maintainer:** DevOps Team

## Recent Changes (2026-01-13)

### ✅ Environment-Specific SOPS Configuration
- **Status**: ✅ IMPLEMENTED for tbi-package-travelbrands production
- **Issue**: SOPS used wrong KMS key when secrets encrypted with incorrect environment key
- **Solution**: Change to tbi-environments/{env} directory before helm secrets to use correct .sops.yaml
- **Benefit**: Ensures environment-specific KMS keys are always used correctly

### ✅ Production KMS Access Fix
- **Status**: ✅ IMPLEMENTED for tbi-package-travelbrands
- **Issue**: SOPS decryption failed due to missing AWS credentials export
- **Solution**: Explicitly export AWS credentials before helm secrets command
- **Root Cause**: Production secrets file encrypted with UAT KMS key instead of production key

### ✅ No-Changes Fix Implementation
- **Status**: ✅ IMPLEMENTED across all environments
- **Issue**: Deployment scripts failed when no changes needed to be committed
- **Solution**: Added `git diff --cached --quiet` check before committing
- **Files Updated**: All `deploy-services.sh` scripts in dev, uat, and prod environments

## Recent Changes (2026-01-12)

### ✅ New Service Added
- **tbi-rlv-profile-frontend**: Added dev environment configuration
  - Created missing values file in tbi-environments repository
  - Fixed CircleCI deployment pipeline

### ✅ CircleCI Workflow Names Standardized
Updated all dev services to use service-specific workflow names instead of generic "build-and-deploy":
- tbi-rlv-profile-frontend ✅
- tbi-rlv-profile-backend-service ✅  
- tbi-redtag-mobile-ms-flights ✅
- tbi-rlv-vacations-frontend ✅
- tbi-redtag-mobile-ms-bookings ✅
- tbi-redtag-mobile-hotel-service ✅
- tbi-redtag-mobile-ms-cars ✅
- tbi-redtag-mobile-ms-activities ✅

### ✅ Deployment Script Fix Applied
- **tbi-rlv-profile-frontend**: Applied "no changes" fix to prevent commit failures on redeployments

---

## Table of Contents

1. [Overview](#overview)
2. [Deployment Script Anatomy](#deployment-script-anatomy)
3. [Environment Setup (dev, qa, uat, prod)](#environment-setup-dev-qa-uat-prod)
4. [Critical Fixes and Patterns](#critical-fixes-and-patterns)
5. [Production Deployments](#production-deployments)
6. [Jenkins Blacklist Configuration](#jenkins-blacklist-configuration)
7. [Troubleshooting Common Issues](#troubleshooting-common-issues)
8. [Testing and Verification](#testing-and-verification)

---

## Overview

### Repository Structure

All deployment scripts follow a consistent pattern across environments:

```
tbi-devops-environments/
├── dev/
│   └── {service-name}/
│       ├── .circleci/config.yml
│       ├── docker/Dockerfile
│       ├── scripts/
│       │   ├── deploy-services.sh       ← Main deployment script
│       │   └── setup-kubectl.sh         ← Kubernetes setup
│       └── charts/{service-name}/
│           ├── Chart.yaml
│           ├── values.yaml
│           └── templates/
├── qa/           # Same structure
├── uat/          # Same structure
└── prod/         # Same structure (with additional safety checks)
```

### Related Repositories

**tbi-environments** (values & secrets):
```
tbi-environments/
├── dev/
│   ├── values/{service-name}.yaml      ← Service configuration
│   ├── requirements.yaml                ← Helm chart versions
│   └── secrets/secrets.{service-name}.yaml  ← Encrypted secrets
```

**tbi-devops-common** (Helm charts):
- Shared Helm chart templates
- Common Kubernetes manifests

---

## Deployment Script Anatomy

### Standard Script Structure

Every `deploy-services.sh` follows this pattern:

```bash
#!/bin/bash
set -e  # Exit on any error

echo "=== Starting deployment of {Service Name} ==="

# ========================================
# CONFIGURATION
# ========================================
SERVICE_NAME="{service-name}"
NAMESPACE="{namespace}"
ENVIRONMENT="${DEPLOY_ENV:-dev}"
AWS_REGION="${AWS_REGION:-ca-central-1}"
TBI_ENVIRONMENTS_BRANCH="${TBI_ENVIRONMENTS_BRANCH:-master}"
HELM_REPO_URL="${HELM_REPO_URL:-https://tbi-charts.s3.amazonaws.com}"
HELM_S3_BUCKET="${HELM_S3_BUCKET:-tbi-charts}"

# For shared image deployments
IMAGE_SERVICE_NAME="${IMAGE_SERVICE_NAME:-${SERVICE_NAME}}"  # Override if reusing image

# ========================================
# VALIDATION
# ========================================
# Validate required environment variables
if [ -z "${CLUSTER_NAME}" ]; then
    echo "ERROR: CLUSTER_NAME environment variable is not set!"
    exit 1
fi

if [ -z "${AWS_ACCOUNT_ID}" ]; then
    echo "ERROR: AWS_ACCOUNT_ID environment variable is not set!"
    exit 1
fi

if [ -z "${GITHUB_TOKEN}" ]; then
    echo "ERROR: GITHUB_TOKEN environment variable is not set!"
    exit 1
fi

# ========================================
# FUNCTIONS
# ========================================

# 1. Package Helm chart
package_helm_chart() { ... }

# 2. Push chart to S3
push_chart_to_s3_repo() { ... }

# 3. Clone tbi-environments
clone_tbi_environments() { ... }

# 4. Update values file with image URL
update_values_file() { ... }

# 5. Update requirements.yaml
update_requirements() { ... }

# 6. Commit and push updates
push_values_update() { ... }

# 7. Deploy service
deploy_service() { ... }

# ========================================
# MAIN DEPLOYMENT FLOW
# ========================================
echo "=== Step 1: Package Helm chart ==="
package_helm_chart

echo "=== Step 2: Initialize Helm S3 repository ==="
helm repo add tbi-charts s3://${HELM_S3_BUCKET}/

echo "=== Step 3: Push chart to S3 repository ==="
push_chart_to_s3_repo

echo "=== Step 4: Clone tbi-environments repository ==="
clone_tbi_environments

echo "=== Step 5: Update values file with image URL ==="
update_values_file

echo "=== Step 6: Update requirements.yaml ==="
update_requirements

echo "=== Step 7: Commit and push values file update ==="
push_values_update

echo "=== Step 8: Update Helm repository ==="
helm repo update

echo "=== Step 9: Deploy service to Kubernetes ==="
deploy_service

echo "=== Deployment Complete ==="
```

---

## Environment Setup (dev, qa, uat, prod)

### Environment Variables by Environment

| Variable | dev | qa | uat | prod |
|----------|-----|----|----|------|
| `DEPLOY_ENV` | `dev` | `qa` | `uat` | `prod` |
| `CLUSTER_NAME` | `dev` | `qa` | `uat` | `tbi-prod` |
| `AWS_ACCOUNT_ID` | `594765360358` | `594765360358` | `594765360358` | `489176460030` |
| `HELM_S3_BUCKET` | `tbi-charts` | `tbi-charts` | `tbi-charts` | `tbi-charts-prod` |
| `NAMESPACE` | Service-specific | Service-specific | Service-specific | Service-specific |

### Namespace Examples

- `access` - Package Travelbrands services
- `privatelabel` - Private label / white-label services
- `redtag` - RedTag mobile services

### CircleCI Context Configuration

Each environment requires specific CircleCI contexts:

**dev environment:**
```yaml
context:
  - devops-mgt  # Base context with shared variables
```

**qa environment:**
```yaml
context:
  - devops-mgt
  - devops-mgt-qa  # Contains DEPLOY_ENV=qa
```

**uat environment:**
```yaml
context:
  - devops-mgt
  - devops-mgt-uat  # Contains DEPLOY_ENV=uat
```

**prod environment:**
```yaml
context:
  - devops-mgt-prod  # Contains DEPLOY_ENV=prod + prod credentials
```

---

## Critical Fixes and Patterns

### Fix 1: Pipeline Failure When Image Already Exists

**Problem:**
When redeploying the same version or when values file is already up-to-date, the pipeline fails with:
```
nothing to commit, working tree clean
Exited with code exit status 1
```

**Solution:**
Check for changes before committing:

```bash
push_values_update() {
    echo "=== Pushing values file updates ==="

    cd /tmp/tbi-environments

    # Configure git
    git config user.name "CircleCI Bot"
    git config user.email "circleci@tbidevelopment.ca"

    # Show changes
    echo "Changes made:"
    git diff ${ENVIRONMENT}/values/
    git diff ${ENVIRONMENT}/requirements.yaml

    # Stage files for commit
    git add ${ENVIRONMENT}/values/${SERVICE_NAME}.yaml
    git add ${ENVIRONMENT}/requirements.yaml

    # ✅ CHECK IF THERE ARE ANY CHANGES BEFORE COMMITTING
    if git diff --cached --quiet; then
        echo "ℹ️  No changes detected - values already up to date with version ${CHART_VERSION}"
        echo "Skipping commit and continuing with deployment..."
    else
        # Commit and push values file AND requirements.yaml
        git commit -m "Update ${SERVICE_NAME} to ${CHART_VERSION}

Deployed by CircleCI build #${CIRCLE_BUILD_NUM}
Branch: ${CIRCLE_BRANCH}
Service: ${SERVICE_NAME}

Changes:
- Updated values file with ECR image URL: ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${IMAGE_SERVICE_NAME}:${CHART_VERSION}
- Updated requirements.yaml with chart version: ${CHART_VERSION}
"
        git push origin ${TBI_ENVIRONMENTS_BRANCH}
        echo "✅ Values file and requirements.yaml updated and pushed"
    fi
}
```

**Key points:**
- ✅ Always stage files first with `git add`
- ✅ Check `git diff --cached --quiet` before committing
- ✅ Continue with deployment even if no changes
- ✅ This prevents pipeline failures on re-deployments

**Implementation Status:**
- ✅ tbi-rlv-profile-frontend (dev) - Fixed 2026-01-12
- ✅ tbi-pl-dynamic (dev, uat, prod) - Previously fixed
- ✅ tbi-package-travelbrands (prod) - Fixed 2026-01-13 (includes SOPS directory fix)
- ✅ All other services - Fixed 2026-01-12

**Where to apply:**
- ✅ All environment deployment scripts (dev, qa, uat, prod)
- ✅ Both standard and shared-image deployment patterns

---

### Fix 2: Always Update requirements.yaml

**Problem:**
Values file gets updated but `requirements.yaml` is not, causing:
- ArgoCD not detecting changes
- Helm chart version tracking out of sync
- Difficult to audit deployments

**Solution:**
Always call `update_requirements()` and commit both files together:

```bash
# Function to update requirements.yaml
update_requirements() {
    echo "=== Updating requirements.yaml for ${SERVICE_NAME} ==="

    local req_file="/tmp/tbi-environments/${ENVIRONMENT}/requirements.yaml"

    if [ ! -f "${req_file}" ]; then
        echo "ERROR: Requirements file not found: ${req_file}"
        return 1
    fi

    # Update version using sed
    sed -i.bak "/name: ${SERVICE_NAME}/,/version:/ s|version:.*|version: ${CHART_VERSION}|" ${req_file}
    rm -f ${req_file}.bak

    echo "✅ Updated ${SERVICE_NAME} to version ${CHART_VERSION} in requirements.yaml"
}

# Main deployment flow
echo "=== Step 6: Update requirements.yaml ==="
update_requirements  # ← MUST CALL THIS

echo "=== Step 7: Commit and push values file update ==="
push_values_update  # ← This commits BOTH values AND requirements.yaml
```

**In push_values_update function:**
```bash
# Stage BOTH files
git add ${ENVIRONMENT}/values/${SERVICE_NAME}.yaml
git add ${ENVIRONMENT}/requirements.yaml  # ← Critical!
```

**Verification:**
```bash
cd /path/to/tbi-environments
git log --oneline dev/requirements.yaml  # Should show recent commits
```

---

### Fix 3: Shared Image Deployment Pattern

**For services that reuse another service's Docker image:**

Example: `tbi-pl-dynamic` uses `tbi-package-travelbrands` image

**Configuration:**
```bash
#!/bin/bash
set -e

SERVICE_NAME="tbi-pl-dynamic"
NAMESPACE="privatelabel"

# IMPORTANT: This service uses the Docker image from tbi-package-travelbrands
IMAGE_SERVICE_NAME="tbi-package-travelbrands"
```

**update_values_file function:**
```bash
update_values_file() {
    echo "=== Updating values file for ${SERVICE_NAME} ==="

    local values_file="/tmp/tbi-environments/${ENVIRONMENT}/values/${SERVICE_NAME}.yaml"

    # Construct ECR image URL - NOTE: Uses IMAGE_SERVICE_NAME (primary service)
    local image_url="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${IMAGE_SERVICE_NAME}:${CHART_VERSION}"

    echo "Shared image URL: ${image_url}"

    # Update image field
    if grep -q "^image:" "${values_file}"; then
        sed -i.bak "s|^image:.*|image: ${image_url}|" ${values_file}
        echo "Updated existing image field"
    else
        sed -i.bak "1 a\\
image: ${image_url}" ${values_file}
        echo "Added new image field"
    fi
    rm -f ${values_file}.bak

    echo "✅ Updated ${SERVICE_NAME} image to ${image_url}"
    echo "NOTE: Image is shared with ${IMAGE_SERVICE_NAME}"
}
```

**Key points:**
- Primary service builds the image
- Secondary service reads image tag and reuses it
- Both services update their own values files
- Both services update requirements.yaml
- Image URL points to primary service's ECR repository

See [SHARED-IMAGE-MIGRATION.md](./SHARED-IMAGE-MIGRATION.md) for complete guide.

---

## Production Deployments

### Production Safety Checks

Production scripts include additional validation:

```bash
# ========================================
# PRODUCTION SAFETY CHECKS
# ========================================

echo "🔒 ========================================="
echo "🔒 PRODUCTION DEPLOYMENT SAFETY CHECKS"
echo "🔒 ========================================="

# Check 1: Verify DEPLOY_ENV is set to prod
if [ -z "${DEPLOY_ENV}" ]; then
    echo "❌ ERROR: DEPLOY_ENV environment variable is not set!"
    echo "Expected: prod"
    echo "This should be set in CircleCI context 'devops-mgt-prod'"
    exit 1
fi

if [ "${DEPLOY_ENV}" != "prod" ]; then
    echo "❌ ERROR: DEPLOY_ENV is not 'prod'!"
    echo "Expected: prod"
    echo "Got: ${DEPLOY_ENV}"
    echo "Production deployments MUST have DEPLOY_ENV=prod"
    exit 1
fi

echo "✅ DEPLOY_ENV verified: ${DEPLOY_ENV}"

# Check 2: Verify cluster name is production
if [[ ! "${CLUSTER_NAME}" =~ prod ]]; then
    echo "❌ ERROR: CLUSTER_NAME does not contain 'prod'!"
    echo "Expected: tbi-prod"
    echo "Got: ${CLUSTER_NAME}"
    echo "Refusing to deploy to non-production cluster in production pipeline!"
    exit 1
fi

echo "✅ Cluster verified: ${CLUSTER_NAME}"

# Check 3: Validate production image tag
echo "Validating PRODUCTION image tag: ${CHART_VERSION}"

# Reject dev/qa/uat tags
if [[ "${CHART_VERSION}" =~ (develop|qa|uat) ]]; then
    echo "❌ ERROR: Cannot deploy dev/qa/uat image tags to production!"
    echo "Image tag: ${CHART_VERSION}"
    echo "Production requires stable release tags only"
    exit 1
fi

# Reject 'latest' tag
if [[ "${CHART_VERSION}" == "latest" ]]; then
    echo "❌ ERROR: 'latest' tag not allowed in production!"
    echo "Production requires explicit version tags"
    exit 1
fi

# Validate production tag format
if [[ "${CHART_VERSION}" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo "✅ Production clean version tag: ${CHART_VERSION}"
elif [[ "${CHART_VERSION}" =~ ^[0-9]+\.[0-9]+\.[0-9]+-master-[0-9]+$ ]]; then
    echo "✅ Production legacy version tag: ${CHART_VERSION}"
else
    echo "⚠️  WARNING: Image tag doesn't match expected production patterns"
    echo "Expected: X.Y.Z or X.Y.Z-master-BUILD"
    echo "Got: ${CHART_VERSION}"
    echo "Continuing anyway... (manual builds may have different patterns)"
fi

echo "✅ Image tag validated for production: ${CHART_VERSION}"
```

### Production Credential Handling

Production uses cross-account KMS access:

**Configuration:**
- Production cluster account: `489176460030`
- KMS key account: `495680544243`
- KMS key must allow cross-account access

**Implementation:**
```bash
deploy_service() {
    echo "=== Deploying ${SERVICE_NAME} using helm secrets (PRODUCTION) ==="

    cd /tmp/tbi-environments

    # Ensure CircleCI context credentials are used (not EC2 instance role)
    if [ -z "${AWS_ACCESS_KEY_ID}" ] || [ -z "${AWS_SECRET_ACCESS_KEY}" ]; then
        echo "❌ ERROR: AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY not set!"
        echo "These must be set in CircleCI context 'devops-mgt-prod'"
        exit 1
    fi

    # Write credentials to file to ensure they take precedence over instance role
    mkdir -p ~/.aws
    cat > ~/.aws/credentials << EOF
[default]
aws_access_key_id = ${AWS_ACCESS_KEY_ID}
aws_secret_access_key = ${AWS_SECRET_ACCESS_KEY}
EOF

    cat > ~/.aws/config << EOF
[default]
region = ${AWS_REGION}
EOF

    # Create .sops.yaml with KMS key (no aws_profile - uses default credentials)
    cat > /tmp/tbi-environments/.sops.yaml << 'SOPSEOF'
creation_rules:
  - kms: arn:aws:kms:ca-central-1:495680544243:key/99547b03-9580-403f-8579-ea1595335830
SOPSEOF

    # Verify production credentials are being used (not instance role)
    echo "Verifying production AWS credentials..."
    PROD_IDENTITY=$(aws sts get-caller-identity --output json)
    PROD_ACCOUNT=$(echo "$PROD_IDENTITY" | jq -r '.Account')
    PROD_ARN=$(echo "$PROD_IDENTITY" | jq -r '.Arn')

    echo "Production identity: $PROD_ARN"
    echo "Production account: $PROD_ACCOUNT"

    # Ensure we're not using an EC2 instance role
    if [[ "$PROD_ARN" == *":assumed-role/"*"/i-"* ]]; then
        echo "❌ ERROR: Using EC2 instance role instead of CircleCI credentials!"
        echo "   Current identity: $PROD_ARN"
        exit 1
    fi

    if [ "$PROD_ACCOUNT" != "489176460030" ]; then
        echo "❌ ERROR: Not using production account credentials!"
        echo "   Expected account: 489176460030"
        echo "   Current account: $PROD_ACCOUNT"
        exit 1
    fi

    echo "✅ Using production credentials for both kubectl and SOPS"
    echo "✅ SOPS will use cross-account KMS access to decrypt secrets"
    echo "   Production account: 489176460030"
    echo "   KMS key account: 495680544243"

    # Deploy using helm secrets with production credentials
    helm secrets upgrade --install \
        --namespace ${NAMESPACE} ${SERVICE_NAME} \
        tbi-charts/${SERVICE_NAME} \
        --set cluster="${CLUSTER_NAME}" \
        --version ${CHART_VERSION} \
        --reset-values \
        -f ./${ENVIRONMENT}/common.yaml \
        -f ./${ENVIRONMENT}/secrets.common.yaml \
        -f ./${ENVIRONMENT}/values/${SERVICE_NAME}.yaml \
        -f ./${ENVIRONMENT}/secrets/secrets.${SERVICE_NAME}.yaml

    # Verify deployment
    echo "Verifying PRODUCTION deployment..."
    kubectl rollout status deployment/${SERVICE_NAME} -n ${NAMESPACE} --timeout=5m

    echo "✅ ${SERVICE_NAME} deployed successfully to PRODUCTION"
}
```

**Why this matters:**
- Production cluster runs in different AWS account
- SOPS needs KMS access from DevOps account
- Must explicitly use CircleCI credentials, not instance role
- Prevents accidental use of wrong credentials

---

## Jenkins Blacklist Configuration

### When to Add Services to Jenkins Blacklist

Add services to Jenkins blacklist when migrating from Jenkins to CircleCI:

**Location:** `tbi-environments/Jenkinsfile`

```groovy
def run_workflow(List<String> list, String commit_message, boolean dry_run) {
    // ========================================
    // CircleCI Migration Blacklist
    // Services migrated to CircleCI - Jenkins will skip deployment
    // ========================================
    def CIRCLECI_MIGRATED_SERVICES = [
        'tbi-redtag-mobile-hotel-service',
        'tbi-redtag-mobile-hotel-sqs-service',
        'tbi-redtag-mobile-hotel-cron-service',
        'tbi-redtag-mobile-hotel-cronone-service',
        'tbi-redtag-mobile-ms-cars',
        'tbi-redtag-mobile-ms-activities',
        'tbi-redtag-mobile-ms-bookings',
        'tbi-redtag-mobile-ms-flights',
        'tbi-package-travelbrands',     // ← Added 2025-12-09
        'tbi-pl-dynamic'                // ← Added 2025-12-09
    ]

    // ... rest of workflow ...

    changes.each { env, components ->
        // Filter out CircleCI-migrated services
        def filtered_components = components.unique().findAll { component ->
            !CIRCLECI_MIGRATED_SERVICES.contains(component)
        }

        def skipped = components.unique() - filtered_components
        if (skipped) {
            println("[${env.toUpperCase()}] Skipping CircleCI-migrated services: ${skipped}")
        }

        filtered_components.each { component ->
            deploy_component(env, component, dry_run)
        }
    }
}
```

**Process:**
1. Migrate service to CircleCI
2. Test CircleCI deployment thoroughly
3. Add service name to `CIRCLECI_MIGRATED_SERVICES` list
4. Commit to `tbi-environments` repository
5. Jenkins will now skip deploying this service

**Verification:**
```bash
# After commit, check Jenkins logs
# Should see: "Skipping CircleCI-migrated services: [service-name]"
```

---

## Troubleshooting Common Issues

### Issue 1: SOPS KMS Key Mismatch

**Problem:** Deployment fails with KMS access denied errors when secrets are encrypted with wrong environment's KMS key.

**Symptoms:**
```
Error: secrets "secrets.tbi-package-travelbrands" is forbidden: 
User "system:serviceaccount:kube-system:aws-load-balancer-controller" 
cannot get resource "secrets" in API group "" in the namespace "access"
```

**Root Cause:** Secrets file encrypted with UAT KMS key but deployed to production environment.

**Solution:**
1. **Immediate Fix:** Use environment-specific SOPS directory
   ```bash
   # Change to correct environment directory before helm secrets
   cd "${TBI_ENVIRONMENTS_DIR}/prod"  # or dev, uat
   helm secrets upgrade --install ...
   ```

2. **Permanent Fix:** Re-encrypt secrets with correct KMS key
   ```bash
   # Decrypt with wrong key (UAT example)
   cd tbi-environments/uat
   sops --decrypt ../prod/secrets/secrets.service.yaml > /tmp/decrypted.yaml
   
   # Re-encrypt with correct key
   cd ../prod
   sops --encrypt /tmp/decrypted.yaml > secrets/secrets.service.yaml
   ```

**Prevention:** Always run SOPS commands from the correct environment directory to use the right `.sops.yaml` configuration.

### Issue 2: "nothing to commit, working tree clean" Error

**Symptom:**
```
Changes made:
On branch master
nothing to commit, working tree clean
Exited with code exit status 1
```

**Cause:**
Script attempts to commit when no changes exist (values file already up-to-date)

**Solution:**
Apply Fix #1 (see above) - check `git diff --cached --quiet` before committing

**Verification:**
```bash
# Re-run deployment - should see:
# "ℹ️  No changes detected - values already up to date with version X.Y.Z"
# "Skipping commit and continuing with deployment..."
```

---

### Issue 2: requirements.yaml Not Updated

**Symptom:**
- Values file shows new version
- requirements.yaml shows old version
- ArgoCD not detecting changes

**Cause:**
- `update_requirements()` function not called
- requirements.yaml not committed

**Solution:**
```bash
# Add to main deployment flow:
echo "=== Step 6: Update requirements.yaml ==="
update_requirements

# Ensure push_values_update commits both files:
git add ${ENVIRONMENT}/values/${SERVICE_NAME}.yaml
git add ${ENVIRONMENT}/requirements.yaml  # ← Add this
```

---

### Issue 3: DEPLOY_ENV Mismatch

**Symptom:**
```
WARNING: DEPLOY_ENV=dev but expected 'uat'
```

**Cause:**
- CircleCI context not properly configured
- DEPLOY_ENV variable missing or wrong

**Solution:**
1. Check CircleCI contexts:
   - `devops-mgt-qa` should have `DEPLOY_ENV=qa`
   - `devops-mgt-uat` should have `DEPLOY_ENV=uat`
   - `devops-mgt-prod` should have `DEPLOY_ENV=prod`

2. Verify workflow contexts in `.circleci/config.yml`:
   ```yaml
   workflows:
     deploy-uat:
       jobs:
         - deploy-to-kubernetes:
             context:
               - devops-mgt
               - devops-mgt-uat  # ← Must include environment-specific context
             filters:
               branches:
                 only:
                   - uat
   ```

---

### Issue 4: Cross-Account KMS Access Denied (Production)

**Symptom:**
```
Error: Failed to decrypt SOPS file
AccessDeniedException: User is not authorized to perform: kms:Decrypt
```

**Cause:**
- Using wrong AWS credentials
- KMS key policy doesn't allow cross-account access
- Using EC2 instance role instead of CircleCI credentials

**Solution:**
1. Verify CircleCI context has correct credentials:
   ```
   AWS_ACCESS_KEY_ID=...
   AWS_SECRET_ACCESS_KEY=...
   ```

2. Ensure deployment script writes credentials to `~/.aws/credentials`

3. Verify identity:
   ```bash
   aws sts get-caller-identity
   # Should show production account: 489176460030
   # Should NOT show instance role (i-xxxxx)
   ```

4. Check KMS key policy allows production IAM principal

---

### Issue 5: Service Still Deploying via Jenkins

**Symptom:**
- Service deployed to CircleCI
- Jenkins still attempting deployments
- Duplicate deployments

**Cause:**
- Service not added to Jenkins blacklist

**Solution:**
Add service to `tbi-environments/Jenkinsfile`:

```groovy
def CIRCLECI_MIGRATED_SERVICES = [
    // ... existing services ...
    'your-service-name'  // ← Add here
]
```

Commit and push to `tbi-environments` repository.

---

## Testing and Verification

### Pre-Deployment Checklist

Before deploying to any environment:

- [ ] Deployment script includes `git diff --cached --quiet` check
- [ ] `update_requirements()` function exists and is called
- [ ] Both values and requirements files are committed together
- [ ] DEPLOY_ENV validation included (for prod/uat)
- [ ] Service name and namespace correctly configured
- [ ] IMAGE_SERVICE_NAME set correctly (for shared images)
- [ ] CircleCI contexts properly configured
- [ ] Secrets files exist in tbi-environments

### Post-Deployment Verification

After successful deployment:

**1. Check tbi-environments commit:**
```bash
cd /path/to/tbi-environments
git log --oneline -n 5 ${ENV}/values/
git log --oneline -n 5 ${ENV}/requirements.yaml
```

Expected:
- Recent commit from CircleCI bot
- Both values and requirements.yaml updated in same commit

**2. Verify values file:**
```bash
cat ${ENV}/values/${SERVICE_NAME}.yaml | grep image:
# Should show: image: {ACCOUNT}.dkr.ecr.{REGION}.amazonaws.com/{SERVICE}:{TAG}
```

**3. Verify requirements.yaml:**
```bash
cat ${ENV}/requirements.yaml | grep -A2 "name: ${SERVICE_NAME}"
# Should show updated version matching image tag
```

**4. Check Kubernetes deployment:**
```bash
kubectl get pods -n ${NAMESPACE} -l app=${SERVICE_NAME}
kubectl logs -n ${NAMESPACE} deployment/${SERVICE_NAME} --tail=50
kubectl rollout status deployment/${SERVICE_NAME} -n ${NAMESPACE}
```

Expected:
- Pods running (not CrashLoopBackOff)
- No errors in logs
- Rollout complete

**5. Verify image in ECR:**
```bash
aws ecr describe-images \
  --repository-name ${SERVICE_NAME} \
  --image-ids imageTag=${IMAGE_TAG} \
  --region ${AWS_REGION}
```

**6. Test application endpoint:**
```bash
# Adjust URL for your environment and service
curl https://${SERVICE_NAME}.${ENV}.tbidevelopment.ca/health
```

---

## Common Deployment Patterns

### Pattern 1: Standard Single-Service Deployment

**Use when:**
- Service builds its own Docker image
- Deploys to one namespace
- Independent from other services

**Example:** `tbi-redtag-mobile-ms-flights`

**Key characteristics:**
- `SERVICE_NAME` == `IMAGE_SERVICE_NAME`
- Builds image in CircleCI
- Updates own values file
- Updates requirements.yaml

---

### Pattern 2: Shared Image Deployment

**Use when:**
- Multiple services use same Docker image
- Different configurations per service
- Different namespaces

**Example:** `tbi-package-travelbrands` (primary) + `tbi-pl-dynamic` (secondary)

**Key characteristics:**
- Primary builds image
- Secondary reuses image
- `IMAGE_SERVICE_NAME` != `SERVICE_NAME` (for secondary)
- Secondary reads image tag from primary's values file
- Both update their own values and requirements files

See [SHARED-IMAGE-MIGRATION.md](./SHARED-IMAGE-MIGRATION.md) for details.

---

### Pattern 3: Multi-Service from Single Repo

**Use when:**
- One repository contains multiple services
- Services share codebase but have different entry points

**Example:** `tbi-redtag-mobile-hotel-service` (4 services: main, sqs, cron, cronone)

**Key characteristics:**
- One Dockerfile, multiple services
- Different command/entrypoint per service
- Separate Helm charts per service
- All use same image tag

---

## Best Practices

### ✅ DO

- ✅ Always check `git diff --cached --quiet` before committing
- ✅ Always update requirements.yaml alongside values file
- ✅ Always validate DEPLOY_ENV in production scripts
- ✅ Always use explicit image tags (never "latest")
- ✅ Always verify credentials in production (prevent instance role usage)
- ✅ Always include SNS notifications for deployment status
- ✅ Always use semantic versioning for image tags

### ❌ DON'T

- ❌ Don't commit without checking for changes first
- ❌ Don't update values file without updating requirements.yaml
- ❌ Don't skip DEPLOY_ENV validation in production
- ❌ Don't use "latest" image tags
- ❌ Don't allow EC2 instance role in production deployments
- ❌ Don't deploy dev/qa/uat tags to production
- ❌ Don't forget to add migrated services to Jenkins blacklist

---

## Reference Scripts

### Complete Example: Standard Deployment

See: `prod/tbi-package-travelbrands/scripts/deploy-services.sh`

Key sections:
- Production safety checks
- Credential verification
- Git commit check
- requirements.yaml update
- Cross-account KMS configuration

### Complete Example: Shared Image Deployment

See: `prod/tbi-pl-dynamic/scripts/deploy-services.sh`

Key sections:
- IMAGE_SERVICE_NAME configuration
- Shared image URL construction
- Same git commit check pattern
- Same requirements.yaml update

---

## Summary

### Key Improvements Implemented

1. **✅ Git Commit Failure Fix**
   - Check for changes before committing
   - Prevents pipeline failures on re-deployments
   - Applied to all environments

2. **✅ requirements.yaml Tracking**
   - Always update alongside values file
   - Enables ArgoCD tracking
   - Provides audit trail

3. **✅ Production Safety**
   - DEPLOY_ENV validation
   - Image tag validation
   - Cluster name verification
   - Credential verification

4. **✅ Cross-Account KMS**
   - Explicit credential handling
   - Instance role prevention
   - Cross-account access verification

5. **✅ Jenkins Migration**
   - Blacklist configuration
   - Prevents duplicate deployments
   - Clean migration path

---

## Additional Resources

- **CLAUDE-CONTEXT.md:** Project overview and architecture
- **MIGRATION-CHECKLIST.md:** Step-by-step migration guide
- **SHARED-IMAGE-MIGRATION.md:** Shared image deployment pattern
- **DEVELOPER-GUIDE.md:** Developer workflows and secrets management
- **QUICK-REFERENCE.md:** Common commands and troubleshooting
- **SECRETS-PIPELINE-SETUP.md:** Secrets management details

---

**Questions or Issues?**

Contact DevOps team or open an issue in `tbi-devops-environments` repository.

**Last Updated:** 2025-12-09