# Claude Context: TBI RedTag CI/CD Centralization

This document provides context for AI assistants working on TBI RedTag microservices migration.

## Project Overview

**Goal**: Centralize all CI/CD configurations from individual service repos into this single `tbi-devops-environments` repository.

**Status**: ✅ ALL REDTAG SERVICES MIGRATED + RLV Services in progress

**Migrated Services**:
- ✅ `tbi-redtag-mobile-hotel-service` (reference implementation with 4 sub-services)
- ✅ `tbi-redtag-mobile-ms-cars`
- ✅ `tbi-redtag-mobile-ms-activities`
- ✅ `tbi-redtag-mobile-ms-bookings`
- ✅ `tbi-redtag-mobile-ms-flights`
- ✅ `tbi-rlv-profile-backend-service` (RLV Profile API - NestJS)
- ✅ `tbi-package-travelbrands` (RLV Package service - primary image builder)
- ✅ `tbi-pl-dynamic` (Private label service - shares image with tbi-package-travelbrands)

## Architecture Decisions

### 1. Centralized vs Decentralized

**Before**:
- Each service repo had: `.circleci/config.yml`, `Dockerfile`, `scripts/`, `Jenkinsfile`, DevOps docs
- Scattered, inconsistent configurations
- Hard to maintain and update

**After**:
- **tbi-devops-environments**: Contains ALL CI/CD configs
- **Service repos**: Contains ONLY application code + `.github/workflows/manage-secrets.yml`

### 2. CircleCI Configuration

CircleCI is configured at the **project level** (not in service repos):

**Pipeline Settings in CircleCI UI**:
- **Config source**: `tbi-devops-environments` (this repo)
- **Config branch**: `master`
- **Config filepath**: `dev/tbi-redtag-mobile-hotel-service/.circleci/config.yml`
- **Checkout source**: `tbi-redtag-mobile-hotel-service` (application repo)

**How it works**:
1. CircleCI fetches `.circleci/config.yml` from tbi-devops-environments
2. Config has `checkout` step that checks out the application code
3. Config clones tbi-devops-environments to get Dockerfile and scripts
4. Builds and deploys using centralized configs

### 3. GitHub Actions (Secrets Management)

Secrets management workflow **stays in service repos** (`.github/workflows/manage-secrets.yml`) because:
- Needs to trigger from service repo events
- Provides UI for developers to manage secrets
- Fetches secrets from `tbi-environments` repo (SOPS encrypted)

### 4. Repository Structure

```
tbi-devops-environments/
├── docs/                                    # Documentation
│   ├── SECRETS-PIPELINE-SETUP.md           # Complete setup guide
│   ├── MIGRATION-CHECKLIST.md              # Step-by-step checklist
│   ├── SHARED-IMAGE-MIGRATION.md           # Shared image deployment pattern
│   ├── DEPLOYMENT-SCRIPT-GUIDE.md          # Deployment script patterns & fixes
│   ├── DEVELOPER-GUIDE.md                  # Developer workflows
│   ├── QUICK-REFERENCE.md                  # Command reference
│   └── CLAUDE-CONTEXT.md                   # This file
├── scripts/
│   └── migrate-service.sh                  # Automation script
└── <env>/                                  # dev, qa, uat, redtag-prod
    └── <service-name>/
        ├── .circleci/
        │   └── config.yml                  # CircleCI config
        ├── .github/
        │   └── workflows/
        │       └── manage-secrets.yml      # GitHub Actions (copied here for reference)
        ├── docker/
        │   └── Dockerfile                  # Docker build config
        ├── scripts/
        │   ├── deploy-services.sh          # Helm deployment
        │   └── setup-kubectl.sh            # Kubectl setup
        └── jenkins/
            └── Jenkinsfile                 # Jenkins (deprecated)
```

## Critical Gotchas

### 🔴 GOTCHA #1: YAML Secrets Format (MOST IMPORTANT)

**Problem**: Node.js apps parse dotenv files line-by-line. Incorrect YAML format causes ALL environment variables to be `undefined`.

**Symptom in pod logs**:
```
wait for config loaded undefined
env stage undefined
redis host undefined
```

**WRONG Formats** ❌:
```yaml
# Single line with spaces
configs:
    dotenv: DB_HOST=example.com DB_USERNAME=admin DB_PASSWORD=secret

# Literal \n characters
configs:
    dotenv: "DB_HOST=example.com\nDB_USERNAME=admin\nDB_PASSWORD=secret"

# Quoted multi-line
configs:
    dotenv: "DB_HOST=example.com
        DB_USERNAME=admin
        DB_PASSWORD=secret"
```

**CORRECT Format** ✅:
```yaml
configs:
    dotenv: |-
        DB_HOST=example.com
        DB_USERNAME=admin
        DB_PASSWORD=secret
        # More variables...
```

**Key points**:
- MUST use `|-` literal block scalar
- Each variable on its own line (actual newlines, not `\n`)
- 8-space indentation for variable lines (2 levels: configs + dotenv)
- No quotes around the entire block
- No duplicate keys

**How to fix if broken**:
See `QUICK-REFERENCE.md` section "Fix Broken Format" for Python script.

### 🔴 GOTCHA #2: SOPS Directory Context

**Problem**: `.sops.yaml` is in environment directories (dev/qa/uat), not repo root.

**WRONG** ❌:
```bash
cd tbi-environments
sops -e -i dev/secrets/secrets.tbi-redtag-mobile-ms-flights.yaml  # FAILS
```

**CORRECT** ✅:
```bash
cd tbi-environments/dev
sops -e -i secrets/secrets.tbi-redtag-mobile-ms-flights.yaml  # WORKS
```

### 🔴 GOTCHA #3: CircleCI Needs to Fetch Centralized Files

CircleCI config MUST clone tbi-devops-environments and copy files:

```yaml
- checkout  # Checks out application code
- run:
    name: Stage Dockerfile from tbi-devops-environments
    command: |
      repoURL="https://${GITHUB_TOKEN}@github.com/redlabelvacations/tbi-devops-environments"
      git clone $repoURL -b master --single-branch tbi-devops-environments

      # Map branch to environment
      case ${CIRCLE_BRANCH} in
        develop) envFolder="dev" ;;
        qa) envFolder="qa" ;;
        uat) envFolder="uat" ;;
        master|main) envFolder="redtag-prod" ;;
      esac

      # Copy Dockerfile
      cp tbi-devops-environments/${envFolder}/<service>/docker/Dockerfile ./
```

### 🔴 GOTCHA #4: AWS Cognito Client ID Mismatches

**Symptom**: 401 Unauthorized errors even with valid JWT

**Cause**: JWT `aud` claim doesn't match `AWS_AUTH_COGNITO_CLIENT_ID`

**Fix**:
1. Decode JWT token (use jwt.io)
2. Check `aud` claim value
3. Verify it matches `AWS_AUTH_COGNITO_CLIENT_ID` in secrets
4. Update if needed and restart pods

### 🔴 GOTCHA #5: Duplicate Environment Variables

**Symptom**: Variables appear twice in secrets file

**Cause**: Update workflow was not checking for duplicate keys before appending to line_order

**Fix Applied**: ✅ Fixed in dev environment (2025-11-28)
- GitHub Actions workflow now properly de-duplicates keys during parsing
- Added check: `if key not in line_order:` before appending keys
- This prevents duplicates from being created when updating secrets

**Manual fix for existing duplicates**: See `QUICK-REFERENCE.md` section "Fix Broken Format"

**Note**: Fix only applied to `dev/` environment workflows. Apply to qa/uat/redtag-prod when those branches are created.

## Related Repositories

### tbi-environments
**Purpose**: Encrypted secrets storage (SOPS)
**URL**: `https://github.com/redlabelvacations/tbi-environments`
**Structure**:
```
tbi-environments/
├── dev/
│   ├── .sops.yaml
│   └── secrets/
│       └── secrets.tbi-redtag-mobile-ms-<service>.yaml
├── qa/
└── uat/
```

### tbi-devops-common
**Purpose**: Shared DevOps resources (Helm charts, K8s configs)
**URL**: `https://github.com/redlabelvacations/tbi-devops-common`
**Contains**:
- Helm charts
- Kubernetes manifests
- Self-hosted GitHub Actions runner chart: `app/mgt/apps/gha-runner/`

### Service Repositories
**Purpose**: Application code ONLY
**Examples**:
- `tbi-redtag-mobile-hotel-service` ✅ (migrated)
- `tbi-redtag-mobile-ms-flights` ⏳
- `tbi-redtag-mobile-ms-cars` ⏳
- `tbi-redtag-mobile-ms-activities` ⏳
- `tbi-redtag-mobile-ms-bookings` ⏳

## Migration Process (Quick Reference)

### For Each New Service:

1. **Create directory structure**:
```bash
for env in dev qa uat redtag-prod; do
  mkdir -p $env/tbi-redtag-mobile-ms-<service>/{.circleci,.github/workflows,docker,scripts,jenkins}
done
```

2. **Copy configs from hotel-service** (reference implementation):
```bash
# Use hotel-service as template
cp -r dev/tbi-redtag-mobile-hotel-service/.circleci dev/tbi-redtag-mobile-ms-<service>/
cp -r dev/tbi-redtag-mobile-hotel-service/docker dev/tbi-redtag-mobile-ms-<service>/
cp -r dev/tbi-redtag-mobile-hotel-service/scripts dev/tbi-redtag-mobile-ms-<service>/
# Repeat for qa, uat, redtag-prod
```

3. **Update service names** in all copied files:
```bash
cd dev/tbi-redtag-mobile-ms-<service>
find . -type f -exec sed -i '' 's/hotel-service/<service>/g' {} +
```

4. **Create secrets files** in tbi-environments:
```bash
cd /path/to/tbi-environments/dev
# Copy from copolo equivalent or create new
sops -e -i secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml
```

5. **Clean up service repo**:
```bash
cd /path/to/tbi-redtag-mobile-ms-<service>
git rm -rf .circleci scripts Dockerfile Jenkinsfile
git rm *.md  # Remove DevOps docs
# Update README to reference tbi-devops-environments
git commit -m "Centralize CI/CD configs to tbi-devops-environments"
```

6. **Configure CircleCI pipeline** (in CircleCI UI):
- Config source: `tbi-devops-environments`
- Config filepath: `dev/tbi-redtag-mobile-ms-<service>/.circleci/config.yml`
- Checkout source: `tbi-redtag-mobile-ms-<service>`

7. **Test**:
- Trigger CircleCI build (push to develop)
- Run GitHub Actions secrets workflow
- Verify pods start correctly
- Check pod logs for config loading

## Key AWS Resources

### KMS
**Dev KMS Key**: `arn:aws:kms:ca-central-1:594765360358:key/e558a999-9f99-432e-9d6b-73f06de7d1a7`

### SNS
**Secrets Pipeline Topic**: `arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices`

### EKS Clusters
- **dev**: TBD
- **qa**: TBD
- **uat**: TBD
- **prod**: TBD

### Namespaces
- Most RedTag services: `redtag`

## Service-Specific Information

### tbi-redtag-mobile-hotel-service ✅
**Status**: Fully migrated (reference implementation)
**Namespace**: `redtag`
**Services**: 4 (main, sqs, cron, cronone)
**Special notes**:
- Multiple services built from same repo
- Secrets format was broken (single-line) - fixed to `|-` format
- Had duplicate `AWS_AUTH_COGNITO_CLIENT_ID` - cleaned up

### tbi-redtag-mobile-ms-flights ✅
**Status**: Fully migrated
**Copolo equivalent**: `tbi-copolo-mobile-ms-flights`
**Namespace**: `redtag`
**Services**: 1 (main)
**Special notes**:
- Flight API credentials (Sabre/Amadeus)
- Migrated 2025-11-19

### tbi-redtag-mobile-ms-cars ✅
**Status**: Fully migrated
**Copolo equivalent**: `tbi-copolo-mobile-ms-cars`
**Namespace**: `redtag`
**Services**: 1 (main)
**Special notes**:
- Car rental API credentials
- Mapping service config
- Migrated 2025-11-19

### tbi-redtag-mobile-ms-activities ✅
**Status**: Fully migrated
**Copolo equivalent**: `tbi-copolo-mobile-ms-activities`
**Namespace**: `redtag`
**Services**: 1 (main)
**Special notes**:
- Activities supplier credentials
- Migrated 2025-11-19

### tbi-redtag-mobile-ms-bookings ✅
**Status**: Fully migrated
**Copolo equivalent**: `tbi-copolo-mobile-ms-bookings`
**Namespace**: `redtag`
**Services**: 1 (main)
**Special notes**:
- Payment gateway credentials
- All other service endpoints
- SNS/SQS configuration
- Migrated 2025-11-19

### tbi-rlv-profile-backend-service ✅
**Status**: Fully migrated
**Technology**: NestJS + TypeScript
**Namespace**: `redtag`
**Services**: 1 (main)
**Special notes**:
- User profile management API
- DynamoDB single-table design
- AWS Cognito authentication
- Kinesis Firehose logging
- Redis caching
- S3 integration
- Travcom API integration
- Health endpoint: `/health`
- Port: 3000
- Ingress: `tbi-rlv-profile-backend.dev.tbidevelopment.ca`
- Migrated 2025-12-02

### tbi-package-travelbrands ✅
**Status**: Fully migrated (all environments: dev, qa, uat, prod)
**Technology**: PHP 8.0 + Apache + React
**Namespace**: `access`
**Services**: 1 (main)
**Special notes**:
- Builds Docker image used by tbi-pl-dynamic
- Production deployments with cross-account KMS access
- Emotion dependencies required for React components
- Shared image pattern (primary service)
- Jenkins blacklist configured
- Migrated 2025-12-09

### tbi-pl-dynamic ✅
**Status**: Fully migrated (all environments: dev, qa, uat, prod)
**Technology**: PHP 8.0 + Apache + React (shared image)
**Namespace**: `privatelabel`
**Services**: 1 (main)
**Special notes**:
- Reuses Docker image from tbi-package-travelbrands
- Deploy-only pipeline (no build step)
- 12-minute image verification retry logic
- Shared image pattern (secondary service)
- Jenkins blacklist configured
- Migrated 2025-12-09

## Environment Mapping

| Git Branch | Environment Folder | Cluster |
|------------|-------------------|---------|
| `develop` | `dev/` | dev |
| `qa` | `qa/` | qa |
| `uat` | `uat/` | uat |
| `master`/`main` | `redtag-prod/` | prod |

## Testing Checklist

After migrating a service:

- [ ] CircleCI build succeeds
- [ ] Docker image built and pushed to ECR
- [ ] Helm deployment succeeds
- [ ] Pods start without CrashLoopBackOff
- [ ] Pod logs show: `env stage dev` (not `undefined`)
- [ ] Pod logs show: `redis host <host>` (not `undefined`)
- [ ] API health endpoint responds 200
- [ ] No 401 Unauthorized errors with valid JWT
- [ ] Secrets can be viewed via GitHub Actions
- [ ] Secrets can be updated via GitHub Actions
- [ ] Pod restart after secret update works

## Common Commands

### View Secrets
```bash
cd tbi-environments/dev
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml
```

### Update Secrets
```bash
cd tbi-environments/dev
sops secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml
# Edit, save (auto-encrypts)
git add . && git commit -m "Update secrets" && git push
```

### Check Pod Logs
```bash
kubectl logs -n redtag deployment/tbi-redtag-mobile-ms-<service> --tail=50
```

### Restart Pods
```bash
kubectl rollout restart deployment/tbi-redtag-mobile-ms-<service> -n redtag
```

### Verify Secret Content
```bash
kubectl get secret -n redtag tbi-redtag-mobile-ms-<service>-secrets \
  -o jsonpath='{.data.dotenv}' | base64 -d | head -20
```

## Self-Hosted GitHub Actions Runner

**Location**: `tbi-devops-common/app/mgt/apps/gha-runner`
**Cluster**: Management cluster
**Labels**: `self-hosted,linux,x64,tbi-github-runner`
**Authentication**: Node instance role (NOT IRSA)
**Access**:
- KMS decrypt via node instance role
- SNS publish
- EKS kubectl access (via aws-auth ConfigMap)

## Company Policies

### Commit Messages
❌ **DO NOT** use Claude signatures:
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
```

✅ **Use plain commit messages**:
```
Add secrets management workflow for flights service
```

### Jenkins
Jenkins is **deprecated**. Jenkinsfiles are kept in `jenkins/` folder for reference only.

## Documentation

- **Complete Setup Guide**: `docs/SECRETS-PIPELINE-SETUP.md`
- **Migration Checklist**: `docs/MIGRATION-CHECKLIST.md`
- **Shared Image Migration**: `docs/SHARED-IMAGE-MIGRATION.md`
- **Deployment Script Guide**: `docs/DEPLOYMENT-SCRIPT-GUIDE.md`
- **Developer Guide**: `docs/DEVELOPER-GUIDE.md`
- **Quick Reference**: `docs/QUICK-REFERENCE.md`
- **Automation Script**: `scripts/migrate-service.sh`
- **This File**: `CLAUDE-CONTEXT.md`

## Next Steps

✅ **All CI/CD configs migrated to centralized repository!**

**For each service repo, complete the following**:
1. Create secrets files in `tbi-environments` repo (if not already done)
2. Configure CircleCI pipeline to use centralized configs
3. Clean up service repos (remove Dockerfile, .circleci, scripts)
4. Test full CI/CD pipeline (build, push, deploy)
5. Verify secrets management workflow
6. Update service README to reference centralized configs

**Future work**:
- Archive/remove Copolo services once RedTag services confirmed working
- Monitor and optimize CI/CD pipelines
- Consider ArgoCD for GitOps-based deployments

## Questions?

Refer to:
1. `SECRETS-PIPELINE-SETUP.md` for detailed technical info
2. `MIGRATION-CHECKLIST.md` for step-by-step process
3. `QUICK-REFERENCE.md` for commands
4. `tbi-redtag-mobile-hotel-service` configs as reference implementation

---

**Last Updated**: 2025-12-09
**Status**: ✅ All 5 RedTag services + 3 RLV services migrated
**Contact**: DevOps team