# Developer Guide: CI/CD Pipeline & Secrets Management

> **Audience:** Developers working on TBI RedTag and RLV microservices
> **Last Updated:** 2025-12-02
> **Maintainer:** DevOps Team

---

## Table of Contents

1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Triggering Deployments](#triggering-deployments)
4. [Managing Secrets](#managing-secrets)
5. [Monitoring Deployments](#monitoring-deployments)
6. [Troubleshooting](#troubleshooting)
7. [FAQ](#faq)

---

## Overview

All CI/CD configurations for TBI services have been centralized into the `tbi-devops-environments` repository. Your service repository now only contains **application code** and a GitHub Actions workflow for secrets management.

### What Changed?

**Before:**
```
your-service-repo/
├── .circleci/config.yml      ❌ Removed
├── Dockerfile                 ❌ Removed
├── scripts/                   ❌ Removed
└── src/                       ✅ Your code
```

**After:**
```
your-service-repo/
├── .github/workflows/
│   └── manage-secrets.yml     ✅ Secrets management only
└── src/                       ✅ Your code
```

**All CI/CD configs are now in:** `tbi-devops-environments` repository

---

## Architecture

### Repository Structure

```
📦 tbi-devops-environments (CI/CD configs)
├── dev/your-service/
│   ├── .circleci/config.yml      # Build & deploy pipeline
│   ├── docker/Dockerfile          # Docker image build
│   ├── scripts/                   # Deployment scripts
│   └── charts/                    # Helm charts
├── qa/your-service/               # Same structure
└── uat/your-service/              # Same structure

📦 tbi-environments (Secrets storage)
├── dev/
│   ├── secrets/secrets.your-service.yaml  # Encrypted with SOPS
│   └── values/your-service.yaml           # Service configuration
└── qa/, uat/                               # Same structure

📦 your-service (Application code)
├── .github/workflows/manage-secrets.yml   # Secrets UI
└── src/                                   # Your application code
```

### Branch to Environment Mapping

| Git Branch | Environment | Deploys To |
|------------|-------------|------------|
| `develop`  | dev         | Dev cluster |
| `qa`       | qa          | QA cluster |
| `uat`      | uat         | UAT cluster |
| `master`   | production  | Prod cluster |

---

## Triggering Deployments

### Automatic Deployment

Pushing to configured branches automatically triggers CircleCI:

```bash
# Deploy to dev
git checkout develop
git push origin develop

# Deploy to QA
git checkout qa
git push origin qa

# Deploy to UAT
git checkout uat
git push origin uat
```

### What Happens During Deployment?

```mermaid
graph LR
    A[Push Code] --> B[CircleCI Triggered]
    B --> C[Lint & Test]
    C --> D[Build Docker Image]
    D --> E[Push to ECR]
    E --> F[Deploy to Kubernetes]
    F --> G[Verify Rollout]
```

**Pipeline Stages:**

1. **Lint & Test** (Node.js 20.4)
   - `npm ci` - Install dependencies
   - `npm run lint` - Linting (non-blocking)
   - `npm run test` - Unit tests
   - `npm run build` - Build validation

2. **Build & Push Images**
   - Fetch Dockerfile from `tbi-devops-environments`
   - Build Docker image with tag: `{VERSION}-{BRANCH}-{BUILD_NUM}`
   - Push to AWS ECR with immutable tags

3. **Deploy to Kubernetes**
   - Fetch deployment scripts from `tbi-devops-environments`
   - Package Helm chart
   - Push chart to S3 Helm repository
   - Update image version in `tbi-environments`
   - Deploy using `helm secrets upgrade`
   - Wait for rollout to complete

### Viewing Build Status

**CircleCI Dashboard:**
- URL: `https://app.circleci.com/pipelines/github/redlabelvacations`
- Find your service project
- Click on the latest build

**Kubernetes Pod Status:**
```bash
# Check pod status
kubectl get pods -n redtag | grep your-service

# View pod logs
kubectl logs -n redtag deployment/your-service --tail=100

# Check deployment status
kubectl get deployment your-service -n redtag
```

---

## Shared Docker Image Deployments

### What is a Shared Image Deployment?

Some services share the same Docker image but deploy to different namespaces with different configurations. This pattern reduces build time and ensures consistency across deployments.

**Example:**
- `tbi-package-travelbrands` builds the image and deploys to `access` namespace
- `tbi-pl-dynamic` reuses the same image and deploys to `privatelabel` namespace

### How It Works

```
Primary Service (tbi-package-travelbrands):
  1. Builds Docker image
  2. Pushes to ECR
  3. Deploys to access namespace
  4. Updates image tag in tbi-environments
  5. Triggers secondary service deployment

Secondary Service (tbi-pl-dynamic):
  1. Reads image tag from primary service's values file
  2. Verifies image exists in ECR (with retry)
  3. Deploys to privatelabel namespace
  4. Uses same image but different configuration
```

### Triggering Deployments

**Primary Service (builds image):**
```bash
cd /path/to/tbi-package-travelbrands
git checkout develop
git push origin develop  # Triggers full build + deploy + secondary trigger
```

**Secondary Service (deploy-only):**
```bash
cd /path/to/tbi-pl-dynamic
git push origin develop  # Triggers deploy with existing image
```

**Automatic Trigger:**
When primary service deploys successfully, it automatically triggers secondary service via empty commit.

### Checking Deployment Status

**Primary Service:**
```bash
# Check CircleCI for full pipeline
# Stages: lint-and-test → build-and-push-images → deploy-to-kubernetes

# Check pods
kubectl get pods -n access -l app=tbi-package-travelbrands
kubectl logs -n access deployment/tbi-package-travelbrands --tail=50
```

**Secondary Service:**
```bash
# Check CircleCI for deploy-only pipeline
# Stages: get-image-tag → verify-image → deploy-to-kubernetes

# Check pods
kubectl get pods -n privatelabel -l app=tbi-pl-dynamic
kubectl logs -n privatelabel deployment/tbi-pl-dynamic --tail=50
```

**Verify Image Sync:**
```bash
cd /path/to/tbi-environments

# Both should use the same image tag
cat dev/values/tbi-package-travelbrands.yaml | grep image:
cat dev/values/tbi-pl-dynamic.yaml | grep image:
```

### Important Notes

- **Wait Time:** Secondary service may wait up to 12 minutes for image availability (handles race conditions)
- **Image Repository:** Both services use the **primary service's ECR repository**
- **Version Tracking:** Both services update `requirements.yaml` for Helm/ArgoCD tracking
- **Notifications:** Both services send SNS notifications on deployment success/failure

For detailed migration instructions, see [SHARED-IMAGE-MIGRATION.md](./SHARED-IMAGE-MIGRATION.md).

---

## Managing Secrets

### GitHub Actions Secrets Workflow

Your service repository has a GitHub Actions workflow that provides a **user-friendly UI** for managing secrets.

### Accessing the Secrets Workflow

1. Go to your service repository on GitHub
2. Click **Actions** tab
3. Select **"Manage Secrets"** workflow
4. Click **"Run workflow"** button

### Workflow Options

#### Option 1: View Secrets

**Use Case:** Check current secret values

**Steps:**
1. Select **Environment:** `dev`, `qa`, or `uat`
2. Select **Action:** `view-secrets`
3. Select **Output Method:**
   - `github-summary` - View in GitHub Actions summary (default)
   - `download-artifact` - Download as file (expires in 1 day)
   - `email-sns` - Receive via email (requires SNS setup)
   - `all` - All of the above

**Result:** Decrypted secrets displayed in the workflow summary

⚠️ **Security:** Only users with repository access can view secrets

---

#### Option 2: Update Secrets

**Use Case:** Change specific secret values without restarting pods

**Steps:**
1. Select **Environment:** `dev`, `qa`, or `uat`
2. Select **Action:** `update-secrets`
3. Enter **Secret Updates** as JSON:
   ```json
   {
     "DATABASE_PASSWORD": "new-secure-password",
     "REDIS_PASSWORD": "new-redis-password",
     "API_KEY": "new-api-key-value"
   }
   ```
4. Click **Run workflow**

**Result:**
- ✅ Secrets file updated and encrypted in `tbi-environments`
- ✅ Committed to Git with audit trail
- ⚠️ **Pods NOT restarted** - changes apply on next deployment

---

#### Option 3: Update and Restart

**Use Case:** Update secrets AND immediately apply them to running pods

**Steps:**
1. Select **Environment:** `dev`, `qa`, or `uat`
2. Select **Action:** `update-and-restart`
3. Enter **Secret Updates** as JSON (same format as above)
4. Click **Run workflow**

**Result:**
- ✅ Secrets file updated and encrypted
- ✅ Committed to Git
- ✅ Deployed to Kubernetes cluster
- ✅ Pods restarted with new secrets

**Timeline:** ~2-5 minutes for complete rollout

---

### Secrets File Format

Secrets are stored as `.env` format with proper YAML multi-line blocks:

```yaml
configs:
    dotenv: |-
        AWS_REGION=ca-central-1
        DATABASE_HOST=db.example.com
        DATABASE_PASSWORD=secret123
        REDIS_HOST=redis.example.com
        API_KEY=abc123xyz
        # Add more environment variables...
```

**⚠️ Important Rules:**
- ✅ Use `|-` literal block scalar
- ✅ Each variable on its own line (actual newlines, not `\n`)
- ✅ 8-space indentation (2 levels: configs + dotenv)
- ❌ No quotes around the entire block
- ❌ No duplicate keys

---

### SNS Email Notifications (Optional)

To receive secrets updates via email:

1. When running the workflow, provide **Email Addresses:**
   ```
   dev1@example.com,dev2@example.com
   ```

2. **First time:** Check your email and confirm the SNS subscription

3. **Future runs:** Automatic email notifications

**SNS Topic:** `arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices`

---

## Monitoring Deployments

### Check Deployment Status

**Using kubectl:**
```bash
# Get pod status
kubectl get pods -n redtag -l app=your-service

# Watch deployment rollout
kubectl rollout status deployment/your-service -n redtag

# Check recent events
kubectl get events -n redtag --sort-by='.lastTimestamp' | grep your-service
```

**Using CircleCI:**
1. Go to CircleCI dashboard
2. Find your service project
3. Click on the build number
4. Review each step's output

### View Pod Logs

```bash
# Live tail logs
kubectl logs -f deployment/your-service -n redtag

# Last 100 lines
kubectl logs deployment/your-service -n redtag --tail=100

# Previous pod (if crashed)
kubectl logs deployment/your-service -n redtag --previous
```

### Check Application Health

**Health Endpoint:**
```bash
# Dev environment
curl https://your-service.dev.tbidevelopment.ca/health

# Expected response
{"status":"ok"}
```

---

## Troubleshooting

### Issue 1: Build Fails at Lint/Test Stage

**Symptoms:**
- CircleCI fails during `npm run lint` or `npm run build`
- Error messages about code issues

**Solution:**
1. Run tests locally:
   ```bash
   npm ci
   npm run lint
   npm run build
   npm run test
   ```
2. Fix any errors
3. Commit and push fixes

---

### Issue 2: Pods in CrashLoopBackOff

**Symptoms:**
```bash
NAME                        READY   STATUS             RESTARTS
your-service-xxx-yyy        0/1     CrashLoopBackOff   5
```

**Diagnosis:**
```bash
# Check pod logs
kubectl logs deployment/your-service -n redtag --previous

# Check events
kubectl describe pod your-service-xxx-yyy -n redtag
```

**Common Causes:**

**A. Missing Environment Variables**
```
Error: Missing required environment variables:
  - DATABASE_PASSWORD
  - API_KEY
```

**Solution:** Update secrets via GitHub Actions workflow

**B. Wrong Environment Variable Names**
- Check `.env.example` in your repo
- Ensure secrets file has matching variable names
- Use GitHub Actions to view current secrets

**C. Database Connection Failed**
```
Error: connect ECONNREFUSED 10.0.1.5:5432
```

**Solution:** Verify database host/credentials in secrets

---

### Issue 3: Secrets Not Loading

**Symptoms:**
- Pod logs show: `env variable undefined`
- Application can't connect to services

**Check Secret Exists:**
```bash
kubectl get secret your-service-secrets -n redtag

# View secret content (base64 encoded)
kubectl get secret your-service-secrets -n redtag -o yaml
```

**Verify Mount Path:**
```bash
# Check if .env file is mounted
kubectl exec -it deployment/your-service -n redtag -- ls -la /app/.env
kubectl exec -it deployment/your-service -n redtag -- cat /app/.env
```

**Solution:**
1. Ensure secrets file exists in `tbi-environments/dev/secrets/`
2. Re-run CircleCI build to redeploy
3. Check Helm chart `deployment.yaml` has correct volumeMounts

---

### Issue 4: Image Not Found

**Symptoms:**
```
Failed to pull image "594765360358.dkr.ecr...": rpc error
```

**Solution:**
1. Check ECR repository exists:
   ```bash
   aws ecr describe-repositories --repository-names your-service
   ```

2. Verify image tag in `tbi-environments/dev/values/your-service.yaml`:
   ```yaml
   image: 594765360358.dkr.ecr.ca-central-1.amazonaws.com/your-service:1.0.3-develop-123
   ```

3. Re-run CircleCI build to push image

---

### Issue 5: CircleCI Config Not Found

**Symptoms:**
```
CircleCI: Config file not found
```

**Solution:**

Check CircleCI project settings:
1. Go to CircleCI → Project Settings → Advanced
2. Verify **Config source:** `tbi-devops-environments`
3. Verify **Config filepath:** `dev/your-service/.circleci/config.yml`
4. Verify **Checkout source:** `your-service` (your app repo)

---

## FAQ

### Q: How do I add a new environment variable?

**A:** Use the GitHub Actions workflow:

1. Go to **Actions** → **Manage Secrets**
2. Select **update-and-restart**
3. Add new variable in JSON:
   ```json
   {
     "NEW_VARIABLE": "new-value"
   }
   ```
4. Run workflow

OR manually edit in `tbi-environments`:
```bash
cd tbi-environments/dev
sops secrets/secrets.your-service.yaml
# Add: NEW_VARIABLE=new-value
# Save (auto-encrypts)
git commit -m "Add NEW_VARIABLE"
git push
```

---

### Q: How do I roll back a deployment?

**A:** Use Helm rollback:

```bash
# List releases
helm list -n redtag

# Show release history
helm history your-service -n redtag

# Rollback to previous revision
helm rollback your-service -n redtag

# Rollback to specific revision
helm rollback your-service 5 -n redtag
```

---

### Q: Can I deploy without running tests?

**A:** Not recommended, but you can modify the CircleCI config:

1. Edit `tbi-devops-environments/dev/your-service/.circleci/config.yml`
2. Change test step to non-blocking:
   ```yaml
   - run:
       name: Run tests
       command: npm run test || echo "Tests failed but continuing"
   ```
3. Commit and push

---

### Q: How do I debug secrets encryption issues?

**A:** Check SOPS configuration:

```bash
cd tbi-environments/dev

# Test SOPS access
sops --version

# Check AWS credentials
aws sts get-caller-identity

# Try decrypting
sops -d secrets/secrets.your-service.yaml
```

**Common Issues:**
- AWS credentials not configured (use `aws configure --profile tbi-dev`)
- KMS key permissions missing
- Wrong directory (must be in `dev/`, `qa/`, or `uat/` for `.sops.yaml` to work)

---

### Q: Where are the Helm charts?

**A:** Helm charts are stored in two places:

**Chart Templates:**
```
tbi-devops-environments/dev/your-service/charts/your-service/
├── Chart.yaml
├── values.yaml
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── ingress.yaml
    └── secrets.yaml
```

**Packaged Charts (S3 Helm Repository):**
```
s3://tbi-charts/your-service-{version}.tgz
```

**Values Files:**
```
tbi-environments/dev/values/your-service.yaml
```

---

### Q: How do I access production secrets?

**A:** Production secrets require special approval:

1. Contact DevOps team
2. Production deployments use `master` branch
3. Secrets workflow supports `uat` environment (pre-prod)
4. Production secrets managed separately for security

---

### Q: What if GitHub Actions workflow is missing?

**A:** Copy from another service:

```bash
cd your-service-repo
mkdir -p .github/workflows
cp ../other-service/.github/workflows/manage-secrets.yml .github/workflows/

# Update service name in the file
# Commit and push
git add .github/
git commit -m "Add secrets management workflow"
git push
```

---

## Getting Help

### Documentation

- **Detailed Setup:** `tbi-devops-environments/docs/SECRETS-PIPELINE-SETUP.md`
- **Migration Guide:** `tbi-devops-environments/docs/MIGRATION-CHECKLIST.md`
- **Shared Image Migration:** `tbi-devops-environments/docs/SHARED-IMAGE-MIGRATION.md`
- **Quick Reference:** `tbi-devops-environments/docs/QUICK-REFERENCE.md`
- **Context Doc:** `tbi-devops-environments/CLAUDE-CONTEXT.md`

### Support Channels

- **DevOps Team:** devops@travelbrands.com
- **GitHub Issues:** Report bugs in `tbi-devops-environments` repo
- **Slack:** #devops-support

### Reference Service

**Use `tbi-redtag-mobile-hotel-service` as reference:**
- ✅ Complete CircleCI config
- ✅ Proper secrets format
- ✅ Working deployment scripts
- ✅ GitHub Actions workflow

---

## Best Practices

### ✅ DO

- ✅ Use GitHub Actions workflow for secrets management
- ✅ Test locally before pushing (`npm run build`, `npm run test`)
- ✅ Use meaningful commit messages
- ✅ Monitor pod logs after deployment
- ✅ Keep secrets encrypted (never commit plaintext)
- ✅ Use proper `.env` file format (multiline with `|-`)

### ❌ DON'T

- ❌ Don't commit secrets to Git (use SOPS encryption)
- ❌ Don't edit secrets directly in Kubernetes
- ❌ Don't skip tests without good reason
- ❌ Don't force push to protected branches
- ❌ Don't use `latest` image tags (use version tags)
- ❌ Don't bypass CircleCI (manual kubectl deployments)

---

**Questions or Issues?** Contact the DevOps team or check the documentation in `tbi-devops-environments/docs/`