# Secrets Pipeline Setup Guide

This guide documents the setup of GitHub Actions secrets management pipeline for TBI RedTag microservices.

## Table of Contents
1. [Prerequisites](#prerequisites)
2. [Secrets File Setup](#secrets-file-setup)
3. [GitHub Actions Workflow Setup](#github-actions-workflow-setup)
4. [Testing the Workflow](#testing-the-workflow)
5. [Troubleshooting](#troubleshooting)

---

## Prerequisites

### 1. Self-Hosted GitHub Actions Runner

A self-hosted runner is required in the management cluster because SOPS decryption needs access to AWS KMS via the EKS node instance role (not IRSA).

**Runner is already deployed at**: `tbi-devops-common/app/mgt/apps/gha-runner`

**Key configuration**:
- Runs in management cluster with node instance role access
- Has tools: kubectl, helm, sops, aws-cli, git, jq
- Labels: `self-hosted,linux,x64,tbi-github-runner`
- Uses organization-level GitHub PAT secret: `TBI_GITHUB_TOKEN`

### 2. GitHub Secrets Required

Each repository needs:
- `TBI_GITHUB_TOKEN`: Organization-level PAT with access to tbi-environments repo
- `AWS_REGION`: `ca-central-1` (if not already set)
- `SNS_TOPIC_ARN`: `arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices`

### 3. AWS Permissions

The management cluster's node instance role needs:
- KMS decrypt permissions for dev/qa/uat KMS keys
- Access to target EKS clusters (configured in aws-auth ConfigMap)
- SNS publish permissions

---

## Secrets File Setup

### Step 1: Create Secrets File in tbi-environments

Navigate to the tbi-environments repo and create secrets for each environment.

**Directory structure**:
```
tbi-environments/
├── dev/
│   ├── .sops.yaml
│   └── secrets/
│       └── secrets.tbi-<service-name>.yaml
├── qa/
│   ├── .sops.yaml
│   └── secrets/
│       └── secrets.tbi-<service-name>.yaml
└── uat/
    ├── .sops.yaml
    └── secrets/
        └── secrets.tbi-<service-name>.yaml
```

**Service name mapping**:
- `tbi-redtag-mobile-hotel-service` → `tbi-redtag-mobile-hotel-service`
- `tbi-redtag-mobile-ms-flights` → `tbi-redtag-mobile-ms-flights`
- `tbi-redtag-mobile-ms-cars` → `tbi-redtag-mobile-ms-cars`
- `tbi-redtag-mobile-ms-activities` → `tbi-redtag-mobile-ms-activities`
- `tbi-redtag-mobile-ms-bookings` → `tbi-redtag-mobile-ms-bookings`

### Step 2: Create .sops.yaml (if not exists)

Each environment directory needs a `.sops.yaml` file:

```yaml
creation_rules:
  - kms: 'arn:aws:kms:ca-central-1:594765360358:key/e558a999-9f99-432e-9d6b-73f06de7d1a7'
```

### Step 3: Create Secrets File

**CRITICAL**: The secrets file MUST use YAML literal block scalar (`|-`) format with actual newlines:

```yaml
configs:
    dotenv: |-
        DB_HOST=example.dev.tbidevelopment.ca
        DB_USERNAME=admin
        DB_PASSWORD=secret123
        DB_PORT=5432
        REDIS_HOST=redis.dev.tbidevelopment.ca
        REDIS_PORT=6379
        AWS_REGION=ca-central-1
        # ... add all environment variables
```

**DO NOT use**:
- Single-line format: `dotenv: "DB_HOST=... DB_USERNAME=..."`
- Literal `\n` characters: `dotenv: "DB_HOST=...\nDB_USERNAME=..."`
- Quoted multi-line strings

**Why this format?**
- The Node.js apps parse dotenv files line-by-line
- Incorrect format causes all environment variables to be `undefined`
- See troubleshooting section for symptoms

### Step 4: Encrypt the Secrets File

```bash
cd tbi-environments/<environment>
sops -e -i secrets/secrets.tbi-<service-name>.yaml
```

### Step 5: Commit to tbi-environments

```bash
git add .
git commit -m "Add secrets for tbi-<service-name> in <environment>"
git push origin master
```

---

## GitHub Actions Workflow Setup

### Step 1: Copy Workflow File

Copy `.github/workflows/manage-secrets.yml` from `tbi-redtag-mobile-hotel-service` to the new repo.

**Key sections to update**:

1. **Service input options** (line ~20):
```yaml
service:
  description: 'Service to update secrets for'
  type: choice
  options:
    - tbi-redtag-mobile-ms-flights      # Update this
    - tbi-redtag-mobile-ms-cars         # Update this
    - tbi-redtag-mobile-ms-activities   # Update this
    - tbi-redtag-mobile-ms-bookings     # Update this
```

2. **Namespace mapping** (if different from `redtag`):
   - Most services use namespace: `redtag`
   - If different, update the kubectl commands

3. **SNS topic** (already configured):
   - `arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices`

### Step 2: Workflow Features

The workflow provides three actions:

#### 1. view-secrets
- Decrypts and displays current secrets
- Output options:
  - `github-summary`: Show in GitHub Actions summary
  - `download-artifact`: Download as artifact
  - `email-sns`: Send via SNS email
  - `all`: All of the above

#### 2. update-secrets
- Update specific secrets via JSON input
- Maintains YAML `|-` literal block format
- Commits changes to tbi-environments
- Applies to Kubernetes cluster

**JSON format**:
```json
{
  "DB_PASSWORD": "new_password_123",
  "REDIS_HOST": "new-redis-host.example.com"
}
```

#### 3. update-and-restart
- Same as update-secrets
- Additionally restarts pods with `kubectl rollout restart`

### Step 3: Commit Workflow

```bash
git add .github/workflows/manage-secrets.yml
git commit -m "Add GitHub Actions secrets management workflow"
git push origin develop
```

---

## CI/CD Pipeline Migration to tbi-devops-environments

### Directory Structure

Move all CI/CD configs to `tbi-devops-environments`:

```
tbi-devops-environments/
├── dev/
│   ├── tbi-redtag-mobile-ms-flights/
│   │   ├── .circleci/
│   │   │   └── config.yml
│   │   ├── .github/
│   │   │   └── workflows/
│   │   │       └── manage-secrets.yml
│   │   └── docker/
│   │       └── Dockerfile
│   ├── tbi-redtag-mobile-ms-cars/
│   │   ├── .circleci/
│   │   ├── .github/
│   │   └── docker/
│   └── ...
├── qa/
│   └── (same structure)
├── uat/
│   └── (same structure)
└── redtag-prod/
    └── (same structure)
```

### Migration Steps

1. **Create directory structure**:
```bash
cd tbi-devops-environments
mkdir -p dev/tbi-redtag-mobile-ms-flights/{.circleci,.github/workflows,docker}
mkdir -p qa/tbi-redtag-mobile-ms-flights/{.circleci,.github/workflows,docker}
mkdir -p uat/tbi-redtag-mobile-ms-flights/{.circleci,.github/workflows,docker}
```

2. **Copy configs from repo**:
```bash
# From the service repo
cp .circleci/config.yml ../tbi-devops-environments/dev/tbi-redtag-mobile-ms-flights/.circleci/
cp .github/workflows/manage-secrets.yml ../tbi-devops-environments/dev/tbi-redtag-mobile-ms-flights/.github/workflows/
cp Dockerfile ../tbi-devops-environments/dev/tbi-redtag-mobile-ms-flights/docker/
```

3. **Update service repo to reference external configs**:
   - Add symlinks or documentation
   - Update README with new config location

4. **Commit to tbi-devops-environments**:
```bash
git add .
git commit -m "Add CI/CD configs for tbi-redtag-mobile-ms-flights"
git push origin master
```

---

## Testing the Workflow

### Test 1: View Secrets

1. Go to GitHub Actions: `https://github.com/redlabelvacations/<repo>/actions/workflows/manage-secrets.yml`
2. Click "Run workflow"
3. Select:
   - Environment: `dev`
   - Service: `tbi-redtag-mobile-ms-<service>`
   - Action: `view-secrets`
   - View output: `github-summary`
4. Verify secrets are decrypted and displayed

### Test 2: Update Single Secret

1. Run workflow with:
   - Action: `update-secrets`
   - Secret updates JSON: `{"TEST_VAR":"test_value"}`
2. Check tbi-environments repo for new commit
3. Verify secret applied to cluster:
```bash
kubectl get secret -n redtag tbi-redtag-mobile-ms-<service>-secrets -o jsonpath='{.data.dotenv}' | base64 -d | grep TEST_VAR
```

### Test 3: Update and Restart Pods

1. Run workflow with:
   - Action: `update-and-restart`
   - Secret updates JSON: `{"DB_PASSWORD":"new_test_password"}`
2. Verify pod restart:
```bash
kubectl get pods -n redtag -l app=tbi-redtag-mobile-ms-<service>
```
3. Check pod logs for config loading:
```bash
kubectl logs -n redtag deployment/tbi-redtag-mobile-ms-<service> --tail=50 | grep -i "wait for config"
```

### Test 4: SNS Notification

1. Run any action with email notification
2. Check SNS topic for message delivery
3. Verify email received at configured addresses

---

## Troubleshooting

### Issue 1: Config Not Loading in Pods

**Symptoms**:
```
wait for config loaded undefined
env stage undefined
redis host undefined
```

**Cause**: Incorrect YAML format in secrets file

**Fix**:
1. Decrypt the secrets file:
```bash
cd tbi-environments/dev
sops -d secrets/secrets.tbi-<service>.yaml > /tmp/decrypted.yaml
```

2. Check format - should see:
```yaml
configs:
    dotenv: |-
        DB_HOST=...
        DB_USERNAME=...
```

3. If single-line or has `\n` literals, reformat using Python:
```python
import re

with open('/tmp/decrypted.yaml', 'r') as f:
    content = f.read()

# Extract single-line dotenv
match = re.search(r'dotenv:\s*(.+)', content)
if match:
    dotenv_line = match.group(1).strip()
    parts = re.split(r'\s+(?=[A-Z_]+=)', dotenv_line)

    formatted = "configs:\n    dotenv: |-\n"
    for part in parts:
        formatted += f"        {part}\n"

    with open('/tmp/fixed.yaml', 'w') as f:
        f.write(formatted)
```

4. Encrypt and commit:
```bash
cp /tmp/fixed.yaml secrets/secrets.tbi-<service>.yaml
sops -e -i secrets/secrets.tbi-<service>.yaml
git add . && git commit -m "Fix dotenv format" && git push
```

5. Re-apply to cluster:
```bash
kubectl create secret generic tbi-redtag-mobile-ms-<service>-secrets \
  --from-file=dotenv=/tmp/fixed-dotenv.txt \
  --dry-run=client -o yaml | kubectl apply -f -
kubectl rollout restart deployment/tbi-redtag-mobile-ms-<service> -n redtag
```

### Issue 2: SOPS "config file not found"

**Symptoms**:
```
Error: config file not found and no keys provided
```

**Cause**: Running `sops -e -i` from repo root instead of environment directory

**Fix**:
```bash
# Wrong
cd tbi-environments
sops -e -i dev/secrets/secrets.tbi-<service>.yaml  # ❌

# Correct
cd tbi-environments/dev
sops -e -i secrets/secrets.tbi-<service>.yaml  # ✅
```

### Issue 3: kubectl Authentication Failed

**Symptoms**:
```
error: You must be logged in to the server (Unauthorized)
```

**Cause**: Management cluster node role not in target cluster's aws-auth

**Fix**:
1. Get management cluster node role ARN:
```bash
kubectl get nodes -o jsonpath='{.items[0].spec.providerID}' | cut -d'/' -f5
aws ec2 describe-instances --instance-ids <instance-id> --query 'Reservations[0].Instances[0].IamInstanceProfile.Arn'
```

2. Add to target cluster's aws-auth:
```bash
kubectl edit configmap aws-auth -n kube-system

# Add under mapRoles:
- rolearn: arn:aws:iam::594765360358:role/<node-role-name>
  username: system:node:{{EC2PrivateDNSName}}
  groups:
    - system:bootstrappers
    - system:nodes
```

### Issue 4: Duplicate Environment Variables

**Symptoms**:
```
AWS_AUTH_COGNITO_CLIENT_ID=value1
AWS_AUTH_COGNITO_CLIENT_ID=value2
```

**Cause**: Multiple updates without de-duplication logic

**Fix**:
The workflow now handles duplicates automatically. To manually clean:

```bash
# Decrypt
sops -d secrets/secrets.tbi-<service>.yaml > /tmp/decrypted.yaml

# Remove duplicates with Python
python3 << 'EOF'
with open('/tmp/decrypted.yaml', 'r') as f:
    lines = f.readlines()

seen = {}
unique_lines = []
in_dotenv = False

for line in lines:
    if 'dotenv: |-' in line:
        in_dotenv = True
        unique_lines.append(line)
    elif in_dotenv and '=' in line:
        key = line.split('=')[0].strip()
        if key not in seen:
            seen[key] = True
            unique_lines.append(line)
    else:
        unique_lines.append(line)

with open('/tmp/cleaned.yaml', 'w') as f:
    f.writelines(unique_lines)
EOF

# Re-encrypt
cp /tmp/cleaned.yaml secrets/secrets.tbi-<service>.yaml
sops -e -i secrets/secrets.tbi-<service>.yaml
```

### Issue 5: 401 Unauthorized After Secret Update

**Symptoms**:
API returns 401 even with valid JWT token

**Possible causes**:
1. `AWS_AUTH_COGNITO_CLIENT_ID` mismatch with JWT `aud` claim
2. `AWS_AUTH_COGNITO_USER_POOL_ID` mismatch with JWT `iss` claim
3. Pod not restarted after secret update

**Fix**:
1. Verify JWT claims match config:
```bash
# Decode JWT (use jwt.io or jwt-cli)
jwt decode <your-token>

# Check aud claim matches:
kubectl get secret -n redtag tbi-redtag-mobile-ms-<service>-secrets \
  -o jsonpath='{.data.dotenv}' | base64 -d | grep AWS_AUTH_COGNITO_CLIENT_ID
```

2. Restart pods:
```bash
kubectl rollout restart deployment/tbi-redtag-mobile-ms-<service> -n redtag
```

---

## Service-Specific Configurations

### tbi-redtag-mobile-ms-flights

**Namespace**: `redtag`

**Key environment variables**:
- Flight API credentials
- Sabre/Amadeus configuration
- Database connection (shared or dedicated)

**Copy from**: `tbi-copolo-mobile-ms-flights`

### tbi-redtag-mobile-ms-cars

**Namespace**: `redtag`

**Key environment variables**:
- Car rental API credentials
- Database connection
- Mapping service configuration

**Copy from**: `tbi-copolo-mobile-ms-cars`

### tbi-redtag-mobile-ms-activities

**Namespace**: `redtag`

**Key environment variables**:
- Activities supplier credentials
- Database connection
- Redis configuration

**Copy from**: `tbi-copolo-mobile-ms-activities`

### tbi-redtag-mobile-ms-bookings

**Namespace**: `redtag`

**Key environment variables**:
- Payment gateway credentials
- Database connection
- SNS/SQS configuration
- All other service endpoints

**Copy from**: `tbi-copolo-mobile-ms-bookings`

---

## Checklist for Each Service

- [ ] Create secrets files in tbi-environments (dev/qa/uat)
- [ ] Encrypt secrets with SOPS
- [ ] Verify YAML `|-` format is correct
- [ ] Create GitHub Actions workflow
- [ ] Add `TBI_GITHUB_TOKEN` secret to repo
- [ ] Test view-secrets action
- [ ] Test update-secrets action
- [ ] Test update-and-restart action
- [ ] Verify pod logs show config loaded correctly
- [ ] Test API endpoints with valid credentials
- [ ] Move CI/CD configs to tbi-devops-environments
- [ ] Update CircleCI pipeline
- [ ] Add service to Jenkins blacklist
- [ ] Remove Copolo references from frontend/docs
- [ ] Update README with new workflow instructions
- [ ] Send SNS notification test

---

## Additional Resources

- **Self-hosted runner**: `tbi-devops-common/app/mgt/apps/gha-runner/README.md`
- **SOPS documentation**: https://github.com/mozilla/sops
- **GitHub Actions**: https://docs.github.com/en/actions
- **EKS aws-auth**: https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html

---

## Support

For issues or questions:
1. Check troubleshooting section above
2. Review GitHub Actions workflow run logs
3. Check pod logs: `kubectl logs -n redtag deployment/tbi-redtag-mobile-ms-<service>`
4. Verify secrets format in tbi-environments repo
5. Contact DevOps team

---

**Last updated**: 2025-11-19
**Version**: 1.0
**Author**: DevOps Team