# Quick Reference Card

Essential commands and patterns for secrets management workflow.

## Common Operations

### View Current Secrets

**Via GitHub Actions UI**:
1. Go to: `https://github.com/redlabelvacations/<repo>/actions/workflows/manage-secrets.yml`
2. Run workflow → Action: `view-secrets`, View output: `github-summary`

**Via Command Line**:
```bash
cd tbi-environments/dev
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml
```

### Update a Secret

**Via GitHub Actions UI**:
1. Run workflow → Action: `update-secrets`
2. Secret updates JSON:
```json
{"DB_PASSWORD":"new_password","REDIS_HOST":"new-redis.example.com"}
```

**Via Command Line**:
```bash
cd tbi-environments/dev

# Decrypt
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml > /tmp/decrypted.yaml

# Edit the file
nano /tmp/decrypted.yaml

# Encrypt back
sops -e /tmp/decrypted.yaml > secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml

# Commit
git add . && git commit -m "Update secrets" && git push

# Apply to cluster
kubectl create secret generic tbi-redtag-mobile-ms-<service>-secrets \
  --from-file=dotenv=<(sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml | yq eval '.configs.dotenv' -r) \
  --dry-run=client -o yaml | kubectl apply -f - -n redtag

# Restart pods
kubectl rollout restart deployment/tbi-redtag-mobile-ms-<service> -n redtag
```

### Update Secret and Restart Pods

**Via GitHub Actions UI**:
1. Run workflow → Action: `update-and-restart`
2. Secret updates JSON: `{"KEY":"value"}`

### Check Pod Configuration

```bash
# Check if config is loading
kubectl logs -n redtag deployment/tbi-redtag-mobile-ms-<service> --tail=50 | grep "wait for config"

# Should see:
# wait for config loaded <db-host>
# env stage dev
# redis host <redis-host>
# NOT: undefined

# Check secret content
kubectl get secret -n redtag tbi-redtag-mobile-ms-<service>-secrets \
  -o jsonpath='{.data.dotenv}' | base64 -d | head -20
```

### Verify YAML Format

```bash
# Decrypt and check format
cd tbi-environments/dev
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml | head -10

# Should see:
# configs:
#     dotenv: |-
#         KEY1=value1
#         KEY2=value2

# NOT:
# configs:
#     dotenv: "KEY1=value1\nKEY2=value2"
# configs:
#     dotenv: KEY1=value1 KEY2=value2
```

## Critical YAML Format

**CORRECT**:
```yaml
configs:
    dotenv: |-
        DB_HOST=example.com
        DB_USERNAME=admin
        DB_PASSWORD=secret
```

**WRONG**:
```yaml
# Single line with spaces
configs:
    dotenv: DB_HOST=example.com DB_USERNAME=admin DB_PASSWORD=secret

# Literal \n characters
configs:
    dotenv: "DB_HOST=example.com\nDB_USERNAME=admin\nDB_PASSWORD=secret"

# Quoted multi-line
configs:
    dotenv: "DB_HOST=example.com
        DB_USERNAME=admin
        DB_PASSWORD=secret"
```

## Fix Broken Format

```bash
# Step 1: Decrypt
cd tbi-environments/dev
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml > /tmp/broken.yaml

# Step 2: Fix with Python
python3 << 'EOF'
import re

with open('/tmp/broken.yaml', 'r') as f:
    content = f.read()

# Extract single-line dotenv
match = re.search(r'dotenv:\s*(.+)', content)
if match:
    dotenv_line = match.group(1).strip()
    # Remove quotes if present
    dotenv_line = dotenv_line.strip('"').strip("'")
    # Split on space followed by KEY= pattern
    parts = re.split(r'\s+(?=[A-Z_]+=)', dotenv_line)

    # Create proper format
    formatted = "configs:\n    dotenv: |-\n"
    for part in parts:
        # Replace literal \n with actual newlines
        part = part.replace('\\n', '\n')
        if '=' in part:
            formatted += f"        {part}\n"

    with open('/tmp/fixed.yaml', 'w') as f:
        f.write(formatted)

    print(f"Fixed {len(parts)} variables")
else:
    print("ERROR: Could not find dotenv line")
EOF

# Step 3: Encrypt
sops -e /tmp/fixed.yaml > secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml

# Step 4: Commit and apply
git add . && git commit -m "Fix dotenv format" && git push

# Step 5: Apply to cluster
kubectl create secret generic tbi-redtag-mobile-ms-<service>-secrets \
  --from-file=dotenv=<(cat /tmp/fixed.yaml | yq eval '.configs.dotenv' -r) \
  --dry-run=client -o yaml | kubectl apply -f - -n redtag

kubectl rollout restart deployment/tbi-redtag-mobile-ms-<service> -n redtag
```

## Kubernetes Commands

```bash
# Get pod status
kubectl get pods -n redtag -l app=tbi-redtag-mobile-ms-<service>

# View pod logs
kubectl logs -n redtag deployment/tbi-redtag-mobile-ms-<service> --tail=100

# Follow logs
kubectl logs -n redtag deployment/tbi-redtag-mobile-ms-<service> -f

# Restart deployment
kubectl rollout restart deployment/tbi-redtag-mobile-ms-<service> -n redtag

# Check rollout status
kubectl rollout status deployment/tbi-redtag-mobile-ms-<service> -n redtag

# View secret (base64 decoded)
kubectl get secret -n redtag tbi-redtag-mobile-ms-<service>-secrets \
  -o jsonpath='{.data.dotenv}' | base64 -d

# Delete and recreate secret
kubectl delete secret tbi-redtag-mobile-ms-<service>-secrets -n redtag
kubectl create secret generic tbi-redtag-mobile-ms-<service>-secrets \
  --from-file=dotenv=/tmp/dotenv.txt -n redtag

# Describe deployment
kubectl describe deployment tbi-redtag-mobile-ms-<service> -n redtag

# Get deployment YAML
kubectl get deployment tbi-redtag-mobile-ms-<service> -n redtag -o yaml

# Check events
kubectl get events -n redtag --sort-by='.lastTimestamp' | grep <service>
```

## SOPS Commands

```bash
# Encrypt a file in-place
cd tbi-environments/dev
sops -e -i secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml

# Decrypt a file
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml

# Decrypt to file
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml > /tmp/decrypted.yaml

# Encrypt from stdin
cat /tmp/decrypted.yaml | sops -e /dev/stdin > secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml

# Edit encrypted file directly (opens in editor, auto-encrypts on save)
sops secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml

# Extract specific value
sops -d secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml | yq eval '.configs.dotenv' -r | grep DB_PASSWORD
```

## Git Commands

```bash
# tbi-environments workflow
cd tbi-environments
git pull origin master
# ... make changes ...
git add dev/secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml
git commit -m "Update <service> secrets in dev"
git push origin master

# Service repo workflow
cd tbi-redtag-mobile-ms-<service>
git checkout develop
git pull origin develop
# ... make changes ...
git add .github/workflows/manage-secrets.yml
git commit -m "Add secrets management workflow"
git push origin develop

# tbi-devops-environments workflow
cd tbi-devops-environments
git pull origin master
# ... make changes ...
git add dev/tbi-redtag-mobile-ms-<service>/
git commit -m "Add CI/CD configs for <service>"
git push origin master
```

## Troubleshooting Quick Fixes

### Pod shows "undefined" env vars
```bash
# Fix YAML format (see "Fix Broken Format" section above)
```

### 401 Unauthorized errors
```bash
# Check Cognito client ID matches JWT
kubectl get secret -n redtag tbi-redtag-mobile-ms-<service>-secrets \
  -o jsonpath='{.data.dotenv}' | base64 -d | grep AWS_AUTH_COGNITO_CLIENT_ID

# Decode JWT token at https://jwt.io
# Verify aud claim matches AWS_AUTH_COGNITO_CLIENT_ID
```

### SOPS "config file not found"
```bash
# Must run from environment directory, not repo root
cd tbi-environments/dev  # ✅
sops -e -i secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml

# NOT from:
cd tbi-environments  # ❌
sops -e -i dev/secrets/secrets.tbi-redtag-mobile-ms-<service>.yaml
```

### Duplicate environment variables
```bash
# Check for duplicates
kubectl get secret -n redtag tbi-redtag-mobile-ms-<service>-secrets \
  -o jsonpath='{.data.dotenv}' | base64 -d | sort | uniq -d

# Remove duplicates (see "Fix Broken Format" section)
```

### GitHub Actions workflow fails
```bash
# Check runner logs
kubectl logs -n mgt -l app=tbi-github-runner

# Check runner status
# Go to: GitHub Org Settings → Actions → Runners

# Verify secrets exist
# Go to: Repo Settings → Secrets and variables → Actions
# Should have: TBI_GITHUB_TOKEN, AWS_REGION, SNS_TOPIC_ARN
```

## Shared Image Deployments

### Check Image Tag Sync

```bash
# Verify both services use the same image tag
cd tbi-environments

# Primary service
cat dev/values/tbi-package-travelbrands.yaml | grep image:

# Secondary service (should match)
cat dev/values/tbi-pl-dynamic.yaml | grep image:
```

### Verify Image in ECR

```bash
# Check if image exists
aws ecr describe-images \
  --repository-name tbi-package-travelbrands \
  --region ca-central-1 \
  --query 'imageDetails[0:5].[imageTags,imagePushedAt]' \
  --output table

# Check specific tag
aws ecr describe-images \
  --repository-name tbi-package-travelbrands \
  --image-ids imageTag=1.0.0-develop-123 \
  --region ca-central-1
```

### Check requirements.yaml Updates

```bash
cd tbi-environments

# View requirements file
cat dev/requirements.yaml

# Check specific service version
cat dev/requirements.yaml | grep -A2 "name: tbi-package-travelbrands"
cat dev/requirements.yaml | grep -A2 "name: tbi-pl-dynamic"

# Check update history
git log --oneline -n 10 dev/requirements.yaml
```

### Trigger Secondary Service Manually

```bash
cd /path/to/tbi-pl-dynamic

# Create empty commit to trigger deployment
git commit --allow-empty -m "Trigger deployment with existing image"
git push origin develop
```

### Check Both Services Status

```bash
# Primary service (access namespace)
kubectl get pods -n access -l app=tbi-package-travelbrands
kubectl logs -n access deployment/tbi-package-travelbrands --tail=50

# Secondary service (privatelabel namespace)
kubectl get pods -n privatelabel -l app=tbi-pl-dynamic
kubectl logs -n privatelabel deployment/tbi-pl-dynamic --tail=50
```

### Rollback Both Services

```bash
# Rollback primary
helm rollback tbi-package-travelbrands -n access

# Rollback secondary
helm rollback tbi-pl-dynamic -n privatelabel

# Verify rollback
kubectl get pods -n access
kubectl get pods -n privatelabel
```

### Fix requirements.yaml Not Updating

```bash
# Edit deployment script
cd tbi-devops-environments/dev/tbi-package-travelbrands/scripts

# Ensure these lines exist in deploy-services.sh:
# echo "=== Step 6: Update requirements.yaml ==="
# update_requirements
#
# In push_values_update():
# git add ${ENVIRONMENT}/values/${SERVICE_NAME}.yaml
# git add ${ENVIRONMENT}/requirements.yaml  # ← Must be here
```

### Verify DEPLOY_ENV Context

```bash
# Check CircleCI build logs for:
# "DEPLOY_ENV is set to: uat"

# If missing, add to CircleCI context:
# Organization Settings → Contexts → devops-mgt-uat
# Add variable: DEPLOY_ENV=uat
```

## Service URLs

- **Dev**: `https://tbi-redtag-<service>-service.dev.tbidevelopment.ca`
- **QA**: `https://tbi-redtag-<service>-service.qa.tbidevelopment.ca`
- **UAT**: `https://tbi-redtag-<service>-service.uat.tbidevelopment.ca`

### Package Travelbrands Services

- **Dev Access**: `https://tbi-package-travelbrands.dev.tbidevelopment.ca`
- **Dev Private Label**: `https://tbi-pl-dynamic.dev.tbidevelopment.ca`
- **UAT Access**: `https://tbi-package-travelbrands.uat.tbidevelopment.ca`
- **UAT Private Label**: `https://tbi-pl-dynamic.uat.tbidevelopment.ca`

## Important ARNs

- **SNS Topic**: `arn:aws:sns:ca-central-1:594765360358:tbi-devops-secrets-pipeline-notices`
- **KMS Key**: `arn:aws:kms:ca-central-1:594765360358:key/e558a999-9f99-432e-9d6b-73f06de7d1a7`

## Repository URLs

- **tbi-environments**: `https://github.com/redlabelvacations/tbi-environments`
- **tbi-devops-environments**: `https://github.com/redlabelvacations/tbi-devops-environments`
- **tbi-devops-common**: `https://github.com/redlabelvacations/tbi-devops-common`

## Contacts

- **DevOps Team**: [Contact info]
- **Documentation**: See SECRETS-PIPELINE-SETUP.md
- **Issues**: Report at GitHub Issues

## Troubleshooting

See [PIPELINE-TROUBLESHOOTING.md](PIPELINE-TROUBLESHOOTING.md) for common pipeline issues and fixes.

---

**Print this page for quick reference during migrations and troubleshooting**