# Production Approval Workflow Implementation - COMPLETE

**Date**: 2026-03-11  
**Status**: ✅ COMPLETE  
**Services Updated**: 20/20 ACC services

---

## Summary

Successfully updated all 20 ACC production CircleCI configs with 2-stage approval workflow and SNS notifications.

---

## What Was Updated

### All 20 Services Now Have:

1. **Two-Stage Approval Workflow**
   - Step 1: Run tests
   - Step 2: Notify BUILD approval needed (SNS)
   - Step 3: Manual approval gate (BUILD)
   - Step 4: Build and push Docker image
   - Step 5: Notify DEPLOY approval needed (SNS)
   - Step 6: Manual approval gate (DEPLOY)
   - Step 7: Deploy to production Kubernetes

2. **SNS Notifications** (4 total per deployment)
   - BUILD approval needed notification
   - DEPLOY approval needed notification
   - Deployment success notification
   - Deployment failure notification (on_fail)

3. **Production Configuration**
   - Context: `devops-mgt-prod`
   - Runner: `tbi-local-runner/prod-win`
   - Branch filter: `master` only
   - Clean version tags (no branch/build suffix)
   - SNS Topic: `arn:aws:sns:ca-central-1:489176460030:DevOps-alert`

4. **Infrastructure-as-Code**
   - Dockerfile staging from tbi-devops-environments
   - Deployment scripts staging from tbi-devops-environments
   - DEPLOY_ENV validation (must be "prod")

---

## Services Updated

### Standard Configs (9 services)
These had the standard Node.js/PHP structure and were updated to add approval workflow:

1. ✅ tbi-acc-booking-service
2. ✅ tbi-acc-ceetiz-connector
3. ✅ tbi-acc-exotiktours-connector
4. ✅ tbi-acc-hotel-mapping-service
5. ✅ tbi-acc-mybooking-service
6. ✅ tbi-acc-notification-service
7. ✅ tbi-acc-profile-service
8. ✅ tbi-acc-profile-sync-service
9. ✅ tbi-acc-quotes-service

### Parameter-Based Configs (11 services)
These had parameter-based multi-environment configs and were converted to prod-only with approval workflow:

10. ✅ tbi-acc-cron-service
11. ✅ tbi-acc-gimmonix-automation
12. ✅ tbi-acc-hotel-service
13. ✅ tbi-acc-hotel-service-connector
14. ✅ tbi-acc-mybff
15. ✅ tbi-acc-mybooking-frontend
16. ✅ tbi-acc-package-cron
17. ✅ tbi-acc-package-service
18. ✅ tbi-acc-sabreapi
19. ✅ tbi-acc-travelbrands
20. ✅ tbi-acc-travelbrandsagent

---

## Implementation Details

### Approach Used

**Standard Configs (9 services)**:
- Updated existing jobs with prod context and branch filters
- Added two new jobs: `notify-await-build-approval` and `notify-await-deploy-approval`
- Updated `build-and-push-images` job with clean version tagging logic
- Updated `deploy-to-kubernetes` job with SNS success/failure notifications
- Replaced workflow with 7-step approval process
- Changed resource_class to `tbi-local-runner/prod-win`

**Parameter-Based Configs (11 services)**:
- Completely replaced with prod-only configs
- Removed pipeline parameters (not needed for prod-only)
- Removed conditional workflows (not needed for prod-only)
- Implemented same 7-step approval workflow as standard configs
- Maintained language-specific lint/test steps (PHP vs Node.js)

### Automation Scripts Created

1. **scripts/update_prod_configs.py**
   - Updated 6 standard configs automatically
   - Applied consistent changes across all services

2. **scripts/convert_parameter_to_prod.py**
   - Converted 11 parameter-based configs to prod-only
   - Detected language (PHP/Node.js) and applied appropriate steps
   - Generated complete configs from template

3. **scripts/update-prod-configs.sh**
   - Helper script for verification

---

## Verification Results

All 20 services verified with:
- ✅ `notify-await-build-approval` job present
- ✅ `notify-await-deploy-approval` job present
- ✅ Context: `devops-mgt-prod`
- ✅ Resource class: `tbi-local-runner/prod-win`
- ✅ Branch filter: `master` only
- ✅ SNS notifications (4 per deployment)

---

## Key Features

### 1. Two-Stage Approval Gates

**BUILD Approval**:
- Triggered after tests pass
- SNS notification sent to DevOps team
- Manual approval required in CircleCI UI
- Prevents unauthorized image builds

**DEPLOY Approval**:
- Triggered after image is built and pushed
- SNS notification sent to DevOps team
- Manual approval required in CircleCI UI
- Prevents unauthorized production deployments

### 2. SNS Notifications

**BUILD Approval Needed**:
```
⏳ PRODUCTION Pipeline - Awaiting BUILD Approval
Service: [service-name]
Environment: PRODUCTION
Build: #[build-number]
✅ Tests have passed successfully
⏳ MANUAL APPROVAL REQUIRED to proceed with Docker image build
```

**DEPLOY Approval Needed**:
```
⏳ PRODUCTION Pipeline - Awaiting DEPLOY Approval
Service: [service-name]
Environment: PRODUCTION
✅ Tests passed
✅ Docker image built and pushed to ECR
⏳ MANUAL APPROVAL REQUIRED to deploy to PRODUCTION cluster
⚠️  WARNING: This will deploy to PRODUCTION Kubernetes cluster!
```

**Deployment Success**:
```
✅ Deployment Successful - [service-name]
Environment: prod
Namespace: access
Image Tag: [version]
Build: #[build-number]
```

**Deployment Failure**:
```
❌ Deployment Failed - [service-name]
Environment: prod
Namespace: access
Image Tag: [version]
Build: #[build-number]
Please check the build logs for error details.
```

### 3. Clean Version Tags

Production uses clean semantic version tags without branch/build suffixes:
- Dev/UAT: `1.2.3-develop-456`
- Production: `1.2.3`

This is enforced in the `build-and-push-images` job:
```bash
if [ "${CIRCLE_BRANCH}" == "master" ] || [ "${CIRCLE_BRANCH}" == "main" ]; then
  IMAGE_TAG="${BASE_VERSION}"
  echo "🔒 PRODUCTION: Using clean version tag: ${IMAGE_TAG}"
else
  IMAGE_TAG="${BASE_VERSION}-${CIRCLE_BRANCH}-${CIRCLE_BUILD_NUM}"
fi
```

### 4. Environment Validation

The deploy job validates that DEPLOY_ENV is set to "prod":
```bash
if [ -z "${DEPLOY_ENV}" ]; then
  echo "ERROR: DEPLOY_ENV not set in CircleCI context!"
  exit 1
fi

if [ "${DEPLOY_ENV}" != "prod" ]; then
  echo "WARNING: DEPLOY_ENV=${DEPLOY_ENV} but expected 'prod'"
  exit 1
fi
```

---

## CircleCI Context Requirements

The `devops-mgt-prod` context must contain:

**AWS Credentials**:
- `AWS_ACCOUNT_ID`: 489176460030
- `AWS_REGION`: ca-central-1
- `AWS_ACCESS_KEY_ID`: [prod credentials]
- `AWS_SECRET_ACCESS_KEY`: [prod credentials]

**GitHub**:
- `GITHUB_TOKEN`: For cloning tbi-devops-environments

**Environment**:
- `DEPLOY_ENV`: prod

**Kubernetes**:
- Kubeconfig or credentials for prod cluster

---

## Self-Hosted Runner

All prod deployments use the self-hosted runner:
- **Resource Class**: `tbi-local-runner/prod-win`
- **Purpose**: Runs on Windows machine with kubectl access to prod cluster
- **Already Configured**: No additional setup needed

---

## Next Steps

### 1. CircleCI Setup (Required)

For each of the 20 services, you need to:

1. **Enable the project in CircleCI**
   - Go to CircleCI dashboard
   - Add project from GitHub
   - Select the service repository

2. **Configure branch triggers**
   - Set up to trigger on `master` branch only
   - Or manually trigger builds

3. **Verify context access**
   - Ensure `devops-mgt-prod` context is accessible
   - Verify all required environment variables are set

### 2. Test the Workflow

**Recommended Testing Approach**:

1. **Pick one service** (e.g., tbi-acc-profile-service)
2. **Trigger a build** on master branch
3. **Verify the workflow**:
   - Tests run successfully
   - BUILD approval notification received via SNS
   - Approve BUILD in CircleCI UI
   - Image builds and pushes to ECR
   - DEPLOY approval notification received via SNS
   - Approve DEPLOY in CircleCI UI
   - Deployment executes
   - Success notification received via SNS

4. **If successful**, roll out to remaining services

### 3. Documentation

Update your team documentation with:
- How to approve builds in CircleCI
- SNS notification format and what to look for
- Rollback procedures
- Emergency deployment procedures

---

## Files Modified

**CircleCI Configs** (20 files):
```
prod/tbi-acc-booking-service/.circleci/config.yml
prod/tbi-acc-ceetiz-connector/.circleci/config.yml
prod/tbi-acc-cron-service/.circleci/config.yml
prod/tbi-acc-exotiktours-connector/.circleci/config.yml
prod/tbi-acc-gimmonix-automation/.circleci/config.yml
prod/tbi-acc-hotel-mapping-service/.circleci/config.yml
prod/tbi-acc-hotel-service/.circleci/config.yml
prod/tbi-acc-hotel-service-connector/.circleci/config.yml
prod/tbi-acc-mybff/.circleci/config.yml
prod/tbi-acc-mybooking-frontend/.circleci/config.yml
prod/tbi-acc-mybooking-service/.circleci/config.yml
prod/tbi-acc-notification-service/.circleci/config.yml
prod/tbi-acc-package-cron/.circleci/config.yml
prod/tbi-acc-package-service/.circleci/config.yml
prod/tbi-acc-profile-service/.circleci/config.yml
prod/tbi-acc-profile-sync-service/.circleci/config.yml
prod/tbi-acc-quotes-service/.circleci/config.yml
prod/tbi-acc-sabreapi/.circleci/config.yml
prod/tbi-acc-travelbrands/.circleci/config.yml
prod/tbi-acc-travelbrandsagent/.circleci/config.yml
```

**Automation Scripts** (3 files):
```
scripts/update_prod_configs.py
scripts/convert_parameter_to_prod.py
scripts/update-prod-configs.sh
```

**Documentation** (1 file):
```
docs/PROD-APPROVAL-WORKFLOW-COMPLETE.md
```

---

## Comparison: Before vs After

### Before
- ❌ No approval gates
- ❌ No SNS notifications
- ❌ Wrong context (devops-mgt)
- ❌ Wrong runner (loadtest)
- ❌ Allowed all branches
- ❌ Version tags had branch suffix

### After
- ✅ Two-stage approval workflow
- ✅ Four SNS notifications per deployment
- ✅ Correct context (devops-mgt-prod)
- ✅ Correct runner (prod-win)
- ✅ Master branch only
- ✅ Clean version tags

---

## Reference Implementation

The reference implementation used as the template:
- **File**: `prod/tbi-package-travelbrands/.circleci/config.yml`
- **Features**: Complete approval workflow with SNS notifications
- **Status**: Production-ready and tested

---

## Support

If you encounter issues:

1. **Check CircleCI logs** for detailed error messages
2. **Verify context variables** are set correctly
3. **Check SNS topic** permissions
4. **Verify runner** is online and accessible
5. **Review this document** for configuration details

---

**Implementation Complete**: 2026-03-11  
**Total Services**: 20/20  
**Total Files Modified**: 24  
**Status**: ✅ Ready for CircleCI setup and testing