# Production and Redtag-Prod Approval Workflow Update - COMPLETE

**Date**: 2026-03-12  
**Status**: ✅ COMPLETE  
**Services Updated**: 7 services (2 prod + 5 redtag-prod)

---

## Summary

Successfully updated all remaining prod and redtag-prod CircleCI configs with 2-stage approval workflow and SNS notifications to match the reference implementation (tbi-package-travelbrands).

---

## Services Updated

### Prod Services (2)

1. ✅ **tbi-itravel-aff-booking-path**
   - Added 2-stage approval workflow (BUILD + DEPLOY)
   - Added 4 SNS notifications
   - Updated context: `devops-mgt-prod`
   - Updated runner: `tbi-local-runner/prod-win`
   - Added clean version tags for production
   - Added DEPLOY_ENV validation (must be "prod")
   - Restricted to `master` branch only

2. ✅ **tbi-copolo-frontend**
   - Added 2-stage approval workflow (BUILD + DEPLOY)
   - Added 4 SNS notifications
   - Updated context: `devops-mgt-prod`
   - Updated runner: `tbi-local-runner/prod-win`
   - Added clean version tags for production
   - Added DEPLOY_ENV validation (must be "prod")
   - Restricted to `master` branch only

### Redtag-Prod Services (5)

3. ✅ **tbi-redtag-mobile-hotel-service**
4. ✅ **tbi-redtag-mobile-ms-activities**
5. ✅ **tbi-redtag-mobile-ms-bookings**
6. ✅ **tbi-redtag-mobile-ms-cars**
7. ✅ **tbi-redtag-mobile-ms-flights**

All redtag-prod services now have:
- 2-stage approval workflow (BUILD + DEPLOY)
- 4 SNS notifications per deployment
- Context: `devops-mgt-prod` (same as prod)
- Runner: `tbi-local-runner/prod-win`
- Clean version tags for production
- DEPLOY_ENV validation (must be "redtag-prod")
- Restricted to `master` branch only
- Cluster: `redtag-prod` (different from prod)
- Namespace: `redtag` (different from prod's `access`)

---

## Key Differences: Prod vs Redtag-Prod

| Feature | Prod Services | Redtag-Prod Services |
|---------|--------------|---------------------|
| Context | `devops-mgt-prod` | `devops-mgt-prod` (same) |
| Runner | `tbi-local-runner/prod-win` | `tbi-local-runner/prod-win` (same) |
| Cluster | `prod` | `redtag-prod` |
| Namespace | `access` | `redtag` |
| DEPLOY_ENV | `prod` | `redtag-prod` |
| envFolder mapping | `master\|main) envFolder="prod"` | `master\|main) envFolder="redtag-prod"` |

---

## Implementation Details

### 2-Stage Approval Workflow

All 7 services now follow this workflow:

1. **Step 1**: Run tests (`lint-and-test`)
2. **Step 2**: Send BUILD approval notification via SNS
3. **Step 3**: Manual approval gate (BUILD)
4. **Step 4**: Build and push Docker image
5. **Step 5**: Send DEPLOY approval notification via SNS
6. **Step 6**: Manual approval gate (DEPLOY)
7. **Step 7**: Deploy to production Kubernetes

### SNS Notifications (4 per deployment)

**1. BUILD Approval Needed**
```
⏳ PRODUCTION Pipeline - Awaiting BUILD Approval
Service: [service-name]
Environment: PRODUCTION
✅ Tests have passed successfully
⏳ MANUAL APPROVAL REQUIRED to proceed with Docker image build
```

**2. DEPLOY Approval Needed**
```
⏳ PRODUCTION Pipeline - Awaiting DEPLOY Approval
Service: [service-name]
Environment: PRODUCTION
✅ Tests passed
✅ Docker image built and pushed to ECR
⏳ MANUAL APPROVAL REQUIRED to deploy to PRODUCTION cluster
⚠️  WARNING: This will deploy to PRODUCTION Kubernetes cluster!
```

**3. Deployment Success**
```
✅ Deployment Successful - [service-name]
Environment: [prod or redtag-prod]
Namespace: [access or redtag]
Image Tag: [version]
Build: #[build-number]
```

**4. Deployment Failure**
```
❌ Deployment Failed - [service-name]
Environment: [prod or redtag-prod]
Namespace: [access or redtag]
Image Tag: [version]
Build: #[build-number]
Please check the build logs for error details.
```

### Clean Version Tags

Production deployments use clean semantic version tags:
- Dev/UAT: `1.2.3-develop-456`
- Production: `1.2.3`

Implemented in `build-and-push-images` job:
```bash
if [ "${CIRCLE_BRANCH}" == "master" ] || [ "${CIRCLE_BRANCH}" == "main" ]; then
  IMAGE_TAG="${BASE_VERSION}"
  echo "🔒 PRODUCTION: Using clean version tag: ${IMAGE_TAG}"
else
  IMAGE_TAG="${BASE_VERSION}-${CIRCLE_BRANCH}-${CIRCLE_BUILD_NUM}"
fi
```

### Environment Validation

**Prod Services** validate DEPLOY_ENV="prod":
```bash
if [ "${DEPLOY_ENV}" != "prod" ]; then
  echo "WARNING: DEPLOY_ENV=${DEPLOY_ENV} but expected 'prod'"
  exit 1
fi
```

**Redtag-Prod Services** validate DEPLOY_ENV="redtag-prod":
```bash
if [ "${DEPLOY_ENV}" != "redtag-prod" ]; then
  echo "WARNING: DEPLOY_ENV=${DEPLOY_ENV} but expected 'redtag-prod'"
  exit 1
fi
```

---

## Automation Scripts Created

### 1. scripts/update_redtag_prod_configs.py
- Updated all 5 redtag-prod configs automatically
- Applied consistent changes across all services
- Added approval workflow, SNS notifications, clean tags
- Updated context, runner, and DEPLOY_ENV validation

**Usage**:
```bash
python3 scripts/update_redtag_prod_configs.py
```

---

## Files Modified

**Prod Configs** (2 files):
```
prod/tbi-itravel-aff-booking-path/.circleci/config.yml
prod/tbi-copolo-frontend/.circleci/config.yml
```

**Redtag-Prod Configs** (5 files):
```
redtag-prod/tbi-redtag-mobile-hotel-service/.circleci/config.yml
redtag-prod/tbi-redtag-mobile-ms-activities/.circleci/config.yml
redtag-prod/tbi-redtag-mobile-ms-bookings/.circleci/config.yml
redtag-prod/tbi-redtag-mobile-ms-cars/.circleci/config.yml
redtag-prod/tbi-redtag-mobile-ms-flights/.circleci/config.yml
```

**Automation Scripts** (1 file):
```
scripts/update_redtag_prod_configs.py
```

**Documentation** (1 file):
```
docs/PROD-REDTAG-APPROVAL-UPDATE-COMPLETE.md
```

---

## Verification Results

All 7 services verified with:
- ✅ `notify-await-build-approval` job present
- ✅ `notify-await-deploy-approval` job present
- ✅ Context: `devops-mgt-prod`
- ✅ Resource class: `tbi-local-runner/prod-win`
- ✅ Branch filter: `master` only
- ✅ SNS notifications (4 per deployment)
- ✅ Clean version tags for production
- ✅ DEPLOY_ENV validation

---

## Complete Status: All Production Services

### ACC Services (20) - Previously Completed
All 20 ACC production services were updated in the previous batch:
- tbi-acc-booking-service
- tbi-acc-ceetiz-connector
- tbi-acc-cron-service
- tbi-acc-exotiktours-connector
- tbi-acc-gimmonix-automation
- tbi-acc-hotel-mapping-service
- tbi-acc-hotel-service
- tbi-acc-hotel-service-connector
- tbi-acc-mybff
- tbi-acc-mybooking-frontend
- tbi-acc-mybooking-service
- tbi-acc-notification-service
- tbi-acc-package-cron
- tbi-acc-package-service
- tbi-acc-profile-service
- tbi-acc-profile-sync-service
- tbi-acc-quotes-service
- tbi-acc-sabreapi
- tbi-acc-travelbrands
- tbi-acc-travelbrandsagent

### Other Prod Services (3) - Now Complete
- ✅ tbi-package-travelbrands (reference implementation)
- ✅ tbi-itravel-aff-booking-path (updated today)
- ✅ tbi-copolo-frontend (updated today)

### Redtag-Prod Services (5) - Now Complete
- ✅ tbi-redtag-mobile-hotel-service (updated today)
- ✅ tbi-redtag-mobile-ms-activities (updated today)
- ✅ tbi-redtag-mobile-ms-bookings (updated today)
- ✅ tbi-redtag-mobile-ms-cars (updated today)
- ✅ tbi-redtag-mobile-ms-flights (updated today)

### Special Case (1) - Already Correct
- ✅ tbi-pl-dynamic (deployment-only service, already has proper approval)

---

## Total Production Services with Approval Workflow

**29 services total** across prod and redtag-prod folders:
- 24 prod services (20 ACC + 3 other + 1 special)
- 5 redtag-prod services

All now have 2-stage approval workflow with SNS notifications.

---

## Next Steps

### 1. Git Commit and Push

Commit all changes to the repository:

```bash
git add prod/tbi-itravel-aff-booking-path/.circleci/config.yml
git add prod/tbi-copolo-frontend/.circleci/config.yml
git add redtag-prod/*/. circleci/config.yml
git add scripts/update_redtag_prod_configs.py
git add docs/PROD-REDTAG-APPROVAL-UPDATE-COMPLETE.md

git commit -m "Add 2-stage approval workflow to remaining prod and redtag-prod services

- Updated tbi-itravel-aff-booking-path with approval workflow
- Updated tbi-copolo-frontend with approval workflow
- Updated 5 redtag-prod mobile services with approval workflow
- Added SNS notifications (4 per deployment)
- Updated context to devops-mgt-prod
- Updated runner to tbi-local-runner/prod-win
- Added clean version tags for production
- Added DEPLOY_ENV validation
- Restricted workflows to master branch only

Services updated:
- prod/tbi-itravel-aff-booking-path
- prod/tbi-copolo-frontend
- redtag-prod/tbi-redtag-mobile-hotel-service
- redtag-prod/tbi-redtag-mobile-ms-activities
- redtag-prod/tbi-redtag-mobile-ms-bookings
- redtag-prod/tbi-redtag-mobile-ms-cars
- redtag-prod/tbi-redtag-mobile-ms-flights

All 29 production services now have 2-stage approval workflow."

git push origin master
```

### 2. Sync Documentation to S3

```bash
bash scripts/sync-to-s3.sh
```

### 3. CircleCI Setup

For each of the 7 newly updated services:

1. **Enable the project in CircleCI**
   - Go to CircleCI dashboard
   - Add project from GitHub
   - Select the service repository

2. **Configure branch triggers**
   - Set up to trigger on `master` branch only

3. **Verify context access**
   - Ensure `devops-mgt-prod` context is accessible
   - Verify DEPLOY_ENV is set correctly:
     - Prod services: `DEPLOY_ENV=prod`
     - Redtag-prod services: `DEPLOY_ENV=redtag-prod`

### 4. Test the Workflow

**Recommended Testing Approach**:

1. **Pick one service** (e.g., tbi-copolo-frontend)
2. **Trigger a build** on master branch
3. **Verify the workflow**:
   - Tests run successfully
   - BUILD approval notification received via SNS
   - Approve BUILD in CircleCI UI
   - Image builds and pushes to ECR
   - DEPLOY approval notification received via SNS
   - Approve DEPLOY in CircleCI UI
   - Deployment executes
   - Success notification received via SNS

4. **If successful**, roll out to remaining services

---

## Reference Implementation

The reference implementation used as the template:
- **File**: `prod/tbi-package-travelbrands/.circleci/config.yml`
- **Features**: Complete approval workflow with SNS notifications
- **Status**: Production-ready and tested

---

## Support

If you encounter issues:

1. **Check CircleCI logs** for detailed error messages
2. **Verify context variables** are set correctly
3. **Check SNS topic** permissions
4. **Verify runner** is online and accessible
5. **Review this document** for configuration details
6. **Check DEPLOY_ENV** matches expected value (prod or redtag-prod)

---

**Implementation Complete**: 2026-03-12  
**Total Services Updated**: 7 (2 prod + 5 redtag-prod)  
**Total Production Services**: 29 (all with approval workflow)  
**Status**: ✅ Ready for commit, push, and CircleCI setup