Operational Excellence
Release Guardrail Template
Use this checklist before, during, and after releases—especially when error budget is constrained. Prevent incidents and ensure safe deployments.
🚀 Release: Payment Service v2.4.0
Feature: Add support for Apple Pay integration + Performance optimizations
Phase 1: Pre-Flight Checks
✓ APPROVED
✓
Error Budget Health Check
Current Status: 62% consumed (38% remaining)
Projection: At current burn rate, 28% will remain after 7-day window
Decision: ✅ Sufficient budget for medium-risk release. Proceed with enhanced monitoring.
Projection: At current burn rate, 28% will remain after 7-day window
Decision: ✅ Sufficient budget for medium-risk release. Proceed with enhanced monitoring.
✓
Risk Assessment & Blast Radius
Feature Type: New payment method integration (Apple Pay) + Database query optimizations
Blast Radius: Payment service only. Checkout flow has fallback to existing methods.
Risk Factors: External dependency (Apple Pay API), DB schema unchanged, feature flag enabled.
Mitigation: Canary deployment (5% → 25% → 100%), circuit breaker configured.
Blast Radius: Payment service only. Checkout flow has fallback to existing methods.
Risk Factors: External dependency (Apple Pay API), DB schema unchanged, feature flag enabled.
Mitigation: Canary deployment (5% → 25% → 100%), circuit breaker configured.
✓
Rollback Plan Validated
Rollback Method: Kubernetes deployment rollback + feature flag disable
Testing: ✅ Tested in staging environment on Jan 12
ETA: < 5 minutes to complete rollback
Owner: DevOps on-call (Jordan Lee)
Testing: ✅ Tested in staging environment on Jan 12
ETA: < 5 minutes to complete rollback
Owner: DevOps on-call (Jordan Lee)
✓
Approval & Stakeholder Sign-Off
Approved By: Alex Chen (SRE Lead), Sarah Kim (Engineering Manager)
Informed: Product team, Customer Support, Executive sponsor
Conditions: Monitor first 2 hours closely; pause at any SLO breach signal.
Informed: Product team, Customer Support, Executive sponsor
Conditions: Monitor first 2 hours closely; pause at any SLO breach signal.
💡 Go/No-Go Decision Matrix
| Error Budget Remaining | Risk Level | Decision |
| > 50% | Low/Medium | GO |
| 30-50% | Medium | CAUTION Require senior approval |
| < 30% | Any | STOP Defer non-critical releases |
Phase 2: During Release (Live Monitoring)
⚡ IN PROGRESS
Canary: 25% traffic routed to v2.4.0 | 35% remaining
Monitor SLO Symptoms in Real-Time
Dashboard: https://monitoring.example.com/payment-service
Key Metrics:
• Availability: Currently 99.94% ✅ (Target: 99.9%)
• Error Rate: 0.4% ✅ (Target: < 1%)
• Latency p95: 340ms ✅ (Target: < 600ms)
• Apple Pay API Success Rate: 98.2% ✅ (Target: > 95%)
Key Metrics:
• Availability: Currently 99.94% ✅ (Target: 99.9%)
• Error Rate: 0.4% ✅ (Target: < 1%)
• Latency p95: 340ms ✅ (Target: < 600ms)
• Apple Pay API Success Rate: 98.2% ✅ (Target: > 95%)
🛑 Abort Criteria - STOP DEPLOYMENT IF:
- Error rate > 2% sustained for 5+ minutes
- Latency p95 > 800ms (33% above target) for 10+ minutes
- Availability drops below 99.5% for any duration
- Apple Pay API failures > 10% indicating integration issue
- Customer support tickets spike > 20% vs. baseline
Action: Immediately trigger rollback and notify incident commander. Do not wait for confirmation.
Canary Deployment Progression
Stage 1: 5% traffic for 30 minutes ✅ Completed @ 10:15 AM
Stage 2: 25% traffic for 1 hour ⚡ In progress (10:45 AM - 11:45 AM)
Stage 3: 100% traffic - Pending approval after Stage 2 validation
Stage 2: 25% traffic for 1 hour ⚡ In progress (10:45 AM - 11:45 AM)
Stage 3: 100% traffic - Pending approval after Stage 2 validation
⚠️ Rollback Procedure (if needed)
# Step 1: Disable feature flag (immediate)
curl -X POST https://flags.internal/payment/apple-pay -d '{"enabled": false}'
# Step 2: Rollback Kubernetes deployment
kubectl rollout undo deployment/payment-service -n production
# Step 3: Verify rollback
kubectl rollout status deployment/payment-service -n production
# Step 4: Monitor metrics for 10 minutes
# Dashboard: https://monitoring.example.com/payment-service
# Step 5: Notify stakeholders in #incidents channel
# ETA: < 5 minutes total rollback time
curl -X POST https://flags.internal/payment/apple-pay -d '{"enabled": false}'
# Step 2: Rollback Kubernetes deployment
kubectl rollout undo deployment/payment-service -n production
# Step 3: Verify rollback
kubectl rollout status deployment/payment-service -n production
# Step 4: Monitor metrics for 10 minutes
# Dashboard: https://monitoring.example.com/payment-service
# Step 5: Notify stakeholders in #incidents channel
# ETA: < 5 minutes total rollback time
Owner: Jordan Lee (DevOps On-Call) | Backup: Alex Chen (SRE Lead)
Phase 3: Post-Release Review
Log Release Outcome
Status: [ ] Success - No incidents [ ] Partial - Minor issues [ ] Failed - Rolled back
Duration: _____ hours from start to 100% rollout
Incidents Triggered: [ ] None [ ] Minor (non-SLO) [ ] Major (SLO breach)
Error Budget Impact: _____ % additional burn during release window
Duration: _____ hours from start to 100% rollout
Incidents Triggered: [ ] None [ ] Minor (non-SLO) [ ] Major (SLO breach)
Error Budget Impact: _____ % additional burn during release window
Incident Response (if applicable)
If rolled back:
• Open postmortem ticket: JIRA template "Release Failure Review"
• Schedule postmortem meeting within 48 hours
• Document: What happened, why, how to prevent
• Action items assigned with owners and due dates
• Open postmortem ticket: JIRA template "Release Failure Review"
• Schedule postmortem meeting within 48 hours
• Document: What happened, why, how to prevent
• Action items assigned with owners and due dates
Update Guardrails & Runbooks
Learnings to Document:
• New failure modes discovered during release
• Monitoring gaps identified (add new alerts)
• Rollback process improvements needed
• Update risk assessment criteria for future similar releases
• New failure modes discovered during release
• Monitoring gaps identified (add new alerts)
• Rollback process improvements needed
• Update risk assessment criteria for future similar releases
✅ Success Criteria
Release is considered successful when:
✓ 100% traffic routed to new version with no rollback
✓ All SLO targets maintained for 24 hours post-release
✓ No increase in customer support tickets or complaints
✓ Error budget burn remains within projected limits
✓ All monitoring and alerting functioning correctly
✓ 100% traffic routed to new version with no rollback
✓ All SLO targets maintained for 24 hours post-release
✓ No increase in customer support tickets or complaints
✓ Error budget burn remains within projected limits
✓ All monitoring and alerting functioning correctly
💡 Best Practices
- Peak hours: Avoid releases during 9am-5pm weekdays unless urgent. Schedule for low-traffic windows.
- Communication: Post release status in #engineering and #product channels. Keep stakeholders informed.
- Blameless culture: If rollback needed, focus on system improvements not individual fault.
- Documentation: Update this template with lessons learned from each release.