Incident Details
| CloudWatch Alarm ARN | arn:aws:cloudwatch:us-east-1:466711825721:alarm:CWT-ALB-5XX-Alarm |
| Load Balancer | app/CWT-ALB/07a5a68474fdff28 |
| Target Group | arn:aws:elasticloadbalancing:us-east-1:466711825721:targetgroup/CWT-Web-Servers/bc45d541b59d03c3 |
| Timestamp | 2026-06-27T20:26:58.942000+00:00 |
| Analysis Window | Last 1 Hour |
| Severity | CRITICAL |
Executive Summary
The ALB app/CWT-ALB/07a5a68474fdff28 is experiencing a complete service outage. HealthyHostCount = 0 — all targets in target group CWT-Web-Servers are unhealthy or deregistered. Of 2,163,788 requests processed in the last hour, 416,720 (19.26%) returned 5XX errors generated entirely by the load balancer itself, with zero successful target responses recorded. All 195,167 TargetConnectionError events correspond directly to HTTP 504 timeouts, confirming that no backend instance is available to serve traffic.
Root Cause Summary: The incident is caused by a complete loss of healthy targets in the CWT-Web-Servers target group. With HealthyHostCount (Maximum) = 0, the ALB has no backend to forward requests to, resulting in self-generated HTTP 502 (Bad Gateway — 221,559 occurrences) and HTTP 504 (Gateway Timeout — 195,151 occurrences) errors. The access log data confirms this: all requests routed to target IP 172.31.10.253 received a target status code of "-" (no response), and the response processing time is uniformly -1.0 across all sampled request groups, indicating the connection was never completed from the target's side. The processing times from the ALB itself (1–4 ms request processing, 0–3 ms target processing attempt) are nominal, ruling out ALB-internal processing issues. The 1,185 requests with all three timing values at -1.0 indicate connection attempts that were dropped before any processing could begin. The HTTPCode_Target_5XX_Count = 0 further confirms that no errors originated from the application layer — all failures are infrastructure-level connection failures. Immediate action is required to restore healthy targets to the target group.
Incident Overview
ALB Metrics
| Metric Name | Statistic | Value | Assessment |
|---|---|---|---|
| RequestCount | Sum | 2,163,788 | Elevated |
| HTTPCode_ELB_5XX_Count | Sum | 416,720 | Critical |
| HTTPCode_ELB_502_Count | Sum | 221,559 | Critical |
| HTTPCode_ELB_503_Count | Sum | 10 | Normal |
| HTTPCode_ELB_504_Count | Sum | 195,151 | Critical |
| HTTPCode_Target_5XX_Count | Sum | 0 | Normal |
| TargetConnectionErrorCount | Sum | 195,167 | Critical |
| TargetResponseTime | p90 | 1.113 s | Warning |
| UnHealthyHostCount | Minimum | 0 | Warning |
| HealthyHostCount | Maximum | 0 | Critical |
| ELB 5XX Error Rate | Derived | 19.26% | Critical |
CloudWatch Graphs
Access Log — Target Response
| Target Group ARN | Target IP | Target Status Code | Request Count |
|---|---|---|---|
| arn:aws:elasticloadbalancing:us-east-1:466711825721:targetgroup/CWT-Web-Servers/bc45d541b59d03c3 | 172.31.10.253 | - (No Response) | 250,190 |
Note: A target status code of - indicates the ALB received no HTTP response from the target. This confirms complete target unavailability during the incident window.
Access Log — Processing Time
| Target Group | Target IP | Request Processing (s) | Target Processing (s) | Response Processing (s) | Request Count |
|---|---|---|---|---|---|
| CWT-Web-Servers | 172.31.10.253 | 0.001 | 0.0 | -1.0 | 5,364 |
| CWT-Web-Servers | 172.31.10.253 | 0.001 | 0.001 | -1.0 | 2,054 |
| CWT-Web-Servers | 172.31.10.253 | 0.002 | 0.0 | -1.0 | 1,745 |
| CWT-Web-Servers | 172.31.10.253 | 0.002 | 0.001 | -1.0 | 1,656 |
| CWT-Web-Servers | 172.31.10.253 | 0.001 | 0.002 | -1.0 | 1,331 |
| CWT-Web-Servers | 172.31.10.253 | -1.0 | -1.0 | -1.0 | 1,185 |
| CWT-Web-Servers | 172.31.10.253 | 0.003 | 0.001 | -1.0 | 1,061 |
| CWT-Web-Servers | 172.31.10.253 | 0.003 | 0.002 | -1.0 | 962 |
| CWT-Web-Servers | 172.31.10.253 | 0.001 | 0.003 | -1.0 | 757 |
| CWT-Web-Servers | 172.31.10.253 | 0.004 | 0.002 | -1.0 | 746 |
Note: A response_processing_time of -1.0 indicates that the ALB did not receive a complete response from the target. Row with all values -1.0 (1,185 requests) represents connections dropped before any processing phase.
Recommendations
-
1ImmediateRestore Healthy Targets to the Target GroupNavigate to EC2 > Target Groups > CWT-Web-Servers and verify the registration and health status of target 172.31.10.253. If the instance is stopped, start it immediately. If it has failed health checks, SSH in or use SSM Session Manager to inspect the application process (e.g.,
systemctl status nginx/apache/httpd), check application logs, and restart the service if needed. Confirm the health check path is returning HTTP 200 before traffic is re-routed. -
2ImmediateInvestigate Target Instance State and System LogsCheck the EC2 instance status for 172.31.10.253 in the AWS Console or via CLI (
aws ec2 describe-instance-status). Review OS-level logs (/var/log/syslog,/var/log/messages) and application logs for crash dumps, OOM kills, or disk-full conditions. Check CPU and memory utilization in CloudWatch to determine if resource exhaustion caused the outage. -
3ImmediateVerify Security Group and Network ACL RulesConfirm that the security group attached to the target instance allows inbound traffic on the health check port from the ALB's security group. Verify that Network ACLs on the subnet containing 172.31.10.253 (subnet for 172.31.10.x) permit both inbound and outbound traffic on the relevant application port. A misconfigured security group rule update could silently block ALB health checks, causing all targets to fail simultaneously.
-
4Short-TermImplement Multi-Instance / Multi-AZ Target RegistrationThe CWT-Web-Servers target group currently relies on a single target (172.31.10.253). This represents a critical single point of failure. Register at least two additional EC2 instances — ideally in different Availability Zones — to the target group. Configure an Auto Scaling Group (ASG) with a minimum capacity of 2 healthy instances to ensure automatic replacement of failed instances and prevent a full outage from any single host failure.
-
5Short-TermConfigure ALB CloudWatch Alarms with Actionable ThresholdsCreate dedicated CloudWatch alarms for: (1) HealthyHostCount < 1 — trigger immediate PagerDuty/SNS alert when all hosts become unhealthy; (2) HTTPCode_ELB_5XX_Count > threshold with a short evaluation period (1–2 minutes); (3) TargetConnectionErrorCount > 0 sustained over 5 minutes. Ensure alarms are tied to an SNS topic with email, SMS, and on-call escalation policies for 24/7 coverage.
-
6Medium-TermImplement an ALB 502/504 Custom Error Page via S3 or CloudFrontWhile backend recovery is in progress, end-users currently receive raw HTTP 502/504 responses with no context. Configure an ALB fixed-response action or integrate a CloudFront distribution with a custom S3-hosted maintenance page as an origin fallback. This ensures users receive a meaningful message during outages and protects brand reputation. Use CloudFront's origin failover feature to automatically serve the maintenance page when the ALB origin returns 5XX.
-
7Long-TermAdopt Infrastructure-as-Code and Immutable Deployment PatternsMigrate the CWT-Web-Servers infrastructure to an IaC-managed stack (AWS CDK, Terraform, or CloudFormation) that enforces minimum healthy host counts, ALB configuration, and Auto Scaling policies as code. Implement blue/green or canary deployment strategies via AWS CodeDeploy or ECS to eliminate manual configuration drift and enable rapid, zero-downtime rollbacks when application failures occur. Establish regular chaos engineering drills (e.g., instance termination tests) to validate recovery procedures and alarm efficacy before a real incident.