DevOps Agent: Automatic Investigation
1. Architecture ▾
At this time, CloudWatch Alarms does not have a native integration to initiate a DevOps Agent investigations. However, alarm can leverage Lambda Alarm Actions to address this gap. The Lambda Function uses a webhook to automatically launch a DevOps Agent investigation to determine the root cause.
In the final scenarion, the collection of CloudWatch metric and composite alarms are deployed as illustrated in the diagram. The alarms monitor both the performance of a load balancers and its targets.
2. Automatic Investigation ▾
Although investigations were initiated by different alarms, DevOps Agents correctly correlated them to the same workload.
3. Root Cause ▾
Root Cause
Distributed Load Test (120 Fargate tasks) overwhelmed CWT Web Servers with ~117K requests/min. At 2026-06-10T20:41:16Z, the Distributed Load Testing (DLT) infrastructure (AWS Solution SO0062) created an ECS service 'dlt-4nCWbwc3nM-us-east-1' with 120 desired Fargate tasks on the 'LaunchWizard-Web-Server-Load-Testing' cluster. The task definition 'dlt-worker-4nCWbwc3nM:1' was triggered by Lambda function 'DLTLambdaFunctionTaskRun' (TestId: 4nCWbwc3nM, TestRunId: v7CLh74Dwa). Within ~2 minutes of launch, the 120 Fargate tasks began sending HTTP traffic to the CWT-ALB, causing request rates to surge from 0 to 117,702 req/min by 20:46, new connections to reach 145,592/min, and active connections to hit 42,769. This is the 3rd such load test cycle in the last hour (previous bursts at ~19:54 and ~20:07). The CWT Web Servers backend (1-2 t3.micro instances running NGiNX) cannot handle this volume, resulting in TCP connection exhaustion (2046 vs 1024 capacity), 28.99s ALB latency, 72,757 ELB 5XX errors/min, and only 23% success response rate.
4. Key Findings ▾
Key Findings
CWT Web Servers ASG did not scale out to handle load test traffic (fixed at 1-2 t3.micro). The CWT-Web-Servers ASG is configured with min 1, max 2 t3.micro instances. During the load test generating 117K+ req/min, no scaling activity occurred - the ASG remained at its current size. A single t3.micro instance running NGiNX has limited connection handling capacity (alarm threshold indicates 1024 connections is the performance degradation point). The combination of massive inbound traffic and an undersized, non-scaling backend fleet means all traffic concentrates on 1-2 small instances, quickly exhausting their TCP connection capacity and causing cascading failures (latency spike, 5XX errors, low success rate).
5. Mitigation ▾
Mitigation
Scale up CWT-Web-Servers Auto Scaling Group from 1-2 instances to 2-10 instances and configure auto-scaling policies to handle traffic surges. The root cause analysis identified that the Distributed Load Testing infrastructure with 120 Fargate tasks overwhelmed the CWT Web Service by generating ~117,000 requests per minute. The CWT-Web-Servers Auto Scaling Group (ASG) in AWS account 466711825721, region us-east-1, is configured with a severely undersized capacity (MinSize=1, MaxSize=2, DesiredCapacity=1) using t3.micro instances. During the load test, a single t3.micro instance running NGiNX hit TCP connection capacity limits (2046 vs 1024 threshold), causing cascading failures: 28.99s ALB latency, 72,757 ELB 5XX errors per minute, 23% success response rate, and unhealthy hosts. The ASG has no scaling policies configured, meaning it cannot automatically respond to traffic increases. Expanding the ASG to MinSize=2, MaxSize=10, DesiredCapacity=4, and adding target tracking auto-scaling policies will provide sufficient capacity headroom to absorb similar traffic patterns in the future and prevent recurrence. Impacted resources: arn:aws:autoscaling:us-east-1:466711825721:autoScalingGroup:77600a79-a7d2-41e7-ab40-ac863057ebc7:autoScalingGroupName/CWT-Web-Servers, arn:aws:elasticloadbalancing:us-east-1:466711825721:loadbalancer/app/CWT-ALB/07a5a68474fdff28