On the night of 19 October 2025, a routine piece of automation inside Amazon DynamoDB raced against itself and left the service’s main address in the US-EAST-1 region pointing at nothing. The record was not wrong. It was empty. For the next fifteen hours, that empty record rippled outward through EC2, Network Load Balancing, Lambda and more than seventy other AWS services, and out into Snapchat, Reddit, Fortnite, Coinbase, Ring, Duolingo, several banks and two US airlines.

AWS has since published a detailed post-event summary. It is worth reading in full. This is the shorter version, and what it says about how large systems actually fail.

The DNS race condition

DynamoDB’s regional endpoint — dynamodb.us-east-1.amazonaws.com — is served by hundreds of load balancers whose addresses change constantly as capacity comes and goes. Two automated components keep DNS in step with reality:

  • The DNS Planner watches load balancer health and produces a “plan”: the current correct set of IP addresses for the endpoint.
  • The DNS Enactor takes the newest plan and applies it to Route 53. It runs redundantly across three Availability Zones for resilience.

On the night of the incident, one Enactor stalled. In AWS’s words, it “experienced unusually high delays” while applying an older plan. While it was stuck, the Planner produced several newer plans, and a second Enactor picked up the newest one and applied it cleanly — then ran its normal cleanup pass, deleting the older plans it believed were now obsolete.

Then the stalled first Enactor woke up and finished its job: it wrote its old plan to the regional endpoint. That plan had just been deleted from under it. The result, per AWS: “all IP addresses for the regional endpoint were immediately removed,” and the system “was left in an inconsistent state that required manual operator intervention to correct.”

That last clause is the important one. The automation could not recover on its own because the safe assumption it was built on — that a plan being applied still exists — no longer held. A human had to log in and put the addresses back.

Why one region’s database took down EC2

An empty DNS record for a database is bad. It should not have been able to break the ability to launch virtual machines. It did, because EC2’s control plane quietly depends on DynamoDB.

EC2 physical servers (“droplets”) are managed by a component called DropletWorkflowManager (DWFM), which holds a lease on each server and renews it by checking in with DynamoDB. With DynamoDB unreachable, the leases “slowly started to time out.” When DNS was restored and DWFM tried to re-establish every lease at once, it “entered a state of congestive collapse”: the work queue grew faster than it could be drained, so nothing completed, so the queue grew. Engineers had to throttle incoming work and selectively restart DWFM hosts before it would converge — roughly six hours after the original DNS problem was fixed.

New EC2 instances then launched but had no network connectivity, because the network-configuration backlog took until mid-morning to clear.

The second-order cascade: health checks

Network Load Balancer added its own twist. As delayed EC2 instances came online, NLB health checks “checked new EC2 instances while network state had not yet fully propagated.” Checks flipped between healthy and failing. NLB interpreted the failures as a dead Availability Zone and automatically failed away from it via DNS — pulling healthy capacity out of service. The fix was to disable automatic health-check failover entirely until the network caught up.

This is a familiar failure shape: a safety mechanism (fail away from an unhealthy AZ) misfiring on bad input (transient health-check noise) and amplifying the outage it was meant to contain. AWS’s headline corrective action for NLB is a “velocity control mechanism to limit the capacity a single NLB can remove when health check failures cause AZ failover.”

Why recovery took as long as it did

The DNS record was restored around 2:25 AM PDT, a little over two hours after the incident began. Full service recovery came at roughly 3 PM. The gap — most of the outage — was spent digging out from the cascade, not fixing the original bug:

  • DWFM congestive collapse: throttle, restart, wait.
  • Network propagation backlog: wait.
  • NLB pulling healthy capacity: disable the failover, wait, re-enable.
  • Redshift clusters whose node credentials had expired during the outage, triggering replacement workflows that were themselves blocked by the EC2 launch impairment.

Each recovery step had to wait for the one beneath it. The system came back up in the reverse of the order it fell down.

The lessons that generalise

Regional isn’t isolated. US-EAST-1 is AWS’s oldest and busiest region, and several global control-plane functions — parts of IAM, some console operations, Redshift’s IAM path — still route through it. A US-EAST-1 event is never only a US-EAST-1 event.

Control planes hide dependencies. DynamoDB taking down EC2 launches surprised a lot of experienced engineers. The data plane (running instances, serving traffic) held up far better than the control plane (creating, configuring, healing). If your disaster-recovery plan assumes you can launch replacement capacity during a major cloud incident, test that assumption.

Automation needs a safe failure mode, not just a fast success path. Both the DNS Enactor and NLB’s failover behaved exactly as written. Neither had a brake for the case where the world changed underneath them. The corrective actions AWS committed to are almost all brakes: disable the DNS automation worldwide pending a fix, rate-limit DWFM by queue depth, cap how much capacity an NLB can shed.

DNS is still the single point of failure it always was. Decades of building redundancy into everything else, and the thing that took down a sizable fraction of the internet was one endpoint resolving to an empty set.

What AWS changed

From the post-event summary, AWS has:

  • Disabled the DynamoDB DNS Planner and Enactor automation worldwide until the race condition is fixed and “additional protections to prevent application of incorrect DNS plans” are in place.
  • Added a velocity control to limit how much capacity an NLB removes on health-check-driven AZ failover.
  • Committed to build a DWFM recovery test suite and to rate-limit its incoming work by queue size.

None of that is exotic. It is the same lesson every large distributed system eventually learns in production: the failure you did not rehearse is the one that keeps you down for fifteen hours.