Amazon Web Services Has 3rd Outage In A Month. Is It Just Too Big To Work?

There are three issues to consider: is it just too big to function reliably? Has the socio-economic system become too dependent on one provider? And if this happens without any proximate cause, what could happen if Russian, Chinese or Iranian hackers actually wanted to shut it down?

Just as with 'just-in-time' supply chain models, there is a point where the potential threat from disruption outweighs the ostensible benefits of scale and efficiency. Imagine being one of those people trying to resolve customer problems or attempting to finish up so you could take some holiday time off but had to wait twelve hours for the problem to be incompletely resolved. JL

James Vincent reports in The Verge:

Amazon’s crucial web services business AWS experienced problems Wednesday due to a power outage. It’s the third time in as many weeks that problems with AWS have had a significant effect on online services. Some Slack users were unable to view or upload images and work management tool Asana also hit by the outages.The official AWS service health dashboard blamed the issues on power outages in a single data center, affecting one Availability Zone. Twelve hours after the problem began, Amazon marked the issue resolved. However, that didn’t necessarily fix everything for all of the servers affected.

Amazon’s crucial web services business AWS has experienced problems today due to a power outage, affecting services like Slack, Imgur, and the Epic Games store for some users. It’s not looking good if you’re working from home, with some Slack users unable to view or upload images and work management tool Asana also hit by the outages.

The official AWS service health dashboard blamed the issues on power outages in a single data center, affecting one Availability Zone (USE1-AZ4) within the US-EAST-1 Region. At 9:13AM ET, Amazon said it had restored power to the affected servers, and by 12:28PM ET, it had “restored underlying connectivity to the majority of the remaining” systems. However, users may still be experiencing issues as services and servers are relaunched.

At 7:22PM ET, just over twelve hours after the problem began, Amazon marked the issue resolved. However, that restoration didn’t necessarily fix everything for all of the servers affected. The final AWS status page update explains Amazon’s issues:

Starting at 4:11 AM PST some EC2 instances and EBS volumes experienced a loss of power in a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region. Instances in other data centers within the affected Availability Zone, and other Availability Zones within the US-EAST-1 Region were not affected by this event.

At 4:55 AM PDT, power was restored to EC2 instances and EBS volumes in the affected data center, which allowed the majority of EC2 instances and EBS volumes to recover. However, due to the nature of the power event, some of the underlying hardware experienced failures, which needed to be resolved by engineers within the facility.

Engineers worked to recover the remaining EC2 instances and EBS volumes affected by the issue. By 2:30 PM PST, we recovered the vast majority of EC2 instances and EBS volumes. However, some of the affected EC2 instances and EBS volumes were running on hardware that has been affected by the loss of power and is not recoverable.

For customers still waiting for recovery of a specific EC2 instance or EBS volume, we recommend that you relaunch the instance or recreate the volume from a snapshot for full recovery. If you need further assistance, please contact AWS Support.

In an incident update at the peak of the outage, Slack said its services were “experiencing issues with file uploads, message editing, and other services.” Asana said the problems constituted a “major outage,” with “many of our users unable to access Asana.” Epic Games Store said “Internet services outages” were “affecting logins, library, purchases, etc.”

Of the affected services, it appears that they are largely back online, even as Amazon’s restoration continues. A Slack page tracking the connection issues marked the problem resolved around 3:13PM ET.

It’s the third time in as many weeks that problems with AWS have had a significant effect on online services. Two incidents earlier this month involving AWS ended up knocking out a huge array of platforms and products, taking out streaming sites like Netflix and Disney Plus, as well as smart home devices like security cameras from Ring and Wyze.

Today’s outages seem less widespread but still notable, with some users unable to access services entirely and others merely experiencing intermittent faults. DownDetector.com shows reports of issues with the platforms mentioned above, as well as news aggregator Flipboard, online learning site Udemy, dating app Grindr, streaming service Hulu, and IoT services from Honeywell, Life360, and Samsung’s SmartThings.