03-02-2017, 11:25 PM
The major outage that hit tens of thousands of websites using Amazon's AWS cloud computing service on Tuesday ends up having been the result of a simple typo — just one incorrectly-entered command.
On Tuesday morning, an Amazon team was investigating a problem that was slowing down [a] billing system.
At 9:37 am Pacific time, one of the team members executed a command that was meant to take a few of the [billing] servers offline.
"Unfortunately," Amazon said in its posting, one part of that command was entered incorrectly — i.e. it had a typo.
That mistake caused a larger number of servers to be taken offline than they'd wanted. Two of those servers ran some important systems for the whole East Coast region, such as the ones [responsible for] trillions of files [on] customers' websites.
To get it back, both systems required a full restart...It wasn't until...four hours and 17 minutes after the mistyped command was first entered that the entire system was back up and running.
Amazon says it'll never happen again. Whew!