Amazon Web Services, Disaster Continuity, Service Alert, Storage

Amazon S3 Outage

Amazon S3 Outage We WILL KeepYouWORKING

Feb 28, 2017 1:15 PM PST.  We are aware of the current Amazon S3 Outage.  Ironic that this happens in the middle of our Disaster Preparedness launch, but it’s proving to be a good test.

Update:  Please follow @BansheeHelpDesk on Twitter for real-time updates.  All status updates will be appended to this blog post!

Here’s what we know:

Amazon Web Services (AWS) Simple Storage Service (S3) is affected in their US-East-1 region.   Since many of their services use S3, these services would be impacted.   Also, since many of our own vendors use S3 in that region, some of the services we offer (like our social media posting capabilities) are also impacted.

BUT….most of our clients are not in that region.  Our policy is to use the AWS region closest to your physical operations, so other regions (notably the western US) are still fully operational.  So if you are in California, your S3 backups are working fine.

Furthermore, since we use multiple vendors for cloud infrastructure (such as Microsoft Azure), much of your operation is not even affected.

So currently, some of our capabilities are affected.  But much of our #KeepYouWORKING Disaster Preparedness systems are still operating without any issues.

In the event of any extreme impact from the S3 outage, we can also switch backup storage to other AWS regions or even to Azure.  There may be a cost impact, but we’ll still be able to maintain uninterrupted backup and be able to keep you operating in the event of a disaster.

Feb 28, 2017 1:35 PM PST:  AWS US-East-1 is partially recovered, but still impacting service.

Our S3-based services in the Western US are still operating, as are Europe and Asia.

Some websites dependent on S3 and related services may see some impact, but most are running fine.  Basically, if you can view this blog and our main site, yours should be fine too!

Currently, the biggest impact to our marketing services is with social media automation.   However, we can continue to use manual posting for the most time-sensitive items.

Feb 28, 2017 1:35 PM PST:  Constant Contact users may be unable to upload images.  We’re about to send a client email, so we shall see firsthand about the impact!

Feb 28, 2017 2:15 PM PST:  We see no issues with composing email and uploading images to Constant Contact, but we suspect the outage is affecting processing and possibly routing of the messages.  If you are planning to send a mailing, please give it time to get out.

Feb 28, 2017 2:29 PM PST:  AWS reports service is operating normally!

However, we are still seeing impact on email marketing and social media automation.

Feb 28, 2017 3:14 PM PST:  Amazon S3 is reported as back to normal, but other services in the US-East-1 region are still impacted.

Constant Contact emails are being received.

We have not seen any significant downtime on websites and applications we manage, including third party-hosted sites, with the notable exception of content delivery of S3-hosted assets in the eastern region.   Since we are based in California, most of our hosting is not in the impacted region.

Automation of social media posts is working to at least some degree.  We are checking on the status of all posts scheduled from this morning.

Feb 28, 2017 3:47 PM PST:  Adobe cloud services appear to be fully up and running.

Feb 28, 2017 3:47 PM PST:  AWS services apparently are back to normal!   We still have some wrapping up to do on this matter, but we are so happy to see that even with a major S3 outage…our Disaster Preparedness system has worked again.

We developed #KeepYouWORKING systems during major natural disasters, but this is actually the first failure of a major cloud infrastructure component.   We’ve helped clients through epic storms, earthquakes, and major disasters…and AWS has always been running through it all.   So thanks Amazon…we were able to get through this man-made disaster together!

Now it’s time for us to look at what can be learned from this experience, so that we can deal with any future cloud outages!




Amazon S3 Outage We WILL KeepYouWORKING

Leave a Reply

Theme by Anders Norén