S3 outage - We weren’t affected
Amazon S3 had an outage today. First I knew about it was reporters emailing and calling me asking if we were knocked out by it.
We weren’t. No customers reported issues, and our systems were all showing typically low and acceptable error rates. To be honest, I’m surprised.
I wasn’t going to blog about it until I understood why we weren’t affected, but I’m really getting inundated with requests now, so I figured this would be a good way to optimize my time rather than spending all day on the phone. 
We’re researching what happened now, but again, I didn’t know about the outage until after it was over, and I haven’t spoken to anyone at Amazon yet. Until I finish my research and speak with Amazon, I’m not going to speculate on what may have happened or why.
I can say, once again, that we pay the same rates everyone else pays and that, other than some early access to upcoming beta services, we don’t get any preferential treatment that I’m aware of.
Some thoughts, though:
- We expect Amazon to have outages. No website I’m aware of doesn’t, whether it’s Google, Amazon, your bank, or SmugMug.
- I’ve written about Amazon S3 outages in the past, but in the last ~12 months, we’ve only seen a single ~2 minute outage window (on January 22nd, 2008 at around 4:38pm Pacific). We also had one recent fairly major hiccup with EC2.
- Yes, I believe there will probably be times where SmugMug is seriously affected, possibly even offline completely, because Amazon (or some other web services provider) is having problems. Today wasn’t that day. Nobody likes outages, especially not us, but we’ve decided the tradeoffs are worth it. You should have your eyes wide open when you make the decision to use AWS or any other external service, though. It will fail from time to time.
- We’ve done our best to build SmugMug in such a way that we handle failures as gracefully as possible. We can’t cover every case, but I think that the fact that we didn’t experience customer-facing outages today is a testament to that. Again, I want to stress that we do expect Amazon to cause us (rare) outages in the future, and that’s unavoidable, but today we dodged that bullet.
- Amazon’s communication about this has been terrible. It took far too long to acknowledge the problem. Fixing a major problem can take forever, which is understandable, but communicating with your customers should happen very rapidly. Amazon’s culture, internally, is very customer focused, so this is a strange anomaly. I will definitely lean on them some about it, and everyone who was affected should rightfully howl too.
- I’ve asked Amazon repeatedly for an “Amazon Web Services Health” page that shows the current expected state of all their services. Then you can tell at a glance (and even poll and work into your own monitoring) whether any of the services are having problems. Something like Keynote’s Internet Health Report would be a good start, but as Jesse Robbins points out, trust.salesforce.com is the gold standard. This page could also double as a mechanism to let customers know what’s being worked on and current ETAs when there are problems.
I’ll try to post a follow-up about why we weren’t affected when I know more. It’s possible that some of the reasons we survived was due to some of our “secret sauce” and I just won’t be able to say, but I kinda doubt it.
Bottom line: While the outage was certainly a big deal to those affected, I think the bigger deal here is how Amazon handled the outage. They need to communicate better about these mission critical services and their health.
If I didn’t answer any questions you’d like me to answer, please post a comment and/or send me an email. I’ll do my best to respond.
UPDATE 1: I’m not sure why there’s all this confusion, but SmugMug *does* use Amazon as our primary data store. We maintain a small “hot cache” in our datacenters of frequently/recently viewed photos and videos, but there are massive numbers of them that are only at Amazon. This is a change from our initial usage of S3, and the change is based on how reliable they’ve been. Yes, we still consider them to be very reliable even after an outage like this. And yes, I suspect our “hot cache” did at least partially enable us to ride out this issue.
Tags: amazon, aws, ec2, outage, s3, smugmug, storage, web services
February 15th, 2008 at 12:52 pm
My understanding is that you simple use S3 as backup storage, so as long as you are queueing your syncs you should be able to handle fairly long outages on their side. I have never noticed an image URL on your site that is actually coming from S3.
February 15th, 2008 at 12:58 pm
@Brad:
Sorry for the confusion, but S3 is our primary data store, not just backup.
February 15th, 2008 at 1:07 pm
I’ve worked with S3 a bit. How are you serving up images from a http://www.smugmug.com address? Do you house them locally for a certain about of time until they get committed?
February 15th, 2008 at 1:12 pm
Hey Don - I’m glad that you (and my photos) weren’t affected by this downtime. I would echo your suggestion that AWS implement some sort of status page. How about leading by example, though? In this dgrin post, I recommended creating a Smugmug “status” blog that’s hosted in a different DC than the rest of your infrastructure. To my knowledge, no one responded to that suggestion.
Sure, outages are usually reported in Dgrin - that’s not an ideal situation, though, IMHO. If you implemented a status blog, it would make it very simple for your operations staff to post quick updates as they work through problems as well as enabling customers to subscribe to the blog’s RSS feed and get regular updates that way.
Thoughts?
February 15th, 2008 at 1:18 pm
@erik:
I heard you loud and clear during our last outage, and we now have an offsite location to post updates: http://smugmug.wordpress.com/
We just haven’t had an outage since then to use the blog on - but we will!
February 15th, 2008 at 1:20 pm
@Don:
Very nice. I’m already subscribed.
Thanks and keep up the great work!
February 15th, 2008 at 2:00 pm
[...] Please read the SmugMug blog on this. The real news isn’t that they weren’t affected, but is very reasoned expectations for [...]
February 15th, 2008 at 2:32 pm
@Don:
I have subscribed also… thanks for letting us know about the site.
February 15th, 2008 at 4:39 pm
According to Amazon, only one out of three S3 server locations was affected. We (gladly) did not experience any outages today but we are sure, as you stated, that inevitably one day we will
February 15th, 2008 at 5:14 pm
Speaking of S3, I was wondering if that article on you EC2 use you hinted at some time ago was coming soon… I’d be interested in hearing how you use it (it seems to me like you wouldn’t have enough resizing load to justify it… but I suppose I must be wrong :))
February 15th, 2008 at 6:36 pm
[...] S3 outage - We weren’t affected [...]
February 15th, 2008 at 10:43 pm
[...] earlier today, which affected KnowledgeTreeLive and its users. The outage was quite widely reported.KnowledgeTreeLive is in beta and so this was a great way to learn about our contingency planning, [...]
February 16th, 2008 at 2:42 am
[...] http://blogs.smugmug.com/don/2008/02/15/s3-outage-we-werent-affected/ Posted in web20. Tags: amazon, aws, ec2, s3, sqs, [...]
February 16th, 2008 at 3:15 am
Well I guess we were lucky enough to have our files at the one that went down. Yippee. Our customers certainly appreciated it. There’s sure nothing like waking up to think all your files are gone to start a day right. Don’t worry about us though our reputation will be fine. About the time it’s back to where it was before this they’ll poop again and it’ll be right back down in the trash.
I’m sure your paying customers will appreciate your sentiments around the fact that Amazon is down less than you would have been if you did your own hosting. Does telling them that really work? If so I want your customers because ours are complete the opposite. Of course yours put up images for fun. Not ever customer of S3 is doing this for jollies. Not to lessen what you are doing, but some things are more mission critical. Maybe shame on us then for having an alternative. Have you looked into this Don? Why not send images another S3 like service. Then at least a simple DNS change or URL filter could be used. It doubles the cost of storage but if Amazon ever goes down for say a day it’s going to be hell of a bad day.
February 16th, 2008 at 7:06 am
[...] SmugMug’s Don MacAskil has it right, and discussed his approach on TechCrunch’s report of the outage: We do rely on S3 for our primary storage, but we do maintain our own “hot cache” of data in our datacenters, too, which is less than 10% of our total storage. Our customers weren’t affected by this morning’s outage. [...]
February 16th, 2008 at 5:22 pm
Don, the reason that people are under the impression that S3 is only secondary storage for Smugmug is this post from you on 12 August 2006:
February 16th, 2008 at 8:16 pm
@SHG:
That’s so strange to me. Do people still really believe that humans will never fly because there’s a lot written about how impossible it is a few centuries ago?
That blog post is ancient history. The Wright Brothers showed us how to fly, and SmugMug has moved on with S3.
Guess I’d better update those old posts, but that seems silly to me that people wouldn’t do a little research rather than trusting some old blog post….
February 17th, 2008 at 12:34 am
[...] My journalist colleagues at Wired.com published some of my comments related to Amazon S3.1 Wired also posted another article titled Customers Shrug Off S3 Service Failure. I agree with the views of many of the customers expressed in the article. Don MacAskill, CEO of the popular photo hosting site Smugmug, wrote an understanding post about it. [...]
February 18th, 2008 at 6:00 am
No wonder I had some problems uploading at times on Friday (Aus) and there was an incident I uploaded successfully 5 pics and they didn’t appear in the gallery.
Keep us posted. Thanks
February 18th, 2008 at 6:47 am
[...] Don indeed responds and promises to provide more details once he found out why they were not affected [...]
February 18th, 2008 at 5:17 pm
@Arg: if you’re hosting mission critical data on something that openly states they only have 99.9% uptime, then you should be fired for incompetence. 99.9% uptime is NOT mission critical. If the service was advertising 5 9’s, you might have something to gripe about. Right now, you’re basically complaining that you’re having troubles getting a screw out of a board with a hammer.
@Don: If everything is on S3 now, how are there 3 copies of every photo? If it’s all sitting on S3, I guess that scares me. You’re basically entrusting them to never have catastrophic data loss… and I guess I don’t trust any one company that much
February 19th, 2008 at 9:36 am
@TimC: S3 has 3 different server locations.
February 19th, 2008 at 10:39 pm
@Jorge:
they could have 10 different server locations, if they have a failure in their software which results in data loss, it doesn’t really matter how many different locations the failure occurs at.
February 20th, 2008 at 3:34 am
@TimC: you asked “how are there 3 copies of every photo?”
February 20th, 2008 at 11:35 pm
^^which would be why I asked originally when it was quoted if they are only using S3 now as I read it to be.
March 3rd, 2008 at 9:44 pm
“That’s so strange to me. Do people still really believe that humans will never fly because there’s a lot written about how impossible it is a few centuries ago?”
That was the reason why I choose Smugmug and I can not find the announcement about the change :(.
It ultimately means that I am now paying only for a gallery and Smugmug resells S3 storage to host my photos.
Are you planning any discount/light packages with external S3 accounts? (I mean I am purchasing only the use of the software from Smugmug and the storage from S3)
March 17th, 2008 at 3:56 pm
[...] SmugBlog: S3 outage - We weren’t affected. some good criticism of amazon from that s3 outage. totally agree what AWS should have a site like this salesforce report card. [...]