Amazon's S3 Storage Chokes, Startups Gag
Amazon's S3 Web-storage-on-demand service has won a lot of mindshare in the Web 2.0/startup world. Which means when the service breaks down, as it did this morning, you're going to hear a lot about it.
Users are complaining that parts of Twitter and Tumblr -- graphics, for example -- are down (see screenshot below, via Allen Stern).
NY Web software startup AdaptiveBlue, which relies on S3 for some of its infrastructure, is also struggling. "Just wanted to let you all know that there are currently some issues that are effecting BlueOrganizer functionality," exec Andy Roth wrote on the company's blog a few minutes ago.
"Amazing how many of the services i use are reliant on S3," Union Square Ventures' Fred Wilson Twitters. "Stuff is broken everywhere this morning."
End of the world? Hardly. There's a big future in distributed storage and computing, and Amazon (AMZN) is on the leading edge. Nimble startups benefit any time they can focus more on building their companies than building their server infrastructure. But just as hosting firm Rackspace has lost business because of poor performance, we expect some companies will rethink some of their hosting (or at least backup hosting) options this afternoon.
What I observe is that most people treat Amazon S3 as a content delivery service. While this is not inherently wrong, one has to notice that S3 was especially designed to be a STORAGE service. S3 does not claim to be a CDN.
The point is, since terabyte hard drives are affordable nowadays and internet traffic grows steadily, the stress goes much more on content delivery and network infrastructure rather than on storage. If you are not concerned about using remote storage, there are much better services especially suited for content delivery.
SteadyOffload.com provides an innovative, subtle and convenient way to offload static content. The whole mechanism there is quite different from Amazon S3. Instead of permanently uploading your files to a third-party host, their cachebot crawls your site and mirrors the content in a temporary cache on their servers. Content remains stored on your server while it is being delivered from the SteadyOffload cache. The URL of the cached object on their server is dynamically generated at page loading time, very scrambled and is changing often, so you don’t have to worry about hotlinking. This means that there is an almost non-existent chance that the cached content gets exposed outside of your web application.
It’s definitely worth trying because it’s not a storage service like S3 but exactly a service for offloading static content.
Watch that:
http://video.google.com/videoplay?docid=-8193919167634099306 (the video shows integration with WordPress, but it is integrable with any other webpage)
http://www.steadyoffload.com/
http://codex.wordpress.org/WordPress_Optimization/Offloading
Cost of bandwidth comes under $0.2 per GB - affordable, efficient and convenient. Looks like a startup but lures me very much. Definitely simpler and safer than Amazon S3.
Not so. S3, like Amazon's other services, is highly distributed. Werner Vogels and colleagues have written an excellent technical paper on the architecture behind this:
http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html
Architecturally, this is a lot like P2P networks, except that all of the peer computers are servers run by Amazon.
Distributed systems aren't immune from failure, and other networks like Akamai have had failures in the past too. This wasn't something like a single server or router going down; we'll have to see what Amazon's explanation is, but I'd guess it has something to do with a bug in the inter-peer protocols that caused crashes or overloads across a broad swathe of the hosts.
http://www.johnmwillis.com/cloud-computing/look-mom-two-nines-amazon-s3-major-outage-today/
Hilarious some of the "important" stuff being hosted: twitter, tumblr, pageflakes. Diaries are important and all, but calm down!