Amazon have announced a pretty interesting low-cost archive solution called Amazon Glacier; certainly the pricing which works out at $120 per terabyte per annum with 11 9’s availability is competitive. For those of us working with large media archives could this be a competitor with tape and all the tape management headaches that tape brings. I look after a fairly large media archive and there are times when I would gladly see the back of tape for-ever.
So is the Amazon Glacier the solution we are looking for? Well for the type of media archives that I manage, unfortunately the answer is not yet or not for all use cases. The 3-4 hour latency introduced on a recall by the Glacier does not fit many media organisations, especially those who might have a news component. At times even the minutes that retrieving from tape take seems to be unacceptable, especially to news editors and the like. And even at $120 per terabyte; when you are growing at multiple petabytes a year, the costs fairly quickly add-up.
Yet, this is the first storage product which has made be sit up and think that we could replace tape. If the access times were reduced substantially and it looked more like a large tape library; this would be an extremely interesting service. I just need the Glacier to move a bit faster.
I’ve been thinking the same thing this morning. I also work for a media organization and found this announcement very intriguing.
We have a large number of assets where there has long been a desire to archive them in perpetuity that we’ve been unable to accommodate due to the costs involved. This would make that possible, and the 3-4 hour access time would not be an issue for most of those assets.
I could also easily make a case for 3-5x the cost if the retrieval time could be lowered to something like 15 minutes or less – particularly if they could provide configurable policies for automatically demoting things from the 15 minute tier to 3-4 hours.
Curious – if you have a large amount of media data that you need to have quicker access, why wouldn’t S3 work for you? At your scales, is tape still cheaper than S3?
Also to Chuck’s point, they are saying there will be a method to migrate S3 data into Glacier, so essentially you could have an cloud-online and cloud-nearline storage tiers. But, what other barriers to entry are there for you folks who work with these massive archives?
S3 is still non-competitive with tape, the costs don’t stack up yet. It is debatable whether the costs for Glacier even if we could live with the access delay would stack up yet but it is closer. Also we have considerable sunk costs in our tape infrastructure and processes, we know how to manage this. Now, I am keeping a close eye on cloud storage and costs but it’s not there yet. There is also the not inconsiderable challenge of integrating with our existing applications.
The benchmark is £5000 to £7000 to store a PETA byte for ever. So while thisis interesting it still is costly relative to an academic archive
I’d like to see the calculations for that; I’m not sure that I can buy the tape-media to store that quantity of data for £5000-£7000, assuming that you are using tape. Then if you start equating in the supporting infrastructure, the refresh points and the integrity testing required to guarantee that the data is good forever. Yes, Glacier is expensive at scale, it is probably not cost effective for the peta-scale archives…yet.
Thanks for the post, Martin. Here’s yet another example of a vendor not really understanding that “media archives” actually often require fast retrieval. But, definitely interesting and a nice data point on the continuing drop in storage costs. Also the best marketing of “spin down” hard drive capabilities yet!
One clarification – the 11 9’s represents durability, not availability. I doubt even Amazon would claim 99.999999999% availability. In a year that would mean that their service was unavailable for 0.03 seconds (if my quick math is correct).
There aren’t just technical considerations in the offerings, there are the political. Glacier looks promising, until companies find themselves caught in legislation that requires them to store their data in the EU, or in their own territory, or anywhere not subject to foreign (ie US) laws.
Turns out that Glacier is in fact…. tape! Still a good fit for research data that we generate.