I was fascinated and indeed slightly surprised at yesterday's announcement that NetApp had bought Bycast, an object storage company. Why so?
Well I distinctly remember Val saying just after the Atmos launch that NetApp had an object storage strategy and something would be coming soon; I'm sure someone can probably find the exact comment. So if NetApp were already working on object storage, why did they purchase Bycast? Now either Val was being economical with the truth and when he said that NetApp were working on object storage, he meant that they were looking at buying something? Or perhaps, NetApp have struggled internally to develop something; certainly there are rumours that the focus was on getting OnTap 8 out the door and all development resource was really focused on that.
Now this purchase leaves NetApp with something of a problem as to what to actually do.
1) Do they integrate Bycast into OnTap? I suspect that this would be the preferred route and would allow them to keep the 'Unified Storage' tag but this route is fraught with difficulty and the Spinnaker experience may well have scared them off.
2) Do they simply continue with Bycast as a separate product? This is a much more pragmatic approach but it leaves them open to criticism that they no longer sell 'Unified Storage'.
It's an interesting position for NetApp to find themselves in.
Object Storage itself is an interesting technology area and one which itself brings a number of challenges. Actually implementing an object store is not especially hard but interfacing the object store with a multitude of different applications is going to be the biggest challenge. Meta-data continues to be challenging with many custom schema being deployed for different uses.
It's still very early days for large-scale deployments of object storage; I suspect that this market has some way to go before any kind of front runner/leader can be declared. I think we are still stumbling over the early fences at the moment.
It will also be interesting to see how NetApp position the product; it could feasibly be positioned as a direct competitor with OnTap 8. Could they fall into some of internal politics that EMC's Atmos has suffered from with sales-guys worrying that it could cannibalise their traditional revenue streams?
Or do they use it as a way to compete with EMC's DaaD strategy?
Do they try to build an application infrastructure around it?
Interesting times….
Hi, Martin.
I’m interested in your comment that “Actually implementing an object store is not especially hard”. Could you expand on your thinking there?
I ask because I’ve worked on 3 OS now in succession (Centera, Atmos and now HCP), learning something new each time, and I’ve found there to be a number of interesting challenges. One of the greater is solving the bookkeeping problem. It’s one thing to be able to ingest billions of user objects, which in turn typically turn into 10’s of billions of internal cluster objects, but keeping track of these over a distributed system that presents as a GNS (global name space) is hard.
In fact, I think two of the harder general problems in CS are scale and concurrency; the latter being the bane of many a grad student tackling a large scale project. 🙂
Object stores, particularly if they are distributed object stores, need to solve both of these. Granted, the concurrency issue isn’t as severe as it is in microchips as there’s a different time scale, but nonetheless it’s non trivial to keep a set of discrete nodes all maintaining a concurrent view of the state of a potentially very large set of objects within the domain.
The second hard problem to solve is scale, which is where the bookkeeping problem arises. How does one keep track of 10’s of billions of fragments (depending on the means for data protection selected and whether it fragments or clones objects) that somehow distill to user objects. Couple this with the expanded metadata requirements that are expected with most enterprise-class OS and the problem set expands. I’d assert you can’t simply use a COTS DBMS with nothing added if you hope to reach very large scale *and* be able to perform operations (e.g. read, garbage collection, auto recovery, etc.) in a reasonably quick manner.
I realize the description I’ve provided here may be too brief to do justice to explaining the challenges of an OS. I’m working on a paper that talks about the principles of operation for a distributed OS, and I’m finding that just culling it down to a reasonable size to be a bigger challenge than I expected – so perhaps my sense of the difficulties of large scale OS is a bit more top of mind for me right now.
Anyway, I’m interested in where you were going with that.
Thanks,
Bob
Bob,
when I said not that hard, I was probably trivialising the technical challenges but I would suggest that the hardest part of implementing an object store is herding all the application vendors and developers and getting them to use it. Build it and they will come…really rarely works in my experience.
But to get all the value of an object store; simply sticking NFS/CIFS/HTTP gateways in front are probably not going to cut it.
Martin
Hi Martin,
Thanks for blogging about this! Note that while NetApp are in the midst of our Q4 quiet period as well as closing the Bycast acquisition itself, I cannot comment in detail at this time.
However this announcement is very consistent with our overall Unified Storage strategy, with various Object Interfaces for our Unified Storage arrays scheduled to appear over time.
Bycast has precious years of battle-hardened real-world experiences in designing, building and operating large-scale multi-site global object namespaces for mission-critical applications. We look forward to harnessing that valuable experience towards a market-leading Object Storage offering for Enterprise Archive and Cloud Storage customers over the short term.
See this excellent blog for more insight into Bycast technology. Note the heavy SNIA CDMI standards-based implementation entries 🙂
http://intotheinfrastructure.blogspot.com/
http://intotheinfrastructure.blogspot.com/2008/11/welcome-emc.html
Val Bercovici
Chair, SNIA Cloud Storage Initiative
Cloud Czar, NetApp Office of the CTO
http://blogs.netapp.com/exposed/