As many of you are aware, my team is part of a major project to build-out a file-based workflow system for a major UK broadcaster. One of the things which is daunting is the sheer amount of data that we are beginning to ingest and the fact that we are currently looking at a ‘grow forever’ archive; everything we ingest, we will keep forever.
Even though we are less than two years into the project, we are already thinking about the next refresh of technology. And what is really daunting is that with our data growth; once we start refreshing, I suspect that we will never stop.
Not only will we be storing petabytes of new content every year; we will be moving even more old content between technologies every year. We are already looking at moving many hundreds of terabytes into the full production system without impacting operations and with little to no downtime.
We might be an edge case as we need to keep everything forever but as big data becomes a way of life for many companies; the migration effort will become considerable and the traditional migration techniques might not scale.
Data is going to have to flow automatically and between silos of storage technologies. Automated Storage Tiering was just the start; like it or not, the HSM problem is probably going to need to be tackled. And I can see some readers of my blog shuddering already!
Ironically this makes me think of how you can migrate data between pools with Tivoli Storage Manager (did a lot with TSM in a previous life). For a recommendation that might be actually helpful, beautiful levels of abstraction for data management and ease around scalability.
Woops…meant to include Isilon in that last sentence.
I think the answer lies in CDMI and what is coming with that. I’ve been hearing some amazing use cases for this over the past couple of weeks and am looking at a couple of deployment scenarios that match what you’re doing. No point migrating with another proprietary technology just to end up in the same trap 2 years later. Open standards is the key!
Shall I read into your comment that SONAS has not tackled the HSM problem? Some of your readers may shudder but I certainly am not!
Don’t believe I mentioned SONAS; we don’t use SONAS, we do use GPFS and TSM but we are not using the HSM function. SONAS does have HSM, how well it works and how well applications cope with having to wait for their files depends.
For HSM to work, both applications and user workflows have to be well documented and understood. Historically for some reason in open-systems, this has been close to an insurmountable problem.
I guess you’ve never heard of F5 ARX? CIFS/NFSv3 seamless tiering between NAS silos. Zero-touch migration forever. Please contact us for a quote. 😉
I’ve head of F5 ARX….it’s not zero-touch and it’s not forever. And it’s not really scale-out.
I guess I’d better know your performance requirements before shouting out. Out of context it seemed obvious. Cheerz!