gestaltit

As follow-up to my blog here; I'd like to share yet more thoughts on availability and the potential negative impacts on some of the new technologies out there.

How many of you run clusters of servers? HA/CMP? Veritas Cluster? Microsoft Cluster? VMWare Clustering? I suspect lots of you do? How many of you cluster NAS heads? Yet again, I suspect lots of you do? How many of you cluster arrays? Not so many I guess? Certainly in my experience, it is uncommon to cluster an array. And when I talk about clustering an array, I don't mean the implementation of replication.

So, if you don't cluster your arrays; how do you protect against the failure of a RAID rank? Statistically unlikely but it is it more or less unlikely than a loss of data-centre? I'm not sure and the failure of a RAID rank for many people could well mean the invocation of the disaster recovery plan. Why? 

The loss of a RAID rank might well lead to the loss of an application/service and if it is an absolutely business critical service, can you bring it up at the remote replication site in isolation? As a discrete component? If you can, can you cope with increased transaction times due to latency? Many applications now have complex interactions with partner applications; these might not be well understood. So the failure of RAID rank could lead to the invocation of the Disaster Recovery Plan. Actually in my experience, this is very nearly always the case unless the service has been designed with recovery in mind; this requires infrastructure and application teams to work together, something which we are not exactly good at.

But you now take the challenge and make sure that every application can be failed over as a discrete component. Excellent, a winner is you! You know the impact of loosing a RAID rank, you know what applications it impacts, you've done your service mappings etc, etc. And you have been very careful to lay things out to minimise a single RAID failure's impact.

And then you implement automated storage tiering. Firstly, you now have no idea in advance what impact a RAID rank failure may have; you have no idea what applications may be impacted. And actually, the failure of a single RAID rank may well have huge impact. We could be looking at restoring many terabytes of data to cope with the failure of a couple of terabytes and many applications failing.

It will depend on the implementation of the automated storage tiering and I am concerned that at present we do not know enough about the various implementations which will hitting our arrays over the next eighteen months. So despite automation making things day-to-day a lot easier, we cannot treat it as Automagic Storage Tiering; we need to know how this works and how we plan to manage this. 

And perhaps for key applications, we will need to cluster storage arrays locally; that in itself will bring challenges.

I'm still a big fan of automated storage tiering but over the next few months, I would like to see the various vendors start talking about how they mitigate some of this risk. Barry Burke has made a big thing about the impact of a double disk failure on an XIV array in the past; in a FAST v2 environment, I would like to see how EMC mitigate against very similar problems.

I would also like to know what impact of a PAM card failure from NetApp is; does the array degrade to the extent where it is not useable? What kind of tools can NetApp give me to assess potential impact. As Preston points out here; failure of individual components within an array could have significant impacts.

We are heading to a situation where technology gets every more complex and arguably ever more reliable. But we rely on it to ever more greater extents; so we must understand risks and mitigations to a much greater amount than we have in the past. 

I've been thinking about FAST and especially FAST v2 but not entirely from a storage point of view. FAST v2 and indeed any automated storage tiering product has some interesting uses beyond storage and could be a basis for a whole new way of managing IT as a service. In fact, it finally enables storage and beyond to managed as a service. BTW I'm going to use FAST as shorthand for any automated storage product; so please don't take this as only being about EMC.

In order for FAST to work, it needs to gather and react to a lot of information from the array itself. In fact for FAST to be truly useful, it needs to gather, react and store alot of information about what is going on the array.

Take a typical corporate accounting application; most of the time it can be pretty quiet and non-performance intensive but at certain times of the year, it will be a very intensive workload. During these times, you might want it all to be on the fastest, most performant tier; now FAST will react to a sudden increase in workload and move the application when it sees the demand increase but will FAST be able to move this quickly enough? So perhaps, we need to give the array some hints as to when to prime the load?

These sort of peaks are very predictable and we know when they will happen but not all peaks are quite as predictable; or at least we don't think they are. FAST will be gathering stats all the time and by analysing this data; it might be able to do the predictive analysis a lot quicker and spot things that we can't or at least don't have the time for. It may pick up on relationships between applications, application X runs hot at a certain time which causes application Y to become busy at some period later; for example, certain types of activity may cause a reporting job to be run at a later date.

You see from our storage infrastructure, we can start to gather a lot of information about our whole estate. But EMC could go further, they have things like nLayers and Smarts to leverage; they could start to pull information from VMware and do a whole lot of analysis on this. NetApp have SanScreen; HP have a zillion tools as do IBM.

Once you've got that information, you need to start turning that into something the business understands so that you can sit with the business and do what-if modelling, show conflicts and clashes where multiple services are demanding the same high-performance infrastructure at the same time. Perhaps the business owner needs to prioritise or purchase more infrastructure. Perhaps they need less, perhaps they can shift some stuff into the Public Cloud and just pull it back when they need too.

So FAST could be rather more than just a way optimising your storage infrastructure; if you data-mine this in the same way Google data-mine statistics, you can find out a lot of stuff which you didn't realise and probably completely change the way you look at your infrastructure.

So when EMC talk about FAST being a foundational technology, they aren't wrong…actually, like Virtual Provisioning, it is so important….it should be Free! Actually they could fund this by getting rid of half their account managers; FAST could literally sell itself.