You can't have it both ways and it appears that some in IBM want to have it both ways. IBM historically have enjoyed benchmarketing and have historically kicked EMC for not submitting Symmetrix for SPC. EMC have responded that customers don't run SPC as a workload and it does not realistically reflect a true customer workload. At which point the whole storage industry blows a raspberry and calls them chicken.
But read this comment from StorageBuddhist on my blog entry 'Wither XIV'
"The closest you will get to an XIV benchmark is probably my blog post from April http://storagebuddhist.wordpress.com/2010/04/06/xiv-ms-exch-2007-replicated-performance/ but being a non-trad box, XIV has a very non-trad performance profile. We recently heard from a customer who has got more than 100,000 IOPs out of their XIV in the course of normal business duties – something I had previously assumed was only possible in the labs."
I'm trying to work out what he is saying; is he saying that XIV is only suited for non-traditional workloads? Why won't IBM publish standardised benchmarks for XIV, they do for everything else they ship? If they could find a way of benchmarking the free pens they hand out, they would! I know he doesn't mention SPC but that's the standard benchmark for storage that IBM support.
You can't call EMC out for not submitting their arrays to SPC and then not submit one of your own because you know it will perform poorly.
Either SPC is a valid storage benchmark or it's not! Or perhaps XIV is only good for running some workloads? You see if that is the case then no matter how simple it is to manage; it does mean that to cover all my workloads, I am going to need multiple types of array and that builds in complexity to the environment. And at that point, does not much of XIV's selling point go down the toilet? I will still need a traditional storage team to tune and manage those applications which need a traditional array to perform and don't only perform well when run from cache.
Do I think EMC should submit their arrays to SPC; actually, I do. Do I think IBM should submit all their arrays to SPC and not cherry-pick which they do, abso-blooming-lutely I do. Or IBM should come very clean as to what applications those arrays they do not submit to SPC are not suited for. If they do not, they call the whole validity of the SPC into question.
Hi Martin
For me, the real question is why should EMC — as a market leader in storage — endorse a benchmark we believe is entirely substandard and misleading?
We have always had serious concerns with the specific tests themselves, as well as flaws in the methodologies used. Lots of evidence over the last decade confirms this perspective.
One take might be “what the hell, caveat emptor”, throw the numbers out there, and let customers attempt to figure out what it all means.
Another take might be “we’re not going to run these tests, but you’re welcome to do so on your own”.
This resulted in one of our competitors pulling a rather adolescent marketing stunt — and not entirely reinforcing our faith in the individual who runs SPC part-time.
So, for the time being, the whole “standard benchmark” thing doesn’t seem very important to most storage customers these days. Seems that they’ve got more pressing challenges on their minds these days 🙂
Thanks for a good post
— Chuck
It would be better if EMC stopped whining about benchmarks being ‘substandard’ or ‘misleading’, and act like a leader they purport to be and propose a standard so that customers can honestly assess performance relative to other choices. 16 years in the industry taught me that this is just *one* measure customers look at.
Lead, Follow or Get Out of the Way are the only options. Whining is not on that list.
At least EMC are consistent and don’t cherry pick, I feel that at least shows some integrity. However, I agree that EMC, as the market leader should show some leadership and propose a better benchmark. I’m not holding my breath at the moment.
Martin, I too wish that IBM didn’t cherry-pick. Maybe that was Moshe’s call. That’s why I pulled together the ESRP comparison to try to plug the information gap. By the way, nothing I said implied that “XIV is only suited for non-traditional workloads”. What I was suggesting is that because XIV is a non-traditional architecture, the performance profile is non-traditional. Anecdotal evidence suggest that it is a draught horse rather than a race horse. Keep adding load and it keeps trucking on, and the more varied the workload the stronger it seems to be in relation to traditional systems, and it especially likes write-heavy workloads compared to trad systems.
EMC has marketing down to a science, and this policy is no exception. As a market leader, it is a good policy to not publish benchmarks. If they were the best, people would expect it anyway. If they weren’t, however, it would cause quite a stir. Chuck is just doing his usual smoke and mirrors thing.
Remember how much publicity NetApp received by forcing EMC’s hand into a CX SPC-1? That was a good move for NetApp. Even if EMC published good numbers today, someone would post better numbers tomorrow and use it against them.
No, EMC likes the ground-game. With 1:1 marketing to each potential customer, they can talk big with their slide decks on whiz-bang hw architectures. They are much better positioned to sell performance there than with the even playing field of the SPC benchmarks.
I’m not saying their performance is bad. I know some of their platforms are smokin’ fast. They would just rather not be pinned down during a sales cycle with details from the SPC.
IBM is just following the same marketing practice, however, they do have a good amount of publicity to gain with actually publishing nice numbers. Their reluctance makes me suspect that maybe the news wouldn’t be very good.
Lets get it right; NetApp actually sponsored the CX SPC benchmark; they did not force EMC into anything. It was a shoddy little piece of marketing and one I hope no vendor ever gets into their tiny little minds to repeat.
You and I may believe that EMC should be involved in industry standard benchmarking; if they are not happy with the current benchmark, they should should work with the community to define a different one. But what NetApp did actually damaged the credibility of the SPC; they should have refused to carry it out. If ever the term ‘benchmarketing’ should apply, what NetApp did, fits the bill! I know there are plenty in NetApp who are really rather embarrassed by that episode.
I don’t understand your objection to Netapp running the CLARiiON SPC-1 benchmark? I thought what Netapp did was entirely reasonable, and personally I’d be quite happy if they did the same with an XIV : ) Why not have more information and more transparency?
It was trying to force EMC into doing something they said they wouldn’t do; we’ll do it and if you aren’t happy with the results, you tune your array and submit it again. It was tantamount to a form of blackmail. Actually, to EMC’s credit; they stuck to their guns and didn’t play.
And I think it’s rather telling that NetApp have not done it again…perhaps they will but I think not. I hope we’ve all moved on from that.
EMC called it upon themselves. I was at NetApp for 5 years and nearly every competitive opportunity we had with EMC, they would fill the customer with a load of WAFL performance FUD while providing no benchmarks to back it up.
You can’t blame NetApp for finally calling them to the carpet–and not with some questionable “independent” report funded by NetApp–but with an actual SPC benchmark.
The only shoddy marketing here was the EMC FUD, not NetApp actually bringing facts and metrics to combat it.
Like I said, EMC prefers the sales 1:1 ground game where cool slide decks and, yes, even FUD can be shared behind closed doors. They won’t do industry standard benchmarks even if they could write the whole benchmark themselves.
Guys, some things about XIV and why IBM is not posting a SPC-1 benchmark. First of all (and quite obvious) – because XIV would not excel in that benchmark. The benchmark does not measure somewhere near the sweet spot of the system, which StorageBhuddist expressed rightly in his post.
In my very own humble opinion, EMC has never had the fastest systems – but how important is the maximum performance of a system to the end user in most cases ?
In my customer situations, I very frequently meet customers who are very sure they need SSD to cover their performance needs – which ironically they cannot even describe. Going the hard way and actually metering what’s going on in their environment often proves the grossly over-estimate the IOPS they are making – which leads to expensively wrong decisions.
(I am working for IBM, but pose my personal opinions only)
So, if you go for the highest IOPS, IBM will not even think to offer XIV but will lead with DS8000 or SVC.
If you look for ‘good enough’ performance in a consolidation environment, IBM will likely lead with XIV – and ask lots of questions concerning your workload in order to size the system big enough.
When buying a tractor, the top speed is usually not what concerns you most – right ?
Yet they will still publish that top speed of a tractor. Sorry, if IBM are unwilling to publish figures for their whole range; we can only assume that they have something to hide.
And lets be honest, if a Ferrari could do the job of a tractor at a comparable cost, you’d buy the Ferrari wouldn’t you?
IBM have been quite willing to talk about XIV doing ridiculous amounts of IOPs but generally unwilling to submit it to any kind of public benchmark. Do I think IBM have run SPC against XIV? Yes, I do and does it expose XIV’s serious performance limitations? Of course it does. Perhaps it is a carthorse or a tractor; time to admit, it is only good for ploughing fields then.