I had hoped we’d moved beyond the SPC-1 benchmarketing but it appears not. If you read Hu’s blog; you will find that the VSP G1000 is
the clear leader in storage performance against the leading all flash storage arrays!
But when you look at the list, there are so many flash arrays missing from the list that it is hardly worth bothering with. No Pure, no Solidfire, no Violin and obviously no EMC (obviously because they don’t play the SPC game). Now, I haven’t spoken to the absentees whether they intend to both with the SPC benchmarketing exercise; I suspect most don’t intend too at the moment as they are too busy trying to improve and iterate their products.
So what we end up with is a pretty meaningless list.
Is it useful to know when your array’s performance falls of a cliff? Yes, it probably is but you might be better trying to get your vendor to sign-up to some performance guarantees as opposed to relying on a benchmark that currently appears to have little value.
I wish we could move away from benchmarketing, magic quadrants and the ‘woo’ that surrounds the storage market. I suspect we won’t anytime soon.
Martin,
To the best of my knowledge, SPC doesn’t validate results run on storage arrays with data reduction technologies enabled. This antiquated restriction is limiting a number of AFAs from participating in the benchmark.
A number of storage vendors are working together to get SPC to update the criteria for validation of results. Hopefully we’ll see this change occur in 2015.
— Cheers,
v
I almost wish that you wouldn’t bother…SPC seems to generate a general willy-waving attitude as opposed to creating capability to improve customer experience. It’s one of the most tedious parts of the storage industry.
Perhaps in future you could have competitions to install your arrays in the fastest possible time?
Or whose array can work in the most inhospitable environment.
Or a competition to see how fast a nun can configure your array…’A nun installing a reasonably priced lun’?
“and obviously no EMC (obviously because they don’t play the SPC game)”
That’s right. Makes it tricky when the RFP wants to see SPC numbers.
But a lot of these benchmarks “suck” for a period of time because it is well worth the investment to game them (especially if a large RFP wants to see SPEC numbers – for instance). Many a compiler tweek to game SPEC fp/int. What happens is the real-world code that can be gamed causes the benchmark to be kicked out of SPEC as the committee reviews, new ones incorporated. Likewise, TPC-C was “broken” long ago, etc. SPC-1 is broken [1], everyone knows it.
Finally.. what do you do short of industry standard benchmarks? POC each box? That works for the big boys, but not practical for SMB.
[1] All cache hits with AFA. Not sure how or what replaces it. Maybe one only focuses on cost and features with reasonable SPC-1 numbers as a check down. And certain vendors tap dance and remind you SPC means little or nothing.