Last week saw IBM crack the 10 million transactions per minute barrier in the TPC-C Benchmark, in the latest round of what has become the server performance equivalent of an arms race. So has IBM’s action put the last nail in the coffin of TPC-C? With such a performance number on the board, which is meaningless for all but the biggest enterprises, are there anymore mountains left to climb with this benchmark?
The latest IBM result has been widely reported already and it has been interesting to see some of the reaction. Given that others have addressed the technical aspects, this blog is going to take a different tack and instead step back and look at the history of the race to be king of this benchmark mountain.
The chart above paints an interesting picture. It plots all TPC-C Results over time and it shows a select few results from companies, who have broken away from the pack, in order to chase the much coveted #1 position and the associated bragging rights.
A Brief History of the Race to Become #1
In many ways the same questions are being asked now as in 2003, when the previous psychological barrier of 1 million transactions per second was broken by HP with an Itanium2 based Integrity Superdome configuration. Back then many wondered if achieving that level of performance would dampen the enthusiasm to continue testing.
In 2004 IBM answered that question, when its POWER5 processor based technology eServer p5 595 configuration cracked the 3 million transactions per minute barrier. This record stood for about 3-years before IBM again upped the ante, this time with its next generation POWER5+ based System p5 595, which managed just over 4 million transactions per minute. HP was obviously waiting in the wings and very quickly edged past this with an Itanium 2 processor based Integrity Superdome configuration a few weeks later.
It was then some time until IBM released its next generation POWER6 processor and the company increased the stakes with a result for a Power 595 in June of 2008 that achieved over 6 million transactions per minute, to once again grab the lead. A Bull Escala model PL6460R result was also released with the same performance as this IBM system. This was the same system and configuration as the IBM machine, but sold under the Bull banner, primarily in Europe.
Then in 2009 Oracle, hot on the back of its acquisition of Sun Microsystems, rejoined the TPC-C fray with a result for a cluster of 12 x T5440 servers, reaching over 7.6 million transactions per minute, trumping IBM. Sun for many years prior to this result was released shunned the TPC-C benchmark and so Oracle’s decision to once again play put new life into the benchmark. But of course, IBM wasn’t going to stand still and it appears bided its time until the release of the POWER7 processor technology and it looks like set its mind on the next psychological barrier of 10 million transactions per minute. Last week the company released a result for a cluster of Power 780 servers to take the current number one performance position in the benchmark.
Can We Expect Testing to Continue?
So, where does the TPC-C Benchmark go from here? Can we expect IBM’s latest result to be overtaken? And does this ever increasing performance ceiling, which is beyond that of the majority of usage applications, render the benchmark irrelevant?
In the computer industry, there is a natural focus on the high-end, be it a new world record performance result, or simply news of a new high-end system. The big system for example tends to get the lion’s share of attention at announcement time, whilst other smaller systems in a set of announcements don’t often get the same level of attention. But that doesn’t mean they are less relevant. Often it is the opposite. And the same can be said of performance benchmarking.
While the race to number one has been going on, as the chart above shows, many other results have continued to be released at lower levels of performance. So, as previous number one results have not seemed to affect the general testing activity, we believe this latest IBM result won’t really dampen the general desire to test either.
The TPC-C Benchmark has been around a long time, which makes the test appealing because it provides some consistency in what is an ever changing market. It enables users to compare performance gains for products over a long period of time. And the recent introduction of the optional energy consumption metrics to the test is likely to breathe some life back into testing as well.
However, one downside of this longevity is that the benchmark hasn’t kept up with general computing workload developments and despite some minor changes to the specification over the years, it is probably fair to say that the benchmark probably doesn’t test today’s computing architectures as much as it did when it was first released back in the early 2000s.
The TPC has created the TPC-E Benchmark in response to such changes, but for various reasons it hasn’t yet taken off as expected and many companies still choose to test with the TPC-C Benchmark specification. So, it appears that TPC-C will be with us for the time being and IDEAS predicts more low, and high-end results will follow.
What is Next For the High-end?
That brings us back to the high-end which is always interesting for its news value and intrigue in the race for the top. It would seem that we may see a change in approach, with more clustered* configurations. This is because it would appear that we have reached a point where the performance levels needed to score a number one result may be beyond single system capabilities. Despite what you may think about the relevance of the actual results, the outcomes and associated bragging rights are still very valuable to the vendors for brand marketing purposes. So, IDEAS believes it would be odd for Oracle to walk away now, given the company made the most of its recent number one positioning. IDEAS anticipates a response from Oracle at some stage. Although as the graph shows, with the time between number one results, that could be a while.
It will be interesting to see what happens in the next instalment of this battle for the performance high ground.
Gary Burgess, SVP & Senior Analyst 
Follow Gary on Twitter
More Information
For more details on the results refered to in this story including the price/performance outcomes, go to TPC-C Benchmark Top Ten on the IDEAS Benchmark Gateway The Benchmark Gateway features the Top Ten results for all the major industry standard benchmarks. In addition, the Competitive Profiles service includes full details for all the major benchmarks in addition to system features, and pricing details.
IDEAS TPC Comparison Advice
Where possible Ideas International recommends that the reader should compare TPC results using similar software environments, such as the operating system and/or relational databases, to minimize differences in performance and price/performance caused by these technologies. However, this is not always possible. Comparisons of unlike software environments are valid under TPC Rules, however the reader should be careful when drawing relativity conclusions under these circumstances.
* Similarly, readers should take care when comparing clustered and non-clustered results. Customers whose databases are not readily partitionable, or whose transaction locality (per node) is unpredictably dynamic, will not be able to accurately predict scalability for their workload by using clustered tpmC results.
It is interesting to note that IBM claim they never lost the record to Oracle. When Oracle had the TPC-C world record, IBM still claimed they had the record - because "IBM had faster cores than Oracle's Sun machines".
IBM reasoning is similar to when all 1.4GHz Niagara T2 record benchmarks where rejected and IBM claimed their POWER6 at 5GHz where faster than T2 (even though T2 won benchmarks). The reasoning from IBM goes likes this:
A) Each POWER6 core is faster than each T2 core.
B) Therefore the POWER6 cpu is faster than the T2 cpu.
While A) is true, it doesnt say anything about the whole cpu. If POWER6 has a faster ALU, or faster core, it does not mean that the whole cpu is faster. In fact, POWER6 had only 2 fast cores, whereas T2 had many slower cores. Each T2 core was slow, but there where many cores so T2 won the benchmarks.
IBM used this to twist their marketing: IBM compared a POWER6 machine with 8 cpus, vs a Sun T2 machine with 2 cpus and concluded the POWER6 is a faster cpu because it won the benchmark. The POWER6 machine had 16 cores, as did the Sun T2 machine. And eight POWER6 cpus is faster than two T2 cpus, but that does not mean that each POWER6 cpu is faster than each Sun T2 cpu.
IBM just lied. As usual. According to wikipedia article on "FUD", IBM was the first company to employ FUD. IBM has always been the big bad company, but Microsoft took that role. But IBM has never abandoned that role.
Posted by: Kebabbert | September 10, 2010 at 06:25 AM