IDEAS Insights

IBM’s Overlapping Design Teams Ensure Flow of New Servers

IBM's recent earnings release - lower revenues but higher earnings - brings to mind an old saying "You can't save your way to prosperity."  Perhaps some cutbacks may turn out to be short-sighted, but those that challenge/change business fundamentals may yield long lasting benefit.

Locales suffering IBM's "stealth layoffs," under the guise of "resource rebalancing," may bemoan the shift of employment offshore.  But, the merging of "z," "p," and "i" design engineers allows more frequent product refreshes across all server platforms.  

Designing a complex system does take many years.  Mainframes used to see significant updates every 5+ years, but now that mainframes leverage POWER building blocks, IBM has indicated that its next mainframe update is targeted for 2010, just a couple years after its 2008 introduction of z10.

Back in the pre-90s mainframe heyday, IBM had multiple overlapped design teams – while one was wrapping up a design, another was in the midst of design and yet another was exploring alternatives for the next, future design.  As one design finished, engineers were shifted onto a future design.  The overlapped teams meant that a new system could be released twice as often than if there were only a single team.  Over the years, cost cutting reduced mainframe engineers from 2.5 teams to something like 1.5 teams, which lengthened the introduction of new architecture.

That overlapping team approach is still used for complex chip designs like IBM's POWER.  The POWER5/5+ team had a different crew of engineers than the POWER6/6+ team.  The POWER7 team picked up some of the POWER5 guys while the POWER6/6+ guys were still hard at work.  POWER8 is now the skeleton exploration team.  Sure, there is a cadre of gurus who oversee all, but the grunt work involves overlapping teams of engineers.  Like clockwork, the overlapped POWER teams have introduced new designs every 3 years (POWER4=2001, POWER5=2004, POWER6=2007, POWER7 promised for 2010) with speed-bump midlife-kickers in between.

The mainframe continues to support an instruction set architecture (ISA) that has evolved over the past 40+ years while steadily maintaining backwards compatibility.  It is clearly not the same ISA as POWER, with its RISC roots.  But, ISA aside, there are a lot of common functions – cache hierarchies, memory interfaces, even execution units (IEEE floating point, Decimal floating point, etc.) – that can be leveraged between POWER RISC and mainframe CISC.  That's just what happened with z10 – a bit of unique mainframe ISA design coupled with some fundamental cache/memory building blocks. 

In 2010, the same design leverage is expected.  POWER7 is promised for staggered 2010 rollout across Power Systems UNIX servers.  And, IBM's recent earnings call indicated that the next mainframe launch will also be in 2010.  Downsizing the number of design teams is truly heartbreaking for now unemployed system designers.  But, combining design teams across products and leveraging fundamental building blocks has allowed IBM to reduce its costs and still deliver timely product refreshes.

Do Economy-Size Servers Make Business Sense?

IT managers constantly seek the optimal size of servers that will most efficiently handle their computing workloads. Clearly, if a particular application needs a large memory and can use a lot of processing cores, then a large server seems like the right choice. However, if individual applications do not need such capacity, then a collection of smaller, less costly servers may seem attractive. On the other side of the coin, the cost to power and manage server sprawl has increased dramatically in recent years, making it unacceptable in many cases. The question ultimately comes down to how much to consolidate into each server. IT managers must carefully balance the higher hardware costs and the reduced number of software images to purchase, manage and patch in a large server, against the lower hardware cost, higher software cost, and higher management costs of a number of smaller servers. Does an “economy size” larger server provide the execution efficiency economies of scale to justify the typically higher cost of the higher-end design?

Here at Ideas International, we deal with server pricing, energy efficiency, and performance, so we thought we would provide some basic calculations to illustrate some of the costs involved in determining the optimal size of servers. Note that we are not attempting a comprehensive TCO analysis, which is beyond the scope of a blog. For this exercise, we selected the HP ProLiant DL785 G6 and four DL385 G6 servers. We could have selected IBM servers because they have an industry-standard server that can scale to eight sockets, but we felt that HP was a better example because the DL785 G6 can scale within the same physical box. Next, we loaded each configuration to its maximum number of six-core “Istanbul” Opteron processors, populated a total of 32GB of memory, added an equal number of 300GB SAS drives, selected the redundant power options, and loaded Windows 2008 Server (Enterprise Edition). Essentially, the stack of four DL385 G6 rack servers had an identical configuration to the single DL785 G6. And these servers where not stripped configurations; they were configured to host real application loads.

When we looked at the overall hardware cost of our configurations, the four DL385 G6 servers cost a total of $39,796 before any discounts and allowances, and the DL785 G6 cost $55,819. That 29% price differential was to be expected because designing and building eight sockets into a single server requires  more technology and engineering than two socket servers. What did surprise us was how small the difference in cost actually was when selecting real-life configurations rather than the low cost entry configurations showcased on HPs’ website. One point worth noting is that the four DL385 G6 servers require four Windows Enterprise licenses at a cost of $2,710 each versus just one license for the DL785 G6. Those license costs have already been factored into the total costs above.

Finally, as a rough gauge of energy costs, we looked at the cost to power the power supplies rather than the more detailed summation of the power consumption of all of the individual server components. In this exercise we ignored the redundant power supplies and concentrated only on the base power options, assuming they were running at 100%. It is important to note that both ProLiants support the optional HP Insight Power Manager to help monitor and regulate power usage. The DL385 G6 has a choice of three power supplies, 460W, 750W and 1200W, and the DL785 G6 only supports three 1200W power supplies. In our case, we selected the 750W power supply for our DL385 G6 configuration due to the two six-core processors, the amount of memory and the two SAS drives. If we had installed more memory and other options, the 1200W would have been a more appropriate choice. Calculations were based on three years of ownership and a commercial power cost of 15.32 cents per kWh (New Hampshire, USA). Power costs were kept constant for all three years, although that is probably an unrealistic assumption.

HP ProLiant DL385 G6 Power Costs

Power Supplies (qty of 4)	36 month power cost (USD)
460W	$6,800
750W *	11,100
1200W	17,800

*Used in our configuration of four DL385 G6 servers

HP ProLiant DL785 G6 Power Costs

Power Supplies (qty of 3)	36 month power cost (USD)
1200W **	$13,300

**Used in our configuration of the DL785 G6

The results above show that there really is not much difference in energy cost between the two stacks. The DL785 G6 power costs at $13,300 are roughly the same as the $11,100 for the four DL385 G6 servers.

We were surprised by the relatively small difference in hardware and energy cost between the DL385 and DL785 alternatives. Once the IT Manager has considered all associated costs, including the management of multiple servers, there may not be much cost difference at all between the two options. What that means is if a large server like the DL785 G6 will handle the application stack more efficiently, then the 29% difference in hardware and software cost may not be significant in our opinion. Other driving factors for selecting the DL785 G6 include its ability to run Windows Datacenter and the fact that there is a single entity to manage versus four servers with the DL385 G6. There is also the issue of virtualization efficiency on a large server versus numerous smaller servers. With a common pool of shared compute “headroom,” the larger, eight-socket server may be able to run at higher overall utilization than the smaller, two socket servers.

As IT managers confront the question of choosing the optimal hardware footprint for x86 workloads, they still have a tough decision to make. This pricing exercise indicates that the hardware and energy costs may not be the deciding factor for selecting a larger x86 server instead of multiple smaller servers.  Rather, it will more likely be the “soft” costs related to management that will determine whether eight-socket x86 servers make economic sense. While these costs are notoriously challenging to quantify with a generalized approach, the relatively modest pricing premium of the DL785 G6 sets a low hurdle for achieving tangible returns on its investment.

Low Latency and Clouds Pervade at High Performance Computing On Wall Street Conference

At the 6th Annual High Performance Computing (HPC) on Wall Street conference, the key words of the day were low latency and cloud computing. The two topics were often combined in displays and talks that interwove both concepts. Despite the collapse of some of Wall Street’s biggest investment banks in 2008, the drive for faster low latency computing not only continues, but is coveted, as microsecond trading continues to reap profits for the ones who survived. 

Most of the session speakers focused on the question of what cloud computing means for the financial sector, while vendors displayed their top-of-the line equipment in the exhibit area. IBM’s booth focused on its Computing on Demand offering. Unlike some cloud competitors, who offer cycles distributed across a myriad of servers, IBM believes that HPC users need to have dedicated servers and the knowledge of exactly what they are getting (IBM’s offering includes some POWER servers, and a good deal of x86 servers). IBM also emphasized its security credentials. Beyond the typical standard offerings related to security (i.e. firewalls), IBM offers separate memory and storage for consumers’ virtual machines.

High performance computing remains a key priority for Wall Street, but so does low-latency networking. As reported in the New York Times, the NASDAQ exchange allows users, for a fee, to obtain stock data 30 milliseconds before it is made public. Although early trading may yield only miniscule profit per share, when performed many times in aggregate, a hefty profit can be made with such "high-frequency" or "flash" trading (note that the practice is now being scrutinized by the Securities and Exchange Comission). In order to beat the competition in this type and other forms of quantitative trading, low network latency is crucial. Fujitsu, Sun, Arista and other vendors all displayed high-end 10 Gigabit Ethernet data center switches to help improve networking performance. Arista showcased hardware that can reduce latency to as little as 600 nanoseconds. For the users who were concerned about how software and hardware components would function together, the Securities Technology Analysis Center (STAC) has been devising performance benchmarks that focus exclusively on evaluating systems for Wall Street requirements. STAC highlights that, unlike other “standard” benchmarks, the end users are the ones who set the specifications. This comes at a price though. It cost $15,000 to have a say in the specifications, while access to the published data costs $5,000.

Even after the collapse of the financial sector, and its continually slow recovery, algorithmic trading is here to stay. This is good news for the vendors as they improve their speed, capabilities, and find more end users in the emerging market of cloud computing. It's also good for the organizers of this event, for they will surely have more conferences in the future, as users continue to seek out new ways to turn computational cycles into wealth.

Intel’s Nehalem Processor Under The SAP Workload

This is the third and final in a series of posts looking at Intel’s Xeon 5500 Series of server processors and how this new technology compares from a performance perspective with older Intel processor generations and with that of IBM’s POWER processor architecture. The previous post looked at Intel vs. IBM using the TPC-C Benchmark outcomes, whilst  the first story examined the current versus previous generation Intel processor performance under the TPC-E Benchmark workload.

This article will look to the SAP 2-Tier Benchmark where IBM has results for its current generation POWER6+ processor, a perspective not available with the TPC-C Benchmark.

The chart below focuses on the competitive hot spot (eight processor cores) for the two processor types, POWER versus Xeon X5570, for the SAP 2-tier Benchmark.  The Nehalem results are represented by the green bars and the IBM POWER results in blue. Light blue represents POWER6 configurations and the darker blue bar represents the POWER6+ test.

 
It is worth noting that the chart groups results into two different revisions of the benchmark: ERP 6.0 2005 and ERP 6.0U 2009. The change in specification had a significant effect on reported performance with outcomes generally lower under the new 2009 specification. This effect is highlighted by the orange bars, which represent HP ProLiant DL380 G6 results under the two specifications. This specification change also probably helps to explain why IBM’s POWER6+ result, which was tested under the 2009 specification, is not substantially different than the POWER6 results tested under the older 2005 rules.

To date there have been seven X5570 results under the 2009 specification. As shown in the Chart  above, the outcomes range from 80% of per-core POWER6+ performance to near parity per-core performance with POWER6+ (Sun Fire X4270).

Two Xeon X5470 results (in gray) are also included to provide a comparison of how the Intel performance expectations have changed between the two generations under this workload.

The Xeon 5500 Series is designed for up to two-socket servers and so has only been tested to a scaling level of eight cores at this time. It is worth noting that IBM has tested its POWER processor servers at 16, 32, and 64 processor-core configurations, which are not shown in this chart and where the Intel technology currently can’t compete. However, new tests will likely emerge once the ‘Nehalem EX’ processors for servers with four or more sockets are introduced in the future.

The IDEAS Bottom Line: What Does All of This Mean?

From the above and the previous two posts the following can be gleaned:

• With the new performance potential of the Intel Xeon Processor 5500 Series, it appears that Intel has elevated itself into a new competitive position and will take the challenge up to IBM.
• IBM currently claims the better scalability, with demonstrated high performance per core all the way up to 64 processor cores.
• It is also fair to say that IBM has probably been relatively conservative of late with its POWER6 architecture and that it will look to make the next significant leap forward soon, thus raising the bar in this extended game of leapfrog. See ‘IBM Reveals A Few POWER7 Details’.
• However, it definitely appears that the new Intel Xeon Processor 5500 Series processors represent a significant generational shift for x86 performance and may start to reset expectations for x86 server technology performance, especially in the in two-socket and four-socket processor server space.

The stage is certainly set for an interesting second half of 2009.

For the full story and some additional analysis, see the August issue of the Tech Trends Monthly Newsletter. This is a free (with registration) on-line newsletter published by Ideas International.

Comprehensive consolidated industry standard performance data is available from Ideas International via two sources. The free Benchmark Gateway service, which features the Top Ten results for major industry standard benchmarks and the subscription service Competitive Profiles that provides comprehensive data coverage of the features, pricing, and performance characteristics of a wide range of enterprise IT products and services.

Gary Burgess
Senior Vice President, Research & Operations
 Follow Gary on Twitter

IDEAS must acknowledge the limited scope of this series of articles; it concentrates on application performance per core, focusing specifically on Intel Xeon versus IBM POWER.  It’s important to note that other companies compete with other processor types and not all compete on raw performance per core. At the end of the day, overall system performance is what matters. If one company uses more processors than another, but provides better or cheaper overall performance, then it offers a valid solution worth considering. Moreover this story concentrates solely on performance; it does not explore in depth any of the price or price/performance angles that may come into play.

 

IBM Reveals a Few POWER7 Details

IEEE’s annual HOT CHIPS conference, gives microprocessor designers an opportunity to be recognized by their peers for innovative design advances. This year’s conference included details on IBM’s POWER7, among other vendors’ server processors. The conference is targeted for those who enjoy exploring implementation details and debating design tradeoffs. Specifics on clock rates and system performance usually wait for actual server announcements. 

Servers employing POWER7 are not expected to become available until sometime in 2010. Furthermore, IBM typically stages the release of various models  – the relatively straight-forward architected, and higher volume, midrange and entry servers, especially those facing competitive offerings, debut the new chips first, while complex high-end systems continue exhaustive testing for a later introduction. Thus, despite the HOT CHIPS buzz, POWER7 is not imminent – various models will be not be unveiled until 2010, presumably split across the Spring and Fall announcement seasons as has been recent IBM practice.  

As with other vendors, IBM faces the dilemma that championing the innovations of the next generation chip may impact sales of servers employing the current chip. So, to reassure customers planning substantial investment in current POWER6/6+ midrange and high-end servers, namely the Power 570 and Power 595, IBM offers an upgrade path that preserves system serial numbers. That is, customers can continue to procure current POWER6/6+ systems and plan to upgrade them to future POWER7 models, without losing financial accounting benefits of maintaining the same serial number.  

Well over a decade ago, IBM significantly intensified its microprocessor design efforts, resulting in a series of POWER chips that were industry leading. The 45 nm POWER 7 should continue that streak. With up to eight cores per chip, compared to the dual-core POWER6, POWER7 could achieve up to four times the performance per chip, depending on the workload. To remain within the same power/thermal envelope, POWER7 clock rates are not expected to be quite as high as POWER6. Four-way Simultaneous Multithreading per core (up from two-way on POWER6), additional execution units, and the return of Out of Order execution (left out of POWER6’s high-clock rate focus) should compensate for POWER7’s slightly slower maximum clock rates. Indeed, IBM indicates that a POWER7 core will have higher performance than a POWER6 core, even at a lower clock frequency.  

With eight cores per chip, the ability to feed data to the cores can become a challenge. Dual DDR3 memory controllers per chip promise 100GB/sec of sustained memory bandwidth per chip. A new “by-eight” ECC allows taking advantage of increasing memory chip density without compromising the Chipkill error correction coverage. 

IBM highlights that POWER7 is the first processor chip with an internal embedded Dynamic RAM, eDRAM, cache. On-chip caches traditionally have employed Static RAM, SRAM, which is faster but consumes more power and is less dense than DRAM. POWER7 Level 1 and 2 caches remain SRAM; what’s new is a 32 MB eDRAM on-chip Level 3 cache. Prior POWER chips used a separate chip for L3 cache. Bringing L3 onto the same chip as the processor reduces latency six-fold and doubles bandwidth, according to IBM. Furthermore, moving L3 on-chip removes the power-consuming off-chip driver circuits and also obviates the need for the specialized multi-chip module previously used for many POWER models. (The ultra-dense “Blue Waters” supercomputer, being built for the National Center for Supercomputing Applications, will employ multi-chip POWER7 modules with water-cooled cold plates, in its quest to achieve one petaflop sustained performance.)  

Other designs have had large on-chip caches – the Montvale Itanium implementation contains up to 24 MB of cache memory and Intel’s upcoming Nehalem-EX Xeon plans to have 24 MB on-chip L3. Per IBM, use of eDRAM requires only one third the chip space of conventional SRAM implementations and uses only one-fifth the standby power. (IBM estimated that the 1.2 B transistor POWER7 would have swollen to 2.7 B transistors if they had implemented a 32 MB SRAM L3 instead of eDRAM.) Such space and power savings permitted IBM to implement an eight-core design with large internal L3 and still stay within power envelope of prior servers. 

IBM says it has enhanced various power saving modes in POWER7, refining the power-saving states of POWER6 to optimize the tradeoffs between energy saving and fast return to full performance. Clearly this is difficult to quantify the benefit. The whole industry is still learning and evolving, but IBM appears to have carefully explored the tradeoffs and this design iteration should balance performance and power-efficiency.  

Until servers are released, during 2010, it will be hard to gauge POWER7’s competitive advantages. Of course, competitors will not stand still and can be expected to release their new chip implementations. Nonetheless, by all indications, POWER7 holds the promise to retain the role of chip to beat.