Maximizing Memory Capacity in Two-Socket Blades

I recently attended the Cisco Live! event in Melbourne, as a guest of Cisco, where executives spoke about the unique Extended Memory Technology used in some of Cisco’s Unified Computing System rack and blade servers. In the age of virtualization, memory capacity is an important enabler for maximizing compute resources. Currently, Intel’s Xeon architecture is the pre-eminent platform for many computing applications and vendors are looking for ways to differentiate their products from those of competitors using the same basic technology. So we thought it would be worthwhile to compare the memory capacities of some of the leading two-socket blade servers in the market today.

* IBM HX5 is scalable to four sockets by joining two blades together; this table shows the capacity of a single two socket blade

** Includes MAX5 memory expansion blade

From this table two things are immediately clear. Firstly the Cisco UCS B250 M2 has the highest number of DIMM slots for any of the blades in the table. Secondly the HP blades have the highest overall capacity due to support for larger capacity DIMMs.

At this point it is a good idea to briefly explain the memory architecture at work in these servers. All of these products are derived from Intel’s “Nehalem” architecture which places memory controllers in the processor itself. Therefore the number of memory channels available directly corresponds with the number of processors installed. The Xeon 5500 and 5600 series processors are designed for dual processor servers. The Xeon 7500 and E7 series are designed for servers with four or more processors but are also used in some higher end dual socket servers.

Xeon 5500/5600 series processors have one memory controller each, these in turn support three channels and each channel can support up to three DIMM slots. So a standard Xeon 5600 based server has up to nine DIMM slots per processor, or a maximum of 18 in total if two processors are installed.

The Xeon 7500/6500 and new E7 processors are more powerful and of course more expensive than their 5500/5600 series counterparts. Each 7500/6500 and E7 series processor has two memory controllers per processor, which in turn connect to two Scalable Memory Buffers (SMB) and then to four DIMM slots per SMB. Therefore each Xeon 7500/6500 and E7 processor can support up to 16 DIMM slots. In a two socket system this means a total of 32 DIMM slots.

A quick glance at the table above shows that with one exception all blade servers using the Xeon 7500 and E7 processors can access 32 DIMM slots which is the same as the standard Intel reference architecture. The total capacities amongst these servers varies because some vendors offer support for larger capacity DIMMs. The highest memory capacity of all servers in the table is attributed to the HP ProLiant BL620c G7 via 32GB DIMMs in all 32 slots. The one exception is the IBM BladeCenter HX5 which can accommodate extra DIMMs through the use of a MAX5 memory expansion blade; however this takes up an extra blade slot in the chassis thereby sacrificing overall density.

The high performance Xeon 7500 and E7 systems bring extra capacity at a greater cost. The processors are designed primarily for systems with four or more sockets, but vendors offer them in two socket servers partly because of the extra memory capacity they offer. However there is a price penalty – a typical dual processor Xeon 5600 blade is less than three quarters of the cost of a similarly configured E7 blade at a similar performance point.

This cost effectiveness is partly why the Xeon 5600 series platform is such a popular choice in the market today. The ability to support more memory than competing systems using the same platform is therefore a key competitive advantage. Amongst the blade servers compared in the table above, the Cisco UCS B250 M2 earns the distinction of offering the greatest number of DIMM slots. Cisco call this Extended Memory Technology and it works by adding additional electronics which allow each channel to support up to eight DIMMs, compared to the standard three. There is a slight latency cost incurred by squeezing in this extra memory, but that is par for the course; premium memory performance in any current Xeon system is obtained by only using one DIMM per channel, extra capacity always means slower speeds.

While Cisco can claim an advantage in offering access to more DIMM slots, the HP BL460c G7 offers the same capacity with only a quarter of the DIMM slots by supporting higher density 32GB DIMMs. Cisco will of course be able to claim a higher memory capacity as soon as support is added in the UCS B250 for the higher density DIMMs used by its rivals.

A large memory capacity is vital for many server applications, particularly virtualization and in-memory database workloads. However, extra capacity comes at a cost both in terms of price per gigabyte and bandwidth. 32GB DIMMs are new technology that some vendors are still testing, which is why not all manufacturers support them. As is often the case with new technology they are also very expensive. For the BL460c G7 the cost per gigabyte of a 32GB DIMM is nearly five times that of an 8GB DIMM. In the end it is a three way trade-off between capacity, performance and price. Achieving the optimum balance between these three criteria is the key for selecting the solution that best meets your needs.

Comments

[Mr Foo]

Don't forget the application. In a vurtuailzed environment, memory and through put are king. CPU is rarely the bottleneck.

[Stevem]

Andrew, nice article. Just a couple thoughts and a comment on Brad's post.

The B250 may have 32 DIMMs sockets and use lots of "cheap" memory, note that the B250 also takes two server bays, doubling the physical infrastructure required. So after off-setting the extra infrastructure costs (twice as many server bays for the same number of servers), that "cheap" memory no longer appears to be cheap. To install the same number of servers using 16 or 32GB DIMMs will take less physical space (server bays, switch ports, power connections etc...). Overall, I think the higher density DIMMs are be a better approach.

The BL460c will run two dual ranked DIMMs per bus at 1333 MHz, with the appropriate CPU. Also important to note, that specific CPUs are required for 1333MHz speeds and many customers don’t use those CPUs, therefore; memory will run at a reduced speed anyway.

With all that said, I think there is way too much focus on memory speeds. In reality, very few applications will benefit from the difference in memory speeds of 1333 vs. 1066 anyways.

Also, the cost delta between Dual ranked 8GB and Quad ranked 16GB DIMMs is very close (cost per GB), so if memory speed is not important, the 16GB Quad ranked make a lot of sense. HP offers both Dual and Quad ranked 16GB DIMMs. In addition, Quad ranked 32GB DIMMs are also available.

Enjoy...

[Brad Hedlund]

Some additional data points also worth discussing here is the memory speed at these maximum capacities and the cost per GB (using street prices for memory).
Sure, it's great that HP supports 32GB DIMMs, but at what cost per GB and how much does it slow down the memory channels? (Hint: 800Mhz).
Note that Cisco UCS B250 operates 384GB at the full 1333Mhz speed using extremely cost effective 8GB DIMMs. Obtaining that balance performance and price is not possible yet with 16GB and 32GB DIMMs.

Cheers,
Brad