Lessons In Building A Truly Massive Web Site

This article in BaseLine Magazine provides a detailed chronicle of how designers of the wildly popular social networking site MySpace overcame a series of technical challenges as they struggled to accommodate the staggering growth of the site’s user base for the last three years. Unlike many high-traffic web sites that primarily deliver content, and can thus be optimized with various read-only caching techniques, interaction with MySpace typically involves considerable input from users, who may submit messages, comments, graphics, and multimedia content to the system. As a result, its underlying web application imposes load conditions that involve both reading and writing, which can stress far more parts of the system as the number of users grows.

The article details the scalability barriers that MySpace hit at successive milestones in its user base (i.e. 500,000 users, 1 million users, 3 million users, 9 million users, and 26 million users), and shows how problems had to be solved in different levels of the application stack at each milestone (including switching from the ColdFusion application server to Microsoft’s .Net platform and C# language; partitioning databases, and moving from a scale-up to a scale-out design; rehosting storage, and finally, deploying a 64-bit operating system and database).

While much of the discussion in the article will be most interesting to developers who are designing scalable web sites, a few points come out that are noteworthy for their strategic relevance. First, at the 10-million user mark, storage became the primary bottleneck. The way to get overcome it turned out to be the implementation of a storage virtualization solution from 3PARdata. Much of the industry’s current interest in virtualization is centered around virtualizing servers, for purposes of consolidation, or improving flexibility and reliability. While many of the same concepts can be applied to storage virtualization, the value proposition for storage virtualization solutions has been more elusive compared to the business case for virtual machines, where consolidation can result in huge hardware cost savings. The MySpace example shows vividly how storage virtualization solves a very real problem that users will encounter as they expose their applications to web-scale user populations. Futher, MySpace learned some painful lessons in implementing effective Disaster Recovery, including the need to avoid single points of failure in all parts of the system, i.e. not just servers, but also SAN infrastructure.

But most significantly, MySpace’s story is a powerful testimonial to the scalability of Microsoft’s server platform, including its .Net application development framework, Windows Server 2003 operating system, and SQL Server database. MySpace clearly pushed Microsoft’s server products to their absolute limits, hitting boundary conditions that triggered the system’s features in unforeseen ways (for example, it produced server loads so high that the system interpreted them as denial-of-service attacks, and took preventative action by automatically shutting down). Traditionally, Microsoft’s server platform has been credited for its ease-of-use and low barriers to entry, while alternatives such as Java have enjoyed more credibility for providing the highest levels of scalability and reliability. However, MySpace’s experience proves that Windows can now support some of the highest interactive workloads in the industry at levels of responsiveness that are acceptable for typical consumers.