The National Center for Supercomputing Applications (NCSA) has selected Linux as the operating system for the multi-petaflop IBM POWER7-based Blue Waters supercomputer that will come online in 2011.

Staff at NCSA and IBM evaluated both the open-source Linux and IBM's AIX as potential operating systems for Blue Waters, extensively testing both on identical hardware at large scale. SARA Computing and Networking Services in the Netherlands and the U.S. National Center for Atmospheric Research (NCAR) shared their experiences and provided access to their large IBM POWER6 systems, both with more than 3,300 cores and both using Infiniband interconnects. Both ran IBM's full HPC Software stack and used the GPFS file system. SARA's system uses Linux, while NCAR uses AIX.

More than 160 features and functions were reviewed, including features for reliability, use of unique hardware features, performance monitoring, accounting, file systems, resource management, memory management, security, programming environments and OS resource use. Because Blue Waters will not enter full service until 2011, and will continue to operate for at least five years, the evaluation took into account the road map of future features for each operating system.

"We looked at the features that will be part of each OS in 2011, not just the features available today," said Blue Waters deputy project director William Kramer.

NCSA also conducted multiple application performance tests, including MILC, PARATEC, four of the NAS Parallel Benchmarks (Fast Fourier Transform, Conjugate Gradient, Lower-Upper symmetric Gauss-Seidel, and MultiGrid) for Class B and Class D problem sizes, HPC Challenge Benchmarks, and parallel 3D Fast Fourier Transform tests.

The team also used an AIX POWER6/Infiniband system at the European Center for Medium-Range Weather Forecasting (ECMWF) to study jitter, the amount of application interference caused by OS functions, which can limit an application's ability to run efficiently at large scale.

"We determined that either OS could ably meet the requirements for Blue Waters and serve the science and engineering research community," Kramer said. "Linux offered some additional advantages, such as being somewhat more familiar to the research community and offering the potential for NCSA to more directly participate in enhancing the OS."

The collaborators plan to publish the OS performance results for both Linux and AIX.

Blue Waters is based on the most advanced technologies under development at IBM, which are embodied in the PERCS (Productive, Easy-to-use, Reliable Computing System) system design. The development of PERCS is supported by the Defense Advanced Research Projects Agency (DARPA) High Productivity Computing Systems (HPCS) program.

Under HPCS, IBM has produced capabilities for both AIX and Linux HPC deployments. DARPA and IBM have also selected Linux for the IBM PERCS prototype.