Sandia National Laboratories’ 8960-processor Thunderbird Linux cluster, developed in collaboration with Dell, Inc. and Cisco, maintained its sixth position in the Top500 Supercomputers by achieving an improved overall performance of 53.0 teraflops, an increase of 15 percent per node over last year’s performance testing. The Top500 ranking of supercomputers is based on the Linpack benchmark, a yardstick of performance to test processor speed, scalability, and accuracy. “This achievement represents a long-term investment to meet our mission to transform engineering and provide greater processing capacity,” says John Zepper, Computing Systems Senior Manager at Sandia. Sandia researchers use Thunderbird to perform a broad range of weapons simulations, including atomistic scale-to-device modeling of radiation effects on semiconductor electronics, assessing weapon-response safety in extreme thermal and impact environments, and quantifying uncertainties in weapon performance. The level of detail being modeled in these assessments was not practical without the new level of scalable capacity that Thunderbird provides. With its 4,480 commodity compute servers linked with an Infiniband message-passing interconnect, Thunderbird is the largest cluster of its type in the world. Sandia is a National Nuclear Security Administration laboratory. The improvement in Thunderbird’s performance were propelled by its switch to OpenFabric Enterprise Distribution (OFED) and OpenMPI — together, a Linux-based open-source software stack qualified by the OpenFabrics Alliance to operate with multi-vendor Infiniband hardware and implement open-source Message Passing Interface (MPI) protocol. The achievement was a joint venture involving Sandia and Cisco. As an active developer in the OFED and OpenMPI projects, Cisco’s engineers were on site at Sandia to assist with monitoring, diagnosing, and fine-tuning Thunderbird’s performance. The new software-stack environment allows for more memory per node to be available for parallel jobs at runtime, as well as an increase in reliability and scalability of users’ jobs. Sandia’s extensive use of the new software ironed out bugs and tweaked performance — improvements that benefit the entire high-performance-computing community. Infiniband is widely regarded as one of the most attractive commodity interconnect technologies, because of its high bandwidth, low latency, and low cost. This is the first time Infiniband, OpenMPI, and OFED have been used in such a massive configuration as Thunderbird.