Voltaire Grid Director Switches Incorporated into New Architecture Employing Latest Processors and GPU-based Computing to Speed Performance 30 Times

The Tokyo Institute of Technology has selected Voltaire’s 40 Gb/s InfiniBand switches for Japan’s first 2.4-petaflop supercomputer, the TSUBAME 2.0.  The new supercomputer will bring powerful research tools to users in both the industrial and academic world.

“TSUBAME 2.0 will continue to push science forward by providing world-class supercomputer facilities that enable research and development to be completed and utilized more quickly than ever before,” said Professor Satoshi Matsuoka of Global Scientific Information and Computing Center (GSIC), Tokyo Institute of Technology. “We designed the system with the latest Intel Westmere-EP and Nehalem-EX CPUs coupled with more than 4,200 of NVIDIA Tesla 20-series GPUs to provide extreme processing capabilities on each node.  To capitalize on this new level of compute power, we implemented a dual-rail, non-blocking fabric that can support throughput up to 80 Gb/s per node, employing two Voltaire 40 Gb/s InfiniBand connections on each node.”

“Since GPUs provide extremely high floating-point capabilities as well as memory bandwidth, Voltaire’s high-speed InfiniBand fabric is needed to achieve the highest throughput and lowest latency possible, in order to sustain performance in large-scale parallel workloads involving thousands of GPUs,” Professor Matsuoka added.

Voltaire is working with NEC on the system that will begin operation in the fall. The TSUBAME 2.0 supercomputer, while being 30 times faster than its predecessor, the TSUBAME 1.0, will require only 200 square meters of floor space, a reduction by two-thirds, and will exhibit comparable power consumption thanks to numerous green technologies that will be incorporated. The performance of the new supercomputer will also be 12 times greater than the current fastest system in Japan, owned by the Japan Atomic Energy Agency – also a Voltaire customer.

“The Tokyo Institute of Technology’s requirements in its design of TSUBAME 2.0 call for extreme levels of high-performance, high scalability, and low-latency in the fabric,” said Asaf Somekh, vice president of marketing, Voltaire. “Voltaire’s 40 Gb/s InfiniBand solutions deliver top performance, and as a result are used within the world’s leading supercomputers. We are pleased to be working with our long-time OEM partner NEC on this project.”

The new TSUBAME 2.0 supercomputer system, which will have more than 1,400 compute nodes, will incorporate Voltaire’s QDR InfiniBand fabric in a fully non-blocking configuration, with 12 Grid Director 4700 40 Gb/s InfiniBand switches, 179 Grid Director 4036 edge switches and 6 Grid Director 4036E switches for high performance bridging to 10 GbE storage.

The TSUBAME 2.0 supercomputer is equipped with cutting-edge technologies including the latest Intel Westmere-EP and Nehalem-EX CPUs combined with over 4,200 NVIDIA Tesla M2050 GPUs, based on the Fermi architecture. This “hybrid scalar-vector architecture” will achieve world-leading performance in both compute and bandwidth, while maintaining the overall system size, power, and cost to be an order of magnitude smaller than its competitors.

The system will also facilitate nearly 3,000 solid-state drives (SSDs) for the world’s fastest total data I/O throughput at 2/3 terabytes per second, augmented with DataDirect Networks storage system with over 7 petabytes of capacity and planned 8 petabytes of tape. The operating system will include both Linux and Microsoft Windows HPC, and will also use cloud and virtual machine technologies in order provide flexible services to its over 2,000 users as well as facilitate advanced machine operation capabilities.