The Pittsburgh Supercomputing Center (PSC) acquired an SGI UV1000 in July 2010 with help from an award from the National Science Foundation (NSF). Named Blacklight, this system has 32 terabytes of memory partitioned into two connected 16-terabyte shared-memory nodes, in effect, the two largest shared-memory systems in the world. With certain programming approaches, furthermore, such as message-passing and partitioned global address space (PGAS), researchers can access all 32 terabytes as a single memory space.

As a resource of XSEDE, the NSF cyberinfrastructure program, Blacklight serves researchers nationwide across many fields of science and engineering. During 2011, to optimize Blacklight’s productivity for these researchers, PSC staff, under the lead of J. Ray Scott, PSC director of systems & operations, worked closely with SGI to enhance a number of Blacklight features.

These enhancements include:

• reducing substantially the boot time of the system,

• enabling high input-output file performance for very large datasets without the need to change application software,

• improving tools to diagnose research application problems, optimize job layout among processors, and track and isolate hardware problems,

• modifying the portable Torque/Maui batch system scheduler, so that resources assigned to one shared-memory job don’t impinge on other jobs, and

• enabling researchers to use Zest, a high-speed memory snapshot utility, and SLASH2, a wide-area data-replicating file system, to ease the burdens of data management for very large, distributed datasets.

“We’ve been at this for 25 years,” said PSC scientific directors Ralph Roskies and Michael Levine, “and we’ve become expert at working out the sometimes perplexing problems that go along with bringing a new system through the process of shakedown and final testing to provide researchers with a maximally productive tool.”

Many of these enhancements, say PSC and SGI officials, will benefit not only XSEDE and the researchers who use Blacklight, but will also benefit the research community at large by enabling improved performance for other large-configuration SGI UV1000 systems.

“Collaborating with PSC on our newest and largest shared-memory system has been of great mutual benefit,” said SGI Chief Technology Officer Eng Lim Goh. “These enhancements have improved the performance and effectiveness of our UV architecture. The PSC staff have years of hands-on experience with our largest SGI systems, and they have become a valuable partner and technical resource as we develop our next designs for our customers.”

“At 32 terabytes, Blacklight has the largest possible memory configuration deployed to date for an SGI UV system,” added Scott. “Because of that, we worked closely with SGI, and the result is that they’ve been able to enhance and improve its operation in several areas beyond what had previously been achieved.”