By Paul Tooby, San Diego Supercomputer Center -- Earthquakes are a fact of life in California. The southern part of the major San Andreas fault, however, has not seen a major earthquake since about 1690, and the accumulated movement may now amount to as much as six meters-setting the stage for an earthquake as large as magnitude 7.7-the "big one." To understand the basic science of earthquakes and to help engineers better prepare for such an event, scientists want to identify which regions are likely to experience the most intense shaking, particularly in the populated sediment-filled basin of Los Angeles and similar areas in Southern California and northern Mexico. This understanding can be used to improve building codes in high-risk areas and to help engineers design safer structures, potentially saving lives and property. But the challenges in modeling earthquakes are daunting. Accurate simulations must span an enormous range of scales, from meters near the earthquake source to hundreds of kilometers across the entire region, and time scales from hundredths of a second-to capture the higher frequencies which have greatest impact on buildings-to hundreds of seconds for the full event. Adding to the challenge, ground motion from earthquake waves is strongly influenced by the complex 3-D subsurface structure of the soil, which is not fully known and scientists can observe only indirectly. Now, based on previous simulations at the San Diego Supercomputer Center (SDSC), earthquake scientists from the Southern California Earthquake Center/Community Modeling Environment (SCEC/CME) have run enhanced simulations at SDSC using the improved TeraShake 2 earthquake model. The new simulations, which used the Anelastic Wave Model (AWM), a fourth-order finite difference code developed by Kim Olsen, associate professor of geological sciences at San Diego State University (SDSU), are the most realistic yet of where the most intense ground motion may occur in Southern California during a magnitude 7.7 San Andreas Fault earthquake. An important scientific goal of SCEC scientists including the TeraShake research group is to improve model realism by incorporating more fundamental physics into earthquake simulations. "To make the TeraShake 2 simulations more realistic, we added a new physics-based dynamic rupture component to the simulation, run at very high 100 m resolution, to create the earthquake source description for the San Andreas Fault," said Steven Day, professor of geological sciences at SDSU. The dynamic rupture component models friction-based slip on the fault surface. This is more physically realistic than the kinematic source descriptions used previously, which were adapted from recorded earthquake data. Using the results of the improved rupture simulation developed by Olsen and Day as the earthquake source, the researchers modeled the 3-D velocity throughout the volume and surface of the simulated region. To fully capture this large-scale natural event, the researchers needed to encompass the entire region in their model - a slab 600 kilometers long by 300 kilometers wide and 80 km deep (a volume of more than 14 million cubic kilometers) that includes all major population centers in Southern California and runs from the Ventura Basin, Tehachapi, and the southern San Joaquin Valley in the north, down to Los Angeles, San Diego, out to Catalina Island, and as far as the Mexican cities of Mexicali, Tijuana, and Ensenada in the south. The highly detailed TeraShake 2 simulation, run at 200 m resolution over the immense volume with some 1.8 billion grid points, ran for four days on 240 processors of the newly-expanded 15.6 Teraflops DataStar supercomputer, requiring a complex choreography of data movement between DataStar, disk, and archival storage. The 10 terabytes of output data is archived in the SCEC Digital Library, managed by the SDSC Storage Resource Broker (SRB) at SDSC, where it is easily available to researchers for further analysis. These improved simulations are giving scientists new insights into where strong ground motions may occur in the event of such an earthquake, which can be especially intense and long-lasting in sediment-filled basins such as the Los Angeles area. Two factors have been especially important to this advance in computational science. One is SDSC's focus on large-scale, end-to-end data cyberinfrastructure, with data-oriented resources such as DataStar supported by a General Parallel File System (GPFS) that makes more than a petabyte of online disk available to users, and reliable data archiving with more than six petabytes of HPSS and SAM-QFS tape. One petabyte is one million gigabytes, about 10,000 times the capacity of today's typical PC hard drives. The second factor in enabling this research is the close, long-term collaboration between SCEC researchers and experts from groups across SDSC in a Strategic Applications Collaboration. "To solve the novel challenges that emerge when scientists run their codes at the largest scales and data sets grow to immense size, we worked closely with the scientists through months of code porting, new feature integration, and optimization," said SDSC computational scientist Yifeng Cui. The collaboration incorporated the new, dynamic rupture feature and modifications to speed up the AWM code, and the optimized dynamic TeraShake 2 code can now scale up to 2,048 processors. During and after the run, many other SDSC staff followed up with data movement, application and data support, SAM-QFS and SANergy targeting, SRB support, I/O and GPFS expertise, and overall run management. SDSC also provided important visualization services to the collaboration, helping SCEC scientists monitor the simulations and find new insights in integrated views of the immense 10 terabyte data set. To produce the visualizations, the researchers used over 30,000 hours on SDSC resources, including 20,000 hours on DataStar alone to create more than 100,000 images. The visualizations also help make the results more understandable to nonspecialists, and dramatic movies of the simulations can be viewed online (see links below). Another product of the SDSC SAC collaboration with the SCEC scientists is the enhanced TeraShake code that is capable of large-scale runs, maintained at SDSC, which is now available to the U.S. earthquake community for future earthquake simulations. TeraShake 2 demonstrates SDSC's capabilities as a leading site for end-to-end data cyberinfrastructure, showing how much the capabilities have grown to support large-scale simulations with correspondingly large data challenges. The SCEC TeraShake 2 project is led by Thomas H. Jordan of the University of Southern California, Jean-Bernard Minster of the Institute of Geophysics and Planetary Physics (IGPP) at SIO/UCSD, Kim Olsen and Steven Day of SDSU, and Reagan Moore of SDSC/UCSD. Simulation movies: Quicktime 15 mb - http://visservices.sdsc.edu/projects/scec/terashake/movies/Terashake2.1-volume-Vy-800-map.mov Windows Media 2 mb - http://visservices.sdsc.edu/projects/scec/terashake/movies/Terashake2.1-volume-Vy-800-map.wmv The Southern California Earthquake Center (SCEC) SCEC Community Modeling Environment