A team of researchers at the San Diego Supercomputer Center (SDSC) and the Department of Electronic and Computer Engineering at the University of California, San Diego, has developed a highly scalable computer code that will significantly cut both research times and energy costs in simulating seismic hazards.
The team, led by Yifeng Cui, a computational scientist at SDSC, developed the scalable GPU (graphical processing units) accelerated code for use in earthquake engineering and disaster management through regional earthquake simulations at the petascale level as part of a larger computational effort coordinated by the Southern California Earthquake Center (SCEC).
“The increased capability of GPUs, combined with the high-level GPU programming language CUDA, has provided tremendous horsepower required for acceleration of numerically intensive 3D simulation of earthquake ground motions,” said Cui.
The accelerated code, targeted at GPUs as opposed to CPUs, or central processing units, is based on a widely-used wave propagation code called AWP-ODC, which stands for Anelastic Wave Propagation by Olsen, Day and Cui.
It was named after Kim Olsen and Steven Day, geological science professors at San Diego State University (SDSU), and SDSC’s Cui. The research team restructured the code to exploit high performance and throughput, memory locality, and overlapping of computation and communication, which made it possible to scale the code linearly to more than 8,000 Nvidia Kepler GPU accelerators.
The team performed GPU-based benchmark simulations of the 5.4 magnitude earthquake that occurred in July 2008 below Chino Hills, near Los Angeles. Compute systems included Keeneland, managed by Georgia Tech, Oak Ridge National Laboratory (ORNL) and the National Institute for Computational Sciences (NICS), and also part of the National Science Foundation’s (NSF) eXtreme Science and Engineering Discovery Environment (XSEDE), and Blue Waters, based at the National Center for Supercomputing Applications (NCSA).
Also used was the Titan supercomputer, based at ORNL and funded by the U.S. Department of Energy. Titan is equipped with Cray XK7 systems and NIVIDIA’s Tesla K20X GPU accelerators.
The benchmarks, run on Titan, showed a five-fold speedup over the heavily optimized CPU code on the same system, and a sustained performance of one petaflop per second (one quadrillion calculations per second) on the tested system. A previous benchmark of the AWP-ODC code reached only 200 teraflops (trillions of calculations per second) of sustained performance.
By delivering a significantly higher level of computational power, researchers can provide more accurate earthquake predictions with increased physical reality and resolution, with the potential of saving lives and minimizing property damage.
“This is an impressive achievement that has made petascale-level computing a reality for us, opening up some new and really interesting possibilities for earthquake research,” said Thomas Jordan, director of SCEC, which has been collaborating with UC San Diego and SDSU researchers on this and other seismic research projects, such as the simulation of a magnitude 8.0 earthquake, the largest ever simulation to-date.
“Substantially faster and more energy-efficient earthquake codes are urgently needed for improved seismic hazard evaluation,” said Cui, citing the recent destructive earthquakes in China, Haiti, Chile, New Zealand, and Japan.
While the GPU-based AWP-ODC code is already in research use, further enhancements are being planned for use on hybrid heterogeneous architectures such as Titan and Blue Waters.
“One goal going forward is to use this code to calculate an improved probabilistic seismic hazard forecast for the California region under a collaborative effort coordinated by SCEC,” explained Cui.
“Our ultimate goal is to support development of a CyberShake model that can assimilate information during earthquake cascades so we can improve our operational forecasting and early warning systems.”
CyberShake is a SCEC project focused on developing new approaches to performing seismic hazard analyses using 3D waveform modeling. As Cui notes, the GPU-based code has potential to save hundreds of millions of CPU-hours required to complete statewide seismic hazard map calculations in planning.
