Next-generation engines and industrial burners may use less fuel, emit fewer pollutants: Two-thirds of the petroleum Americans use goes for transportation. The remaining one-third heats buildings and generates electricity in steam turbines. In the not-so-distant future, engines, furnaces, and power-generation devices may burn alternative fuels and employ advanced technology. Supercomputers at the National Center for Computational Sciences (NCCS) are hastening the arrival of advanced combustion devices that will consume less energy and emit fewer pollutants. On these machines mechanical engineer Jacqueline Chen of Sandia National Laboratories (SNL) leads an effort to simulate the combustion of diverse fuels. The result is a library of science data that captures complex aero-thermo-chemical interactions and provides insight into how flames stabilize, extinguish, and reignite. Chen’s data libraries will assist engineers in the development of models that will be used to design next-generation combustion devices burning alternative fuels. “If low-temperature compression ignition concepts employing dilute fuel mixtures at high pressure are widely adopted in next-generation autos, fuel efficiency could increase by as much as 25 to 50 percent,” Chen says. With mechanical engineer Chun Sang Yoo of SNL and computational scientist Ramanan Sankaran of Oak Ridge National Laboratory (ORNL), Chen recently used Jaguar, a Cray XT supercomputer at the NCCS, to simulate combustion of ethylene, a hydrocarbon fuel. Their simulation generated more than 120 terabytes (120 trillion bytes) of data—more than ten times as much as contained in the printed contents of the Library of Congress. That flood of data was temporarily stored on the NCCS’s Lustre file system, which can hold up to 284 terabytes. Because processing big data sets requires a lot of memory, this fall the researchers will analyze and visualize the data using computer clusters at the NCCS and SNL, such as the NCCS’s Lens, a Linux cluster with a 677-terabyte capacity and 32 nodes, each of which contains four quad-core processors and 64 gigabytes of memory. Later the data will be archived in the NCCS’s High-Performance Storage System, which currently stores more than 2.86 petabytes (quadrillion bytes) in 10.6 million files, and is configured to grow this archive capability into many tens of petabytes. Exploring lifted flames Advanced combustion technology depends on lifted flames, which result when cold fuel and hot air mix and ignite in a high-speed jet. If the speed increases too much, lifted flames can blow out. For flames to stabilize, or continue to burn downstream from the burner, turbulence, which mixes fuel with air to enable burning, must exist in balance with key ignition reactions that occur upstream of where the flames appear. In industrial burners used for power generation, lifted flames reduce thermal stresses to nozzles by minimizing contact between the flame and the nozzle. Lifted flames are also integral to the workings of direct-injection gasoline engines, compression-ignition diesel engines, and gas turbines, in which streams of cold fuel and hot oxidizer are partially premixed prior to combustion. The position downstream of a fuel injector at which a diesel fuel jet establishes a flame influences the degree of premixing needed and affects combustion and soot formation. Proper positioning of lifted flames in advanced engines could burn fuel so cleanly that emissions of nitrogen oxide, a major contributor to smog, would be nearly undetectable, Chen says. To explore processes underlying ethylene combustion, the group uses direct numerical simulation (DNS), a technique that solves equations governing viscous, heat-conducting fluids without using turbulence models. DNS uses a computational mesh to reveal a turbulent flame’s physical characteristics, such as temperatures and chemical species, on spatial and temporal scales ranging from the smallest to the largest details. Using terascale computing, DNS is feasible for canonical flows with a moderate Reynolds number (an indication of the range of scales in a system), where the dynamic range between the largest and smallest features is approximately 10,000 units. “Direct numerical simulation is our numerical probe to measure, understand, or see things in great detail at the finest scales where chemical reactions occur,” Chen says. “That’s particularly important for combustion because reactions occurring at the finest molecular scales impact global properties like burning rates and emissions.” The simulation ran a software application developed at Sandia called S3D, which runs on multiple processing cores to model compressible, reacting flows with detailed chemistry. S3D was one of six applications recently selected to run pioneering “science-at-scale” simulations efficiently employing most or all processing cores of Jaguar, which was upgraded in May 2008 to perform 263 trillion calculations per second, or teraflops. The simulation used 30,000 of Jaguar’s 31,000 processing cores and 4.5 million processor hours. Running computationally demanding applications after a major machine upgrade is part of a transition-to-operations activity that begins when a commissioned NCCS system passes a formal acceptance test and its performance is monitored and assessed. Science-at-scale simulations run on leadership machines like Jaguar help advance critical scientific application areas to efficiently exploit petascale computing systems, which are capable of a quadrillion calculations per second and will be available to the scientific community in 2009. The most recent DNS calculations required a three-dimensional grid with more than a billion points spaced 15 microns apart. Armed with such a grid, Chen and her colleagues created the world’s first fully resolved simulation of small lifted autoigniting hydrocarbon jet flames designed to represent some of the underlying physics in a direct-injection diesel engine. The sheer scale of the simulation has enabled an unprecedented level of quantitative detail in the description of both turbulence and chemistry and their interactions. The researchers simulated the unsteady distributions of chemical composition, temperature, and reacting flow characteristics using one of the simplest hydrocarbon fuels, ethylene, as a first step toward a progression of more complex, larger hydrocarbon fuels. Ethylene is not a common transportation fuel but is an important reference fuel in research. Unlike common fuels, which are long-chain molecules whose chemical behaviors are difficult to compute, ethylene, which contains two carbon and four hydrogen atoms per molecule, is relatively simple, meaning it is a good candidate for DNS. Moreover, it is easy to work with in the lab and its chemical properties are well known, so it is feasible to test results from experiments against predictions from numerical simulations. Analysis and visualization of the large volume of simulation data have been a challenge. Through collaborations with computer scientists, new methods are being developed to enable researchers to gain physical insight from the multiscale, time-varying data. Valerio Pascucci of the Scientific Discovery through Advanced Computing (SciDAC) Visualization and Analytics Center for Enabling Technologies developed topology-based feature segmentation and software to track the temporal evolution of intermittent ignition and mixing structures that describe the stabilization of the lifted flame. Kwan-Liu Ma of the SciDAC Institute for Ultrascale Visualization performed volume rendering of key scalars, which varied over time in the simulation, and Ramanan Sankaran of ORNL developed in situ particle tracking to follow the mixing rate and thermochemical trajectories of individual ignition structures. What’s next Future simulations of turbulence–chemistry interactions in jet flames will target fuels of increasing complexity and diversity. The Department of Energy’s (DOE’s) Innovative and Novel Computational Impact on Theory and Experiment (or INCITE) program has allocated Chen and SNL mechanical engineer Joe Oefelein 18 million hours on Jaguar in 2008 to simulate autoignition and injection processes with alternative fuels. The researchers will model fuels with a wide range of ignition characteristics—oxygenated fuels such as dimethyl ether, diesel surrogates such as n-heptane, and renewable biofuels such as ethanol—and explore high-pressure, low-temperature environments representative of advanced compression-ignition engines. In addition to the S3D DNS code, Oefelein will use a large-eddy simulation (LES) code called RAPTOR to investigate turbulent reacting flow processes in an actual internal combustion engine. The DNS and LES approaches are complementary. LES captures large-scale, high-Reynolds-number mixing and combustion processes that occur over full engine cycles (that are dominated by the geometric features of the engine), whereas DNS captures details of small-scale mixing and chemistry associated with combustion. Given these unique attributes, the combination of LES and DNS running on petascale computers can provide a nearly complete picture of internal-combustion-engine processes well beyond the current state of the art. The DOE Office of Basic Energy Sciences and Office of Advanced Scientific Computing Research supported this research.—Dawn Levy