High Performance Computing
In the area of High Performance Computing, the GEaRS group continued to expand its partnerships and collaborations with faculty members, assisting them to take advantage of computational resources. This year, staff in GEaRS focused on expanding existing systems and installing new ones to meet the growing demand of its partners' research needs. GEaRS group staff continue to maintain a strong commitment to teaching the use of application packages and libraries and supporting the same in research across many academic disciplines.
LION-XA Installation
LION-XB Installation
LION-XO Upgrade and Expansion
Hammer System Upgrade
Unisys System Expansion
IBM Power System Installation
Spotlight on Systems Expansion
LION-XA Installation
LION-XA is a new system currently being used as a
testbed for parallel filesystems. Parallel filesystems differ from
traditional filesystems; rather than have a single filesystem
served from a single fileserver, the filesystem is distributed across
multiple servers so that filesystem operations can take advantage of the
aggregate performance of all of the servers in the filesystem. Parallel
filesystems also help overcome the filesystem size limitations, thus
helping to address significant increase in demand for storage space from
faculty members in science, engineering, and other disciplines.
The GEaRS group is currently investigating several of the most promising
parallel filesystems available today to find one whose implementation will
best serve the needs of the Penn State research community. These include
GPFS, IBRIX, iGrid, Lustre, and CXFS. When the parallel filesystem
investigation is completed, Lion-XA will become available as another
computational cluster. Lion-XA currently consists of 32 servers, each
with dual 64-bit Intel Xeon Processors and 8 gigabytes (GB) of RAM.
LION-XB
LION-XB is a new system that blends two distinct paradigms of high performance computing: multiprocessor systems with large shared memory and distributed memory compute engines in a cluster of servers where each server has fewer processors. With the availability of multi-core processors, a hybrid is now possible where a cluster model can still be followed, but the building blocks (the individual nodes) of the cluster are mid-sized shared-memory systems. Lion-XB is an example of such a hybrid: it is built from 4-socket servers using dual-core AMD Opteron processors, yielding 8 processing elements in each server, each server having 32GB of shared memory. The Lion-XB cluster has 16 such servers, with an aggregate of 128 processing cores and 512GB of memory.
GEaRS' goal is to simultaneously accommodate two distinct styles of programming: shared memory parallel programming (mostly using compiler directives) and distributed memory programming (using MPI library). Different programming approaches are needed depending on the numerical algorithms and solution techniques necessary for problems in various academic disciplines. LION-XB makes use of the PathScale (now QLogic) HTX InfiniBand adapters. HyperTransport is the native bus used for communication between processors on AMD Opteron systems and HTX is a connector that gives the ability to plug an external adapter directly into that bus. By having the InfiniBand adapter talk directly to the HTX bus, overhead is greatly reduced and performance increases tremendously. These InfiniBand adapters have a measured latency of 1.3 microseconds and a bandwidth of 930 megabytes per second (MB/s). The cards are so fast that they reach half of their maximum bandwidth at a packet size of only 88 bytes. They also scale with the number of processors within a system, reaching 11 million packets per second on an 8 core system, such as LION-XB.
LION-XO Upgrade and Expansion
LION-XO was upgraded to include an additional 44 Sun SunFire v40z servers; 32 of which had quad AMD Opteron processors and 16GB of RAM and 12 of which had quad AMD Opteron processors and 32GB of RAM. This brought the final size of LION-XO to 124 compute servers with a total of 336 processors and 1.5 terabytes (TB) of RAM. This also increased the aggregate local storage available as scratch space for running jobs to 25TB.
The first 32 of the new servers were funded through a new partnership between GEaRS and the Center for Environmental Kinetics Analysis (CEKA), a multidisciplinary initiative to better understand issues related to environmental chemical kinetics, especially as related to geochemical cycling of elements, fate and transport of contaminants, and carbon sequestration within the critical zone.
The last 12 of the new servers were installed with 32GB of RAM to address the needs of users who had research problems that required a large amount of memory to solve.
During this expansion, LION-XO was upgraded to a newer version of RedHat Enterprise Linux, which better supported its AMD Opteron Processors.
Hammer System Upgrade
Hammer was moved from four old, 2 processor AMD Opteron blades with 4GB of RAM to eight new Sun SunFire v40z servers, each with 32GB of RAM. This upgrade allowed new research problems to be completed that were previously unable to be done due to the limited processing power and memory capacity of the old Hammer machines.
Unisys System Expansion
A new Unisys ES7000 system was installed to better address the needs of researchers with large-scale shared-memory parallel code and also of those researchers who need to address an extraordinarily large amount of memory in a single system. These needs were met by configuring the new Unisys ES7000 to have 32 Intel Itanium 2 processors sharing 128GB of globally addressable memory in a single system image.
IBM Power System Installation
A new IBM p570 system was installed in partnership with the University's Institute for Computational Science (ICS). This system was made possible through an IBM SUR (Shared University Research) grant. SUR grants are given by IBM to promote research in areas of mutual interest between IBM and the receiving institution. This system consists of 6 dual-core IBM Power5 processors with 112GB of RAM. It will be used in support of the University's graduate minor in Computational Science.
Spotlight on Systems Expansion
Many of the system upgrades performed this year were targeted at enabling new classes of problems that required too much memory to be solved previously. The following are a few examples of such challenges:
- One of the many things that Dr. Joseph Rose, Paul Morrow Professor of Engineering Design and Manufacturing, studies is the behavior of nanostructures under thermal and structural loading. These studies use finite element calculations to simulate this behavior. Finite element calculations use a mesh to approximate structures and the finer-grained the mesh is, the greater the amount of detail there is of the structures to be studied. As the mesh gets more fine-grained, the amount of memory in a typical system can be quickly exhausted. Dr. Rose's group used ABAQUS on the upgraded Hammer and Unisys systems to perform finite element calculations of a finer scale and larger scope than were previously possible.
- The Protective Technology Center (PTC), of the Department of Civil Engineering in the College of Engineering, is involved with designing structures for post 9/11 that can handle extreme thermal conditions and explosions. Several researchers of the PTC are using LS-DYNA on Hammer and LION-XO to explore the FSI (fluid structure interaction) of a shock wave from an explosion with critical infrastructure. FSI calculations inherently require a large amount of memory to solve.
- Graduate student Azam Thatte of Professor James Brasseur's research group in Mechanical and Bio Engineering is designing a Finite Element Model of Peristalic Waves in the Human Stomach using COMSOL Multiphysics on Hammer and LION-XO. Peristaltic waves are waves created by contractions in hollow muscular structures within the body, such as the esophagus and intestines. The fine-grained grid required for this calculation, due to the irregular geometries being studied, leads to large memory requirements.
- Dr. James Kubicki of the Center for Environmental Kinetics Analysis took advantage of the large memory available in the new Unisys system for his studies regarding uranium complexation to biological functional groups. Uranium and bacteria interact in the environment, and this can affect the transport of uranium in groundwater. This large and complex calculation required the use of the large shared-memory Unisys system. Utilizing the large memory address space of the machine, Dr. Kubicki was able to achieve a significant improvement in time to solution over a traditional distributed memory cluster. Performance increased an average of over four times using a similar number of processors.