Computing Resources
Rapid advancements in imaging technology have provided researchers with the ability to produce very high-resolution, time-varying, multi-dimensional data sets of the human brain. The complexity of the new data, however, requires immense computing and storage capabilities. Institutions and scientists worldwide rely on the Core Sites' computing resources to conduct research.
Computing at LONI
LONI relies on a 306-node, dual-processor SUN Microsystems V20z cluster, one of the largest V20z installations in the world. Each compute node has dual 64-bit 2.4 gigahertz AMD Opteron CPUs with 4 gigabytes of memory. In addition, LONI has a 64-node Dell development cluster, with each node using dual 64-bit 3.6 gigahertz Intel EM64T processors and 4 gigabytes of memory. To augment the facility's cluster resources, LONI has a 64-processor SGI Origin 3800 SMP supercomputer with 32 gigabytes of memory. A comparable 32-processor SGI Onyx2 Reality Monster with 16 gigabytes of memory and a 6-processor SGI Onyx2 with 8 gigabytes of memory SMP supercomputers are utilized to drive graphics-intensive applications and interactive real-time multidimensional visualization of structural brain models and volumetric datasets. To facilitate the submission and execution of compute jobs in this heterogeneous compute environment, SUN's Grid
Engine (SGE) is used to virtualize the resources above into a compute service. A grid layer sits atop the compute resources and submits jobs to available resources according to user-defined criteria such as CPU type, processor count, etc.
Storage at LONI
Institutions and scientists worldwide rely on the facility’s resources to conduct research. LONI has made a decisive move towards fault-tolerant, high-availability systems design to ensure 24/7 functionality. Concurrent with its graphics and computation systems, the laboratory uses a fault-tolerant storage area network (SAN) to accommodate current and projected storage requirements. The SAN hardware infrastructure is composed of the cluster and SMP supercomputers previously mentioned, RAID storage, dual robotic tape silos and a full complement of Brocade fibrechannel switches, delivering up to 800 megabytes per second data throughput.
Alternate paths exist throughout the fabric so that no single point of failure exists, guaranteeing access to critical data and processing power. Two quad-processor 500MHz MIPS-R14000 processor SGI Origin300 servers mediate all data transactions and provide networking services. A high availability application ties both servers together and ensures failover in the case of hardware failure. LONI relies on two silos, a Storagetek SL8500 and a Powderhorn 9310, to store mirrored copies of the facility's offline tape data. These tape robots are housed in two different locations, ensuring that catastrophic events in any one data center will leave a copy of all tape data intact in another data center. Seven high-speed 40-gigabyte and two high-capacity 400-gigabyte tape drives provide LONI's tape services. To leverage the available SAN throughput, an SGI TP9500, TP9400 and a TP9100 RAID5 arrays as well as SUN 3510s provide nearly 50 terabytes of fault-tolerant disk storage.
