The Beginning of NSF Petascale Computing

Deploying the Sun Magnum System:
The Beginning of NSF Petascale Computing
Jay Boisseau, Director Texas Advanced Computing Center
The University of Texas at Austin October 24, 2006

First, Some “Corrections”
• There are no ClearSpeed (or any other) accelerators in this system!

First, Some “Corrections”
• There are no ClearSpeed (or any other) accelerators in this system
• This is a capability system, not only a capacity system: balanced & tightly coupled

First, Some “Corrections”
• There are no ClearSpeed (or any other) accelerators in this system
• This is a capability system, not only a capacity system: balanced & tightly coupled
• Jobs will run on the entire system; it will not be partitioned into smaller systems

First, Some “Corrections”
• There are no ClearSpeed (or any other) accelerators in this system
• This is a capability system, not only a capacity system: balanced & tightly coupled
• Jobs will run on the entire system; it will not be partitioned into smaller systems
• There was no unique deal from AMD (but we’ll take one if they offer!)

And Some Presentation Caveats
• The system does not exist yet
– It is not yet doing science or even drawing power! – Key components are to be delivered in 2007
• Some system details are still non-disclosure
– Ask Giri Chukkapalli
• Our experience with 50K general-purpose CPUs is zero; even with 5K, it’s only 3 weeks
– This is new territory—none of us are experts yet

NSF HPC Vision & Strategy
• Context: NSF Cyberinfrastructure Strategic Plan • NSF now investing in world-class HPC systems
– Annual track 2 solicitations ($30M) – Single five-year Track1 solicitation ($200M)
• Complementary solicitations forthcoming
– petascale applications development solicitations – Software Development for CI has an HPC component – Etc.

Sun System Configuration
[Some data non-disclosure] • Compute power
– 13152 Opteron “Deerhound” processors
• Quad-core, four flops/cycle (dual pipelines) • Initial deployment with SantaRosa processors
– 421 teraflops aggregate peak (at least)
• Memory
– 2GB/core – 105 TB total

Sun System Configuration
[Some data non-disclosure] • Interconnect
– Sun proprietary switch based on IB
• Minimum cabling: robustness and simplicity!
– PathScale adapters (PCI-Express) – MPI latency: 1.6-1.8 microsec – Peak bi-directional b/w: ~ 1 GB/sec – Total backplane b/w: 13.8 TB/sec

Sun System Configuration
[Some data non-disclosure] • File system
– 72 Sun X4500s (“Thumper”)
• 48 500GB disks per 4U!
– 1.7 PB total disk
• 1 PB in largest /work file system
– Lustre file system – Aggregate b/w: 40 GB/s