Cray XK7

XK7 is a supercomputing platform, produced by Cray, launched on October 29, 2012. XK7 is the second platform from Cray to use a combination of central processing units ("CPUs") and graphical processing units ("GPUs") for computing; the hybrid architecture requires a different approach to programming to that of CPU-only supercomputers. Laboratories that host XK7 machines host workshops to train researchers in the new programming languages needed for XK7 machines. The platform is used in Titan, the world's second fastest supercomputer in the November 2013 list as ranked by the TOP500 organization. Other customers include the Swiss National Supercomputing Centre which has a 272 node machine and Blue Waters has a machine that has Cray XE6 and XK7 nodes that performs at approximately 1 petaFLOPS (10¹⁵ floating-point operations per second).[1]

Cray XK7

Active	October 29, 2012
Operators	Cray Inc.
Architecture	AMD Opteron 6200 Interlagos series 16-core CPUs Nvidia Tesla K20 Kepler series GPUs
Power	45 to 54.1 kW per cabinet
Operating system	Cray Linux Environment
Memory	16 or 32 GB ECC DDR3 per CPU 5 or 6 GB ECC GDDR5 per GPU
Speed	Theoretical peak of 50 petaFLOPS
Ranking	TOP500
Web site	www.cray.com/Products/Computing/XK7.aspx

Overview

XK7 is scalable up to 500 cabinets, each contains 24 blades and each blade contains 4 nodes (1 CPU and 1 GPU per node).[2] The CPUs available are of the 16-core AMD Opteron 6200 Interlagos series and the GPUs are of the Nvidia Tesla K20 Kepler series.[2] Each CPU can be paired with either 16 or 32 GB of error-correcting code memory (ECC) while the GPUs have either 5 or 6 GB of ECC memory depending on the model of GPU used.[2] The nodes communicate with each other via the Gemini Interconnect; each Gemini chip services 2 nodes with a capacity of 160 GB/s.[2] Depending on the components used, a full cabinet will consume between 45 and 54.1 kW of electricity which is converted into heat; thus the cabinets need cooling, either by air or water.[2]

XK7 based machines run the Cray Linux Environment which incorporates SUSE Linux Enterprise Server.[2] Code to run on an XK7 machine can be written in a range of programming languages. The hybrid architecture requires different programming to conventional CPU-only supercomputers; Oak Ridge National Laboratory and the Swiss National Supercomputing Centre hold workshops to educate researchers on the new programming approach.[3][4]

Usage

The XK7 platform was announced on October 29, 2012 to coincide with the completion of Titan at Oak Ridge National Laboratory (ORNL).[5] Titan has 18,688 XK7 nodes, each containing an Opteron 6274 CPU with 32 GB of memory and a K20X GPU with 6 GB.[6] The computer has a theoretical peak performance of 27.1 petaFLOPS but in the LINPACK benchmark used by the TOP500 organisation to rank supercomputers it performed at 17.59 petaFLOPS, enough to take first place on the November 2012 list.[7][8] Titan uses 8.2 MW of electricity and is third on the Green500 list which ranks supercomputers by their energy efficiency.[9]

The National Center for Supercomputing Applications (NCSA) in Illinois has a machine, Blue Waters, using a combination of Cray XE6 and XK7 nodes. The machine has 3072 XK7 nodes and 22,752 XE6 nodes.[10] Each XE6 node has two Opteron 6276 and 32 GB of memory per CPU.[11] The XK7 nodes also have Opteron 6276 CPUs with 32 GB of memory and a K20X GPU with 6 GB.[11] Blue Waters has performed at over 1 petaFLOPS in benchmarks; however, the project managers do not believe in the relevance of the LINPACK benchmark used by the TOP500 organisation and therefore did not submit a benchmark test for ranking.[10]

The Swiss National Supercomputing Centre (CSCS) machine named Todi was upgraded to XK7 on October 22, 2012.[12] Todi has 272 nodes with Opteron 6272 CPUs with 32 GB of memory and a K20X GPU with 6 GB.[13][14] Todi has a theoretical peak performance of 393 teraFLOPS and performed at 274 teraFLOPS in the November 2012 TOP500 list taking 91st place.[14] Todi consumes 122 kW and is ranked fourth, one behind Titan, on the November 2012 Green500 list.[9][14]

References

1. Schulten, Klaus (November 29, 2012). "Blue Waters petascale supercomputer now in friendly user phase". Outlook Series. Retrieved January 16, 2013.
2. "Cray XK7 Specifications". Cray. Archived from the original on January 6, 2013. Retrieved January 15, 2013.
3. "Introduction to OpenACC and CUDA Programming on the Cray XK7 Platform". Swiss National Supercomputing Centre. January 10, 2013. Retrieved January 15, 2013.
4. Gajus, Brian (December 17, 2012). "Titan Trainers Take Road Trip". Oak Ridge Leadership Computing Facility. Retrieved January 16, 2013.
5. "Cray Launches XK7 Hybrid Super with Titan Debut at ORNL". Inside HPC. October 29, 2012. Retrieved January 15, 2013.
6. "Introducing Titan". Oak Ridge Leadership Computing Facility. Retrieved January 16, 2013.
7. Gajus, Brian (November 12, 2012). "ORNL Supercomputer Named World's Most Powerful". Oak Ridge Leadership Computing Facility. Retrieved January 16, 2013.
8. "November 2012". TOP500. Retrieved January 15, 2013.
9. "Heterogeneous Systems Re-Claim Green500 List Dominance". Green500. November 14, 2012. Archived from the original on February 5, 2013. Retrieved January 16, 2013.
10. "One Year Later: Delivering Sustained Petascale Science" (PDF). National Center for Supercomputing Applications. December 2012. Retrieved January 16, 2013.
11. "Brief Blue Waters System Summary". National Center for Supercomputing Applications. Retrieved January 16, 2013.
12. "Todi Upgrade to Xk7". Swiss National Supercomputing Centre. October 15, 2012. Archived from the original on February 19, 2013. Retrieved January 16, 2013.
13. "Todi Cray XK7". Swiss National Supercomputing Centre. Archived from the original on January 21, 2013. Retrieved January 16, 2013.
14. "Todi". TOP500. Retrieved January 16, 2013.