System Overview¶
This system consists of 400 compute nodes equipped with NVIDIA GB200 NVL4, shared storage composed of 2 PB of high-speed storage (SSD) and 10 PB of large-capacity storage (HDD), and an InfiniBand XDR network that connects them at high speed.
Compute Node Performance¶
| Item | Value |
|---|---|
| CPU/GPU model | NVIDIA GB200 NVL4 (Grace CPU × 2, B200 GPU × 4) |
| GPU theoretical performance (FP64) | 160.4 TFLOPS (40.1 TFLOPS × 4 GPUs) |
| GPU theoretical performance (FP8) | 38.84 PFLOPS (9.712 PFLOPS × 4 GPUs) |
| CPU memory performance | LPDDR5X, 960 GiB, 768 GB/s (480 GiB, 384 GB/s × 2 CPUs) |
| GPU memory performance | HBM3e, 692.8 GiB, 31.6 TB/s (173.2 GiB, 7.9 TB/s × 4 GPUs) |
| GPU-GPU bandwidth (bidirectional) | NVLink, 600 GB/s (300 GB/s × 2 (bidirectional)) |
| CPU-GPU bandwidth (bidirectional) | NVLink-C2C, 450 GB/s (225 GB/s × 2 (bidirectional)) |
| CPU-CPU bandwidth (bidirectional) | CLink, 300 GB/s (150 GB/s × 2 (bidirectional)) |
| Network | InfiniBand XDR, 800 Gbps × 4 |
| Local storage | Kioxia XD7P NVMe SSD 7.68 TB, bandwidth: Read 7.2 GB/s, Write 4.8 GB/s, IOPS (4 KB): Read 1,550 K, Write 200 K |
For details on CPU/GPU connections and bandwidth, see the following figure.
Overall System Performance¶
| Item | Value |
|---|---|
| Number of compute nodes | 400 nodes |
| GPU theoretical performance (FP64) | 64.160 PFLOPS (160.4 TFLOPS × 400 nodes) |
| GPU theoretical performance (FP8) | 15.539 EFLOPS (38.84 PFLOPS × 400 nodes) |
| GPU memory performance | 270.625 TiB, 12.640 PB/s (692.8 GiB, 31.60 TB/s × 400 nodes) |
| CPU memory performance | 375 TiB, 307.2 TB/s (960 GiB, 768 GB/s × 400 nodes) |
| Shared storage | High-speed storage 2 PB (SSD), large-capacity storage 10 PB (HDD) |
Network Topology¶
The network is a two-layer Fat Tree composed of 6 Spine switches and 17 Leaf switches. Of the 17 Leaf switches, 16 are connected to 24 compute nodes each, and the remaining Leaf switch is connected to 16 compute nodes (16 × 24 nodes + 1 × 16 nodes = 400 nodes). Communication between compute nodes under the same Leaf switch is fast, but communication that crosses Spine switches may experience somewhat lower performance depending on the communication path and the amount of simultaneous traffic.
