GPU Cluster Deneb¶
Overview¶
Planned to be usable in Q2 2027, Deneb will be the next GPU cluster at ZIH. It will incorporate
46 compute nodes based on the Nvidia GB200 NVL4 architecture, for a total of 184 available GPUs. As
the GPUs yield similar FP64 and FP32 performance to those in Capella while raising the
performance of smaller data types ~5-fold, this cluster is primarily useful for machine learning
applications. An SSD-based global Lustre filesystem will be made available specifically for Deneb.
Hardware Details¶
A brief overview of the hardware specification is documented on the page HPC Resources.
Processor
Deneb integrates two Grace Blackwell Superchips per node, each of which hosts one Nvidia
Grace CPU with 72 cores. Thus, each node incorporates a total of 144 CPU cores. Much of the
system architecture details in the context of GB200 NVL4 are not disclosed and can best be
inferred from Grace-only or Grace Hopper systems. Note that Grace is an ARM Neoverse V2
microarchitecture and is incompatible with existing x86 code.
Graphics Processing Unit
- Nvidia GB200 NVL4 Superchip (Data Sheet)
- Memory: 186 GB HBM3e, 8000 GB/s theoretical peak bandwidth
- Interfaces: SXM6 package, PCIe 5.0 x16, interconnected with other GPUs via 1.8 TB/s NVLink, with CPU via 450 GB/s NVLink
- Power limit: 1200 W
System Memory
All Deneb nodes will feature 960 GB of LPDDR5X RAM across two sockets with an aggregate
bandwidth of 1024 GB/s.
Local Storage
Deneb nodes will each include an array of four 3.84 TB SSDs. Their exact format and performance
remain to be determined.
See also: Node-Local Storage in Jobs
Networking
Deneb nodes will feature four Nvidia ConnectX-8 InfiniBand devices (data sheet)
for networking. All four network interfaces will operate at 4xNDR (400 Gb/s, 50 GB/s),
providing the node with 1.6 Tb/s of network connectivity. Each GPU will be local to one NIC and
connect to it using PCIe 5.0 x16. All components will support RDMA.
System Architecture
The system is based on Bull Sequana XH3515B compute blades.
Preliminary system architecture:
Operating System
Deneb will run a latest version of Red Hat Enterprise Linux.
Backlinks¶
The following pages link to this page: