Tier1 Hardware Procurements

Introduction

This page aims to collect experience of the various hardware related to the Tier1 NDGF activity.

Disk pools

HPC2N - Dell R760xd2

7 machines for Tier1 delivered December 2025, in service from December 2025.

• Dell PowerEdge R760xd2, LFF chassis "2U double-deep"
• 2 x Intel Xeon Silver 4510 CPU, each 12 cores @ 2.40GHz (max turbo 4.1 GHz)
  • Two sockets populated due to price/performance compared to single socket CPU with more cores
• 128G RAM
• Data storage
  • PERC H965i RAID controller, 8G non-volatile (flash-backed) cache
    • SAS4116W based
    • mpi3mr driver
  • In total 28 x 24T SAS LFF HDDs
    • 12 x 24T SAS LFF hot-swap HDDs in the front visible
    • 12 x 24T SAS LFF hot-swap HDDs in the midplane behind visible HDDs
    • 4 x 24T SAS LFF hot-swap HDDs in the back
  • Two separate RAID 6 devices/file systems, 256k strip-size:
    • perccli2 /c0 add vd r6 size=all name=dCacheVD-A drives=304:0-13 wb strip=256
    • perccli2 /c0 add vd r6 size=all name=dCacheVD-B drives=304:14-27 wb strip=256
    • Much better bandwidth than a single RAID60 with multiple LUNs or multiple RAID6:s with multiple LUNs
  • XFS, created aligned directly on disk device:
    • mkfs.xfs -f -d su=256k,sw=12 -L grid-poolA /dev/sda
    • mkfs.xfs -f -d su=256k,sw=12 -L grid-poolB /dev/sdb
  • Mounted using labels in fstab:
    • LABEL=grid-poolA /grid/poolA xfs rw,swalloc,largeio,x-systemd.mount-timeout=10min 0 2
    • LABEL=grid-poolB /grid/poolB xfs rw,swalloc,largeio,x-systemd.mount-timeout=10min 0 2
• OS storage
  • Dell BOSS, OS storage - two mirrored M.2 cards in a PCIe slot providing an AHCI device.
• Intel E810-XXV 2x25G Ethernet NIC
  • Both ports used in LACP setup with TransmitHashPolicy=layer3+4
• iDRAC (management) with dedicated network port
• Extra tuning applied via udev rules (also see Operations Tuning Linux)
  • Increase read-ahead to match workload:
    • ATTR{bdi/read_ahead_kb}="16384"
  • Tunings for mq-deadline scheduler (default in recent kernels), carried over from our previous deadline scheduler tunings:
    • ATTR{queue/nr_requests}="128"
    • ATTR{queue/iosched/writes_starved}="10"

HPC2N notes:

• Cable arm and rack rails needed! To hot-swap midplane drives you need to pull out the server and tilt up the midplane to access the drives.
• There seems to be something funky going on with the fan speed regulation, fans seem to be going either full-tilt or slow instead of finding a suitable speed in between.

NDGF shakedown notes:

• Tried a raid60 but got only 2.5-3GB/s speeds, 2 x raid6 got 4GB/s reads and 3-3.5 GB/s writes
• Not very sensitive to harsh load, hundreds of parallel threads only reduced performance by 30-40%

HPC2N - Dell R740xd2 - LFF chassis - purchase 2

7 machines for Tier1 delivered January 2022, in service from XXXX 2022.

• Dell 740 xd2, LFF chassis "2U double-deep"
• 2 x Intel(R) Xeon(R) Silver 4215 CPU, each 8 cores @ 2.50GHz
  • Two sockets populated due to PCIe slot assignment
• 96G RAM
• Data storage
  • H730P RAID controller, 2G non-volatile (flash-backed) cache
    • We wanted the better H740 RAID controller, but Dell couldn't deliver that
  • In total 26 18T SAS LFF HDDs
    • 12 x 18T SAS LFF hot-swap HDDs in the front visible
    • 12 x 18T SAS LFF hot-swap HDDs in the front behind visible HDDs
    • 2 x 18T SAS LFF hot-swap HDDs in the back
  • RAID 60 with two parity groups, 1024k strip-size
    • perccli /c0 add vd r60 size=all name=dCacheVD drives=32:0-25 pdperarray=13 wb strip=1024
  • XFS, created aligned with -d su=1024k,sw=11 directly on disk device.
    • Label set with -L labelname and mounted using LABEL=labelname in fstab
• OS storage
  • Dell BOSS - two mirrored M.2 cards in a PCIe slot providing an AHCI device.
• Intel XXV710 25G Ethernet NIC (1 port used)
  • Long delivery times due to component shortage, NICs will arrive later sometime...
  • 2 machines initially deployed with spare 25G NICs, 5 machines with borrowed 10G NICs.
  • NICs arrived 1st week of February, painless operation to swap (same interface name, so no changes needed to UEFI and OS setup other than change MAC addresses for network boot DHCP).
• iDRAC (management) with dedicated network port
• Extra tuning applied via udev rules (also see Operations Tuning Linux)
  • Increase read-ahead to match workload:
    • ATTR{bdi/read_ahead_kb}="16384"
  • Tunings for mq-deadline scheduler (default in recent kernels), carried over from our previous deadline scheduler tunings:
    • ATTR{queue/nr_requests}="128"
    • ATTR{queue/iosched/writes_starved}="10"

HPC2N notes:

• No cable arm needed!
• Sits on shelves, not rack rails, so need to be 2 persons to handle due to weight.
• H730P RAID controller or SAS expander cabling/setup seems a bit underprovisioned, but we knew that from previous purchase
  • Really recommend going for the better H740 RAID controller if possible.
  • Meets performance requirements set when procuring, so more of a notice on machine design and what to expect.

NDGF shakedown notes:

• See HPC2N R740xd2 purchase 1.

• Pure read load with a concurrency of 10-100 in the 2.3-2.5GB/s range (but on a partially filled pool)

Tape pools

HPC2N - Dell R6615 - U.2 chassis

4 machines for Tier1 delivered October 2024, in service from November 2024.

• Dell PowerEdge R6615, U.2 chassis/backplane
• 1x AMD EPYC 9124, 16 cores @ 3 GHz (max turbo 3.7 GHz)
• 64 GB RAM
• Mellanox ConnectX-6 Lx 25G Ethernet NIC
• PERC H965i RAID controller, 8G non-volatile (flash-backed) cache
  • SAS4116W based
  • mpi3mr driver
  • 5x Dell DC NVMe ISE 7450 RI U.2 3.84TB, 1 DWPD, 7008 TBW
    • perccli2 /c0 add vd r5 size=all name=dCacheVD drives=284:0-4 wt nora strip=64
  • Dell BOSS, OS storage - two mirrored M.2 cards in a PCIe slot providing an AHCI device.

HPC2N notes: Hardware raid, balanced 3 GB/s read performance when doing 3 GB/s write at the same time. Endurance on fs-level is 4*7008 = 28032 TBW minus fs overhead.

ARC Cache

HPC2N - HPE DL325

4 machines delivered during July 2021, partly taken into production during October 2021

• HPE DL325 Gen10+
• 1 x AMD® EPYC 7420P @
• 128G RAM
• Data Storage
  • 8 × 1.6 TB NVMe U.3
  • ZFS RAID0
    • ashift=12 recordsize=1Mb
    • 11.6 TB usable space per machine
    • split into two datasets cache and session
    • session has lz4 compression enabled and cache not
• OS storage
  • HW RAID1
    • 2 × 480 GB NVMe M.2
  • xfs
• 2 × 25 Gb
  • 1 port on LHC OPN
• 200 Gb Mellanox
  • Used at 40Gb on cluster network
• ILO5 (management) with dedicated network port