Tests of a Budget 1U AMD Epyc Server for 4 NVMe Gen 4.0 Drives (Will the BPN-SAS3-815TQ-N4 Support NVMe Gen 4.0 Drives?)
| Supermicro news • Tests • NewsTests of a Budget 1U AMD Epyc Server for 4 NVMe Gen 4.0 Drives (Will the BPN-SAS3-815TQ-N4 Support NVMe Gen 4.0 Drives?)
We built a 1U ITC-RR3R server for a client, equipped with an AMD Epyc Rome 7542 32C/64T 2.9G 128MB processor, and tested whether the BPN-SAS3-815TQ-N4 backplane, which is included in the CSE-813MF2TQC4-R407CB chassis, supports NVMe Gen 4.0 drives.
HDS-SUN0-MZQL2960HCJRA7 Samsung PM9A3 960GB NVMe PCIe4x4 U.2 7mm 1DWPD 5YR SED Opal
fio seq read (Samsung MZQL2960HCJRA7)
fio --filename=/dev/nvme1n1 --rw=read --direct=1 --bs=1M --ioengine=libaio --runtime=60 --numjobs=1 --time_based --group_reporting --name=seq_read --iodepth=16
seq_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=6922MiB/s][r=6921 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=22263: Thu Mar 7 15:36:19 2024
read: IOPS=6913, BW=6913MiB/s (7249MB/s)(405GiB/60003msec)
slat (usec): min=82, max=967, avg=112.97, stdev=15.55
clat (usec): min=1730, max=11732, avg=2200.57, stdev=44.01
lat (usec): min=1837, max=12700, avg=2313.54, stdev=45.45
clat percentiles (usec):
| 1.00th=[ 2147], 5.00th=[ 2180], 10.00th=[ 2180], 20.00th=[ 2180],
| 30.00th=[ 2180], 40.00th=[ 2180], 50.00th=[ 2212], 60.00th=[ 2212],
| 70.00th=[ 2212], 80.00th=[ 2212], 90.00th=[ 2212], 95.00th=[ 2212],
| 99.00th=[ 2212], 99.50th=[ 2245], 99.90th=[ 2278], 99.95th=[ 2802],
| 99.99th=[ 2933]
bw ( MiB/s): min= 6604, max= 6924, per=100.00%, avg=6919.13, stdev=29.14, samples=119
iops : min= 6604, max= 6924, avg=6919.13, stdev=29.14, samples=119
lat (msec) : 2=0.05%, 4=99.94%, 10=0.01%, 20=0.01%
cpu : usr=1.29%, sys=88.29%, ctx=373362, majf=0, minf=4110
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=414807,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=6913MiB/s (7249MB/s), 6913MiB/s-6913MiB/s (7249MB/s-7249MB/s), io=405GiB (435GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=3311288/0, merge=0/0, ticks=7132629/0, in_queue=7132629, util=99.92%
HDS-MMN-MTFDKBA480TFR1BC - Micron 7450 PRO 480GB NVMe PCIe 4.0 M.2 22x80mm 3D TLC
fio seq read (Micron MTFDKBA480TFR1BC)
fio --filename=/dev/nvme2n1 --rw=read --direct=1 --bs=1M --ioengine=libaio --runtime=60 --numjobs=1 --time_based --group_reporting --name=seq_read --iodepth=16
seq_read: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=5042MiB/s][r=5042 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=22330: Thu Mar 7 15:37:59 2024
read: IOPS=5000, BW=5001MiB/s (5244MB/s)(293GiB/60002msec)
slat (usec): min=64, max=848, avg=102.37, stdev=20.28
clat (usec): min=967, max=11332, avg=3096.30, stdev=225.10
lat (usec): min=1062, max=12181, avg=3198.67, stdev=224.67
clat percentiles (usec):
| 1.00th=[ 2933], 5.00th=[ 2999], 10.00th=[ 3032], 20.00th=[ 3032],
| 30.00th=[ 3064], 40.00th=[ 3064], 50.00th=[ 3064], 60.00th=[ 3097],
| 70.00th=[ 3097], 80.00th=[ 3097], 90.00th=[ 3130], 95.00th=[ 3163],
| 99.00th=[ 4113], 99.50th=[ 4948], 99.90th=[ 5800], 99.95th=[ 6128],
| 99.99th=[ 7439]
bw ( MiB/s): min= 3970, max= 5044, per=100.00%, avg=5005.58, stdev=158.93, samples=119
iops : min= 3970, max= 5044, avg=5005.58, stdev=158.93, samples=119
lat (usec) : 1000=0.01%
lat (msec) : 2=0.03%, 4=98.86%, 10=1.11%, 20=0.01%
cpu : usr=0.65%, sys=58.17%, ctx=276012, majf=0, minf=4108
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=300053,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=5001MiB/s (5244MB/s), 5001MiB/s-5001MiB/s (5244MB/s-5244MB/s), io=293GiB (315GB), run=60002-60002msec
Disk stats (read/write):
nvme2n1: ios=2396174/3, merge=0/1, ticks=7200202/0, in_queue=7200202, util=99.92%
Test 2
fio random read (Kioxia KCD6XLUL960G)
fio --filename=/dev/nvme0n1 --rw=randread --direct=1 --bs=1M --ioengine=libaio --runtime=60 --numjobs=1 --time_based --group_reporting --name=seq_read --iodepth=16
seq_read: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=2853MiB/s][r=2852 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=22407: Thu Mar 7 15:41:23 2024
read: IOPS=2848, BW=2849MiB/s (2987MB/s)(167GiB/60006msec)
slat (usec): min=111, max=948, avg=224.10, stdev=36.10
clat (usec): min=2371, max=11871, avg=5390.63, stdev=66.05
lat (usec): min=2526, max=12819, avg=5614.72, stdev=57.78
clat percentiles (usec):
| 1.00th=[ 5276], 5.00th=[ 5342], 10.00th=[ 5342], 20.00th=[ 5342],
| 30.00th=[ 5342], 40.00th=[ 5407], 50.00th=[ 5407], 60.00th=[ 5407],
| 70.00th=[ 5407], 80.00th=[ 5407], 90.00th=[ 5473], 95.00th=[ 5473],
| 99.00th=[ 5473], 99.50th=[ 5473], 99.90th=[ 5538], 99.95th=[ 5538],
| 99.99th=[ 6783]
bw ( MiB/s): min= 2808, max= 2854, per=100.00%, avg=2851.13, stdev= 4.24, samples=119
iops : min= 2808, max= 2854, avg=2851.14, stdev= 4.24, samples=119
lat (msec) : 4=0.01%, 10=99.99%, 20=0.01%
cpu : usr=0.80%, sys=74.14%, ctx=168755, majf=0, minf=4106
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=170938,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=2849MiB/s (2987MB/s), 2849MiB/s-2849MiB/s (2987MB/s-2987MB/s), io=167GiB (179GB), run=60006-60006msec
Disk stats (read/write):
nvme0n1: ios=1364996/0, merge=0/0, ticks=6979131/0, in_queue=6979131, util=99.91%
fio random read (Samsung MZQL2960HCJRA7)
fio --filename=/dev/nvme1n1 --rw=randread --direct=1 --bs=1M --ioengine=libaio --runtime=60 --numjobs=1 --time_based --group_reporting --name=seq_read --iodepth=16
seq_read: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=6922MiB/s][r=6921 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=22522: Thu Mar 7 15:42:48 2024
read: IOPS=6913, BW=6913MiB/s (7249MB/s)(405GiB/60003msec)
slat (usec): min=86, max=958, avg=109.43, stdev=13.04
clat (usec): min=1711, max=11925, avg=2204.04, stdev=43.59
lat (usec): min=1853, max=12884, avg=2313.47, stdev=45.94
clat percentiles (usec):
| 1.00th=[ 2147], 5.00th=[ 2180], 10.00th=[ 2180], 20.00th=[ 2180],
| 30.00th=[ 2212], 40.00th=[ 2212], 50.00th=[ 2212], 60.00th=[ 2212],
| 70.00th=[ 2212], 80.00th=[ 2212], 90.00th=[ 2212], 95.00th=[ 2212],
| 99.00th=[ 2212], 99.50th=[ 2245], 99.90th=[ 2376], 99.95th=[ 2802],
| 99.99th=[ 2835]
bw ( MiB/s): min= 6608, max= 6924, per=100.00%, avg=6919.16, stdev=28.78, samples=119
iops : min= 6608, max= 6924, avg=6919.16, stdev=28.78, samples=119
lat (msec) : 2=0.05%, 4=99.94%, 10=0.01%, 20=0.01%
cpu : usr=1.50%, sys=86.22%, ctx=382785, majf=0, minf=4107
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=414816,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=6913MiB/s (7249MB/s), 6913MiB/s-6913MiB/s (7249MB/s-7249MB/s), io=405GiB (435GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=3311521/0, merge=0/0, ticks=7133786/0, in_queue=7133786, util=99.91%
fio random read (Micron MTFDKBA480TFR1BC)
fio --filename=/dev/nvme2n1 --rw=randread --direct=1 --bs=1M --ioengine=libaio --runtime=60 --numjobs=1 --time_based --group_reporting --name=seq_read --iodepth=16
seq_read: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=16
fio-3.35
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=5043MiB/s][r=5043 IOPS][eta 00m:00s]
seq_read: (groupid=0, jobs=1): err= 0: pid=22590: Thu Mar 7 15:44:09 2024
read: IOPS=5037, BW=5037MiB/s (5282MB/s)(295GiB/60002msec)
slat (usec): min=66, max=920, avg=105.58, stdev=25.21
clat (usec): min=1572, max=12070, avg=3069.73, stdev=110.75
lat (usec): min=1687, max=12816, avg=3175.32, stdev=106.67
clat percentiles (usec):
| 1.00th=[ 2835], 5.00th=[ 2933], 10.00th=[ 2966], 20.00th=[ 2999],
| 30.00th=[ 3032], 40.00th=[ 3064], 50.00th=[ 3064], 60.00th=[ 3097],
| 70.00th=[ 3097], 80.00th=[ 3130], 90.00th=[ 3130], 95.00th=[ 3195],
| 99.00th=[ 3490], 99.50th=[ 3589], 99.90th=[ 3785], 99.95th=[ 3851],
| 99.99th=[ 4113]
bw ( MiB/s): min= 4930, max= 5046, per=100.00%, avg=5041.65, stdev=10.39, samples=119
iops : min= 4930, max= 5046, avg=5041.65, stdev=10.39, samples=119
lat (msec) : 2=0.01%, 4=99.98%, 10=0.02%, 20=0.01%
cpu : usr=0.81%, sys=59.64%, ctx=247929, majf=0, minf=4106
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=302253,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=5037MiB/s (5282MB/s), 5037MiB/s-5037MiB/s (5282MB/s-5282MB/s), io=295GiB (317GB), run=60002-60002msec
Disk stats (read/write):
nvme2n1: ios=2413346/154, merge=0/29, ticks=7210715/18, in_queue=7210733, util=99.92%
You can find more information about the server, the conducted tests, and their results on our YouTube channel.
1/5 Unboxing:
2/4 First boot
3/5 Power consumption: IDLE and STRESS
4/5 Performance Tests of AMD Epyc Rome 7542 32C/64T
5/5 Performance Tests of NVMe
Related pages:
- Test platform Supermicro NVMe 1029U-TN10RT + Intel Virtual RAID on CPU (VROC)
- "Phoronix Test Suite - a platform for benchmark testing."
- Supermicro AOC-SLG3-2M2 Controller Only recognizes One NVMe Drive, Why? (PCIe Bifurcation)
- Savings from using new CPUs in hybrid installations using renewable energy sources (RES)
- Updating NVMe Drive Firmware Using nvme-cli: Example with Samsung PM9A3
- How to Format an NVMe Drive? How to Permanently Erase Data from an NVMe Drive?