Performance of PCIe Gen 4.0 NVMe Drives on the Supermicro AOC-S3816L-L8IR Controller in JBOD, mdadm, and RAID 10 Configurations

| Tests
Facebook Twitter Clipboard

PCIe Gen 4.0 NVMe Drive Performance on the Supermicro AOC-S3816L-L8IR Controller in JBOD, mdadm, and RAID 10 Configurations

In this article, a performance comparison of PCIe Gen 4.0 NVMe drives connected to the AOC-S3816L-L8IR RAID controller was conducted in three different configurations:

  • JBOD mode,
  • RAID 10 managed via mdadm,
  • RAID 10 implemented via hardware controller.

In JBOD mode, the performance of two drives was compared: Intel HDS-IUN0-SSDPF2KE016T1 and Kioxia HDS-TUN0-KCD6XLUL7T68.

The tests were performed on the ITC R4R test server, equipped with an AMD Raphael 16C/32T processor running at 4.5 GHz, 64 MB of cache, and 192 GB of RAM.

Performance tests were conducted using fio, one of the most popular programs for testing disks and storage systems. Thanks to its versatility and precision, fio enables the simulation of various workload scenarios, both for random and sequential operations, providing a thorough understanding of drive performance in different RAID configurations. The tool generates detailed reports covering parameters such as bandwidth, IO operations, and latency, allowing for a comprehensive analysis of the results in the context of different RAID modes and disk configurations.

Conclusions for Intel /dev/sda and Kioxia /dev/sdd in JBOD Mode:

  1. Random Write IOPS (4 KB per single I/O):
    Intel /dev/sda: 1698 MB/s (99.5 GB/s)
    Kioxia /dev/sdd: 1874 MB/s (110 GB/s)
    Conclusion: Kioxia performs better in random write operations, both in bandwidth and I/O operations.

  2. Random Read IOPS (4 KB per single I/O):
    Intel /dev/sda: 1199 MB/s (70.3 GB/s)
    Kioxia /dev/sdd: 1947 MB/s (114 GB/s)
    Conclusion: Kioxia offers better performance in random read operations.

  3. Sequential Write Throughput (128 KB per single I/O):
    Intel /dev/sda: 1190 MB/s (69.7 GB/s)
    Kioxia /dev/sdd: 1204 MB/s (70.5 GB/s)
    Conclusion: Kioxia shows slightly better sequential write performance, but the difference is minimal.

  4. Sequential Read Throughput (128 KB per single I/O):
    Intel /dev/sda: 629 MB/s (36.8 GB/s)
    Kioxia /dev/sdd: 892 MB/s (52.3 GB/s)
    Conclusion: Kioxia achieves better performance in sequential read operations.

  5. Random Write Latency (4 KB per single I/O):
    Intel /dev/sda: 160 MB/s (9.61 GB/s)
    Kioxia /dev/sdd: 56.5 MB/s (3.39 GB/s)
    Conclusion: Intel has higher bandwidth, meaning it is faster in random write operations.

  6. Random Read Latency (4 KB per single I/O):
    Intel /dev/sda: 10.9 MB/s (0.65 GB/s)
    Kioxia /dev/sdd: 10.8 MB/s (0.64 GB/s)
    Conclusion: Intel has slightly higher bandwidth, indicating better performance in random read operations.

  7. Sequential Write Latency (4 KB per single I/O):
    Intel /dev/sda: 145 MB/s (8.73 GB/s)
    Kioxia /dev/sdd: 57.3 MB/s (3.43 GB/s)
    Conclusion: Intel shows better performance in sequential write operations.

  8. Sequential Read Latency (4 KB per single I/O):
    Intel /dev/sda: 105 MB/s (6.3 GB/s)
    Kioxia /dev/sdd: 10.8 MB/s (0.64 GB/s)
    Conclusion: Intel offers better bandwidth in sequential read operations.

Conclusions for RAID 10 via mdadm and RAID 10 via RAID Controller:

  1. Random Write IOPS (4 KB per single I/O):
    RAID 10 via mdadm: 636 MB/s (37.7 GB/s)
    RAID 10 via RAID controller: 1736 MB/s (102 GB/s)
    Conclusion: RAID 10 via RAID controller is significantly faster in random write operations, achieving better results.

  2. Random Read IOPS (4 KB per single I/O):
    RAID 10 via mdadm: 1770 MB/s (104 GB/s)
    RAID 10 via RAID controller: 2124 MB/s (124 GB/s)
    Conclusion: RAID 10 via RAID controller also performs better in random read operations.

  3. Sequential Write Throughput (128 KB per single I/O):
    RAID 10 via mdadm: 957 MB/s (56.1 GB/s)
    RAID 10 via RAID controller: 3718 MB/s (218 GB/s)
    Conclusion: RAID 10 via RAID controller has significantly higher throughput in sequential writes.

  4. Sequential Read Throughput (128 KB per single I/O):
    RAID 10 via mdadm: 1200 MB/s (70.3 GB/s)
    RAID 10 via RAID controller: 12390.4 MB/s (725 GB/s)
    Conclusion: RAID 10 via RAID controller has an astonishingly higher throughput in sequential read operations.

  5. Random Write Latency (4 KB per single I/O):
    RAID 10 via mdadm: 12.5 MB/s (0.74 GB/s)
    RAID 10 via RAID controller: 190 MB/s (11.1 GB/s)
    Conclusion: RAID 10 via RAID controller shows significantly higher bandwidth.

  6. Random Read Latency (4 KB per single I/O):
    RAID 10 via mdadm: 9.13 MB/s (0.53 GB/s)
    RAID 10 via RAID controller: 10.6 MB/s (0.63 GB/s)
    Conclusion: RAID 10 via RAID controller has slightly higher bandwidth.

  7. Sequential Write Latency (4 KB per single I/O):
    RAID 10 via mdadm: 13.6 MB/s (0.81 GB/s)
    RAID 10 via RAID controller: 166 MB/s (0.99 GB/s)
    Conclusion: RAID 10 via RAID controller has significantly better throughput in sequential write operations.

  8. Sequential Read Latency (4 KB per single I/O):
    RAID 10 via mdadm: 9.88 MB/s (0.59 GB/s)
    RAID 10 via RAID controller: 90.2 MB/s (0.54 GB/s)
    Conclusion: RAID 10 via RAID controller shows better throughput in sequential read operations.

Summary:

Choosing Between JBOD, mdadm, and RAID 10 via RAID Controller:

  • JBOD (Intel and Kioxia):
    Pros: Drives in JBOD mode offer varying benefits depending on the type of operation. Kioxia excels in random (both write and read) and sequential (write and read) operations, while Intel offers better latency, which can be beneficial in scenarios where response time is critical.
    Cons: Lack of redundancy, which may lead to data loss if one of the drives fails.

  • RAID 10 via mdadm:
    Pros: Offers better latency in random and sequential operations, especially in write operations. It may be a better choice if performance with lower latency is a priority.
    Cons: Lower throughput compared to hardware RAID 10.

  • RAID 10 via RAID Controller:
    Pros: Highest performance, both in terms of bandwidth and I/O operations. Ideal for environments that demand high data transfer speeds, such as database servers or virtualization environments.
    Cons: Higher cost for hardware solutions, which may not be necessary in less demanding environments.

Recommendation for the Client:

  • If maximum performance in bandwidth and I/O operations (especially sequential read and write) is crucial, RAID 10 via RAID controller is the best solution.
  • JBOD may be a cost-effective option, but it does not offer redundancy and can be risky in the event of a disk failure.

Related pages:

Related Pages

  1. Intel Virtual RAID CPU (KEY VROC) - AOC-VROCINTMOD & AOC-VROCSTNMOD & AOC-VROCPREMOD
  2. RAID Types 0, 1, 5, 6, 10, 50, 60 & RAID Calculator (What RAID level to choose for a server?)
  3. RAID solution with GPU acceleration - GRAID SupremeRAID™ (RAID 0/1/10/5/6 on NVMe drives)
  4. "Phoronix Test Suite - a platform for benchmark testing."
  5. NVMe optimized Supermicro Chassis (Native NVMe)(Matrix)
  6. Tests of a Budget 1U AMD Epyc Server for 4 NVMe Gen 4.0 Drives (Will the BPN-SAS3-815TQ-N4 Support NVMe Gen 4.0 Drives?)
  7. Savings from using new CPUs in hybrid installations using renewable energy sources (RES)