Re: Can't get RAID5/RAID6 NVMe randomread IOPS - AMD ROME what am I missing?????

[Matt Wallis <mattw@madmonks.org>]

Hi Jim,

> On 28 Jul 2021, at 06:32, Finlayson, James M CIV (USA) <james.m.finlayson4.civ@mail.mil> wrote:
> 
> Sorry, this will be a long email with everything I find to be relevant, but I can get over 110GB/s of 4kB random reads from individual NVMe SSDs, but I'm at a loss why mdraid can only do a very small  fraction of it.   I'm at my "organizational world record" for sustained IOPS, but I need protected IOPS to do something useful.     This is everything I do to a server to make the I/O crank.....My role is that of a lab researcher/resident expert/consultant.   I'm just stumped why I can't do better.   If there is a fine manual that somebody can point me to, I'm happy to read it…

I am probably going to get corrected on some if not all of this, but from what I understand, and from my own little experiments on a similar Intel based system…
1. NVMe is stupid fast, you need a good chunk of CPU performance to max it out.
2. Most block IO in the kernel is limited in terms of threading, it may even be essentially single threaded. (This is where I will get corrected)
3. AFAICT, this includes mdraid, there’s a single thread per RAID device handling all the RAID calculations. (mdX_raid6)

What I did to get IOPs up in a system with 24 NVMe, split up into 12 per NUMA domain.
1. Create 8 partitions on each drive (this may be overkill, I just started here for some reason)
2. Create 8 RAID6 arrays with 1 partition per drive.
3. Use LVM to create a single striped logical volume over all 8 RAID volumes. RAID 0+6 as it were.

You now have an LVM thread that is basically doing nothing more than chunking the data as it comes in, then, sending the chunks to 8 separate RAID devices, each with their own threads, buffers, queues etc, all of which can be spread over more cores.

I saw a significant (for me, significant is >20%) increase in IOPs doing this. 

You still have RAID6 protection, but you might want to write a couple of scripts to help you manage the arrays, because a single failed drive now needs to be removed from 8 RAID volumes. 

There’s not a lot of capacity lost doing this, pretty sure I lost less than 100MB to the partitions and the RAID overhead.

You would never consider this on spinning disk of course, way to slow and you’re just going to make it slower, NVMe as you noticed has the IOPs to spare, so I’m pretty sure it’s just that we’re not able to get the data to it fast enough.

Matt