Re: [Non-DoD Source] Can't get RAID5/RAID6 NVMe randomread IOPS - AMD ROME what am I missing?????

[Wols Lists <antlists@youngman.org.uk>]

On 29/07/21 01:54, Matt Wallis wrote:
> Hi Jim,
> 
> Totally get the Frankenstein’s monster aspect, I try not to build those where I can, but at the moment I don’t think there’s much that can be done about it.
> Not sure if LVM is better than MDRAID 0, it just gives you more control over the volumes that can be created, instead of having it all in one big chunk. If you just need one big chunk, then MDRAID 0 is probably fine.
> 
sticking raid 0 on top of raid 6 sounds an extremely weird thing to do.

What I guess you might be wanting to do instead is write a partition
table to the raid-6? That's perfectly normal if, imho, a bit unusual?

And LVM would be MUCH better than raid-0, I'm sure, because it addresses
this very issue by design, rather than by accident.

> I think if you can create a couple of scripts that allows the admin to fail a drive out of all the arrays that it’s in at once, then it's not that much worse than managing an MDRAID is normally. 

Is that wise? KISS.
> 
> Matt.
> 
>> On 28 Jul 2021, at 20:43, Finlayson, James M CIV (USA) <james.m.finlayson4.civ@mail.mil> wrote:
>>
>> Matt,
>> I have put as many as 32 partitions on a drive (based upon great advice from this list) and done RAID6 over them, but I was concerned about our sustainability long term.     As a researcher, I can do these cool science experiments, but I still have to hand designs  to sustainment folks.  I was also running into an issue of doing a mdraid RAID0 on top of the RAID6's so I could toss one  xfs file system on top each of the numa node's drives and the last RAID0 stripe of all of the RAID6's couldn't generate the queue depth needed.    We even recompiled the kernel to change the mdraid nr_request max from 128 to 1023.  
>>
>> I will have to try the LVM experiment.  I'm an LVM  neophyte, so it might take me the rest of today/tomorrow to get new results as I tend to let mdraid do all of its volume builds without forcing, so that will take a bit of time also.  Once might be able to argue that configuration isn't too much of a "Frankenstein's monster" for me to hand it off.
>>
Do. If it solves what you want, then it's worth it. I'm moving my stuff
over to LVM.

To throw something else into the mix, you've gone for raid 6, which
enables you to lose two drives, or corrupt one drive. Do you need the
two-drive redundancy? The calculations are a lot more expensive than
raid-5 if you're worried over write speed. I don't know the impact of it
but I'm playing with dm-integrity which provides some protection against
corruption.

>> Thanks,
>> Jim
>>
Cheers,
Wol