Re: Nr_requests mdraid

[Nikolay Kichukov <nikolay@oldum.net>]

Hello all,

On Mon, 2020-11-16 at 16:51 +0000, Finlayson, James M CIV (USA) wrote:
> On Wed, Oct 28, 2020 at 6:39 PM Vitaly Mayatskih <
> v.mayatskih@gmail.com> wrote
> > 
> > On Thu, Oct 22, 2020 at 2:56 AM Finlayson, James M CIV (USA) <
> > james.m.finlayson4.civ@mail.mil> wrote:
> > > 
> > > All,
> > > I'm working on creating raid5 or raid6 arrays of 800K IOP nvme
> > > drives.   Each of \
> > > the drives performs well with a queue depth of 128 and I set to
> > > 1023 if allowed.   \
> > > In order for me to try to max out the queue depth on each RAID
> > > member, so I'd like \
> > > to set the sysfs nr_requests on the md device to something greater
> > > than 128, like \
> > > #raid members * 128.   Even though
> > > /sys/block/md127/queue/nr_requests is mode 644, \
> > > when I try to change nr_requests in any way as root, I get write
> > > error: invalid \
> > > argument.  When I'm hitting the md device with random reads, my
> > > nvme drives are \
> > > 100% utilized, but only doing 160K IOPS because they have no queue
> > > depth. 
> > > Am I doing something silly?
> > 
> > It only works for blk-mq block devices. MD is not blk-mq.

Would it be possible to implement something similar to the use_blk_mq of
dm_mod on md_mod?

> > 
> > You can exchange simplicity for performance: instead of creating one
> > RAID-5/6 array you can partition drives in N equal sized partitions,
> > create N RAID-5/6 arrays using one partition from every disk, then
> > stripe them into top-level RAID-0. So that would be RAID-5+0 (or
> > 6+0).
> > 
> > It is awful, but simulates multiqueue and performs better in
> > parallel
> > loads. Especially for writes (on RAID-5/6).
> > 
> > 
> > -- 
> > wbr, Vitaly
> 
> Vitaly,
> Thank you for the tip.  My raid5 performance (after creating 32
> partitions per SSD) and running 64  9+1 (2 in reality) stripes is up
> to 11.4M 4K random read IOPS, out of 17M that the box is capable,
> which I'm happy with, because I can't NUMA the raid stripes as I would
> the individual SSDs themselves.   However, when I perform the RAID0
> striping to make the "RAID50 from hell", my performance drops to 7.1M
> 4K random read IOPS.   Any suggestions?  The last RAID50, again won't
> let me generate the queue depth.
> 
> Thanks in advance,
> Jim
> 
> 
>