>
> On 12/01/2021 11:44, Kashyap Desai wrote:
> >>>> loops = 10000
> >>> Is there any effect on random read IOPS when you decrease sdev queue
> >>> depth? For sequential IO, IO merge can be enhanced by that way.
> >>>
> >> Let me check...
> > John - Can you check your test with rq_affinity = 2  and nomerges=2 ?
> >
> > I have noticed similar drops (whatever you have reported) -  "once we
> > reach near pick of sdev or host queue depth, sometimes performance
> > drop due to contention."
> > But this behavior keep changing since kernel changes in this area is
> > very actively happened in past. So I don't know the exact details
> > about kernel version etc.
> > I have similar setup (16 SSDs) and I will try similar test on latest
> > kernel.
> >
> > BTW - I remember that rq_affinity=2 play major role in such issue.  I
> > usually do testing with rq_affinity = 2.
> >
>
> Hi Kashyap,
>
> As requested:
>
> rq_affinity=1, nomerges=0 (default)
>
> sdev queue depth	num jobs=1
> 8			1650
> 16			1638
> 32			1612
> 64			1573
> 254			1435 (default for LLDD)
>
> rq_affinity=2, nomerges=2
>
> sdev queue depth	num jobs=1
> 8			1236
> 16			1423
> 32			1438
> 64			1438
> 254			1438 (default for LLDD)
>
> Setup as original: fio read, 12x SAS SSDs

What is an issue with rq_affinity=2 and nomerges=2 ?
It looks like - "Dropping IOPs from peak (1.6M) is not an issue here but we
are not able to reach peak IOPs. IOPS increase gradually and it saturate."

>
> So, again, we see that performance changes from changing sdev queue depth
> depends on workload and then also other queue config.

Can you send me "dmesg" logs of your setup. What is host_busy counter while
you run your test ? I am not able to duplicate the issue on my setup.

To avoid any io merge issue, can we use rand_read data ?

Kashyap

>
> Thanks,
> John

-- 
This electronic communication and the information and any files transmitted 
with it, or attached to it, are confidential and are intended solely for 
the use of the individual or entity to whom it is addressed and may contain 
information that is confidential, legally privileged, protected by privacy 
laws, or otherwise restricted from disclosure to anyone else. If you are 
not the intended recipient or the person responsible for delivering the 
e-mail to the intended recipient, you are hereby notified that any use, 
copying, distributing, dissemination, forwarding, printing, or copying of 
this e-mail is strictly prohibited. If you received this e-mail in error, 
please return the e-mail to the sender, delete it from your computer, and 
destroy any printed copy of it.