Re: About scsi device queue depth

[James Bottomley <jejb@linux.ibm.com>]

On Mon, 2021-01-11 at 16:21 +0000, John Garry wrote:
> Hi,
> 
> I was looking at some IOMMU issue on a LSI RAID 3008 card, and
> noticed that performance there is not what I get on other SAS HBAs -
> it's lower.
> 
> After some debugging and fiddling with sdev queue depth in mpt3sas 
> driver, I am finding that performance changes appreciably with sdev 
> queue depth:
> 
> sdev qdepth	fio number jobs* 	1	10	20
> 16					1590	1654	1660
> 32					1545	1646	1654
> 64					1436	1085	1070
> 254 (default)				1436	1070	1050
> 
> fio queue depth is 40, and I'm using 12x SAS SSDs.
> 
> I got comparable disparity in results for fio queue depth = 128 and
> num jobs = 1:
> 
> sdev qdepth	fio number jobs* 	1	
> 16					1640
> 32					1618	
> 64					1577	
> 254 (default)				1437	
> 
> IO sched = none.
> 
> That driver also sets queue depth tracking = 1, but never seems to
> kick in.
> 
> So it seems to me that the block layer is merging more bios per
> request, as averge sg count per request goes up from 1 - > upto 6 or
> more. As I see, when queue depth lowers the only thing that is really
> changing is that we fail more often in getting the budget in 
> scsi_mq_get_budget()->scsi_dev_queue_ready().
> 
> So initial sdev queue depth comes from cmd_per_lun by default or 
> manually setting in the driver via scsi_change_queue_depth(). It
> seems to me that some drivers are not setting this optimally, as
> above.
> 
> Thoughts on guidance for setting sdev queue depth? Could blk-mq
> changed this behavior?

In general, for spinning rust, you want the minimum queue depth
possible for keeping the device active because merging is a very
important performance enhancement and once the drive is fully occupied
simply sending more tags won't improve latency.  We used to recommend a
depth of about 4 for this reason.  A co-operative device can help you
find the optimal by returning QUEUE_FULL when it's fully occupied so we
have a mechanism to track the queue full returns and change the depth
interactively.

For high iops devices, these considerations went out of the window and
it's generally assumed (without varying evidence) the more tags the
better.  SSDs have a peculiar lifetime problem in that when they get
erase block starved they start behaving more like spinning rust in that
they reach a processing limit but only for writes, so lowering the
write queue depth (which we don't even have a knob for) might be a good
solution.  Trying to track the erase block problem has been a constant
bugbear.

I'm assuming you're using spinning rust in the above, so it sounds like
the firmware in the card might be eating the queue full returns.  I
could see this happening in RAID mode, but it shouldn't happen in jbod
mode.

James