Question regarding output of btrace and use of blk-mq

From: "Karan Tilak Kumar (kartilak)" <kartilak@cisco.com>
To: linux-btrace@vger.kernel.org
Subject: Question regarding output of btrace and use of blk-mq
Date: Thu, 18 Nov 2021 02:17:23 +0000	[thread overview]
Message-ID: <SJ0PR11MB58966D3BE29649A350F098AEC39B9@SJ0PR11MB5896.namprd11.prod.outlook.com> (raw)

Hello btrace community,

I came across blktrace and blkparse tools as a part of my learning on blk-mq. 
I'm trying to test some code in my driver and I want to test a few things by turning multiqueue on and off, and compare results.
I'm running IOs using medusa on my multipath device /dev/dm-2

When I turn off mq, I still see IOs queued from each CPU:

CPU0 (dm-2):
Reads Queued:       1,980,  199,438KiB  Writes Queued:       1,344,  172,032KiB
Read Dispatches:    3,429,  199,894KiB  Write Dispatches:    2,688,  172,032KiB
Reads Requeued:         0               Writes Requeued:         0
Reads Completed:    6,826,  199,130KiB  Writes Completed:    5,376,  172,032KiB
Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
Read depth:            70               Write depth:            64
IO unplugs:         3,177               Timer unplugs:           0
CPU1 (dm-2):
Reads Queued:       2,101,  247,313KiB  Writes Queued:       1,856,  237,568KiB
Read Dispatches:    3,986,  247,177KiB  Write Dispatches:    3,709,  237,376KiB
Reads Requeued:         0               Writes Requeued:         0
Reads Completed:    7,975,  247,177KiB  Writes Completed:    7,545,  241,344KiB
Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
Read depth:            70               Write depth:            64
IO unplugs:         3,899               Timer unplugs:           0
CPU2 (dm-2):
Reads Queued:       1,907,  224,925KiB  Writes Queued:       1,760,  225,280KiB
Read Dispatches:    3,635,  224,793KiB  Write Dispatches:    3,524,  225,536KiB
Reads Requeued:         0               Writes Requeued:         0
Reads Completed:    7,273,  224,793KiB  Writes Completed:    7,044,  225,536KiB
Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
Read depth:            70               Write depth:            64
IO unplugs:         3,625               Timer unplugs:           1
CPU3 (dm-2):
Reads Queued:       1,572,  173,940KiB  Writes Queued:       1,408,  180,224KiB
Read Dispatches:    2,885,  173,753KiB  Write Dispatches:    2,815,  180,160KiB
Reads Requeued:         0               Writes Requeued:         0
Reads Completed:    5,766,  173,541KiB  Writes Completed:    5,631,  180,160KiB
Read Merges:            0,        0KiB  Write Merges:            0,        0KiB
Read depth:            70               Write depth:            64
IO unplugs:         2,922               Timer unplugs:           0

#cat /sys/module/scsi_mod/parameters/use_blk_mq
N
#cat /sys/module/dm_mod/parameters/use_blk_mq
N

When multiqueue is off, we don't expect IOs from all CPUs. Is this correct?
I've tried running medusa with one thread, and with eight threads on separate occasions, and I see IOs from all CPUs each time.
Is this expected? Please advise.

Thanks,
Karan