* [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix
@ 2020-07-09 12:23 John Garry
2020-07-09 12:23 ` [PATCH v2 1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init John Garry
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: John Garry @ 2020-07-09 12:23 UTC (permalink / raw)
To: jejb, martin.petersen
Cc: linux-scsi, hare, dgilbert, ming.lei, kashyap.desai, John Garry
This series includes hostwide tags support, so we can mimic some SCSI HBAs,
like megaraid sas or hisi_sas.
There is also a fix for an out-of-range module param.
Differences to v1:
- add Doug's Ack for patch 1/2
- sort params alphabetically
- fix max queue at host max queue (when non-zero)
- drop host max queue file write permission
John Garry (2):
scsi: scsi_debug: Add check for sdebug_max_queue during module init
scsi: scsi_debug: Support hostwide tags
drivers/scsi/scsi_debug.c | 86 +++++++++++++++++++++++++++++++++------
1 file changed, 74 insertions(+), 12 deletions(-)
--
2.26.2
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH v2 1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init
2020-07-09 12:23 [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix John Garry
@ 2020-07-09 12:23 ` John Garry
2020-07-09 13:18 ` Ming Lei
2020-07-09 12:23 ` [PATCH v2 2/2] scsi: scsi_debug: Support hostwide tags John Garry
2020-07-14 4:58 ` [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix Martin K. Petersen
2 siblings, 1 reply; 7+ messages in thread
From: John Garry @ 2020-07-09 12:23 UTC (permalink / raw)
To: jejb, martin.petersen
Cc: linux-scsi, hare, dgilbert, ming.lei, kashyap.desai, John Garry
sdebug_max_queue should not exceed SDEBUG_CANQUEUE, otherwise crashes like
this can be triggered by passing an out-of-range value:
Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI RC0 - V1.16.01 03/15/2019
pstate: 20400009 (nzCv daif +PAN -UAO BTYPE=--)
pc : schedule_resp+0x2a4/0xa70 [scsi_debug]
lr : schedule_resp+0x52c/0xa70 [scsi_debug]
sp : ffff800022ab36f0
x29: ffff800022ab36f0 x28: ffff0023a935a610
x27: ffff800008e0a648 x26: 0000000000000003
x25: ffff0023e84f3200 x24: 00000000003d0900
x23: 0000000000000000 x22: 0000000000000000
x21: ffff0023be60a320 x20: ffff0023be60b538
x19: ffff800008e13000 x18: 0000000000000000
x17: 0000000000000000 x16: 0000000000000000
x15: 0000000000000000 x14: 0000000000000000
x13: 0000000000000000 x12: 0000000000000000
x11: 0000000000000000 x10: 0000000000000000
x9 : 0000000000000001 x8 : 0000000000000000
x7 : 0000000000000000 x6 : 00000000000000c1
x5 : 0000020000200000 x4 : dead0000000000ff
x3 : 0000000000000200 x2 : 0000000000000200
x1 : ffff800008e13d88 x0 : 0000000000000000
Call trace:
schedule_resp+0x2a4/0xa70 [scsi_debug]
scsi_debug_queuecommand+0x2c4/0x9e0 [scsi_debug]
scsi_queue_rq+0x698/0x840
__blk_mq_try_issue_directly+0x108/0x228
blk_mq_request_issue_directly+0x58/0x98
blk_mq_try_issue_list_directly+0x5c/0xf0
blk_mq_sched_insert_requests+0x18c/0x200
blk_mq_flush_plug_list+0x11c/0x190
blk_flush_plug_list+0xdc/0x110
blk_finish_plug+0x38/0x210
blkdev_direct_IO+0x450/0x4d8
generic_file_read_iter+0x84/0x180
blkdev_read_iter+0x3c/0x50
aio_read+0xc0/0x170
io_submit_one+0x5c8/0xc98
__arm64_sys_io_submit+0x1b0/0x258
el0_svc_common.constprop.3+0x68/0x170
do_el0_svc+0x24/0x90
el0_sync_handler+0x13c/0x1a8
el0_sync+0x158/0x180
Code: 528847e0 72a001e0 6b00003f 540018cd (3941c340)
In addition, it should not be less than 1.
So add checks for these, and fail the module init for those cases.
Fixes: c483739430f10 ("scsi_debug: add multiple queue support")
Acked-by: Douglas Gilbert <dgilbert@interlog.com>
Signed-off-by: John Garry <john.garry@huawei.com>
---
drivers/scsi/scsi_debug.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
index 4692f5b6ad13..68534a23866e 100644
--- a/drivers/scsi/scsi_debug.c
+++ b/drivers/scsi/scsi_debug.c
@@ -6613,6 +6613,12 @@ static int __init scsi_debug_init(void)
pr_err("submit_queues must be 1 or more\n");
return -EINVAL;
}
+
+ if ((sdebug_max_queue > SDEBUG_CANQUEUE) || (sdebug_max_queue <= 0)) {
+ pr_err("max_queue must be in range [1, %d]\n", SDEBUG_CANQUEUE);
+ return -EINVAL;
+ }
+
sdebug_q_arr = kcalloc(submit_queues, sizeof(struct sdebug_queue),
GFP_KERNEL);
if (sdebug_q_arr == NULL)
--
2.26.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH v2 2/2] scsi: scsi_debug: Support hostwide tags
2020-07-09 12:23 [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix John Garry
2020-07-09 12:23 ` [PATCH v2 1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init John Garry
@ 2020-07-09 12:23 ` John Garry
2020-07-09 15:25 ` Douglas Gilbert
2020-07-14 4:58 ` [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix Martin K. Petersen
2 siblings, 1 reply; 7+ messages in thread
From: John Garry @ 2020-07-09 12:23 UTC (permalink / raw)
To: jejb, martin.petersen
Cc: linux-scsi, hare, dgilbert, ming.lei, kashyap.desai, John Garry
Many SCSI HBAs support a hostwide tagset, whereby each command submitted
to the HW from all submission queues must have a unique tag identifier.
Normally this unique tag will be in the range [0, max queue), where "max
queue" is the depth of each of the submission queues.
Add support for this hostwide tag feature, via module parameter
"host_max_queue". A non-zero value means that the feature is enabled. In
this case, the submission queues are not exposed to upper layer, i.e. from
blk-mq prespective, the device has a single hw queue. There are 2 reasons
for this:
a. it is assumed that the host can support nr_hw_queues * can_queue
commands, but this is not true for hostwide tags
b. for nr_hw_queues != 0, the request tag is not unique over all HW queues,
and some HBA drivers want to use this tag for the hostwide tag
However, like many SCSI HBA drivers today - megaraid sas being an example -
the full set of HW submission queues are still used in the LLDD driver. So
instead of using a complicated "reply_map" to create a per-CPU submission
queue mapping like megaraid sas (as it depends on a PCI device + MSIs) -
use a simple algorithm:
hwq = cpu % queue count
If the host max queue param is set non-zero, then the max queue depth is
fixed at this value also.
If and when hostwide shared tags are supported in blk-mq/scsi mid-layer,
then the policy to set nr_hw_queues = 0 for hostwide tags can be revised.
Signed-off-by: John Garry <john.garry@huawei.com>
---
drivers/scsi/scsi_debug.c | 80 +++++++++++++++++++++++++++++++++------
1 file changed, 68 insertions(+), 12 deletions(-)
diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
index 68534a23866e..2246d721ec88 100644
--- a/drivers/scsi/scsi_debug.c
+++ b/drivers/scsi/scsi_debug.c
@@ -344,6 +344,7 @@ struct sdebug_defer {
struct execute_work ew;
int sqa_idx; /* index of sdebug_queue array */
int qc_idx; /* index of sdebug_queued_cmd array within sqa_idx */
+ int hc_idx; /* hostwide tag index */
int issuing_cpu;
bool init_hrt;
bool init_wq;
@@ -759,6 +760,7 @@ static int sdebug_dsense = DEF_D_SENSE;
static int sdebug_every_nth = DEF_EVERY_NTH;
static int sdebug_fake_rw = DEF_FAKE_RW;
static unsigned int sdebug_guard = DEF_GUARD;
+static int sdebug_host_max_queue; /* per host */
static int sdebug_lowest_aligned = DEF_LOWEST_ALIGNED;
static int sdebug_max_luns = DEF_MAX_LUNS;
static int sdebug_max_queue = SDEBUG_CANQUEUE; /* per submit queue */
@@ -4707,15 +4709,28 @@ static int resp_rwp_zone(struct scsi_cmnd *scp, struct sdebug_dev_info *devip)
static struct sdebug_queue *get_queue(struct scsi_cmnd *cmnd)
{
- u32 tag = blk_mq_unique_tag(cmnd->request);
- u16 hwq = blk_mq_unique_tag_to_hwq(tag);
+ u16 hwq;
- pr_debug("tag=%#x, hwq=%d\n", tag, hwq);
- if (WARN_ON_ONCE(hwq >= submit_queues))
- hwq = 0;
+ if (sdebug_host_max_queue) {
+ /* Provide a simple method to choose the hwq */
+ hwq = smp_processor_id() % submit_queues;
+ } else {
+ u32 tag = blk_mq_unique_tag(cmnd->request);
+
+ hwq = blk_mq_unique_tag_to_hwq(tag);
+
+ pr_debug("tag=%#x, hwq=%d\n", tag, hwq);
+ if (WARN_ON_ONCE(hwq >= submit_queues))
+ hwq = 0;
+ }
return sdebug_q_arr + hwq;
}
+static u32 get_tag(struct scsi_cmnd *cmnd)
+{
+ return blk_mq_unique_tag(cmnd->request);
+}
+
/* Queued (deferred) command completions converge here. */
static void sdebug_q_cmd_complete(struct sdebug_defer *sd_dp)
{
@@ -4747,8 +4762,8 @@ static void sdebug_q_cmd_complete(struct sdebug_defer *sd_dp)
scp = sqcp->a_cmnd;
if (unlikely(scp == NULL)) {
spin_unlock_irqrestore(&sqp->qc_lock, iflags);
- pr_err("scp is NULL, sqa_idx=%d, qc_idx=%d\n",
- sd_dp->sqa_idx, qc_idx);
+ pr_err("scp is NULL, sqa_idx=%d, qc_idx=%d, hc_idx=%d\n",
+ sd_dp->sqa_idx, qc_idx, sd_dp->hc_idx);
return;
}
devip = (struct sdebug_dev_info *)scp->device->hostdata;
@@ -5451,6 +5466,10 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip,
new_sd_dp = false;
}
+ /* Set the hostwide tag */
+ if (sdebug_host_max_queue)
+ sd_dp->hc_idx = get_tag(cmnd);
+
if (ndelay > 0 && ndelay < INCLUSIVE_TIMING_MAX_NS)
ns_from_boot = ktime_get_boottime_ns();
@@ -5572,6 +5591,7 @@ module_param_named(every_nth, sdebug_every_nth, int, S_IRUGO | S_IWUSR);
module_param_named(fake_rw, sdebug_fake_rw, int, S_IRUGO | S_IWUSR);
module_param_named(guard, sdebug_guard, uint, S_IRUGO);
module_param_named(host_lock, sdebug_host_lock, bool, S_IRUGO | S_IWUSR);
+module_param_named(host_max_queue, sdebug_host_max_queue, int, S_IRUGO);
module_param_string(inq_product, sdebug_inq_product_id,
sizeof(sdebug_inq_product_id), S_IRUGO | S_IWUSR);
module_param_string(inq_rev, sdebug_inq_product_rev,
@@ -5642,6 +5662,8 @@ MODULE_PARM_DESC(every_nth, "timeout every nth command(def=0)");
MODULE_PARM_DESC(fake_rw, "fake reads/writes instead of copying (def=0)");
MODULE_PARM_DESC(guard, "protection checksum: 0=crc, 1=ip (def=0)");
MODULE_PARM_DESC(host_lock, "host_lock is ignored (def=0)");
+MODULE_PARM_DESC(host_max_queue,
+ "host max # of queued cmds (0 to max(def) [max_queue fixed equal for !0])");
MODULE_PARM_DESC(inq_product, "SCSI INQUIRY product string (def=\"scsi_debug\")");
MODULE_PARM_DESC(inq_rev, "SCSI INQUIRY revision string (def=\""
SDEBUG_VERSION "\")");
@@ -6141,7 +6163,8 @@ static ssize_t max_queue_store(struct device_driver *ddp, const char *buf,
struct sdebug_queue *sqp;
if ((count > 0) && (1 == sscanf(buf, "%d", &n)) && (n > 0) &&
- (n <= SDEBUG_CANQUEUE)) {
+ (n <= SDEBUG_CANQUEUE) &&
+ (sdebug_host_max_queue == 0)) {
block_unblock_all_queues(true);
k = 0;
for (j = 0, sqp = sdebug_q_arr; j < submit_queues;
@@ -6164,6 +6187,17 @@ static ssize_t max_queue_store(struct device_driver *ddp, const char *buf,
}
static DRIVER_ATTR_RW(max_queue);
+static ssize_t host_max_queue_show(struct device_driver *ddp, char *buf)
+{
+ return scnprintf(buf, PAGE_SIZE, "%d\n", sdebug_host_max_queue);
+}
+
+/*
+ * Since this is used for .can_queue, and we get the hc_idx tag from the bitmap
+ * in range [0, sdebug_host_max_queue), we can't change it.
+ */
+static DRIVER_ATTR_RO(host_max_queue);
+
static ssize_t no_uld_show(struct device_driver *ddp, char *buf)
{
return scnprintf(buf, PAGE_SIZE, "%d\n", sdebug_no_uld);
@@ -6503,6 +6537,7 @@ static struct attribute *sdebug_drv_attrs[] = {
&driver_attr_ptype.attr,
&driver_attr_dsense.attr,
&driver_attr_fake_rw.attr,
+ &driver_attr_host_max_queue.attr,
&driver_attr_no_lun_0.attr,
&driver_attr_num_tgts.attr,
&driver_attr_dev_size_mb.attr,
@@ -6619,6 +6654,20 @@ static int __init scsi_debug_init(void)
return -EINVAL;
}
+ if ((sdebug_host_max_queue > SDEBUG_CANQUEUE) ||
+ (sdebug_host_max_queue < 0)) {
+ pr_err("host_max_queue must be in range [0 %d]\n",
+ SDEBUG_CANQUEUE);
+ return -EINVAL;
+ }
+
+ if (sdebug_host_max_queue &&
+ (sdebug_max_queue != sdebug_host_max_queue)) {
+ sdebug_max_queue = sdebug_host_max_queue;
+ pr_warn("fixing max submit queue depth to host max queue depth, %d\n",
+ sdebug_max_queue);
+ }
+
sdebug_q_arr = kcalloc(submit_queues, sizeof(struct sdebug_queue),
GFP_KERNEL);
if (sdebug_q_arr == NULL)
@@ -7257,7 +7306,10 @@ static int sdebug_driver_probe(struct device *dev)
sdbg_host = to_sdebug_host(dev);
- sdebug_driver_template.can_queue = sdebug_max_queue;
+ if (sdebug_host_max_queue)
+ sdebug_driver_template.can_queue = sdebug_host_max_queue;
+ else
+ sdebug_driver_template.can_queue = sdebug_max_queue;
if (!sdebug_clustering)
sdebug_driver_template.dma_boundary = PAGE_SIZE - 1;
@@ -7272,9 +7324,13 @@ static int sdebug_driver_probe(struct device *dev)
my_name, submit_queues, nr_cpu_ids);
submit_queues = nr_cpu_ids;
}
- /* Decide whether to tell scsi subsystem that we want mq */
- /* Following should give the same answer for each host */
- hpnt->nr_hw_queues = submit_queues;
+ /*
+ * Decide whether to tell scsi subsystem that we want mq. The
+ * following should give the same answer for each host. If the host
+ * has a limit of hostwide max commands, then do not set.
+ */
+ if (!sdebug_host_max_queue)
+ hpnt->nr_hw_queues = submit_queues;
sdbg_host->shost = hpnt;
*((struct sdebug_host_info **)hpnt->hostdata) = sdbg_host;
--
2.26.2
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH v2 1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init
2020-07-09 12:23 ` [PATCH v2 1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init John Garry
@ 2020-07-09 13:18 ` Ming Lei
0 siblings, 0 replies; 7+ messages in thread
From: Ming Lei @ 2020-07-09 13:18 UTC (permalink / raw)
To: John Garry
Cc: jejb, martin.petersen, linux-scsi, hare, dgilbert, kashyap.desai
On Thu, Jul 09, 2020 at 08:23:19PM +0800, John Garry wrote:
> sdebug_max_queue should not exceed SDEBUG_CANQUEUE, otherwise crashes like
> this can be triggered by passing an out-of-range value:
>
> Hardware name: Huawei D06 /D06, BIOS Hisilicon D06 UEFI RC0 - V1.16.01 03/15/2019
> pstate: 20400009 (nzCv daif +PAN -UAO BTYPE=--)
> pc : schedule_resp+0x2a4/0xa70 [scsi_debug]
> lr : schedule_resp+0x52c/0xa70 [scsi_debug]
> sp : ffff800022ab36f0
> x29: ffff800022ab36f0 x28: ffff0023a935a610
> x27: ffff800008e0a648 x26: 0000000000000003
> x25: ffff0023e84f3200 x24: 00000000003d0900
> x23: 0000000000000000 x22: 0000000000000000
> x21: ffff0023be60a320 x20: ffff0023be60b538
> x19: ffff800008e13000 x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000
> x15: 0000000000000000 x14: 0000000000000000
> x13: 0000000000000000 x12: 0000000000000000
> x11: 0000000000000000 x10: 0000000000000000
> x9 : 0000000000000001 x8 : 0000000000000000
> x7 : 0000000000000000 x6 : 00000000000000c1
> x5 : 0000020000200000 x4 : dead0000000000ff
> x3 : 0000000000000200 x2 : 0000000000000200
> x1 : ffff800008e13d88 x0 : 0000000000000000
> Call trace:
> schedule_resp+0x2a4/0xa70 [scsi_debug]
> scsi_debug_queuecommand+0x2c4/0x9e0 [scsi_debug]
> scsi_queue_rq+0x698/0x840
> __blk_mq_try_issue_directly+0x108/0x228
> blk_mq_request_issue_directly+0x58/0x98
> blk_mq_try_issue_list_directly+0x5c/0xf0
> blk_mq_sched_insert_requests+0x18c/0x200
> blk_mq_flush_plug_list+0x11c/0x190
> blk_flush_plug_list+0xdc/0x110
> blk_finish_plug+0x38/0x210
> blkdev_direct_IO+0x450/0x4d8
> generic_file_read_iter+0x84/0x180
> blkdev_read_iter+0x3c/0x50
> aio_read+0xc0/0x170
> io_submit_one+0x5c8/0xc98
> __arm64_sys_io_submit+0x1b0/0x258
> el0_svc_common.constprop.3+0x68/0x170
> do_el0_svc+0x24/0x90
> el0_sync_handler+0x13c/0x1a8
> el0_sync+0x158/0x180
> Code: 528847e0 72a001e0 6b00003f 540018cd (3941c340)
>
> In addition, it should not be less than 1.
>
> So add checks for these, and fail the module init for those cases.
>
> Fixes: c483739430f10 ("scsi_debug: add multiple queue support")
> Acked-by: Douglas Gilbert <dgilbert@interlog.com>
> Signed-off-by: John Garry <john.garry@huawei.com>
> ---
> drivers/scsi/scsi_debug.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
> index 4692f5b6ad13..68534a23866e 100644
> --- a/drivers/scsi/scsi_debug.c
> +++ b/drivers/scsi/scsi_debug.c
> @@ -6613,6 +6613,12 @@ static int __init scsi_debug_init(void)
> pr_err("submit_queues must be 1 or more\n");
> return -EINVAL;
> }
> +
> + if ((sdebug_max_queue > SDEBUG_CANQUEUE) || (sdebug_max_queue <= 0)) {
> + pr_err("max_queue must be in range [1, %d]\n", SDEBUG_CANQUEUE);
> + return -EINVAL;
> + }
> +
> sdebug_q_arr = kcalloc(submit_queues, sizeof(struct sdebug_queue),
> GFP_KERNEL);
> if (sdebug_q_arr == NULL)
> --
> 2.26.2
>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
--
Ming
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 2/2] scsi: scsi_debug: Support hostwide tags
2020-07-09 12:23 ` [PATCH v2 2/2] scsi: scsi_debug: Support hostwide tags John Garry
@ 2020-07-09 15:25 ` Douglas Gilbert
2020-07-09 15:42 ` John Garry
0 siblings, 1 reply; 7+ messages in thread
From: Douglas Gilbert @ 2020-07-09 15:25 UTC (permalink / raw)
To: John Garry, jejb, martin.petersen
Cc: linux-scsi, hare, ming.lei, kashyap.desai
On 2020-07-09 8:23 a.m., John Garry wrote:
> Many SCSI HBAs support a hostwide tagset, whereby each command submitted
> to the HW from all submission queues must have a unique tag identifier.
>
> Normally this unique tag will be in the range [0, max queue), where "max
> queue" is the depth of each of the submission queues.
>
> Add support for this hostwide tag feature, via module parameter
> "host_max_queue". A non-zero value means that the feature is enabled. In
> this case, the submission queues are not exposed to upper layer, i.e. from
> blk-mq prespective, the device has a single hw queue. There are 2 reasons
> for this:
> a. it is assumed that the host can support nr_hw_queues * can_queue
> commands, but this is not true for hostwide tags
> b. for nr_hw_queues != 0, the request tag is not unique over all HW queues,
> and some HBA drivers want to use this tag for the hostwide tag
>
> However, like many SCSI HBA drivers today - megaraid sas being an example -
> the full set of HW submission queues are still used in the LLDD driver. So
> instead of using a complicated "reply_map" to create a per-CPU submission
> queue mapping like megaraid sas (as it depends on a PCI device + MSIs) -
> use a simple algorithm:
>
> hwq = cpu % queue count
>
> If the host max queue param is set non-zero, then the max queue depth is
> fixed at this value also.
>
> If and when hostwide shared tags are supported in blk-mq/scsi mid-layer,
> then the policy to set nr_hw_queues = 0 for hostwide tags can be revised.
>
> Signed-off-by: John Garry <john.garry@huawei.com>
Acked-by: Douglas Gilbert <dgilbert@interlog.com>
I have updated this page to reflect this addition:
http://sg.danny.cz/sg/scsi_debug.html
> ---
> drivers/scsi/scsi_debug.c | 80 +++++++++++++++++++++++++++++++++------
> 1 file changed, 68 insertions(+), 12 deletions(-)
>
> diff --git a/drivers/scsi/scsi_debug.c b/drivers/scsi/scsi_debug.c
> index 68534a23866e..2246d721ec88 100644
> --- a/drivers/scsi/scsi_debug.c
> +++ b/drivers/scsi/scsi_debug.c
> @@ -344,6 +344,7 @@ struct sdebug_defer {
> struct execute_work ew;
> int sqa_idx; /* index of sdebug_queue array */
> int qc_idx; /* index of sdebug_queued_cmd array within sqa_idx */
> + int hc_idx; /* hostwide tag index */
> int issuing_cpu;
> bool init_hrt;
> bool init_wq;
> @@ -759,6 +760,7 @@ static int sdebug_dsense = DEF_D_SENSE;
> static int sdebug_every_nth = DEF_EVERY_NTH;
> static int sdebug_fake_rw = DEF_FAKE_RW;
> static unsigned int sdebug_guard = DEF_GUARD;
> +static int sdebug_host_max_queue; /* per host */
> static int sdebug_lowest_aligned = DEF_LOWEST_ALIGNED;
> static int sdebug_max_luns = DEF_MAX_LUNS;
> static int sdebug_max_queue = SDEBUG_CANQUEUE; /* per submit queue */
> @@ -4707,15 +4709,28 @@ static int resp_rwp_zone(struct scsi_cmnd *scp, struct sdebug_dev_info *devip)
>
> static struct sdebug_queue *get_queue(struct scsi_cmnd *cmnd)
> {
> - u32 tag = blk_mq_unique_tag(cmnd->request);
> - u16 hwq = blk_mq_unique_tag_to_hwq(tag);
> + u16 hwq;
>
> - pr_debug("tag=%#x, hwq=%d\n", tag, hwq);
> - if (WARN_ON_ONCE(hwq >= submit_queues))
> - hwq = 0;
> + if (sdebug_host_max_queue) {
> + /* Provide a simple method to choose the hwq */
> + hwq = smp_processor_id() % submit_queues;
> + } else {
> + u32 tag = blk_mq_unique_tag(cmnd->request);
> +
> + hwq = blk_mq_unique_tag_to_hwq(tag);
> +
> + pr_debug("tag=%#x, hwq=%d\n", tag, hwq);
> + if (WARN_ON_ONCE(hwq >= submit_queues))
> + hwq = 0;
> + }
> return sdebug_q_arr + hwq;
> }
>
> +static u32 get_tag(struct scsi_cmnd *cmnd)
> +{
> + return blk_mq_unique_tag(cmnd->request);
> +}
> +
> /* Queued (deferred) command completions converge here. */
> static void sdebug_q_cmd_complete(struct sdebug_defer *sd_dp)
> {
> @@ -4747,8 +4762,8 @@ static void sdebug_q_cmd_complete(struct sdebug_defer *sd_dp)
> scp = sqcp->a_cmnd;
> if (unlikely(scp == NULL)) {
> spin_unlock_irqrestore(&sqp->qc_lock, iflags);
> - pr_err("scp is NULL, sqa_idx=%d, qc_idx=%d\n",
> - sd_dp->sqa_idx, qc_idx);
> + pr_err("scp is NULL, sqa_idx=%d, qc_idx=%d, hc_idx=%d\n",
> + sd_dp->sqa_idx, qc_idx, sd_dp->hc_idx);
> return;
> }
> devip = (struct sdebug_dev_info *)scp->device->hostdata;
> @@ -5451,6 +5466,10 @@ static int schedule_resp(struct scsi_cmnd *cmnd, struct sdebug_dev_info *devip,
> new_sd_dp = false;
> }
>
> + /* Set the hostwide tag */
> + if (sdebug_host_max_queue)
> + sd_dp->hc_idx = get_tag(cmnd);
> +
> if (ndelay > 0 && ndelay < INCLUSIVE_TIMING_MAX_NS)
> ns_from_boot = ktime_get_boottime_ns();
>
> @@ -5572,6 +5591,7 @@ module_param_named(every_nth, sdebug_every_nth, int, S_IRUGO | S_IWUSR);
> module_param_named(fake_rw, sdebug_fake_rw, int, S_IRUGO | S_IWUSR);
> module_param_named(guard, sdebug_guard, uint, S_IRUGO);
> module_param_named(host_lock, sdebug_host_lock, bool, S_IRUGO | S_IWUSR);
> +module_param_named(host_max_queue, sdebug_host_max_queue, int, S_IRUGO);
> module_param_string(inq_product, sdebug_inq_product_id,
> sizeof(sdebug_inq_product_id), S_IRUGO | S_IWUSR);
> module_param_string(inq_rev, sdebug_inq_product_rev,
> @@ -5642,6 +5662,8 @@ MODULE_PARM_DESC(every_nth, "timeout every nth command(def=0)");
> MODULE_PARM_DESC(fake_rw, "fake reads/writes instead of copying (def=0)");
> MODULE_PARM_DESC(guard, "protection checksum: 0=crc, 1=ip (def=0)");
> MODULE_PARM_DESC(host_lock, "host_lock is ignored (def=0)");
> +MODULE_PARM_DESC(host_max_queue,
> + "host max # of queued cmds (0 to max(def) [max_queue fixed equal for !0])");
> MODULE_PARM_DESC(inq_product, "SCSI INQUIRY product string (def=\"scsi_debug\")");
> MODULE_PARM_DESC(inq_rev, "SCSI INQUIRY revision string (def=\""
> SDEBUG_VERSION "\")");
> @@ -6141,7 +6163,8 @@ static ssize_t max_queue_store(struct device_driver *ddp, const char *buf,
> struct sdebug_queue *sqp;
>
> if ((count > 0) && (1 == sscanf(buf, "%d", &n)) && (n > 0) &&
> - (n <= SDEBUG_CANQUEUE)) {
> + (n <= SDEBUG_CANQUEUE) &&
> + (sdebug_host_max_queue == 0)) {
> block_unblock_all_queues(true);
> k = 0;
> for (j = 0, sqp = sdebug_q_arr; j < submit_queues;
> @@ -6164,6 +6187,17 @@ static ssize_t max_queue_store(struct device_driver *ddp, const char *buf,
> }
> static DRIVER_ATTR_RW(max_queue);
>
> +static ssize_t host_max_queue_show(struct device_driver *ddp, char *buf)
> +{
> + return scnprintf(buf, PAGE_SIZE, "%d\n", sdebug_host_max_queue);
> +}
> +
> +/*
> + * Since this is used for .can_queue, and we get the hc_idx tag from the bitmap
> + * in range [0, sdebug_host_max_queue), we can't change it.
> + */
> +static DRIVER_ATTR_RO(host_max_queue);
> +
> static ssize_t no_uld_show(struct device_driver *ddp, char *buf)
> {
> return scnprintf(buf, PAGE_SIZE, "%d\n", sdebug_no_uld);
> @@ -6503,6 +6537,7 @@ static struct attribute *sdebug_drv_attrs[] = {
> &driver_attr_ptype.attr,
> &driver_attr_dsense.attr,
> &driver_attr_fake_rw.attr,
> + &driver_attr_host_max_queue.attr,
> &driver_attr_no_lun_0.attr,
> &driver_attr_num_tgts.attr,
> &driver_attr_dev_size_mb.attr,
> @@ -6619,6 +6654,20 @@ static int __init scsi_debug_init(void)
> return -EINVAL;
> }
>
> + if ((sdebug_host_max_queue > SDEBUG_CANQUEUE) ||
> + (sdebug_host_max_queue < 0)) {
> + pr_err("host_max_queue must be in range [0 %d]\n",
> + SDEBUG_CANQUEUE);
> + return -EINVAL;
> + }
> +
> + if (sdebug_host_max_queue &&
> + (sdebug_max_queue != sdebug_host_max_queue)) {
> + sdebug_max_queue = sdebug_host_max_queue;
> + pr_warn("fixing max submit queue depth to host max queue depth, %d\n",
> + sdebug_max_queue);
> + }
> +
> sdebug_q_arr = kcalloc(submit_queues, sizeof(struct sdebug_queue),
> GFP_KERNEL);
> if (sdebug_q_arr == NULL)
> @@ -7257,7 +7306,10 @@ static int sdebug_driver_probe(struct device *dev)
>
> sdbg_host = to_sdebug_host(dev);
>
> - sdebug_driver_template.can_queue = sdebug_max_queue;
> + if (sdebug_host_max_queue)
> + sdebug_driver_template.can_queue = sdebug_host_max_queue;
> + else
> + sdebug_driver_template.can_queue = sdebug_max_queue;
> if (!sdebug_clustering)
> sdebug_driver_template.dma_boundary = PAGE_SIZE - 1;
>
> @@ -7272,9 +7324,13 @@ static int sdebug_driver_probe(struct device *dev)
> my_name, submit_queues, nr_cpu_ids);
> submit_queues = nr_cpu_ids;
> }
> - /* Decide whether to tell scsi subsystem that we want mq */
> - /* Following should give the same answer for each host */
> - hpnt->nr_hw_queues = submit_queues;
> + /*
> + * Decide whether to tell scsi subsystem that we want mq. The
> + * following should give the same answer for each host. If the host
> + * has a limit of hostwide max commands, then do not set.
> + */
> + if (!sdebug_host_max_queue)
> + hpnt->nr_hw_queues = submit_queues;
>
> sdbg_host->shost = hpnt;
> *((struct sdebug_host_info **)hpnt->hostdata) = sdbg_host;
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 2/2] scsi: scsi_debug: Support hostwide tags
2020-07-09 15:25 ` Douglas Gilbert
@ 2020-07-09 15:42 ` John Garry
0 siblings, 0 replies; 7+ messages in thread
From: John Garry @ 2020-07-09 15:42 UTC (permalink / raw)
To: dgilbert, jejb, martin.petersen; +Cc: linux-scsi, hare, ming.lei, kashyap.desai
On 09/07/2020 16:25, Douglas Gilbert wrote:
>>
>> hwq = cpu % queue count
>>
>> If the host max queue param is set non-zero, then the max queue depth is
>> fixed at this value also.
>>
>> If and when hostwide shared tags are supported in blk-mq/scsi mid-layer,
>> then the policy to set nr_hw_queues = 0 for hostwide tags can be revised.
>>
>> Signed-off-by: John Garry <john.garry@huawei.com>
>
> Acked-by: Douglas Gilbert <dgilbert@interlog.com>
>
> I have updated this page to reflect this addition:
> http://sg.danny.cz/sg/scsi_debug.html
ok, thanks.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix
2020-07-09 12:23 [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix John Garry
2020-07-09 12:23 ` [PATCH v2 1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init John Garry
2020-07-09 12:23 ` [PATCH v2 2/2] scsi: scsi_debug: Support hostwide tags John Garry
@ 2020-07-14 4:58 ` Martin K. Petersen
2 siblings, 0 replies; 7+ messages in thread
From: Martin K. Petersen @ 2020-07-14 4:58 UTC (permalink / raw)
To: John Garry, jejb
Cc: Martin K . Petersen, hare, linux-scsi, kashyap.desai, dgilbert, ming.lei
On Thu, 9 Jul 2020 20:23:18 +0800, John Garry wrote:
> This series includes hostwide tags support, so we can mimic some SCSI HBAs,
> like megaraid sas or hisi_sas.
>
> There is also a fix for an out-of-range module param.
>
> Differences to v1:
> - add Doug's Ack for patch 1/2
> - sort params alphabetically
> - fix max queue at host max queue (when non-zero)
> - drop host max queue file write permission
>
> [...]
Applied to 5.9/scsi-queue, thanks!
[1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init
https://git.kernel.org/mkp/scsi/c/c87bf24cfb60
[2/2] scsi: scsi_debug: Support hostwide tags
https://git.kernel.org/mkp/scsi/c/c10fa55f5e7a
--
Martin K. Petersen Oracle Linux Engineering
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-07-14 4:59 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-09 12:23 [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix John Garry
2020-07-09 12:23 ` [PATCH v2 1/2] scsi: scsi_debug: Add check for sdebug_max_queue during module init John Garry
2020-07-09 13:18 ` Ming Lei
2020-07-09 12:23 ` [PATCH v2 2/2] scsi: scsi_debug: Support hostwide tags John Garry
2020-07-09 15:25 ` Douglas Gilbert
2020-07-09 15:42 ` John Garry
2020-07-14 4:58 ` [PATCH v2 0/2] scsi: scsi_debug: Support hostwide tags and a fix Martin K. Petersen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).