From: Michael Kelley <mikelley@microsoft.com>
To: John Garry <john.garry@huawei.com>,
Dexuan Cui <decui@microsoft.com>,
KY Srinivasan <kys@microsoft.com>,
Stephen Hemminger <sthemmin@microsoft.com>,
"wei.liu@kernel.org" <wei.liu@kernel.org>,
"jejb@linux.ibm.com" <jejb@linux.ibm.com>,
"martin.petersen@oracle.com" <martin.petersen@oracle.com>,
Haiyang Zhang <haiyangz@microsoft.com>,
"ming.lei@redhat.com" <ming.lei@redhat.com>,
"bvanassche@acm.org" <bvanassche@acm.org>,
"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
"linux-hyperv@vger.kernel.org" <linux-hyperv@vger.kernel.org>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: RE: [PATCH] scsi: storvsc: Cap scsi_driver.can_queue to fix a hang issue during boot
Date: Wed, 6 Oct 2021 15:01:20 +0000 [thread overview]
Message-ID: <MWHPR21MB159368D7BAAD90E19F31D1C6D7B09@MWHPR21MB1593.namprd21.prod.outlook.com> (raw)
In-Reply-To: <e36619df-652d-3550-cb4d-9b65b2f5faee@huawei.com>
From: John Garry <john.garry@huawei.com> Sent: Wednesday, October 6, 2021 1:17 AM
>
> On 06/10/2021 08:03, Dexuan Cui wrote:
> > After commit ea2f0f77538c, a 416-CPU VM running on Hyper-V hangs during
> > boot because scsi_add_host_with_dma() sets shost->cmd_per_lun to a
> > negative number:
> > 'max_outstanding_req_per_channel' is 352,
> > 'max_sub_channels' is (416 - 1) / 4 = 103, so in storvsc_probe(),
> > scsi_driver.can_queue = 352 * (103 + 1) * (100 - 10) / 100 = 32947, which
> > is bigger than SHRT_MAX (i.e. 32767).
>
> Out of curiosity, are these values realistic? You're capping can_queue
> just because of a data size issue, so, if these values are realistic,
> seems a weak reason.
>
The calculated value of can_queue is not realistic. The blk-mq layer
caps the number of tags at 10240, so the excessively large value
calculated here didn't definitively break anything, though it can be
poor from a performance tuning standpoint. The algorithm used here
is fairly broken, particularly in VMs with large CPU counts. I have an
effort underway to fix it, but its part of a bigger set of changes to also
do a better job on the perf tuning aspects.
> >
> > Fix the hang issue by capping scsi_driver.can_queue.
> >
> > Add the below Fixed tag though ea2f0f77538c itself is good.
> >
> > Fixes: ea2f0f77538c ("scsi: core: Cap scsi_host cmd_per_lun at can_queue")
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Dexuan Cui <decui@microsoft.com>
> > ---
> > drivers/scsi/storvsc_drv.c | 10 ++++++++++
> > 1 file changed, 10 insertions(+)
> >
> > diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> > index ebbbc1299c62..ba374908aec2 100644
> > --- a/drivers/scsi/storvsc_drv.c
> > +++ b/drivers/scsi/storvsc_drv.c
> > @@ -1976,6 +1976,16 @@ static int storvsc_probe(struct hv_device *device,
> > (max_sub_channels + 1) *
> > (100 - ring_avail_percent_lowater) / 100;
> >
> > + /*
> > + * v5.14 (see commit ea2f0f77538c) implicitly requires that
> > + * scsi_driver.can_queue should not exceed SHRT_MAX, otherwise
> > + * scsi_add_host_with_dma() sets shost->cmd_per_lun to a negative
> > + * number (note: the type of the "cmd_per_lun" field is "short"), and
> > + * the system may hang during early boot.
> > + */
>
> The different data sizes for cmd_per_lun and can_queue are problematic here.
>
> I'd be more inclined to set cmd_per_lun to the same data size as
> can_queue. We did discuss this when ea2f0f77538c was upstreamed
> (actually it was the other way around - setting can_queue to 16b).
I can see that making can_queue be 16 bits would make sense.
And it also seems that both cmd_per_lun and can_queue should be
unsigned, though I don't the implications of making such a change.
But in today's world where cmd_per_lun is "short" and can_queue
is "int", ea2f0f77538c seems incorrect to me. The comparison should
be done as "int", not "short", in order to prevent the truncation
problem with can_queue that Dexuan's patch is trying to address.
The result will always fit in back into the "short" cmd_per_lun since
it is calculating a "min" function.
>
> Thanks,
> John
>
>
> > + if (scsi_driver.can_queue > SHRT_MAX)
> > + scsi_driver.can_queue = SHRT_MAX;
> > +
This fix works, but is a more of a temporary hack until I can finish
a larger overhaul of the algorithm. But for now, I think the better
fix is for ea2f0f77538c to do the comparison as "int" instead of "short".
Michael
> > host = scsi_host_alloc(&scsi_driver,
> > sizeof(struct hv_host_device));
> > if (!host)
> >
next prev parent reply other threads:[~2021-10-06 15:01 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-06 7:03 [PATCH] scsi: storvsc: Cap scsi_driver.can_queue to fix a hang issue during boot Dexuan Cui
2021-10-06 7:17 ` Greg KH
2021-10-06 8:17 ` John Garry
2021-10-06 15:01 ` Michael Kelley [this message]
2021-10-06 16:03 ` John Garry
2021-10-06 16:08 ` Michael Kelley
2021-10-06 16:24 ` John Garry
2021-10-06 17:28 ` Dexuan Cui
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=MWHPR21MB159368D7BAAD90E19F31D1C6D7B09@MWHPR21MB1593.namprd21.prod.outlook.com \
--to=mikelley@microsoft.com \
--cc=bvanassche@acm.org \
--cc=decui@microsoft.com \
--cc=haiyangz@microsoft.com \
--cc=jejb@linux.ibm.com \
--cc=john.garry@huawei.com \
--cc=kys@microsoft.com \
--cc=linux-hyperv@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=martin.petersen@oracle.com \
--cc=ming.lei@redhat.com \
--cc=stable@vger.kernel.org \
--cc=sthemmin@microsoft.com \
--cc=wei.liu@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).