Re: [PATCH V3 0/5] blk-mq: improvement on handling IO during CPU hotplug

[Ming Lei <tom.leiming@gmail.com>]

On Fri, Oct 11, 2019 at 10:10 PM John Garry <john.garry@huawei.com> wrote:
>
> On 11/10/2019 12:55, Ming Lei wrote:
> > On Fri, Oct 11, 2019 at 4:54 PM John Garry <john.garry@huawei.com> wrote:
> >>
> >> On 10/10/2019 12:21, John Garry wrote:
> >>>
> >>>>
> >>>> As discussed before, tags of hisilicon V3 is HBA wide. If you switch
> >>>> to real hw queue, each hw queue has to own its independent tags.
> >>>> However, that isn't supported by V3 hardware.
> >>>
> >>> I am generating the tag internally in the driver now, so that hostwide
> >>> tags issue should not be an issue.
> >>>
> >>> And, to be clear, I am not paying too much attention to performance, but
> >>> rather just hotplugging while running IO.
> >>>
> >>> An update on testing:
> >>> I did some scripted overnight testing. The script essentially loops like
> >>> this:
> >>> - online all CPUS
> >>> - run fio binded on a limited bunch of CPUs to cover a hctx mask for 1
> >>> minute
> >>> - offline those CPUs
> >>> - wait 1 minute (> SCSI or NVMe timeout)
> >>> - and repeat
> >>>
> >>> SCSI is actually quite stable, but NVMe isn't. For NVMe I am finding
> >>> some fio processes never dying with IOPS @ 0. I don't see any NVMe
> >>> timeout reported. Did you do any NVMe testing of this sort?
> >>>
> >>
> >> Yeah, so for NVMe, I see some sort of regression, like this:
> >> Jobs: 1 (f=1): [_R] [0.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta
> >> 1158037877d:17h:18m:22s]
> >
> > I can reproduce this issue, and looks there are requests in ->dispatch.
>
> OK, that may match with what I see:
> - the problem occuring coincides with this callpath with
> BLK_MQ_S_INTERNAL_STOPPED set:

Good catch, these requests should have been re-submitted in
blk_mq_hctx_notify_dead() too.

Will do it in V4.

Thanks,
Ming Lei