Re: [PATCH RFC v7 10/12] megaraid_sas: switch fusion adapters to MQ

From: Ming Lei <ming.lei@redhat.com>
To: Kashyap Desai <kashyap.desai@broadcom.com>
Cc: John Garry <john.garry@huawei.com>,
	axboe@kernel.dk, jejb@linux.ibm.com, martin.petersen@oracle.com,
	don.brace@microsemi.com, Sumit Saxena <sumit.saxena@broadcom.com>,
	bvanassche@acm.org, hare@suse.com, hch@lst.de,
	Shivasharan Srikanteshwara 
	<shivasharan.srikanteshwara@broadcom.com>,
	linux-block@vger.kernel.org, linux-scsi@vger.kernel.org,
	esc.storagedev@microsemi.com, chenxiang66@hisilicon.com,
	"PDL,MEGARAIDLINUX" <megaraidlinux.pdl@broadcom.com>
Subject: Re: [PATCH RFC v7 10/12] megaraid_sas: switch fusion adapters to MQ
Date: Thu, 6 Aug 2020 23:29:39 +0800	[thread overview]
Message-ID: <20200806152939.GA2062348@T590> (raw)
In-Reply-To: <f1ac35dfca34193e6c9bcedbc11911d2@mail.gmail.com>

On Thu, Aug 06, 2020 at 08:07:38PM +0530, Kashyap Desai wrote:
> > > Hi Ming -
> > >
> > > There is still some race which is not handled.  Take a case of IO is
> > > not able to get budget and it has already marked <restarts> flag.
> > > <restarts> flag will be seen non-zero in completion path and
> > > completion path will attempt h/w queue run. (But this particular IO is
> > > still not in s/w queue.).
> > > Attempt of running h/w queue from completion path will not flush any
> > > IO since there is no IO in s/w queue.
> >
> > Then where is the IO to be submitted in case of running out of budget?
> 
> Typical race in your latest patch is - (Lets consider command A,B and C)
> Command A did not receive budget. Command B completed  (which was already

Command A doesn't get budget, and A is still in sw/scheduler queue
because we try to acquire budget before dequeuing request from sw/scheduler queue,
see __blk_mq_do_dispatch_sched() and blk_mq_do_dispatch_ctx().

Not consider direct issue, because the hw queue will be run explicitly
when not getting budget, see __blk_mq_try_issue_directly.

Not consider command A being added to hctx->dispatch too, because blk-mq will
re-run the queue, see blk_mq_dispatch_rq_list().

> submitted earlier) at the same time and it make sdev->device_busy = 0 from
> " scsi_finish_command".
> Command B has still not called "scsi_end_request". Command C get the
> budget and it will make sdev->device_busy = 1. Now, Command A set  set
> sdev->restarts flags but will not run h/w queue since sdev->device_busy =
> 1.

Right.

> Command B run h/w queue (make sdev->restart = 0) from completion path, but
> command -A is still not in the s/w queue.

Then you didn't answer my question about where A is, did you?

> Command-A is in now in s/w queue. Command-C completed but it will not run h/w queue because
> sdev->restarts = 0.

Why does command-A become in sw/queue now?

> 
> 
> >
> > Any IO request which is going to be added to hctx->dispatch, the queue
> will be
> > re-run via blk-mq core.
> >
> > Any IO request being issued directly when running out of budget will be
> insert
> > to hctx->dispatch or sw/scheduler queue, will be run in the submission
> path.
> 
> I have *not* included below changes we discussed in my testing - If I
> include below patch, it is correct that queue will be run in submission
> path (at least the path which is impacted in my testing). You have already
> mentioned that most of the submission path has fix now in latest kernel
> w.r.t running h/w queue from submission path.  Below path is missing for
> running h/w queue from submission path.
> 
> diff --git a/block/blk-mq-sched.c b/block/blk-mq-sched.c index
> 54f9015..bcfd33a 100644
> --- a/block/blk-mq-sched.c
> +++ b/block/blk-mq-sched.c
> @@ -173,8 +173,10 @@ static int blk_mq_do_dispatch_ctx(struct
> blk_mq_hw_ctx *hctx)
>                 if (!sbitmap_any_bit_set(&hctx->ctx_map))
>                         break;
> 
> -               if (!blk_mq_get_dispatch_budget(hctx))
> +               if (!blk_mq_get_dispatch_budget(hctx)) {
> +                       blk_mq_delay_run_hw_queue(hctx,
> + BLK_MQ_BUDGET_DELAY);
>                         break;
> +               }
> 
>                 rq = blk_mq_dequeue_from_ctx(hctx, ctx);
>                 if (!rq) {
> 
> Are you saying above fix should be included along with your latest patch ?

No.

Thanks,
Ming