Re: [PATCH 2/3] blk-mq: Always complete remote completions requests in softirq

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2020-12-08 13:41:48 [+0100], Daniel Wagner wrote:
> On Tue, Dec 08, 2020 at 12:49:36PM +0100, Sebastian Andrzej Siewior wrote:
> > On 2020-12-08 12:36:53 [+0100], Daniel Wagner wrote:
> > > Obvious in this configuration there are no remote completions (verified
> > > it).
> > 
> > do you complete on a remote CPU if you limit the queues to one (this is
> > untested of course)?
> 
> nvme0n1/ completed   11913011 remote    6718563 56.40%
> 
> yes, but how is this relevant? I thought Jens complain was about the
> additional indirection via the softirq context
> 
> -		rq->q->mq_ops->complete(rq);
> +	blk_mq_trigger_softirq(rq);
> 
> and not the remote completion path. I can benchmark it out but I don't
> know if it's really helping in the discussion.

The only additional softirq path is for cross-CPU completion. If I
understood you correctly then your NVME device always completes locally
because the queue interrupt fires on the correct CPU.
If you take away the queues then you should have cross-CPU completion
since you have only one queue and this will now complete on the remote
CPU in softirq context (and not in IRQ as it used to).
If this single queue NVME device, which may complete on another CPU, is
not an issue / interesting because it is already limited then ignore
this.

Sebastian