[LSF/MM/BPF TOPIC] Storage: generic completion polling

[LSF/MM/BPF TOPIC] Storage: generic completion polling

[Hannes Reinecke]

Hi all,

there had been quite some discussion around completion polling and the
fact that for high-performance devices it might be a performance benefit
[1][2]. And during discussion with other people (hello tglx) the
reaction always had been "Can't you do NAPI?"

So the question is: Can we?
IE is it possible to have a generic framework for handling polled
completiona and interrupt completions, shifting between them depending
on the load?

My idea is to have a sequence like
completion polling -> interrupt handling -> threaded irq/polling
IE invoke completion polling directly from the submission path, enable
interrupts to handle completions from the interrupt handler, and finally
shift to completion polling again if too many completions are present.
Clearly this approach involves quite some tunables (like how many
completions before enabling polling from interrupt context, how long to
wait for completions before enabling interrupts etc), but I thing it
would be worthwhile having this as a generic framework as then one could
start experimenting with the various tunables to see which works best
for the individual hardware.
And it would lift the burden from the hardware vendors to implement a
similar mechanism on their own.

Proposed participants:
Martin K. Petersen
Jens Axboe
Christoph Hellwig
Keith Busch
Kashyap Desai
James Smart
Himanshu Madhani
Sagi Grimberg

[1]http://lists.infradead.org/pipermail/linux-nvme/2020-February/028961.html
[2]https://lore.kernel.org/linux-nvme/20191209175622.1964-1-kbusch@kernel.org/

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

[LSF/MM/BPF TOPIC] Storage: generic completion polling

[Hannes Reinecke]

Hi all,

there had been quite some discussion around completion polling and the
fact that for high-performance devices it might be a performance benefit
[1][2]. And during discussion with other people (hello tglx) the
reaction always had been "Can't you do NAPI?"

So the question is: Can we?
IE is it possible to have a generic framework for handling polled
completiona and interrupt completions, shifting between them depending
on the load?

My idea is to have a sequence like
completion polling -> interrupt handling -> threaded irq/polling
IE invoke completion polling directly from the submission path, enable
interrupts to handle completions from the interrupt handler, and finally
shift to completion polling again if too many completions are present.
Clearly this approach involves quite some tunables (like how many
completions before enabling polling from interrupt context, how long to
wait for completions before enabling interrupts etc), but I thing it
would be worthwhile having this as a generic framework as then one could
start experimenting with the various tunables to see which works best
for the individual hardware.
And it would lift the burden from the hardware vendors to implement a
similar mechanism on their own.

Proposed participants:
Martin K. Petersen
Jens Axboe
Christoph Hellwig
Keith Busch
Kashyap Desai
James Smart
Himanshu Madhani
Sagi Grimberg

[1]http://lists.infradead.org/pipermail/linux-nvme/2020-February/028961.html
[2]https://lore.kernel.org/linux-nvme/20191209175622.1964-1-kbusch@kernel.org/

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		           Kernel Storage Architect
hare@suse.de			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

Re: [LSF/MM/BPF TOPIC] Storage: generic completion polling

[Bart Van Assche]

On 3/3/20 11:17 PM, Hannes Reinecke wrote:
> there had been quite some discussion around completion polling and the
> fact that for high-performance devices it might be a performance benefit
> [1][2]. And during discussion with other people (hello tglx) the
> reaction always had been "Can't you do NAPI?"
> 
> So the question is: Can we?
> IE is it possible to have a generic framework for handling polled
> completiona and interrupt completions, shifting between them depending
> on the load?
> 
> My idea is to have a sequence like
> completion polling -> interrupt handling -> threaded irq/polling
> IE invoke completion polling directly from the submission path, enable
> interrupts to handle completions from the interrupt handler, and finally
> shift to completion polling again if too many completions are present.
> Clearly this approach involves quite some tunables (like how many
> completions before enabling polling from interrupt context, how long to
> wait for completions before enabling interrupts etc), but I thing it
> would be worthwhile having this as a generic framework as then one could
> start experimenting with the various tunables to see which works best
> for the individual hardware.
> And it would lift the burden from the hardware vendors to implement a
> similar mechanism on their own.

To me the above sounds like a description of the lib/irq_poll mechanism? 
 From the top of lib/irq_poll.c:

/*
  * Functions related to interrupt-poll handling in the block layer. This
  * is similar to NAPI for network devices.
  */

Anyway, I'm interested in participating in the discussion of this topic.

Bart.

Re: [LSF/MM/BPF TOPIC] Storage: generic completion polling

[Bart Van Assche]

On 3/3/20 11:17 PM, Hannes Reinecke wrote:
> there had been quite some discussion around completion polling and the
> fact that for high-performance devices it might be a performance benefit
> [1][2]. And during discussion with other people (hello tglx) the
> reaction always had been "Can't you do NAPI?"
> 
> So the question is: Can we?
> IE is it possible to have a generic framework for handling polled
> completiona and interrupt completions, shifting between them depending
> on the load?
> 
> My idea is to have a sequence like
> completion polling -> interrupt handling -> threaded irq/polling
> IE invoke completion polling directly from the submission path, enable
> interrupts to handle completions from the interrupt handler, and finally
> shift to completion polling again if too many completions are present.
> Clearly this approach involves quite some tunables (like how many
> completions before enabling polling from interrupt context, how long to
> wait for completions before enabling interrupts etc), but I thing it
> would be worthwhile having this as a generic framework as then one could
> start experimenting with the various tunables to see which works best
> for the individual hardware.
> And it would lift the burden from the hardware vendors to implement a
> similar mechanism on their own.

To me the above sounds like a description of the lib/irq_poll mechanism? 
 From the top of lib/irq_poll.c:

/*
  * Functions related to interrupt-poll handling in the block layer. This
  * is similar to NAPI for network devices.
  */

Anyway, I'm interested in participating in the discussion of this topic.

Bart.


_______________________________________________
linux-nvme mailing list
linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme