From: Thomas Gleixner <tglx@linutronix.de>
To: Bjorn Helgaas <helgaas@kernel.org>
Cc: Ming Lei <ming.lei@redhat.com>, Christoph Hellwig <hch@lst.de>,
Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org, Sagi Grimberg <sagi@grimberg.me>,
linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org,
linux-pci@vger.kernel.org, Keith Busch <keith.busch@intel.com>
Subject: Re: [PATCH V3 4/5] nvme-pci: avoid irq allocation retrying via .calc_sets
Date: Wed, 13 Feb 2019 22:26:45 +0100 (CET) [thread overview]
Message-ID: <alpine.DEB.2.21.1902132159230.1659@nanos.tec.linutronix.de> (raw)
In-Reply-To: <20190213151339.GE96272@google.com>
On Wed, 13 Feb 2019, Bjorn Helgaas wrote:
> On Wed, Feb 13, 2019 at 06:50:40PM +0800, Ming Lei wrote:
> > Currently pre-caculate each set vectors, and this way requires same
> > 'max_vecs' and 'min_vecs' passed to pci_alloc_irq_vectors_affinity(),
> > then nvme_setup_irqs() has to retry in case of allocation failure.
>
> s/pre-caculate/precalculate/
> My usual "set vectors" question as on other patches.
>
> > This usage & interface is a bit awkward because the retry should have
> > been avoided by providing one reasonable 'min_vecs'.
> >
> > Implement the callback of .calc_sets, so that pci_alloc_irq_vectors_affinity()
> > can calculate each set's vector after IRQ vectors is allocated and
> > before spread IRQ, then NVMe's retry in case of irq allocation failure
> > can be removed.
>
> s/irq/IRQ/
Let me rephrase that thing as well
Subject: nvme-pci: Simplify interrupt allocation
The NVME PCI driver contains a tedious mechanism for interrupt
allocation, which is necessary to adjust the number and size of interrupt
sets to the maximum available number of interrupts which depends on the
underlying PCI capabilities and the available CPU resources.
It works around the former short comings of the PCI and core interrupt
allocation mechanims in combination with interrupt sets.
The PCI interrupt allocation function allows to provide a maximum and a
minimum number of interrupts to be allocated and tries to allocate as
many as possible. This worked without driver interaction as long as there
was only a single set of interrupts to handle.
With the addition of support for multiple interrupt sets in the generic
affinity spreading logic, which is invoked from the PCI interrupt
allocation, the adaptive loop in the PCI interrupt allocation did not
work for multiple interrupt sets. The reason is that depending on the
total number of interrupts which the PCI allocation adaptive loop tries
to allocate in each step, the number and the size of the interrupt sets
need to be adapted as well. Due to the way the interrupt sets support was
implemented there was no way for the PCI interrupt allocation code or the
core affinity spreading mechanism to invoke a driver specific function
for adapting the interrupt sets configuration.
As a consequence the driver had to implement another adaptive loop around
the PCI interrupt allocation function and calling that with maximum and
minimum interrupts set to the same value. This ensured that the
allocation either succeeded or immediately failed without any attempt to
adjust the number of interrupts in the PCI code.
The core code now allows drivers to provide a callback to recalculate the
number and the size of interrupt sets during PCI interrupt allocation,
which in turn allows the PCI interrupt allocation function to be called
in the same way as with a single set of interrupts. The PCI code handles
the adaptive loop and the interrupt affinity spreading mechanism invokes
the driver callback to adapt the interrupt set configuration to the
current loop value. This replaces the adaptive loop in the driver
completely.
Implement the NVME specific callback which adjusts the interrupt sets
configuration and remove the adaptive allocation loop.
Thanks,
tglx
WARNING: multiple messages have this Message-ID (diff)
From: tglx@linutronix.de (Thomas Gleixner)
Subject: [PATCH V3 4/5] nvme-pci: avoid irq allocation retrying via .calc_sets
Date: Wed, 13 Feb 2019 22:26:45 +0100 (CET) [thread overview]
Message-ID: <alpine.DEB.2.21.1902132159230.1659@nanos.tec.linutronix.de> (raw)
In-Reply-To: <20190213151339.GE96272@google.com>
On Wed, 13 Feb 2019, Bjorn Helgaas wrote:
> On Wed, Feb 13, 2019@06:50:40PM +0800, Ming Lei wrote:
> > Currently pre-caculate each set vectors, and this way requires same
> > 'max_vecs' and 'min_vecs' passed to pci_alloc_irq_vectors_affinity(),
> > then nvme_setup_irqs() has to retry in case of allocation failure.
>
> s/pre-caculate/precalculate/
> My usual "set vectors" question as on other patches.
>
> > This usage & interface is a bit awkward because the retry should have
> > been avoided by providing one reasonable 'min_vecs'.
> >
> > Implement the callback of .calc_sets, so that pci_alloc_irq_vectors_affinity()
> > can calculate each set's vector after IRQ vectors is allocated and
> > before spread IRQ, then NVMe's retry in case of irq allocation failure
> > can be removed.
>
> s/irq/IRQ/
Let me rephrase that thing as well
Subject: nvme-pci: Simplify interrupt allocation
The NVME PCI driver contains a tedious mechanism for interrupt
allocation, which is necessary to adjust the number and size of interrupt
sets to the maximum available number of interrupts which depends on the
underlying PCI capabilities and the available CPU resources.
It works around the former short comings of the PCI and core interrupt
allocation mechanims in combination with interrupt sets.
The PCI interrupt allocation function allows to provide a maximum and a
minimum number of interrupts to be allocated and tries to allocate as
many as possible. This worked without driver interaction as long as there
was only a single set of interrupts to handle.
With the addition of support for multiple interrupt sets in the generic
affinity spreading logic, which is invoked from the PCI interrupt
allocation, the adaptive loop in the PCI interrupt allocation did not
work for multiple interrupt sets. The reason is that depending on the
total number of interrupts which the PCI allocation adaptive loop tries
to allocate in each step, the number and the size of the interrupt sets
need to be adapted as well. Due to the way the interrupt sets support was
implemented there was no way for the PCI interrupt allocation code or the
core affinity spreading mechanism to invoke a driver specific function
for adapting the interrupt sets configuration.
As a consequence the driver had to implement another adaptive loop around
the PCI interrupt allocation function and calling that with maximum and
minimum interrupts set to the same value. This ensured that the
allocation either succeeded or immediately failed without any attempt to
adjust the number of interrupts in the PCI code.
The core code now allows drivers to provide a callback to recalculate the
number and the size of interrupt sets during PCI interrupt allocation,
which in turn allows the PCI interrupt allocation function to be called
in the same way as with a single set of interrupts. The PCI code handles
the adaptive loop and the interrupt affinity spreading mechanism invokes
the driver callback to adapt the interrupt set configuration to the
current loop value. This replaces the adaptive loop in the driver
completely.
Implement the NVME specific callback which adjusts the interrupt sets
configuration and remove the adaptive allocation loop.
Thanks,
tglx
next prev parent reply other threads:[~2019-02-13 21:26 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-02-13 10:50 [PATCH V3 0/5] genirq/affinity: add .calc_sets for improving IRQ allocation & spread Ming Lei
2019-02-13 10:50 ` Ming Lei
2019-02-13 10:50 ` [PATCH V3 1/5] genirq/affinity: don't mark 'affd' as const Ming Lei
2019-02-13 10:50 ` Ming Lei
2019-02-13 15:04 ` Bjorn Helgaas
2019-02-13 15:04 ` Bjorn Helgaas
2019-02-13 20:56 ` Thomas Gleixner
2019-02-13 20:56 ` Thomas Gleixner
2019-02-13 21:31 ` Keith Busch
2019-02-13 21:31 ` Keith Busch
2019-02-13 21:41 ` Thomas Gleixner
2019-02-13 21:41 ` Thomas Gleixner
2019-02-13 22:37 ` Keith Busch
2019-02-13 22:37 ` Keith Busch
2019-02-14 8:50 ` Thomas Gleixner
2019-02-14 8:50 ` Thomas Gleixner
2019-02-14 13:04 ` 陈华才
2019-02-14 13:04 ` 陈华才
2019-02-14 13:31 ` Thomas Gleixner
2019-02-14 13:31 ` Thomas Gleixner
2019-02-19 0:42 ` 陈华才
2019-02-19 0:42 ` 陈华才
2019-02-19 6:19 ` Thomas Gleixner
2019-02-19 6:19 ` Thomas Gleixner
2019-02-19 16:12 ` Keith Busch
2019-02-19 16:12 ` Keith Busch
2019-02-13 10:50 ` [PATCH V3 2/5] genirq/affinity: store irq set vectors in 'struct irq_affinity' Ming Lei
2019-02-13 10:50 ` Ming Lei
2019-02-13 15:07 ` Bjorn Helgaas
2019-02-13 15:07 ` Bjorn Helgaas
2019-02-13 10:50 ` [PATCH V3 3/5] genirq/affinity: add new callback for caculating set vectors Ming Lei
2019-02-13 10:50 ` Ming Lei
2019-02-13 15:11 ` Bjorn Helgaas
2019-02-13 15:11 ` Bjorn Helgaas
2019-02-13 20:58 ` Thomas Gleixner
2019-02-13 20:58 ` Thomas Gleixner
2019-02-13 10:50 ` [PATCH V3 4/5] nvme-pci: avoid irq allocation retrying via .calc_sets Ming Lei
2019-02-13 10:50 ` Ming Lei
2019-02-13 15:13 ` Bjorn Helgaas
2019-02-13 15:13 ` Bjorn Helgaas
2019-02-13 21:26 ` Thomas Gleixner [this message]
2019-02-13 21:26 ` Thomas Gleixner
2019-02-13 10:50 ` [PATCH V3 5/5] genirq/affinity: Document .calc_sets as required in case of multiple sets Ming Lei
2019-02-13 10:50 ` Ming Lei
2019-02-13 15:16 ` Bjorn Helgaas
2019-02-13 15:16 ` Bjorn Helgaas
2019-02-13 14:36 ` [PATCH V3 0/5] genirq/affinity: add .calc_sets for improving IRQ allocation & spread Jens Axboe
2019-02-13 14:36 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.21.1902132159230.1659@nanos.tec.linutronix.de \
--to=tglx@linutronix.de \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=helgaas@kernel.org \
--cc=keith.busch@intel.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.