From: Ming Lei <ming.lei@redhat.com>
To: Keith Busch <keith.busch@intel.com>
Cc: Jens Axboe <axboe@kernel.dk>, Christoph Hellwig <hch@lst.de>,
Sagi Grimberg <sagi@grimberg.me>,
linux-nvme@lists.infradead.org,
Bjorn Helgaas <helgaas@kernel.org>,
linux-pci@vger.kernel.org
Subject: Re: [PATCHv2 2/4] nvme-pci: Distribute io queue types after creation
Date: Fri, 4 Jan 2019 15:21:07 +0800 [thread overview]
Message-ID: <20190104072106.GA9948@ming.t460p> (raw)
In-Reply-To: <20190104023121.GB31330@ming.t460p>
On Fri, Jan 04, 2019 at 10:31:21AM +0800, Ming Lei wrote:
> On Thu, Jan 03, 2019 at 03:50:31PM -0700, Keith Busch wrote:
> > The dev->io_queues types were set based on the results of the nvme set
> > feature "number of queues" and the IRQ allocation. This result does not
> > mean we're going to successfully allocate and create those IO queues,
> > though. A failure there will cause blk-mq to have NULL hctx's because the
> > map's nr_hw_queues accounts for more queues than were actually created.
> >
> > Adjust the io_queue types after we've created them when we have less than
> > originally desired.
> >
> > Fixes: 3b6592f70ad7b ("nvme: utilize two queue maps, one for reads and one for writes")
> > Signed-off-by: Keith Busch <keith.busch@intel.com>
> > ---
> > drivers/nvme/host/pci.c | 46 ++++++++++++++++++++++++++++++++++++++++------
> > 1 file changed, 40 insertions(+), 6 deletions(-)
> >
> > diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> > index 98332d0a80f0..1481bb6d9c42 100644
> > --- a/drivers/nvme/host/pci.c
> > +++ b/drivers/nvme/host/pci.c
> > @@ -1733,6 +1733,30 @@ static int nvme_pci_configure_admin_queue(struct nvme_dev *dev)
> > return result;
> > }
> >
> > +static void nvme_distribute_queues(struct nvme_dev *dev, unsigned int io_queues)
> > +{
> > + unsigned int irq_queues, this_p_queues = dev->io_queues[HCTX_TYPE_POLL],
> > + this_w_queues = dev->io_queues[HCTX_TYPE_DEFAULT];
> > +
> > + if (!io_queues) {
> > + dev->io_queues[HCTX_TYPE_POLL] = 0;
> > + dev->io_queues[HCTX_TYPE_DEFAULT] = 0;
> > + dev->io_queues[HCTX_TYPE_READ] = 0;
> > + return;
> > + }
> > +
> > + if (this_p_queues >= io_queues)
> > + this_p_queues = io_queues - 1;
> > + irq_queues = io_queues - this_p_queues;
> > +
> > + if (this_w_queues > irq_queues)
> > + this_w_queues = irq_queues;
> > +
> > + dev->io_queues[HCTX_TYPE_POLL] = this_p_queues;
> > + dev->io_queues[HCTX_TYPE_DEFAULT] = this_w_queues;
> > + dev->io_queues[HCTX_TYPE_READ] = irq_queues - this_w_queues;
> > +}
> > +
> > static int nvme_create_io_queues(struct nvme_dev *dev)
> > {
> > unsigned i, max, rw_queues;
> > @@ -1761,6 +1785,13 @@ static int nvme_create_io_queues(struct nvme_dev *dev)
> > break;
> > }
> >
> > + /*
> > + * If we've created less than expected io queues, redistribute the
> > + * dev->io_queues[] types accordingly.
> > + */
> > + if (dev->online_queues - 1 != dev->max_qid)
> > + nvme_distribute_queues(dev, dev->online_queues - 1);
> > +
> > /*
> > * Ignore failing Create SQ/CQ commands, we can continue with less
> > * than the desired amount of queues, and even a controller without
> > @@ -2185,11 +2216,6 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
> > result = max(result - 1, 1);
> > dev->max_qid = result + dev->io_queues[HCTX_TYPE_POLL];
> >
> > - dev_info(dev->ctrl.device, "%d/%d/%d default/read/poll queues\n",
> > - dev->io_queues[HCTX_TYPE_DEFAULT],
> > - dev->io_queues[HCTX_TYPE_READ],
> > - dev->io_queues[HCTX_TYPE_POLL]);
> > -
> > /*
> > * Should investigate if there's a performance win from allocating
> > * more queues than interrupt vectors; it might allow the submission
> > @@ -2203,7 +2229,15 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
> > return result;
> > }
> > set_bit(NVMEQ_ENABLED, &adminq->flags);
> > - return nvme_create_io_queues(dev);
> > + result = nvme_create_io_queues(dev);
> > +
> > + if (!result)
> > + dev_info(dev->ctrl.device, "%d/%d/%d default/read/poll queues\n",
> > + dev->io_queues[HCTX_TYPE_DEFAULT],
> > + dev->io_queues[HCTX_TYPE_READ],
> > + dev->io_queues[HCTX_TYPE_POLL]);
> > + return result;
> > +
> > }
> >
> > static void nvme_del_queue_end(struct request *req, blk_status_t error)
> > --
> > 2.14.4
> >
>
> This way should be better given it covers irq allocation failure and
> queue creating/initialization failure.
>
> Reviewed-by: Ming Lei <ming.lei@redhat.com>
Thinking about the patch further: after pci_alloc_irq_vectors_affinity()
is returned, queue number for non-polled queues can't be changed at will,
because we have to make sure to spread all CPUs on each queue type, and
the mapping has been fixed by pci_alloc_irq_vectors_affinity() already.
So looks the approach in this patch may be wrong.
Thanks,
Ming
next prev parent reply other threads:[~2019-01-04 7:21 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-03 22:50 [PATCHv2 0/4] NVMe IRQ sets fixups Keith Busch
2019-01-03 22:50 ` [PATCHv2 1/4] nvme-pci: Set tagset nr_maps just once Keith Busch
2019-01-04 1:46 ` Ming Lei
2019-01-03 22:50 ` [PATCHv2 2/4] nvme-pci: Distribute io queue types after creation Keith Busch
2019-01-04 2:31 ` Ming Lei
2019-01-04 7:21 ` Ming Lei [this message]
2019-01-04 15:53 ` Keith Busch
2019-01-04 18:17 ` Christoph Hellwig
2019-01-04 18:35 ` Keith Busch
2019-01-06 2:56 ` Ming Lei
2019-01-03 22:50 ` [PATCHv2 3/4] PCI/MSI: Handle vector reduce and retry Keith Busch
2019-01-04 2:45 ` Ming Lei
2019-01-04 22:35 ` Bjorn Helgaas
2019-01-04 22:56 ` Keith Busch
2019-01-03 22:50 ` [PATCHv2 4/4] nvme-pci: Use PCI to handle IRQ " Keith Busch
2019-01-04 2:41 ` Ming Lei
2019-01-04 18:19 ` Christoph Hellwig
2019-01-04 18:33 ` Keith Busch
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190104072106.GA9948@ming.t460p \
--to=ming.lei@redhat.com \
--cc=axboe@kernel.dk \
--cc=hch@lst.de \
--cc=helgaas@kernel.org \
--cc=keith.busch@intel.com \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).