* [PATCH 0/2] nvme-pic: improve max I/O queue handling
@ 2020-11-12 8:23 Niklas Schnelle
2020-11-12 8:23 ` [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment Niklas Schnelle
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Niklas Schnelle @ 2020-11-12 8:23 UTC (permalink / raw)
To: linux-nvme
Cc: linux-kernel, Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg
Hi,
while searching for a bug around zPCI + NVMe IRQ handling on a distro
kernel, I got confused around handling of the maximum number
of I/O queues in the NVMe driver.
I think I groked it in the end but would like to propose the following
improvements, that said I'm quite new to this code.
I tested both patches on s390x (with a debug config) and x86_64 so
with both data center and consumer NVMes.
For the second patch, since I don't own a device with the quirk, I tried
always returning 1 from nvme_max_io_queues() and confirmed that on my
Evo 970 Pro this resulted in about half the performance in a fio test
but did not otherwise break things. I couldn't find a reason why
allocating only the I/O queues we actually use would be problematic in
the code either but I might have missed something of course.
Best regards,
Niklas Schnelle
Niklas Schnelle (2):
nvme-pci: drop min() from nr_io_queues assignment
nvme-pci: don't allocate unused I/O queues
drivers/nvme/host/pci.c | 17 +++++++----------
1 file changed, 7 insertions(+), 10 deletions(-)
--
2.17.1
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment
2020-11-12 8:23 [PATCH 0/2] nvme-pic: improve max I/O queue handling Niklas Schnelle
@ 2020-11-12 8:23 ` Niklas Schnelle
2020-11-14 9:06 ` Christoph Hellwig
2020-11-12 8:23 ` [PATCH 2/2] nvme-pci: don't allocate unused I/O queues Niklas Schnelle
2020-11-12 14:53 ` [PATCH 0/2] nvme-pic: improve max I/O queue handling Keith Busch
2 siblings, 1 reply; 10+ messages in thread
From: Niklas Schnelle @ 2020-11-12 8:23 UTC (permalink / raw)
To: linux-nvme
Cc: linux-kernel, Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg
in nvme_setup_io_queues() the number of I/O queues is set to either 1 in
case of a quirky Apple device or to the min of nvme_max_io_queues() or
dev->nr_allocated_queues - 1.
This is unnecessarily complicated as dev->nr_allocated_queues is only
assigned once and is nvme_max_io_queues() + 1.
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
drivers/nvme/host/pci.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 0578ff253c47..b56250b83bdf 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2114,8 +2114,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
nr_io_queues = 1;
else
- nr_io_queues = min(nvme_max_io_queues(dev),
- dev->nr_allocated_queues - 1);
+ nr_io_queues = dev->nr_allocated_queues - 1;
result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues);
if (result < 0)
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH 2/2] nvme-pci: don't allocate unused I/O queues
2020-11-12 8:23 [PATCH 0/2] nvme-pic: improve max I/O queue handling Niklas Schnelle
2020-11-12 8:23 ` [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment Niklas Schnelle
@ 2020-11-12 8:23 ` Niklas Schnelle
2020-11-12 14:53 ` [PATCH 0/2] nvme-pic: improve max I/O queue handling Keith Busch
2 siblings, 0 replies; 10+ messages in thread
From: Niklas Schnelle @ 2020-11-12 8:23 UTC (permalink / raw)
To: linux-nvme
Cc: linux-kernel, Keith Busch, Jens Axboe, Christoph Hellwig, Sagi Grimberg
currently the NVME_QUIRK_SHARED_TAGS quirk for Apple devices is handled
during the assignment of nr_io_queues in nvme_setup_io_queues().
This however means that for these devices nvme_max_io_queues() will
actually not return the supported maximum which is confusing and
unexpected and also means that in nvme_probe() we are allocating
for I/O queues that will never be used.
Fix this by moving the quirk handling into nvme_max_io_queues().
Signed-off-by: Niklas Schnelle <schnelle@linux.ibm.com>
---
drivers/nvme/host/pci.c | 16 +++++++---------
1 file changed, 7 insertions(+), 9 deletions(-)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index b56250b83bdf..0f8cea5b30eb 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2089,6 +2089,12 @@ static void nvme_disable_io_queues(struct nvme_dev *dev)
static unsigned int nvme_max_io_queues(struct nvme_dev *dev)
{
+ /*
+ * If tags are shared with admin queue (Apple bug), then
+ * make sure we only use one IO queue.
+ */
+ if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
+ return 1;
return num_possible_cpus() + dev->nr_write_queues + dev->nr_poll_queues;
}
@@ -2107,15 +2113,7 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)
dev->nr_write_queues = write_queues;
dev->nr_poll_queues = poll_queues;
- /*
- * If tags are shared with admin queue (Apple bug), then
- * make sure we only use one IO queue.
- */
- if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS)
- nr_io_queues = 1;
- else
- nr_io_queues = dev->nr_allocated_queues - 1;
-
+ nr_io_queues = dev->nr_allocated_queues - 1;
result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues);
if (result < 0)
return result;
--
2.17.1
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling
2020-11-12 8:23 [PATCH 0/2] nvme-pic: improve max I/O queue handling Niklas Schnelle
2020-11-12 8:23 ` [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment Niklas Schnelle
2020-11-12 8:23 ` [PATCH 2/2] nvme-pci: don't allocate unused I/O queues Niklas Schnelle
@ 2020-11-12 14:53 ` Keith Busch
2020-11-12 15:45 ` Niklas Schnelle
2 siblings, 1 reply; 10+ messages in thread
From: Keith Busch @ 2020-11-12 14:53 UTC (permalink / raw)
To: Niklas Schnelle
Cc: linux-nvme, linux-kernel, Jens Axboe, Christoph Hellwig, Sagi Grimberg
On Thu, Nov 12, 2020 at 09:23:00AM +0100, Niklas Schnelle wrote:
> while searching for a bug around zPCI + NVMe IRQ handling on a distro
> kernel, I got confused around handling of the maximum number
> of I/O queues in the NVMe driver.
> I think I groked it in the end but would like to propose the following
> improvements, that said I'm quite new to this code.
> I tested both patches on s390x (with a debug config) and x86_64 so
> with both data center and consumer NVMes.
> For the second patch, since I don't own a device with the quirk, I tried
> always returning 1 from nvme_max_io_queues() and confirmed that on my
> Evo 970 Pro this resulted in about half the performance in a fio test
> but did not otherwise break things. I couldn't find a reason why
> allocating only the I/O queues we actually use would be problematic in
> the code either but I might have missed something of course.
I don't think you missed anything, and the series looks like a
reasonable cleanup. I suspect the code was left over from a time when we
didn't allocate the possible queues up-front.
Reviewed-by: Keith Busch <kbusch@kernel.org>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling
2020-11-12 14:53 ` [PATCH 0/2] nvme-pic: improve max I/O queue handling Keith Busch
@ 2020-11-12 15:45 ` Niklas Schnelle
2020-11-12 17:36 ` Keith Busch
0 siblings, 1 reply; 10+ messages in thread
From: Niklas Schnelle @ 2020-11-12 15:45 UTC (permalink / raw)
To: Keith Busch
Cc: linux-nvme, linux-kernel, Jens Axboe, Christoph Hellwig, Sagi Grimberg
On 11/12/20 3:53 PM, Keith Busch wrote:
> On Thu, Nov 12, 2020 at 09:23:00AM +0100, Niklas Schnelle wrote:
>> while searching for a bug around zPCI + NVMe IRQ handling on a distro
>> kernel, I got confused around handling of the maximum number
>> of I/O queues in the NVMe driver.
>> I think I groked it in the end but would like to propose the following
>> improvements, that said I'm quite new to this code.
>> I tested both patches on s390x (with a debug config) and x86_64 so
>> with both data center and consumer NVMes.
>> For the second patch, since I don't own a device with the quirk, I tried
>> always returning 1 from nvme_max_io_queues() and confirmed that on my
>> Evo 970 Pro this resulted in about half the performance in a fio test
>> but did not otherwise break things. I couldn't find a reason why
>> allocating only the I/O queues we actually use would be problematic in
>> the code either but I might have missed something of course.
>
> I don't think you missed anything, and the series looks like a
> reasonable cleanup. I suspect the code was left over from a time when we
> didn't allocate the possible queues up-front.
>
> Reviewed-by: Keith Busch <kbusch@kernel.org>
>
You got to get something wrong, I hope in this case it's just the subject
of the cover letter :D
Thanks for the review, I appreciate it. Might be getting ahead of
myself but I'm curious who would take this change through their
tree if accepted?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling
2020-11-12 15:45 ` Niklas Schnelle
@ 2020-11-12 17:36 ` Keith Busch
2020-11-13 13:15 ` Niklas Schnelle
0 siblings, 1 reply; 10+ messages in thread
From: Keith Busch @ 2020-11-12 17:36 UTC (permalink / raw)
To: Niklas Schnelle
Cc: linux-nvme, linux-kernel, Jens Axboe, Christoph Hellwig, Sagi Grimberg
On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
> You got to get something wrong, I hope in this case it's just the subject
> of the cover letter :D
I suppose the change logs could be worded a little better :)
> Thanks for the review, I appreciate it. Might be getting ahead of
> myself but I'm curious who would take this change through their
> tree if accepted?
The linux-nvme tree is located here:
http://git.infradead.org/nvme.git
Christoph is currently handling patch commits there.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling
2020-11-12 17:36 ` Keith Busch
@ 2020-11-13 13:15 ` Niklas Schnelle
2020-11-13 16:25 ` Christoph Hellwig
0 siblings, 1 reply; 10+ messages in thread
From: Niklas Schnelle @ 2020-11-13 13:15 UTC (permalink / raw)
To: Keith Busch
Cc: linux-nvme, linux-kernel, Jens Axboe, Christoph Hellwig, Sagi Grimberg
On 11/12/20 6:36 PM, Keith Busch wrote:
> On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
>> You got to get something wrong, I hope in this case it's just the subject
>> of the cover letter :D
>
> I suppose the change logs could be worded a little better :)
Do you think I should send a v2 with an improved message?
I just realized I'm pretty alone (and wrong) in starting
the commit message body lower case too.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling
2020-11-13 13:15 ` Niklas Schnelle
@ 2020-11-13 16:25 ` Christoph Hellwig
2020-11-13 16:52 ` Niklas Schnelle
0 siblings, 1 reply; 10+ messages in thread
From: Christoph Hellwig @ 2020-11-13 16:25 UTC (permalink / raw)
To: Niklas Schnelle
Cc: Keith Busch, linux-nvme, linux-kernel, Jens Axboe,
Christoph Hellwig, Sagi Grimberg
On Fri, Nov 13, 2020 at 02:15:59PM +0100, Niklas Schnelle wrote:
>
>
> On 11/12/20 6:36 PM, Keith Busch wrote:
> > On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
> >> You got to get something wrong, I hope in this case it's just the subject
> >> of the cover letter :D
> >
> > I suppose the change logs could be worded a little better :)
>
> Do you think I should send a v2 with an improved message?
> I just realized I'm pretty alone (and wrong) in starting
> the commit message body lower case too.
I'll pull in soon, no need to resend.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 0/2] nvme-pic: improve max I/O queue handling
2020-11-13 16:25 ` Christoph Hellwig
@ 2020-11-13 16:52 ` Niklas Schnelle
0 siblings, 0 replies; 10+ messages in thread
From: Niklas Schnelle @ 2020-11-13 16:52 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Keith Busch, linux-nvme, linux-kernel, Jens Axboe, Sagi Grimberg
On 11/13/20 5:25 PM, Christoph Hellwig wrote:
> On Fri, Nov 13, 2020 at 02:15:59PM +0100, Niklas Schnelle wrote:
>>
>>
>> On 11/12/20 6:36 PM, Keith Busch wrote:
>>> On Thu, Nov 12, 2020 at 04:45:35PM +0100, Niklas Schnelle wrote:
>>>> You got to get something wrong, I hope in this case it's just the subject
>>>> of the cover letter :D
>>>
>>> I suppose the change logs could be worded a little better :)
>>
>> Do you think I should send a v2 with an improved message?
>> I just realized I'm pretty alone (and wrong) in starting
>> the commit message body lower case too.
>
> I'll pull in soon, no need to resend.
>
Thanks, appreciate your effort!
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment
2020-11-12 8:23 ` [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment Niklas Schnelle
@ 2020-11-14 9:06 ` Christoph Hellwig
0 siblings, 0 replies; 10+ messages in thread
From: Christoph Hellwig @ 2020-11-14 9:06 UTC (permalink / raw)
To: Niklas Schnelle
Cc: linux-nvme, linux-kernel, Keith Busch, Jens Axboe,
Christoph Hellwig, Sagi Grimberg
Thanks,
applied both patches to nvme-5.11.
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-11-14 9:06 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-12 8:23 [PATCH 0/2] nvme-pic: improve max I/O queue handling Niklas Schnelle
2020-11-12 8:23 ` [PATCH 1/2] nvme-pci: drop min() from nr_io_queues assignment Niklas Schnelle
2020-11-14 9:06 ` Christoph Hellwig
2020-11-12 8:23 ` [PATCH 2/2] nvme-pci: don't allocate unused I/O queues Niklas Schnelle
2020-11-12 14:53 ` [PATCH 0/2] nvme-pic: improve max I/O queue handling Keith Busch
2020-11-12 15:45 ` Niklas Schnelle
2020-11-12 17:36 ` Keith Busch
2020-11-13 13:15 ` Niklas Schnelle
2020-11-13 16:25 ` Christoph Hellwig
2020-11-13 16:52 ` Niklas Schnelle
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).