From: Benjamin Herrenschmidt <benh@kernel.crashing.org> To: linux-nvme@lists.infradead.org Cc: linux-kernel@vger.kernel.org, Paul Pawlowski <paul@mrarm.io>, Jens Axboe <axboe@fb.com>, Keith Busch <kbusch@kernel.org>, Christoph Hellwig <hch@lst.de>, Minwoo Im <minwoo.im.dev@gmail.com>, Damien Le Moal <Damien.LeMoal@wdc.com> Subject: [PATCH v3] nvme-pci: Support shared tags across queues for Apple 2018 controllers Date: Fri, 19 Jul 2019 15:31:02 +1000 [thread overview] Message-ID: <b1f9bdf0294b8d87d292de3c7462c8e99551b02d.camel@kernel.crashing.org> (raw) From 8dcba2ef5b1466b023b88b4eca463b30de78d9eb Mon Sep 17 00:00:00 2001 From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: Fri, 19 Jul 2019 15:03:06 +1000 Subject: Another issue with the Apple T2 based 2018 controllers seem to be that they blow up (and shut the machine down) if there's a tag collision between the IO queue and the Admin queue. My suspicion is that they use our tags for their internal tracking and don't mix them with the queue id. They also seem to not like when tags go beyond the IO queue depth, ie 128 tags. This adds a quirk that marks tags 0..31 of the IO queue reserved Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org> --- Thanks Damien, reserved tags work and make this a lot simpler ! drivers/nvme/host/nvme.h | 5 +++++ drivers/nvme/host/pci.c | 19 ++++++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index ced0e0a7e039..8732da6df555 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -102,6 +102,11 @@ enum nvme_quirks { * Use non-standard 128 bytes SQEs. */ NVME_QUIRK_128_BYTES_SQES = (1 << 11), + + /* + * Prevent tag overlap between queues + */ + NVME_QUIRK_SHARED_TAGS = (1 << 12), }; /* diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7088971d4c42..fc74395a028b 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2106,6 +2106,14 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) unsigned long size; nr_io_queues = max_io_queues(); + + /* + * If tags are shared with admin queue (Apple bug), then + * make sure we only use one IO queue. + */ + if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS) + nr_io_queues = 1; + result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues); if (result < 0) return result; @@ -2278,6 +2286,14 @@ static int nvme_dev_add(struct nvme_dev *dev) dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE; dev->tagset.driver_data = dev; + /* + * Some Apple controllers requires tags to be unique + * across admin and IO queue, so reserve the first 32 + * tags of the IO queue. + */ + if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS) + dev->tagset.reserved_tags = NVME_AQ_DEPTH; + ret = blk_mq_alloc_tag_set(&dev->tagset); if (ret) { dev_warn(dev->ctrl.device, @@ -3057,7 +3073,8 @@ static const struct pci_device_id nvme_id_table[] = { { PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2003) }, { PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2005), .driver_data = NVME_QUIRK_SINGLE_VECTOR | - NVME_QUIRK_128_BYTES_SQES }, + NVME_QUIRK_128_BYTES_SQES | + NVME_QUIRK_SHARED_TAGS }, { 0, } }; MODULE_DEVICE_TABLE(pci, nvme_id_table);
WARNING: multiple messages have this Message-ID (diff)
From: benh@kernel.crashing.org (Benjamin Herrenschmidt) Subject: [PATCH v3] nvme-pci: Support shared tags across queues for Apple 2018 controllers Date: Fri, 19 Jul 2019 15:31:02 +1000 [thread overview] Message-ID: <b1f9bdf0294b8d87d292de3c7462c8e99551b02d.camel@kernel.crashing.org> (raw) >From 8dcba2ef5b1466b023b88b4eca463b30de78d9eb Mon Sep 17 00:00:00 2001 From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: Fri, 19 Jul 2019 15:03:06 +1000 Subject: Another issue with the Apple T2 based 2018 controllers seem to be that they blow up (and shut the machine down) if there's a tag collision between the IO queue and the Admin queue. My suspicion is that they use our tags for their internal tracking and don't mix them with the queue id. They also seem to not like when tags go beyond the IO queue depth, ie 128 tags. This adds a quirk that marks tags 0..31 of the IO queue reserved Signed-off-by: Benjamin Herrenschmidt <benh at kernel.crashing.org> --- Thanks Damien, reserved tags work and make this a lot simpler ! drivers/nvme/host/nvme.h | 5 +++++ drivers/nvme/host/pci.c | 19 ++++++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index ced0e0a7e039..8732da6df555 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -102,6 +102,11 @@ enum nvme_quirks { * Use non-standard 128 bytes SQEs. */ NVME_QUIRK_128_BYTES_SQES = (1 << 11), + + /* + * Prevent tag overlap between queues + */ + NVME_QUIRK_SHARED_TAGS = (1 << 12), }; /* diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7088971d4c42..fc74395a028b 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -2106,6 +2106,14 @@ static int nvme_setup_io_queues(struct nvme_dev *dev) unsigned long size; nr_io_queues = max_io_queues(); + + /* + * If tags are shared with admin queue (Apple bug), then + * make sure we only use one IO queue. + */ + if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS) + nr_io_queues = 1; + result = nvme_set_queue_count(&dev->ctrl, &nr_io_queues); if (result < 0) return result; @@ -2278,6 +2286,14 @@ static int nvme_dev_add(struct nvme_dev *dev) dev->tagset.flags = BLK_MQ_F_SHOULD_MERGE; dev->tagset.driver_data = dev; + /* + * Some Apple controllers requires tags to be unique + * across admin and IO queue, so reserve the first 32 + * tags of the IO queue. + */ + if (dev->ctrl.quirks & NVME_QUIRK_SHARED_TAGS) + dev->tagset.reserved_tags = NVME_AQ_DEPTH; + ret = blk_mq_alloc_tag_set(&dev->tagset); if (ret) { dev_warn(dev->ctrl.device, @@ -3057,7 +3073,8 @@ static const struct pci_device_id nvme_id_table[] = { { PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2003) }, { PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2005), .driver_data = NVME_QUIRK_SINGLE_VECTOR | - NVME_QUIRK_128_BYTES_SQES }, + NVME_QUIRK_128_BYTES_SQES | + NVME_QUIRK_SHARED_TAGS }, { 0, } }; MODULE_DEVICE_TABLE(pci, nvme_id_table);
next reply other threads:[~2019-07-19 5:31 UTC|newest] Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-07-19 5:31 Benjamin Herrenschmidt [this message] 2019-07-19 5:31 ` [PATCH v3] nvme-pci: Support shared tags across queues for Apple 2018 controllers Benjamin Herrenschmidt 2019-07-19 5:45 ` Balbir Singh 2019-07-19 5:45 ` Balbir Singh 2019-07-19 12:28 ` Christoph Hellwig 2019-07-19 12:28 ` Christoph Hellwig 2019-07-19 13:51 ` Benjamin Herrenschmidt 2019-07-19 13:51 ` Benjamin Herrenschmidt 2019-07-22 21:54 ` Benjamin Herrenschmidt 2019-07-22 21:54 ` Benjamin Herrenschmidt 2019-07-23 1:36 ` Ming Lei 2019-07-23 1:36 ` Ming Lei 2019-07-30 15:30 ` Keith Busch 2019-07-30 15:30 ` Keith Busch 2019-07-30 20:15 ` Benjamin Herrenschmidt 2019-07-30 20:15 ` Benjamin Herrenschmidt 2019-07-30 20:28 ` Benjamin Herrenschmidt 2019-07-30 20:28 ` Benjamin Herrenschmidt 2019-08-05 6:49 ` Benjamin Herrenschmidt 2019-08-05 6:49 ` Benjamin Herrenschmidt 2019-08-05 13:49 ` Keith Busch 2019-08-05 13:49 ` Keith Busch 2019-08-05 18:27 ` Sagi Grimberg 2019-08-05 18:27 ` Sagi Grimberg 2019-08-05 18:35 ` Keith Busch 2019-08-05 18:35 ` Keith Busch 2019-08-05 19:07 ` Jens Axboe 2019-08-05 19:07 ` Jens Axboe 2019-08-05 19:56 ` Sagi Grimberg 2019-08-05 19:56 ` Sagi Grimberg 2019-08-05 20:07 ` Sagi Grimberg 2019-08-05 20:07 ` Sagi Grimberg 2019-08-06 5:26 ` Benjamin Herrenschmidt 2019-08-06 5:26 ` Benjamin Herrenschmidt 2019-08-06 5:26 ` Benjamin Herrenschmidt 2019-08-06 5:26 ` Benjamin Herrenschmidt 2019-08-06 5:24 ` Benjamin Herrenschmidt 2019-08-06 5:24 ` Benjamin Herrenschmidt
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=b1f9bdf0294b8d87d292de3c7462c8e99551b02d.camel@kernel.crashing.org \ --to=benh@kernel.crashing.org \ --cc=Damien.LeMoal@wdc.com \ --cc=axboe@fb.com \ --cc=hch@lst.de \ --cc=kbusch@kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=minwoo.im.dev@gmail.com \ --cc=paul@mrarm.io \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.