All of lore.kernel.org
 help / color / mirror / Atom feed
From: Sven Peter <sven@svenpeter.dev>
To: Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@fb.com>,
	Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>
Cc: Sven Peter <sven@svenpeter.dev>, Hector Martin <marcan@marcan.st>,
	Alyssa Rosenzweig <alyssa@rosenzweig.io>,
	Rob Herring <robh+dt@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
	Marc Zyngier <maz@kernel.org>,
	devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
	Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 7/9] nvme-apple: Serialize command issue
Date: Mon, 21 Mar 2022 17:50:47 +0100	[thread overview]
Message-ID: <20220321165049.35985-8-sven@svenpeter.dev> (raw)
In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev>

From: Jens Axboe <axboe@kernel.dk>

This controller shouldn't need serialization of command issue since
the SQ is replaced by a simple array and commands are issued by writing
the array index to a MMIO register.
Without serialization however sometimes commands are still executed
correctly and appear in the CQ but never trigger an interrupt.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
[sven: added our best guess why this needs to be done]
Signed-off-by: Sven Peter <sven@svenpeter.dev>
---
 drivers/nvme/host/apple.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c
index 587d6c7014a0..a4193429564e 100644
--- a/drivers/nvme/host/apple.c
+++ b/drivers/nvme/host/apple.c
@@ -292,6 +292,7 @@ static void apple_nvmmu_inval(struct apple_nvme_queue *q, unsigned int tag)
 static void apple_nvme_submit_cmd(struct apple_nvme_queue *q,
 				  struct nvme_command *cmd)
 {
+	struct apple_nvme *anv = queue_to_apple_nvme(q);
 	u32 tag = nvme_tag_from_cid(cmd->common.command_id);
 	struct apple_nvmmu_tcb *tcb = &q->tcbs[tag];
 
@@ -308,7 +309,18 @@ static void apple_nvme_submit_cmd(struct apple_nvme_queue *q,
 		tcb->dma_flags |= APPLE_ANS_TCB_DMA_FROM_DEVICE;
 
 	memcpy(&q->sqes[tag], cmd, sizeof(*cmd));
+
+	/*
+	 * This lock here doesn't make much sense at a first glace but
+	 * removing it will result in occasional missed completetion
+	 * interrupts even though the commands still appear on the CQ.
+	 * It's unclear why this happens but our best guess is that
+	 * there is a bug in the firmware triggered when a write to the
+	 * CQ and the SQ happen simultaneously.
+	 */
+	spin_lock_irq(&anv->lock);
 	writel(tag, q->sq_db);
+	spin_unlock_irq(&anv->lock);
 }
 
 /*
-- 
2.25.1


WARNING: multiple messages have this Message-ID (diff)
From: Sven Peter <sven@svenpeter.dev>
To: Keith Busch <kbusch@kernel.org>, Jens Axboe <axboe@fb.com>,
	Christoph Hellwig <hch@lst.de>, Sagi Grimberg <sagi@grimberg.me>
Cc: Sven Peter <sven@svenpeter.dev>, Hector Martin <marcan@marcan.st>,
	Alyssa Rosenzweig <alyssa@rosenzweig.io>,
	Rob Herring <robh+dt@kernel.org>, Arnd Bergmann <arnd@arndb.de>,
	Marc Zyngier <maz@kernel.org>,
	devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org,
	Jens Axboe <axboe@kernel.dk>
Subject: [PATCH 7/9] nvme-apple: Serialize command issue
Date: Mon, 21 Mar 2022 17:50:47 +0100	[thread overview]
Message-ID: <20220321165049.35985-8-sven@svenpeter.dev> (raw)
In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev>

From: Jens Axboe <axboe@kernel.dk>

This controller shouldn't need serialization of command issue since
the SQ is replaced by a simple array and commands are issued by writing
the array index to a MMIO register.
Without serialization however sometimes commands are still executed
correctly and appear in the CQ but never trigger an interrupt.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
[sven: added our best guess why this needs to be done]
Signed-off-by: Sven Peter <sven@svenpeter.dev>
---
 drivers/nvme/host/apple.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c
index 587d6c7014a0..a4193429564e 100644
--- a/drivers/nvme/host/apple.c
+++ b/drivers/nvme/host/apple.c
@@ -292,6 +292,7 @@ static void apple_nvmmu_inval(struct apple_nvme_queue *q, unsigned int tag)
 static void apple_nvme_submit_cmd(struct apple_nvme_queue *q,
 				  struct nvme_command *cmd)
 {
+	struct apple_nvme *anv = queue_to_apple_nvme(q);
 	u32 tag = nvme_tag_from_cid(cmd->common.command_id);
 	struct apple_nvmmu_tcb *tcb = &q->tcbs[tag];
 
@@ -308,7 +309,18 @@ static void apple_nvme_submit_cmd(struct apple_nvme_queue *q,
 		tcb->dma_flags |= APPLE_ANS_TCB_DMA_FROM_DEVICE;
 
 	memcpy(&q->sqes[tag], cmd, sizeof(*cmd));
+
+	/*
+	 * This lock here doesn't make much sense at a first glace but
+	 * removing it will result in occasional missed completetion
+	 * interrupts even though the commands still appear on the CQ.
+	 * It's unclear why this happens but our best guess is that
+	 * there is a bug in the firmware triggered when a write to the
+	 * CQ and the SQ happen simultaneously.
+	 */
+	spin_lock_irq(&anv->lock);
 	writel(tag, q->sq_db);
+	spin_unlock_irq(&anv->lock);
 }
 
 /*
-- 
2.25.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

  parent reply	other threads:[~2022-03-21 16:52 UTC|newest]

Thread overview: 114+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-03-21 16:50 [PATCH 0/9] Apple M1 (Pro/Max) NVMe driver Sven Peter
2022-03-21 16:50 ` Sven Peter
2022-03-21 16:50 ` Sven Peter
2022-03-21 16:50 ` [PATCH 1/9] dt-bindings: soc: apple: Add Apple SART Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-31 21:23   ` Rob Herring
2022-03-31 21:23     ` Rob Herring
2022-04-02 12:58     ` Sven Peter
2022-04-02 12:58       ` Sven Peter
2022-03-21 16:50 ` [PATCH 2/9] dt-bindings: soc: apple: Add ANS NVMe Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-23 11:14   ` Krzysztof Kozlowski
2022-03-23 11:14     ` Krzysztof Kozlowski
2022-04-02 13:05     ` Sven Peter
2022-04-02 13:05       ` Sven Peter
2022-04-02 16:06       ` Krzysztof Kozlowski
2022-04-02 16:06         ` Krzysztof Kozlowski
2022-03-21 16:50 ` [PATCH 3/9] soc: apple: Always include Makefile Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-21 16:50 ` [PATCH 4/9] soc: apple: Add SART driver Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-21 17:07   ` Arnd Bergmann
2022-03-21 17:07     ` Arnd Bergmann
2022-04-02 12:38     ` Sven Peter
2022-04-02 12:38       ` Sven Peter
2022-04-02 19:07       ` Arnd Bergmann
2022-04-02 19:07         ` Arnd Bergmann
2022-04-04 14:58         ` Rob Herring
2022-04-04 14:58           ` Rob Herring
2022-04-04 15:01           ` Hector Martin
2022-04-04 15:01             ` Hector Martin
2022-04-05 15:37           ` Sven Peter
2022-04-05 15:37             ` Sven Peter
2022-03-21 17:17   ` Alyssa Rosenzweig
2022-03-21 17:17     ` Alyssa Rosenzweig
2022-04-02 12:40     ` Sven Peter
2022-04-02 12:40       ` Sven Peter
2022-03-21 16:50 ` [PATCH 5/9] soc: apple: Add RTKit IPC library Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-22 13:13   ` Arnd Bergmann
2022-03-22 13:13     ` Arnd Bergmann
2022-03-22 17:41     ` Robin Murphy
2022-03-22 17:41       ` Robin Murphy
2022-04-02 12:56     ` Sven Peter
2022-04-02 12:56       ` Sven Peter
2022-04-02 18:30       ` Arnd Bergmann
2022-04-02 18:30         ` Arnd Bergmann
2022-04-03 10:45         ` Sven Peter
2022-04-03 10:45           ` Sven Peter
2022-03-22 17:23   ` Alyssa Rosenzweig
2022-03-22 17:23     ` Alyssa Rosenzweig
2022-04-02 12:50     ` Sven Peter
2022-04-02 12:50       ` Sven Peter
2022-03-23 11:19   ` Krzysztof Kozlowski
2022-03-23 11:19     ` Krzysztof Kozlowski
2022-04-02 13:51     ` Sven Peter
2022-04-02 13:51       ` Sven Peter
2022-04-02 16:08       ` Krzysztof Kozlowski
2022-04-02 16:08         ` Krzysztof Kozlowski
2022-04-04 15:02       ` Rob Herring
2022-04-04 15:02         ` Rob Herring
2022-04-04 15:47         ` Hector Martin
2022-04-04 15:47           ` Hector Martin
2022-03-21 16:50 ` [PATCH 6/9] nvme-apple: Add initial Apple SoC NVMe driver Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-21 17:01   ` Keith Busch
2022-03-21 17:01     ` Keith Busch
2022-04-02 13:10     ` Sven Peter
2022-04-02 13:10       ` Sven Peter
2022-03-22 13:38   ` Arnd Bergmann
2022-03-22 13:38     ` Arnd Bergmann
2022-04-02 13:34     ` Sven Peter
2022-04-02 13:34       ` Sven Peter
2022-04-02 13:58       ` Janne Grunau
2022-04-02 13:58         ` Janne Grunau
2022-04-02 14:02         ` Sven Peter
2022-04-02 14:02           ` Sven Peter
2022-04-02 21:26       ` Arnd Bergmann
2022-04-02 21:26         ` Arnd Bergmann
2022-03-24  6:16   ` Christoph Hellwig
2022-03-24  6:16     ` Christoph Hellwig
2022-04-02 12:47     ` Sven Peter
2022-04-02 12:47       ` Sven Peter
2022-04-04 15:57     ` Hector Martin
2022-04-04 15:57       ` Hector Martin
2022-04-04 15:59       ` Christoph Hellwig
2022-04-04 15:59         ` Christoph Hellwig
2022-04-04 16:03         ` Sven Peter
2022-04-04 16:03           ` Sven Peter
2022-04-04 16:05           ` hch
2022-04-04 16:05             ` hch
2022-04-04 16:05         ` Hector Martin
2022-04-04 16:05           ` Hector Martin
2022-04-04 18:29         ` Jens Axboe
2022-04-04 18:29           ` Jens Axboe
2022-04-05  6:24           ` Christoph Hellwig
2022-04-05  6:24             ` Christoph Hellwig
2022-03-21 16:50 ` Sven Peter [this message]
2022-03-21 16:50   ` [PATCH 7/9] nvme-apple: Serialize command issue Sven Peter
2022-03-24  6:16   ` Christoph Hellwig
2022-03-24  6:16     ` Christoph Hellwig
2022-04-02 12:42     ` Sven Peter
2022-04-02 12:42       ` Sven Peter
2022-03-21 16:50 ` [PATCH 8/9] nvme-apple: Add support for multiple power domains Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-21 16:50 ` [PATCH 9/9] nvme-apple: Add support for suspend/resume Sven Peter
2022-03-21 16:50   ` Sven Peter
2022-03-24  6:17   ` Christoph Hellwig
2022-03-24  6:17     ` Christoph Hellwig
2022-03-22 17:26 ` [PATCH 0/9] Apple M1 (Pro/Max) NVMe driver Alyssa Rosenzweig
2022-03-22 17:26   ` Alyssa Rosenzweig

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20220321165049.35985-8-sven@svenpeter.dev \
    --to=sven@svenpeter.dev \
    --cc=alyssa@rosenzweig.io \
    --cc=arnd@arndb.de \
    --cc=axboe@fb.com \
    --cc=axboe@kernel.dk \
    --cc=devicetree@vger.kernel.org \
    --cc=hch@lst.de \
    --cc=kbusch@kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-nvme@lists.infradead.org \
    --cc=marcan@marcan.st \
    --cc=maz@kernel.org \
    --cc=robh+dt@kernel.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.