From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1F64C433F5 for ; Mon, 21 Mar 2022 16:52:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351494AbiCUQyC (ORCPT ); Mon, 21 Mar 2022 12:54:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351475AbiCUQxv (ORCPT ); Mon, 21 Mar 2022 12:53:51 -0400 Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 755FA174BA1 for ; Mon, 21 Mar 2022 09:52:23 -0700 (PDT) Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 8343E3201DA2; Mon, 21 Mar 2022 12:52:21 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 21 Mar 2022 12:52:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=svenpeter.dev; h=cc:cc:content-transfer-encoding:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm1; bh=5nQpEZL6uoaGwo qyCOgh7yCZkXCs5xzAO7/dEUKTT+0=; b=CAtQijKbAa1HWvg5Tpk7zi2d5FkuFF aLTNGI76WDQ22B00KJKLI34LbvcaU1EIdqWbga6GcEcquNCAtkIuNQnX1gkJ0CfS qeynsyLAicgu0w6mh4UFOfN4I+UhwfFJ6fQ/dGF4xV1lCG2CeB8L3oXexxLX3MTu cFsXC7a+o3WyWjek76mwkgfXU4lLpPkcRT3U4zEk+V+aYop+0oDBoyXy6oXm5Jx8 CN75O1hNTC0zvOKD+4Gn/4VmVnzKoa+5X+2lNikK/I/AQutTfItUqs3hTJF6LRo9 zvrtvBug2La3VEmVJbGH2BYeE0yKN2OvU+augQNt/FhqElCdFoxaAL6w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=5nQpEZ L6uoaGwoqyCOgh7yCZkXCs5xzAO7/dEUKTT+0=; b=E9MPCUh24viES5QQonaStG Emdx3GaIzqVI3u1pDCc4LgmYtxl6uSuFQVewDbo4g96i/1oEtGVwFHToqzhEXq5d pB/oaaimaAOP6G80vau2wYTzqllX1aS2GOzf+KvyJJ+MH6EBMAy/WcSLEbgjcQyG KQwCWEI9v0vLJAHDRN5Xmt8yjxyttI0VX1ZRMwdKQc5loImHSCA3h3jFsQU8jyt5 GZXH4ofkItVZUff2bF34rt21WcQheTlId9TDDPKi5GR/Z22LmeiqSEdKErISxFSk xmunL5eQz6XaZNcMv+1hhlXEsBa61MwIxsOKhRUfnx5OIL9ekpR6CrEMuFYIfHdQ == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvvddrudegfedgleehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvufffkffojghfggfgsedtkeertdertddtnecuhfhrohhmpefuvhgvnhcu rfgvthgvrhcuoehsvhgvnhesshhvvghnphgvthgvrhdruggvvheqnecuggftrfgrthhtvg hrnheptedvkeetleeuffffhfekteetffeggffgveehieelueefvddtueffveevlefhfeej necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepshhvvg hnsehsvhgvnhhpvghtvghrrdguvghv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 21 Mar 2022 12:52:18 -0400 (EDT) From: Sven Peter To: Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg Cc: Sven Peter , Hector Martin , Alyssa Rosenzweig , Rob Herring , Arnd Bergmann , Marc Zyngier , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Jens Axboe Subject: [PATCH 7/9] nvme-apple: Serialize command issue Date: Mon, 21 Mar 2022 17:50:47 +0100 Message-Id: <20220321165049.35985-8-sven@svenpeter.dev> X-Mailer: git-send-email 2.30.1 (Apple Git-130) In-Reply-To: <20220321165049.35985-1-sven@svenpeter.dev> References: <20220321165049.35985-1-sven@svenpeter.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Jens Axboe This controller shouldn't need serialization of command issue since the SQ is replaced by a simple array and commands are issued by writing the array index to a MMIO register. Without serialization however sometimes commands are still executed correctly and appear in the CQ but never trigger an interrupt. Signed-off-by: Jens Axboe [sven: added our best guess why this needs to be done] Signed-off-by: Sven Peter --- drivers/nvme/host/apple.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index 587d6c7014a0..a4193429564e 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -292,6 +292,7 @@ static void apple_nvmmu_inval(struct apple_nvme_queue *q, unsigned int tag) static void apple_nvme_submit_cmd(struct apple_nvme_queue *q, struct nvme_command *cmd) { + struct apple_nvme *anv = queue_to_apple_nvme(q); u32 tag = nvme_tag_from_cid(cmd->common.command_id); struct apple_nvmmu_tcb *tcb = &q->tcbs[tag]; @@ -308,7 +309,18 @@ static void apple_nvme_submit_cmd(struct apple_nvme_queue *q, tcb->dma_flags |= APPLE_ANS_TCB_DMA_FROM_DEVICE; memcpy(&q->sqes[tag], cmd, sizeof(*cmd)); + + /* + * This lock here doesn't make much sense at a first glace but + * removing it will result in occasional missed completetion + * interrupts even though the commands still appear on the CQ. + * It's unclear why this happens but our best guess is that + * there is a bug in the firmware triggered when a write to the + * CQ and the SQ happen simultaneously. + */ + spin_lock_irq(&anv->lock); writel(tag, q->sq_db); + spin_unlock_irq(&anv->lock); } /* -- 2.25.1