From: "Sven Peter" <sven@svenpeter.dev> To: "Arnd Bergmann" <arnd@arndb.de> Cc: "Keith Busch" <kbusch@kernel.org>, "axboe@fb.com" <axboe@fb.com>, "hch@lst.de" <hch@lst.de>, "sagi@grimberg.me" <sagi@grimberg.me>, "Hector Martin" <marcan@marcan.st>, "Alyssa Rosenzweig" <alyssa@rosenzweig.io>, "Rob Herring" <robh+dt@kernel.org>, "Marc Zyngier" <maz@kernel.org>, DTML <devicetree@vger.kernel.org>, "Linux ARM" <linux-arm-kernel@lists.infradead.org>, "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>, linux-nvme@lists.infradead.org Subject: Re: [PATCH 6/9] nvme-apple: Add initial Apple SoC NVMe driver Date: Sat, 02 Apr 2022 15:34:55 +0200 [thread overview] Message-ID: <7876eb84-9803-48dc-ab6b-27d49858e27a@www.fastmail.com> (raw) In-Reply-To: <CAK8P3a27-eWP=krGQOp29krcc7bVME9=MbN6B3Rs7Q4Ran_VFw@mail.gmail.com> On Tue, Mar 22, 2022, at 14:38, Arnd Bergmann wrote: > On Mon, Mar 21, 2022 at 5:50 PM Sven Peter <sven@svenpeter.dev> wrote: > >> +static int apple_nvme_sart_dma_setup(void *cookie, struct apple_rtkit_shmem *bfr, >> + dma_addr_t iova, size_t size) >> +{ >> + struct apple_nvme *anv = cookie; >> + int ret; >> + >> + if (iova) >> + return -EINVAL; >> + >> + bfr->buffer = dma_alloc_coherent(anv->dev, size, &iova, GFP_KERNEL); >> + if (!bfr->buffer) >> + return -ENOMEM; > > You pass 'iova' as an argument, but then replace it with the address > returned by dma_alloc_coherent(). Can you remove the function > argument? Yup, will remove it. > >> +static void apple_nvmmu_inval(struct apple_nvme_queue *q, unsigned int tag) >> +{ >> + struct apple_nvme *anv = queue_to_apple_nvme(q); >> + >> + writel(tag, anv->mmio_nvme + APPLE_NVMMU_TCB_INVAL); >> + if (readl_relaxed(anv->mmio_nvme + APPLE_NVMMU_TCB_STAT)) >> + dev_warn(anv->dev, "NVMMU TCB invalidation failed\n"); >> +} > > I don't like to see the _relaxed() accessors used without an explanation > about why that helps. Please use the non-relaxed version, or make sure > it's obvious here why you use it. Ok, I'll either use the non-relaxed ones or add a comment whenever I use the relaxed version. In this case here there's no write to any DMA buffers that needs to be visible to the device. That writel there could actually be a writel_relaxed as well. There just used to be a write to a buffer above (which is another good reason to always comment when using the non-relaxed ones, maybe then I would've noticed then and updated it). > >> +bad_sgl: >> + WARN(DO_ONCE(apple_nvme_print_sgl, iod->sg, iod->nents), >> + "Invalid SGL for payload:%d nents:%d\n", blk_rq_payload_bytes(req), >> + iod->nents); > > I think you mean WARN_ONCE() here? This is taken from pci.c which used to use WARN_ONCE but was replaced in d08774738446e77734777adcf5d1045237b4475a with this construction here. The commit message mentions The WARN_ONCE macro returns true if the condition is true, not if the warn was raised, so we're printing the scatter list every time it's invalid. This is excessive and makes debugging harder, so this patch prints it just once. > >> + writel_relaxed(0, anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); >> + (void)readl_relaxed(anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); > > What is the purpose of the readl_relaxed() here? It looks like you are > trying to flush > the write to the hardware, but then again > > a) on Apple hardware, the registers are mapped using PROT_DEVICE_nGnRnE, > so MMIO writes are never posted > > b) the read is "_relaxed", so there is no barrier, and the result is > unused, so > it would appear that the CPU can just keep executing code anyway. > > Since this is all the initialization path, I can't imagine what the > relaxation of > the barriers helps with. Agreed, I've actually tried replacing all non-relaxed ones with the normal accessors (even those inside the hot path) and didn't see any performance difference. I can use the normal ones here and I'll consider using the non-relaxed ones in the hot path together with a comment why they are safe in those places. > >> +static int apple_nvme_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val) >> +{ >> + *val = readl_relaxed(ctrl_to_apple_nvme(ctrl)->mmio_nvme + off); >> + return 0; >> +} >> + >> +static int apple_nvme_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val) >> +{ >> + writel_relaxed(val, ctrl_to_apple_nvme(ctrl)->mmio_nvme + off); >> + return 0; >> +} > > If you have generic register access functions, don't make them use > _relaxed internally. If there are instances that need to be _relaxed, > add another version of the accessor that spells this out in the caller. Ok, there are used internally by the nvme core and can't do any kind of DMA right now but I agree that it's better to use the non-relaxed ones here to prevent surprises in the future. Sven
WARNING: multiple messages have this Message-ID (diff)
From: "Sven Peter" <sven@svenpeter.dev> To: "Arnd Bergmann" <arnd@arndb.de> Cc: "Keith Busch" <kbusch@kernel.org>, "axboe@fb.com" <axboe@fb.com>, "hch@lst.de" <hch@lst.de>, "sagi@grimberg.me" <sagi@grimberg.me>, "Hector Martin" <marcan@marcan.st>, "Alyssa Rosenzweig" <alyssa@rosenzweig.io>, "Rob Herring" <robh+dt@kernel.org>, "Marc Zyngier" <maz@kernel.org>, DTML <devicetree@vger.kernel.org>, "Linux ARM" <linux-arm-kernel@lists.infradead.org>, "Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>, linux-nvme@lists.infradead.org Subject: Re: [PATCH 6/9] nvme-apple: Add initial Apple SoC NVMe driver Date: Sat, 02 Apr 2022 15:34:55 +0200 [thread overview] Message-ID: <7876eb84-9803-48dc-ab6b-27d49858e27a@www.fastmail.com> (raw) In-Reply-To: <CAK8P3a27-eWP=krGQOp29krcc7bVME9=MbN6B3Rs7Q4Ran_VFw@mail.gmail.com> On Tue, Mar 22, 2022, at 14:38, Arnd Bergmann wrote: > On Mon, Mar 21, 2022 at 5:50 PM Sven Peter <sven@svenpeter.dev> wrote: > >> +static int apple_nvme_sart_dma_setup(void *cookie, struct apple_rtkit_shmem *bfr, >> + dma_addr_t iova, size_t size) >> +{ >> + struct apple_nvme *anv = cookie; >> + int ret; >> + >> + if (iova) >> + return -EINVAL; >> + >> + bfr->buffer = dma_alloc_coherent(anv->dev, size, &iova, GFP_KERNEL); >> + if (!bfr->buffer) >> + return -ENOMEM; > > You pass 'iova' as an argument, but then replace it with the address > returned by dma_alloc_coherent(). Can you remove the function > argument? Yup, will remove it. > >> +static void apple_nvmmu_inval(struct apple_nvme_queue *q, unsigned int tag) >> +{ >> + struct apple_nvme *anv = queue_to_apple_nvme(q); >> + >> + writel(tag, anv->mmio_nvme + APPLE_NVMMU_TCB_INVAL); >> + if (readl_relaxed(anv->mmio_nvme + APPLE_NVMMU_TCB_STAT)) >> + dev_warn(anv->dev, "NVMMU TCB invalidation failed\n"); >> +} > > I don't like to see the _relaxed() accessors used without an explanation > about why that helps. Please use the non-relaxed version, or make sure > it's obvious here why you use it. Ok, I'll either use the non-relaxed ones or add a comment whenever I use the relaxed version. In this case here there's no write to any DMA buffers that needs to be visible to the device. That writel there could actually be a writel_relaxed as well. There just used to be a write to a buffer above (which is another good reason to always comment when using the non-relaxed ones, maybe then I would've noticed then and updated it). > >> +bad_sgl: >> + WARN(DO_ONCE(apple_nvme_print_sgl, iod->sg, iod->nents), >> + "Invalid SGL for payload:%d nents:%d\n", blk_rq_payload_bytes(req), >> + iod->nents); > > I think you mean WARN_ONCE() here? This is taken from pci.c which used to use WARN_ONCE but was replaced in d08774738446e77734777adcf5d1045237b4475a with this construction here. The commit message mentions The WARN_ONCE macro returns true if the condition is true, not if the warn was raised, so we're printing the scatter list every time it's invalid. This is excessive and makes debugging harder, so this patch prints it just once. > >> + writel_relaxed(0, anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); >> + (void)readl_relaxed(anv->mmio_coproc + APPLE_ANS_COPROC_CPU_CONTROL); > > What is the purpose of the readl_relaxed() here? It looks like you are > trying to flush > the write to the hardware, but then again > > a) on Apple hardware, the registers are mapped using PROT_DEVICE_nGnRnE, > so MMIO writes are never posted > > b) the read is "_relaxed", so there is no barrier, and the result is > unused, so > it would appear that the CPU can just keep executing code anyway. > > Since this is all the initialization path, I can't imagine what the > relaxation of > the barriers helps with. Agreed, I've actually tried replacing all non-relaxed ones with the normal accessors (even those inside the hot path) and didn't see any performance difference. I can use the normal ones here and I'll consider using the non-relaxed ones in the hot path together with a comment why they are safe in those places. > >> +static int apple_nvme_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val) >> +{ >> + *val = readl_relaxed(ctrl_to_apple_nvme(ctrl)->mmio_nvme + off); >> + return 0; >> +} >> + >> +static int apple_nvme_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val) >> +{ >> + writel_relaxed(val, ctrl_to_apple_nvme(ctrl)->mmio_nvme + off); >> + return 0; >> +} > > If you have generic register access functions, don't make them use > _relaxed internally. If there are instances that need to be _relaxed, > add another version of the accessor that spells this out in the caller. Ok, there are used internally by the nvme core and can't do any kind of DMA right now but I agree that it's better to use the non-relaxed ones here to prevent surprises in the future. Sven _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2022-04-02 13:35 UTC|newest] Thread overview: 114+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-03-21 16:50 [PATCH 0/9] Apple M1 (Pro/Max) NVMe driver Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 16:50 ` [PATCH 1/9] dt-bindings: soc: apple: Add Apple SART Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-31 21:23 ` Rob Herring 2022-03-31 21:23 ` Rob Herring 2022-04-02 12:58 ` Sven Peter 2022-04-02 12:58 ` Sven Peter 2022-03-21 16:50 ` [PATCH 2/9] dt-bindings: soc: apple: Add ANS NVMe Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-23 11:14 ` Krzysztof Kozlowski 2022-03-23 11:14 ` Krzysztof Kozlowski 2022-04-02 13:05 ` Sven Peter 2022-04-02 13:05 ` Sven Peter 2022-04-02 16:06 ` Krzysztof Kozlowski 2022-04-02 16:06 ` Krzysztof Kozlowski 2022-03-21 16:50 ` [PATCH 3/9] soc: apple: Always include Makefile Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 16:50 ` [PATCH 4/9] soc: apple: Add SART driver Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 17:07 ` Arnd Bergmann 2022-03-21 17:07 ` Arnd Bergmann 2022-04-02 12:38 ` Sven Peter 2022-04-02 12:38 ` Sven Peter 2022-04-02 19:07 ` Arnd Bergmann 2022-04-02 19:07 ` Arnd Bergmann 2022-04-04 14:58 ` Rob Herring 2022-04-04 14:58 ` Rob Herring 2022-04-04 15:01 ` Hector Martin 2022-04-04 15:01 ` Hector Martin 2022-04-05 15:37 ` Sven Peter 2022-04-05 15:37 ` Sven Peter 2022-03-21 17:17 ` Alyssa Rosenzweig 2022-03-21 17:17 ` Alyssa Rosenzweig 2022-04-02 12:40 ` Sven Peter 2022-04-02 12:40 ` Sven Peter 2022-03-21 16:50 ` [PATCH 5/9] soc: apple: Add RTKit IPC library Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-22 13:13 ` Arnd Bergmann 2022-03-22 13:13 ` Arnd Bergmann 2022-03-22 17:41 ` Robin Murphy 2022-03-22 17:41 ` Robin Murphy 2022-04-02 12:56 ` Sven Peter 2022-04-02 12:56 ` Sven Peter 2022-04-02 18:30 ` Arnd Bergmann 2022-04-02 18:30 ` Arnd Bergmann 2022-04-03 10:45 ` Sven Peter 2022-04-03 10:45 ` Sven Peter 2022-03-22 17:23 ` Alyssa Rosenzweig 2022-03-22 17:23 ` Alyssa Rosenzweig 2022-04-02 12:50 ` Sven Peter 2022-04-02 12:50 ` Sven Peter 2022-03-23 11:19 ` Krzysztof Kozlowski 2022-03-23 11:19 ` Krzysztof Kozlowski 2022-04-02 13:51 ` Sven Peter 2022-04-02 13:51 ` Sven Peter 2022-04-02 16:08 ` Krzysztof Kozlowski 2022-04-02 16:08 ` Krzysztof Kozlowski 2022-04-04 15:02 ` Rob Herring 2022-04-04 15:02 ` Rob Herring 2022-04-04 15:47 ` Hector Martin 2022-04-04 15:47 ` Hector Martin 2022-03-21 16:50 ` [PATCH 6/9] nvme-apple: Add initial Apple SoC NVMe driver Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 17:01 ` Keith Busch 2022-03-21 17:01 ` Keith Busch 2022-04-02 13:10 ` Sven Peter 2022-04-02 13:10 ` Sven Peter 2022-03-22 13:38 ` Arnd Bergmann 2022-03-22 13:38 ` Arnd Bergmann 2022-04-02 13:34 ` Sven Peter [this message] 2022-04-02 13:34 ` Sven Peter 2022-04-02 13:58 ` Janne Grunau 2022-04-02 13:58 ` Janne Grunau 2022-04-02 14:02 ` Sven Peter 2022-04-02 14:02 ` Sven Peter 2022-04-02 21:26 ` Arnd Bergmann 2022-04-02 21:26 ` Arnd Bergmann 2022-03-24 6:16 ` Christoph Hellwig 2022-03-24 6:16 ` Christoph Hellwig 2022-04-02 12:47 ` Sven Peter 2022-04-02 12:47 ` Sven Peter 2022-04-04 15:57 ` Hector Martin 2022-04-04 15:57 ` Hector Martin 2022-04-04 15:59 ` Christoph Hellwig 2022-04-04 15:59 ` Christoph Hellwig 2022-04-04 16:03 ` Sven Peter 2022-04-04 16:03 ` Sven Peter 2022-04-04 16:05 ` hch 2022-04-04 16:05 ` hch 2022-04-04 16:05 ` Hector Martin 2022-04-04 16:05 ` Hector Martin 2022-04-04 18:29 ` Jens Axboe 2022-04-04 18:29 ` Jens Axboe 2022-04-05 6:24 ` Christoph Hellwig 2022-04-05 6:24 ` Christoph Hellwig 2022-03-21 16:50 ` [PATCH 7/9] nvme-apple: Serialize command issue Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-24 6:16 ` Christoph Hellwig 2022-03-24 6:16 ` Christoph Hellwig 2022-04-02 12:42 ` Sven Peter 2022-04-02 12:42 ` Sven Peter 2022-03-21 16:50 ` [PATCH 8/9] nvme-apple: Add support for multiple power domains Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-21 16:50 ` [PATCH 9/9] nvme-apple: Add support for suspend/resume Sven Peter 2022-03-21 16:50 ` Sven Peter 2022-03-24 6:17 ` Christoph Hellwig 2022-03-24 6:17 ` Christoph Hellwig 2022-03-22 17:26 ` [PATCH 0/9] Apple M1 (Pro/Max) NVMe driver Alyssa Rosenzweig 2022-03-22 17:26 ` Alyssa Rosenzweig
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=7876eb84-9803-48dc-ab6b-27d49858e27a@www.fastmail.com \ --to=sven@svenpeter.dev \ --cc=alyssa@rosenzweig.io \ --cc=arnd@arndb.de \ --cc=axboe@fb.com \ --cc=devicetree@vger.kernel.org \ --cc=hch@lst.de \ --cc=kbusch@kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-nvme@lists.infradead.org \ --cc=marcan@marcan.st \ --cc=maz@kernel.org \ --cc=robh+dt@kernel.org \ --cc=sagi@grimberg.me \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.