From: Benjamin Herrenschmidt <benh@kernel.crashing.org> To: Jason Gunthorpe <jgg@nvidia.com> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>, Clint Sbisa <csbisa@amazon.com>, linux-pci@vger.kernel.org, Bjorn Helgaas <helgaas@kernel.org>, linux-arm-kernel@lists.infradead.org, will@kernel.org, catalin.marinas@arm.com, Leon Romanovsky <leon@kernel.org> Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs Date: Wed, 16 Sep 2020 17:59:24 +1000 [thread overview] Message-ID: <701012f288231d0d0733bf1c2c8fdbd9caa074fd.camel@kernel.crashing.org> (raw) In-Reply-To: <20200915234006.GI1573713@nvidia.com> On Tue, 2020-09-15 at 20:40 -0300, Jason Gunthorpe wrote: > Not quite, upstream kernel will never use WC on those > devices. DEVICE_GRE is not supported in upstream, > arch_can_pci_mmap_wc() is always false and the WC tester will always > fail. > > > With the patch, those device will now use MT_DEVICE_NC. > > Which doesn't do WC at all on some ARM implementations. Lovely... this is arm64 btw, still the case ? Also we could make this a variable rather than a constant and choose a more appropriate set of flags at boot time.... > > Why would that be a regression ? > > Using the WC submission flow when it doesn't work costs something like > 10% performance vs using the non-WC flow. You mean the driver uses a different path to the HW which ahs that overhead, not that MMIOs have that overhead right ? > Like I said, the case where the driver can't self test probably > doesn't intersect with the ARM implementations that can't do write > combining, and if it did, the users probably run the out of tree > driver that has the hacky stuff to make it use DEVICE_GRE. Ok. So you are saying to go for it and ignore that Mellanox case then ? :-) > > BTW. Lorenzo, why don't we use MT_DEVICE_GRE for pgprot_writecombine ? > > Its not supported on some chips ? > > It has alignment requirements drivers don't meet. We need a new > concept of "write combining and I promise to do aligned access" Ah yes, I remember. Right, we would need to provide new/better accessors for these kind of things. It's going to be a mess to find a common set that works for all archs. > > What on earth is pgprot_device() ? This is new ? On ARM it will be > > MT_DEVICE_nGnRE, so it allows posted write. It seems to match what > > ioremap does. Should then ioremap use it as well ? > > > > But it's only ever used for PCI mmap. Why is it different from > > pgprot_noncached() which disables posted writes (nE) ? > > > > Because a whole lot of drivers will use pgprot_noncached() explicitly > > in either mmap or vmap, with the expectation that it's somewhat the > > same as what ioremap does... > > *boggle* > > Only sysfs uses pci_mmap_resource_range() any other driver exposing > BAR pages, like VFIO dies not. Makes no sense at all it is different. > > Delete the ill defined pgprot_device() ? Nobody has complained > something is wrong with VFIO in the 6 years since it was added... I was wondering what it was, that's it ... Cheers, Ben.
WARNING: multiple messages have this Message-ID (diff)
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> To: Jason Gunthorpe <jgg@nvidia.com> Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>, Leon Romanovsky <leon@kernel.org>, linux-pci@vger.kernel.org, Bjorn Helgaas <helgaas@kernel.org>, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, Clint Sbisa <csbisa@amazon.com> Subject: Re: [PATCH] arm64: Enable PCI write-combine resources under sysfs Date: Wed, 16 Sep 2020 17:59:24 +1000 [thread overview] Message-ID: <701012f288231d0d0733bf1c2c8fdbd9caa074fd.camel@kernel.crashing.org> (raw) In-Reply-To: <20200915234006.GI1573713@nvidia.com> On Tue, 2020-09-15 at 20:40 -0300, Jason Gunthorpe wrote: > Not quite, upstream kernel will never use WC on those > devices. DEVICE_GRE is not supported in upstream, > arch_can_pci_mmap_wc() is always false and the WC tester will always > fail. > > > With the patch, those device will now use MT_DEVICE_NC. > > Which doesn't do WC at all on some ARM implementations. Lovely... this is arm64 btw, still the case ? Also we could make this a variable rather than a constant and choose a more appropriate set of flags at boot time.... > > Why would that be a regression ? > > Using the WC submission flow when it doesn't work costs something like > 10% performance vs using the non-WC flow. You mean the driver uses a different path to the HW which ahs that overhead, not that MMIOs have that overhead right ? > Like I said, the case where the driver can't self test probably > doesn't intersect with the ARM implementations that can't do write > combining, and if it did, the users probably run the out of tree > driver that has the hacky stuff to make it use DEVICE_GRE. Ok. So you are saying to go for it and ignore that Mellanox case then ? :-) > > BTW. Lorenzo, why don't we use MT_DEVICE_GRE for pgprot_writecombine ? > > Its not supported on some chips ? > > It has alignment requirements drivers don't meet. We need a new > concept of "write combining and I promise to do aligned access" Ah yes, I remember. Right, we would need to provide new/better accessors for these kind of things. It's going to be a mess to find a common set that works for all archs. > > What on earth is pgprot_device() ? This is new ? On ARM it will be > > MT_DEVICE_nGnRE, so it allows posted write. It seems to match what > > ioremap does. Should then ioremap use it as well ? > > > > But it's only ever used for PCI mmap. Why is it different from > > pgprot_noncached() which disables posted writes (nE) ? > > > > Because a whole lot of drivers will use pgprot_noncached() explicitly > > in either mmap or vmap, with the expectation that it's somewhat the > > same as what ioremap does... > > *boggle* > > Only sysfs uses pci_mmap_resource_range() any other driver exposing > BAR pages, like VFIO dies not. Makes no sense at all it is different. > > Delete the ill defined pgprot_device() ? Nobody has complained > something is wrong with VFIO in the 6 years since it was added... I was wondering what it was, that's it ... Cheers, Ben. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-09-16 7:59 UTC|newest] Thread overview: 117+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-08-31 15:18 [PATCH] arm64: Enable PCI write-combine resources under sysfs Clint Sbisa 2020-08-31 23:24 ` Benjamin Herrenschmidt 2020-09-01 18:37 ` Bjorn Helgaas 2020-09-01 23:22 ` Benjamin Herrenschmidt 2020-09-02 8:57 ` Will Deacon 2020-09-02 11:32 ` Lorenzo Pieralisi 2020-09-02 14:29 ` Clint Sbisa 2020-09-02 16:47 ` Lorenzo Pieralisi 2020-09-02 16:47 ` Lorenzo Pieralisi 2020-09-02 17:21 ` Catalin Marinas 2020-09-02 17:21 ` Catalin Marinas 2020-09-02 17:54 ` Lorenzo Pieralisi 2020-09-02 17:54 ` Lorenzo Pieralisi 2020-09-02 23:03 ` Benjamin Herrenschmidt 2020-09-02 23:03 ` Benjamin Herrenschmidt 2020-09-02 23:08 ` Benjamin Herrenschmidt 2020-09-02 23:08 ` Benjamin Herrenschmidt 2020-09-02 23:08 ` Benjamin Herrenschmidt 2020-09-02 23:08 ` Benjamin Herrenschmidt 2020-09-02 23:07 ` Benjamin Herrenschmidt 2020-09-02 23:07 ` Benjamin Herrenschmidt 2020-09-03 11:08 ` Lorenzo Pieralisi 2020-09-03 11:08 ` Lorenzo Pieralisi 2020-09-03 14:36 ` Clint Sbisa 2020-09-03 14:36 ` Clint Sbisa 2020-09-03 22:26 ` Benjamin Herrenschmidt 2020-09-03 22:26 ` Benjamin Herrenschmidt 2020-09-07 23:33 ` Benjamin Herrenschmidt 2020-09-07 23:33 ` Benjamin Herrenschmidt 2020-09-10 9:46 ` Lorenzo Pieralisi 2020-09-10 9:46 ` Lorenzo Pieralisi 2020-09-10 10:54 ` Leon Romanovsky 2020-09-10 10:54 ` Leon Romanovsky 2020-09-10 12:37 ` Jason Gunthorpe 2020-09-10 12:37 ` Jason Gunthorpe 2020-09-10 15:17 ` Lorenzo Pieralisi 2020-09-10 15:17 ` Lorenzo Pieralisi 2020-09-10 17:10 ` Jason Gunthorpe 2020-09-10 17:10 ` Jason Gunthorpe 2020-09-10 21:46 ` Benjamin Herrenschmidt 2020-09-10 21:46 ` Benjamin Herrenschmidt 2020-09-10 23:29 ` Jason Gunthorpe 2020-09-10 23:29 ` Jason Gunthorpe 2020-09-11 0:39 ` Benjamin Herrenschmidt 2020-09-11 0:39 ` Benjamin Herrenschmidt 2020-09-11 14:21 ` Jason Gunthorpe 2020-09-11 14:21 ` Jason Gunthorpe 2020-09-11 21:42 ` Clint Sbisa 2020-09-11 21:42 ` Clint Sbisa 2020-09-14 14:17 ` Jason Gunthorpe 2020-09-14 14:17 ` Jason Gunthorpe 2020-09-14 14:24 ` Clint Sbisa 2020-09-14 14:24 ` Clint Sbisa 2020-09-14 14:38 ` Jason Gunthorpe 2020-09-14 14:38 ` Jason Gunthorpe 2020-09-14 21:42 ` Benjamin Herrenschmidt 2020-09-14 21:42 ` Benjamin Herrenschmidt 2020-09-14 22:00 ` Benjamin Herrenschmidt 2020-09-14 22:00 ` Benjamin Herrenschmidt 2020-09-14 22:32 ` Clint Sbisa 2020-09-14 22:32 ` Clint Sbisa 2020-09-14 22:57 ` Jason Gunthorpe 2020-09-14 22:57 ` Jason Gunthorpe 2020-09-14 23:25 ` Benjamin Herrenschmidt 2020-09-14 23:25 ` Benjamin Herrenschmidt 2020-09-15 10:18 ` Lorenzo Pieralisi 2020-09-15 10:18 ` Lorenzo Pieralisi 2020-09-15 11:05 ` Jason Gunthorpe 2020-09-15 11:05 ` Jason Gunthorpe 2020-09-15 23:17 ` Benjamin Herrenschmidt 2020-09-15 23:17 ` Benjamin Herrenschmidt 2020-09-15 23:40 ` Jason Gunthorpe 2020-09-15 23:40 ` Jason Gunthorpe 2020-09-16 7:59 ` Benjamin Herrenschmidt [this message] 2020-09-16 7:59 ` Benjamin Herrenschmidt 2020-09-16 12:12 ` Jason Gunthorpe 2020-09-16 12:12 ` Jason Gunthorpe 2020-09-16 14:09 ` Lorenzo Pieralisi 2020-09-16 14:09 ` Lorenzo Pieralisi 2020-09-16 14:14 ` Jason Gunthorpe 2020-09-16 14:14 ` Jason Gunthorpe 2020-09-16 23:59 ` Benjamin Herrenschmidt 2020-09-16 23:59 ` Benjamin Herrenschmidt 2020-09-17 10:28 ` Lorenzo Pieralisi 2020-09-17 10:28 ` Lorenzo Pieralisi 2020-09-17 11:32 ` Jason Gunthorpe 2020-09-17 11:32 ` Jason Gunthorpe 2020-09-17 14:01 ` Lorenzo Pieralisi 2020-09-17 14:01 ` Lorenzo Pieralisi 2020-09-17 16:08 ` Will Deacon 2020-09-17 16:08 ` Will Deacon 2020-09-16 12:48 ` Leon Romanovsky 2020-09-16 12:48 ` Leon Romanovsky 2020-09-16 8:33 ` Will Deacon 2020-09-16 8:33 ` Will Deacon 2020-09-16 8:48 ` Catalin Marinas 2020-09-16 8:48 ` Catalin Marinas 2020-09-16 14:15 ` Lorenzo Pieralisi 2020-09-16 14:15 ` Lorenzo Pieralisi 2020-09-16 17:00 ` Catalin Marinas 2020-09-16 17:00 ` Catalin Marinas 2020-09-16 21:29 ` Benjamin Herrenschmidt 2020-09-16 21:29 ` Benjamin Herrenschmidt 2020-09-16 12:08 ` Jason Gunthorpe 2020-09-16 12:08 ` Jason Gunthorpe 2020-09-15 23:00 ` Benjamin Herrenschmidt 2020-09-15 23:00 ` Benjamin Herrenschmidt 2020-09-15 23:12 ` Clint Sbisa 2020-09-15 23:12 ` Clint Sbisa 2020-09-14 21:41 ` Benjamin Herrenschmidt 2020-09-14 21:41 ` Benjamin Herrenschmidt -- strict thread matches above, loose matches on Subject: below -- 2020-08-21 15:51 Clint Sbisa 2020-08-21 15:51 ` Clint Sbisa 2020-08-27 14:41 ` Clint Sbisa 2020-08-27 14:41 ` Clint Sbisa 2020-08-31 15:22 ` Clint Sbisa 2020-08-31 15:22 ` Clint Sbisa
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=701012f288231d0d0733bf1c2c8fdbd9caa074fd.camel@kernel.crashing.org \ --to=benh@kernel.crashing.org \ --cc=catalin.marinas@arm.com \ --cc=csbisa@amazon.com \ --cc=helgaas@kernel.org \ --cc=jgg@nvidia.com \ --cc=leon@kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-pci@vger.kernel.org \ --cc=lorenzo.pieralisi@arm.com \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.