From: Jason Gunthorpe <jgg@ziepe.ca> To: Robin Murphy <robin.murphy@arm.com> Cc: Joerg Roedel <joro@8bytes.org>, Christoph Hellwig <hch@lst.de>, Vineet Gupta <vgupta@kernel.org>, Russell King <linux@armlinux.org.uk>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Huacai Chen <chenhuacai@kernel.org>, WANG Xuerui <kernel@xen0n.name>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, Paul Walmsley <paul.walmsley@sifive.com>, Palmer Dabbelt <palmer@dabbelt.com>, Albert Ou <aou@eecs.berkeley.edu>, Lorenzo Pieralisi <lpieralisi@kernel.org>, Hanjun Guo <guohanjun@huawei.com>, Sudeep Holla <sudeep.holla@arm.com>, "K. Y. Srinivasan" <kys@microsoft.com>, Haiyang Zhang <haiyangz@microsoft.com>, Wei Liu <wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>, Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>, David Woodhouse <dwmw2@infradead.org>, Lu Baolu <baolu.lu@linux.intel.com>, Niklas Schnelle <schnelle@linux.ibm.com>, Matthew Rosato <mjrosato@linux.ibm.com>, Gerald Schaefer <gerald.schaefer@linux.ibm.com>, Jean-Philippe Brucker <jean-philippe@linaro.org>, Rob Herring <robh+dt@kernel.org>, Frank Rowand <frowand.list@gmail.com>, Marek Szyprowski <m.szyprowski@samsung.com>, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-acpi@vger.kernel.org, iommu@lists.linux.dev, devicetree@vger.kernel.org Subject: Re: [PATCH 0/7] dma-mapping: Clean up arch_setup_dma_ops() Date: Fri, 1 Dec 2023 09:57:43 -0400 [thread overview] Message-ID: <20231201135743.GI1394392@ziepe.ca> (raw) In-Reply-To: <ae27768f-a6fa-4971-b44c-92899a81a2b7@arm.com> On Fri, Dec 01, 2023 at 01:07:36PM +0000, Robin Murphy wrote: > On 29/11/2023 8:36 pm, Jason Gunthorpe wrote: > > On Wed, Nov 29, 2023 at 05:42:57PM +0000, Robin Murphy wrote: > > > Hi all, > > > > > > Prompted by Jason's proposal[1], here's a first step towards truly > > > unpicking the dma_configure vs. IOMMU mess. As I commented before, we > > > have an awful lot of accumulated cruft and technical debt here making > > > things more complicated than they need to be, and we already have hacks > > > on top of hacks trying to work around it, so polishing those hacks even > > > further is really not a desirable direction of travel. And I do know > > > they're hacks, because I wrote most of them and still remember enough of > > > the context of the time ;) > > > > I quite like this, I was also looking at getting rid of those other > > parameters. > > > > I wanted to take smaller steps because it is all pretty hairy. > > > > One thing that still concerns me is if the FW data restricts the valid > > IOVA window that really should be reflected into the reserved ranges > > and not just dumped into the struct device for use by the DMA API. > > > > Or, perhaps, viof/iommufd should be using the struct device data to > > generate some additional reserved ranges? > > > > Either way, I would like to see the dma_iommu and the rest of the > > subsystem agree on what the valid IOVA ranges actually are. > > Note that there is some intentional divergence where iommu-dma reserves > IOVAs matching PCI outbound windows because it knows it wants to avoid > clashing with potential peer-to-peer addresses and doesn't want to have to > get into the details of ACS redirect etc., but we don't expose those as > generic reserved regions because they're firmly a property of the PCI host > bridge, not of the IOMMU group (and more practically, because we did do so > briefly and it made QEMU unhappy). I think there may also have been some > degree of conclusion that it's not the IOMMU API's place to get in the way > of other domain users trying to do weird P2P stuff if they really want to. I'm not sure this is the fully correct conclusion - eg if today we take a NIC device on a non-ACS topology and run DPDK through VFIO it has a chance of failure because some IOVA simply cannot be used by DPDK for DMA at all. qemu and kvm are a different situation that want different things. Eg it would want to identity map the PCI BAR spaces to the IOVA they are claiming. It should still somehow carve out any other IOVA that is unusable due to guest-invisible ACS and reflect it through FW tables into the VM. I'm starting to see people build non-ACS systems and want it to work with VFIO and I'm a little worried we have been too loose here. > bridge and so inherits its restrictions. However I don't recall any > conscious decision for inbound windows to only be considered for DMA domain > reservations rather than for proper reserved regions - pretty sure that's > just a case of that code being added in the place where it seemed to fit > best at the time (because hey it's more host bridge windows and we already > have a thing for host bridge windows...) Yeah, and I don't think anyone actually cared much.. At least as a step it would be nice if the DMA API only restrictions can come out as a special type of reserved region. Then the caller could decide if they want to follow them or not. iommufd could provide an opt-in API to DPDK that matches DMA API's safe IOVA allocator. Jason
WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgg@ziepe.ca> To: Robin Murphy <robin.murphy@arm.com> Cc: Joerg Roedel <joro@8bytes.org>, Christoph Hellwig <hch@lst.de>, Vineet Gupta <vgupta@kernel.org>, Russell King <linux@armlinux.org.uk>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Huacai Chen <chenhuacai@kernel.org>, WANG Xuerui <kernel@xen0n.name>, Thomas Bogendoerfer <tsbogend@alpha.franken.de>, Paul Walmsley <paul.walmsley@sifive.com>, Palmer Dabbelt <palmer@dabbelt.com>, Albert Ou <aou@eecs.berkeley.edu>, Lorenzo Pieralisi <lpieralisi@kernel.org>, Hanjun Guo <guohanjun@huawei.com>, Sudeep Holla <sudeep.holla@arm.com>, "K. Y. Srinivasan" <kys@microsoft.com>, Haiyang Zhang <haiyangz@microsoft.com>, Wei Liu <wei.liu@kernel.org>, Dexuan Cui <decui@microsoft.com>, Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>, David Woodhouse <dwmw2@infradead.org>, Lu Baolu <baolu.lu@linux.intel.com>, Niklas Schnelle <schnelle@linux.ibm.com>, Matthew Rosato <mjrosato@linux.ibm.com>, Gerald Schaefer <gerald.schaefer@linux.ibm.com>, Jean-Philippe Brucker <jean-philippe@linaro.org>, Rob Herring <robh+dt@kernel.org>, Frank Rowand <frowand.list@gmail.com>, Marek Szyprowski <m.szyprowski@samsung.com>, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-acpi@vger.kernel.org, iommu@lists.linux.dev, devicetree@vger.kernel.org Subject: Re: [PATCH 0/7] dma-mapping: Clean up arch_setup_dma_ops() Date: Fri, 1 Dec 2023 09:57:43 -0400 [thread overview] Message-ID: <20231201135743.GI1394392@ziepe.ca> (raw) In-Reply-To: <ae27768f-a6fa-4971-b44c-92899a81a2b7@arm.com> On Fri, Dec 01, 2023 at 01:07:36PM +0000, Robin Murphy wrote: > On 29/11/2023 8:36 pm, Jason Gunthorpe wrote: > > On Wed, Nov 29, 2023 at 05:42:57PM +0000, Robin Murphy wrote: > > > Hi all, > > > > > > Prompted by Jason's proposal[1], here's a first step towards truly > > > unpicking the dma_configure vs. IOMMU mess. As I commented before, we > > > have an awful lot of accumulated cruft and technical debt here making > > > things more complicated than they need to be, and we already have hacks > > > on top of hacks trying to work around it, so polishing those hacks even > > > further is really not a desirable direction of travel. And I do know > > > they're hacks, because I wrote most of them and still remember enough of > > > the context of the time ;) > > > > I quite like this, I was also looking at getting rid of those other > > parameters. > > > > I wanted to take smaller steps because it is all pretty hairy. > > > > One thing that still concerns me is if the FW data restricts the valid > > IOVA window that really should be reflected into the reserved ranges > > and not just dumped into the struct device for use by the DMA API. > > > > Or, perhaps, viof/iommufd should be using the struct device data to > > generate some additional reserved ranges? > > > > Either way, I would like to see the dma_iommu and the rest of the > > subsystem agree on what the valid IOVA ranges actually are. > > Note that there is some intentional divergence where iommu-dma reserves > IOVAs matching PCI outbound windows because it knows it wants to avoid > clashing with potential peer-to-peer addresses and doesn't want to have to > get into the details of ACS redirect etc., but we don't expose those as > generic reserved regions because they're firmly a property of the PCI host > bridge, not of the IOMMU group (and more practically, because we did do so > briefly and it made QEMU unhappy). I think there may also have been some > degree of conclusion that it's not the IOMMU API's place to get in the way > of other domain users trying to do weird P2P stuff if they really want to. I'm not sure this is the fully correct conclusion - eg if today we take a NIC device on a non-ACS topology and run DPDK through VFIO it has a chance of failure because some IOVA simply cannot be used by DPDK for DMA at all. qemu and kvm are a different situation that want different things. Eg it would want to identity map the PCI BAR spaces to the IOVA they are claiming. It should still somehow carve out any other IOVA that is unusable due to guest-invisible ACS and reflect it through FW tables into the VM. I'm starting to see people build non-ACS systems and want it to work with VFIO and I'm a little worried we have been too loose here. > bridge and so inherits its restrictions. However I don't recall any > conscious decision for inbound windows to only be considered for DMA domain > reservations rather than for proper reserved regions - pretty sure that's > just a case of that code being added in the place where it seemed to fit > best at the time (because hey it's more host bridge windows and we already > have a thing for host bridge windows...) Yeah, and I don't think anyone actually cared much.. At least as a step it would be nice if the DMA API only restrictions can come out as a special type of reserved region. Then the caller could decide if they want to follow them or not. iommufd could provide an opt-in API to DPDK that matches DMA API's safe IOVA allocator. Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2023-12-01 13:57 UTC|newest] Thread overview: 61+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-11-29 17:42 [PATCH 0/7] dma-mapping: Clean up arch_setup_dma_ops() Robin Murphy 2023-11-29 17:42 ` Robin Murphy 2023-11-29 17:42 ` [PATCH 1/7] OF: Retire dma-ranges mask workaround Robin Murphy 2023-11-29 17:42 ` Robin Murphy 2023-11-30 14:46 ` Rob Herring 2023-11-30 14:46 ` Rob Herring 2023-11-29 17:42 ` [PATCH 2/7] OF: Simplify DMA range calculations Robin Murphy 2023-11-29 17:42 ` Robin Murphy 2023-11-30 0:46 ` Jason Gunthorpe 2023-11-30 0:46 ` Jason Gunthorpe 2023-11-30 14:56 ` Rob Herring 2023-11-30 14:56 ` Rob Herring 2023-11-29 17:43 ` [PATCH 3/7] ACPI/IORT: Handle memory address size limits as limits Robin Murphy 2023-11-29 17:43 ` Robin Murphy 2023-11-30 0:39 ` Jason Gunthorpe 2023-11-30 0:39 ` Jason Gunthorpe 2023-12-11 13:27 ` Will Deacon 2023-12-11 13:27 ` Will Deacon 2023-12-11 15:01 ` Robin Murphy 2023-12-11 15:01 ` Robin Murphy 2023-12-11 15:30 ` Will Deacon 2023-12-11 15:30 ` Will Deacon 2023-12-11 15:36 ` Jason Gunthorpe 2023-12-11 15:36 ` Jason Gunthorpe 2023-12-11 15:37 ` Robin Murphy 2023-12-11 15:37 ` Robin Murphy 2023-12-11 15:39 ` Mark Rutland 2023-12-11 15:39 ` Mark Rutland 2023-12-11 16:13 ` Robin Murphy 2023-12-11 16:13 ` Robin Murphy 2023-12-11 15:37 ` Mark Rutland 2023-12-11 15:37 ` Mark Rutland 2023-11-29 17:43 ` [PATCH 4/7] dma-mapping: Add helpers for dma_range_map bounds Robin Murphy 2023-11-29 17:43 ` Robin Murphy 2023-11-29 20:40 ` Jason Gunthorpe 2023-11-29 20:40 ` Jason Gunthorpe 2023-11-30 6:11 ` kernel test robot 2023-11-30 6:11 ` kernel test robot 2023-12-04 8:43 ` Christoph Hellwig 2023-12-04 8:43 ` Christoph Hellwig 2023-11-29 17:43 ` [PATCH 5/7] iommu/dma: Make limit checks self-contained Robin Murphy 2023-11-29 17:43 ` Robin Murphy 2023-11-29 20:43 ` Jason Gunthorpe 2023-11-29 20:43 ` Jason Gunthorpe 2023-11-29 17:43 ` [PATCH 6/7] iommu/dma: Centralise iommu_setup_dma_ops() Robin Murphy 2023-11-29 17:43 ` Robin Murphy 2023-11-29 20:50 ` Jason Gunthorpe 2023-11-29 20:50 ` Jason Gunthorpe 2023-11-29 17:43 ` [PATCH 7/7] dma-mapping: Simplify arch_setup_dma_ops() Robin Murphy 2023-11-29 17:43 ` Robin Murphy 2023-11-30 5:23 ` kernel test robot 2023-12-04 8:44 ` Christoph Hellwig 2023-12-04 8:44 ` Christoph Hellwig 2023-12-04 12:54 ` Robin Murphy 2023-12-04 12:54 ` Robin Murphy 2023-11-29 20:36 ` [PATCH 0/7] dma-mapping: Clean up arch_setup_dma_ops() Jason Gunthorpe 2023-11-29 20:36 ` Jason Gunthorpe 2023-12-01 13:07 ` Robin Murphy 2023-12-01 13:07 ` Robin Murphy 2023-12-01 13:57 ` Jason Gunthorpe [this message] 2023-12-01 13:57 ` Jason Gunthorpe
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20231201135743.GI1394392@ziepe.ca \ --to=jgg@ziepe.ca \ --cc=aou@eecs.berkeley.edu \ --cc=baolu.lu@linux.intel.com \ --cc=catalin.marinas@arm.com \ --cc=chenhuacai@kernel.org \ --cc=decui@microsoft.com \ --cc=devicetree@vger.kernel.org \ --cc=dwmw2@infradead.org \ --cc=frowand.list@gmail.com \ --cc=gerald.schaefer@linux.ibm.com \ --cc=guohanjun@huawei.com \ --cc=haiyangz@microsoft.com \ --cc=hch@lst.de \ --cc=iommu@lists.linux.dev \ --cc=jean-philippe@linaro.org \ --cc=joro@8bytes.org \ --cc=kernel@xen0n.name \ --cc=kys@microsoft.com \ --cc=linux-acpi@vger.kernel.org \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux@armlinux.org.uk \ --cc=lpieralisi@kernel.org \ --cc=m.szyprowski@samsung.com \ --cc=mjrosato@linux.ibm.com \ --cc=palmer@dabbelt.com \ --cc=paul.walmsley@sifive.com \ --cc=robh+dt@kernel.org \ --cc=robin.murphy@arm.com \ --cc=schnelle@linux.ibm.com \ --cc=sudeep.holla@arm.com \ --cc=suravee.suthikulpanit@amd.com \ --cc=tsbogend@alpha.franken.de \ --cc=vgupta@kernel.org \ --cc=wei.liu@kernel.org \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.