* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance [not found] ` <3e0b91be-e4a4-4ea5-7d58-6e71b8d51932@marek.ca> @ 2020-10-06 7:23 ` Christoph Hellwig 2020-10-06 13:19 ` Jonathan Marek 2020-10-08 8:27 ` Joerg Roedel 0 siblings, 2 replies; 10+ messages in thread From: Christoph Hellwig @ 2020-10-06 7:23 UTC (permalink / raw) To: Jonathan Marek Cc: David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, Christoph Hellwig, iommu, Daniel Vetter, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul, Robin Murphy On Mon, Oct 05, 2020 at 10:35:43AM -0400, Jonathan Marek wrote: > The cache synchronization doesn't have anything to do with IOMMU (for > example: cache synchronization would be useful in cases where drm/msm > doesn't use IOMMU). It has to do with doing DMA. And we have two frameworks for doing DMA: either the DMA API which is for general driver use, and which as part of the design includes cache maintainance hidden behind the concept of ownership transfers. And we have the much more bare bones IOMMU API. If people want to use the "raw" IOMMU API with not cache coherent devices we'll need a cache maintainance API that goes along with it. It could either be formally part of the IOMMU API or be separate. > What is needed is to call arch_sync_dma_for_{cpu,device} (which is what I > went with initially, but then decided to re-use drm/msm's > sync_for_{cpu,device}). But you are also saying those functions aren't for > driver use, and I doubt IOMMU maintainers will want to add wrappers for > these functions just to satisfy this "not for driver use" requirement. arch_sync_dma_for_{cpu,device} are low-level helpers (and not very great ones at that). The definitively should not be used by drivers. They would be very useful buildblocks for a IOMMU cache maintainance API. Of course the best outcome would be if we could find a way for the MSM drm driver to just use DMA API and not deal with the lower level abstractions. Do you remember why the driver went for use of the IOMMU API? _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-06 7:23 ` [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance Christoph Hellwig @ 2020-10-06 13:19 ` Jonathan Marek 2020-10-07 6:25 ` Christoph Hellwig 2020-10-08 8:27 ` Joerg Roedel 1 sibling, 1 reply; 10+ messages in thread From: Jonathan Marek @ 2020-10-06 13:19 UTC (permalink / raw) To: Christoph Hellwig Cc: David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, iommu, Daniel Vetter, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul, Robin Murphy On 10/6/20 3:23 AM, Christoph Hellwig wrote: > On Mon, Oct 05, 2020 at 10:35:43AM -0400, Jonathan Marek wrote: >> The cache synchronization doesn't have anything to do with IOMMU (for >> example: cache synchronization would be useful in cases where drm/msm >> doesn't use IOMMU). > > It has to do with doing DMA. And we have two frameworks for doing DMA: > either the DMA API which is for general driver use, and which as part of > the design includes cache maintainance hidden behind the concept of > ownership transfers. And we have the much more bare bones IOMMU API. > > If people want to use the "raw" IOMMU API with not cache coherent > devices we'll need a cache maintainance API that goes along with it. > It could either be formally part of the IOMMU API or be separate. > >> What is needed is to call arch_sync_dma_for_{cpu,device} (which is what I >> went with initially, but then decided to re-use drm/msm's >> sync_for_{cpu,device}). But you are also saying those functions aren't for >> driver use, and I doubt IOMMU maintainers will want to add wrappers for >> these functions just to satisfy this "not for driver use" requirement. > > arch_sync_dma_for_{cpu,device} are low-level helpers (and not very > great ones at that). The definitively should not be used by drivers. > They would be very useful buildblocks for a IOMMU cache maintainance > API. > > Of course the best outcome would be if we could find a way for the MSM > drm driver to just use DMA API and not deal with the lower level > abstractions. Do you remember why the driver went for use of the IOMMU > API? > One example why drm/msm can't use DMA API is multiple page table support (that is landing in 5.10), which is something that definitely couldn't work with DMA API. Another one is being able to choose the address for mappings, which AFAIK DMA API can't do (somewhat related to this: qcom hardware often has ranges of allowed addresses, which the dma_mask mechanism fails to represent, what I see is drivers using dma_mask as a "maximum address", and since addresses are allocated from the top it generally works) But let us imagine drm/msm switches to using DMA API. a2xx GPUs have their own very basic MMU (implemented by msm_gpummu.c), that will need to implement dma_map_ops, which will have to call arch_sync_dma_for_{cpu,device}. So drm/msm still needs to call arch_sync_dma_for_{cpu,device} in that scenario. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-06 13:19 ` Jonathan Marek @ 2020-10-07 6:25 ` Christoph Hellwig 2020-10-13 13:42 ` Robin Murphy 0 siblings, 1 reply; 10+ messages in thread From: Christoph Hellwig @ 2020-10-07 6:25 UTC (permalink / raw) To: Jonathan Marek Cc: David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, Christoph Hellwig, iommu, Daniel Vetter, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul, Robin Murphy On Tue, Oct 06, 2020 at 09:19:32AM -0400, Jonathan Marek wrote: > One example why drm/msm can't use DMA API is multiple page table support > (that is landing in 5.10), which is something that definitely couldn't work > with DMA API. > > Another one is being able to choose the address for mappings, which AFAIK > DMA API can't do (somewhat related to this: qcom hardware often has ranges > of allowed addresses, which the dma_mask mechanism fails to represent, what > I see is drivers using dma_mask as a "maximum address", and since addresses > are allocated from the top it generally works) That sounds like a good enough rason to use the IOMMU API. I just wanted to make sure this really makes sense. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-07 6:25 ` Christoph Hellwig @ 2020-10-13 13:42 ` Robin Murphy 2020-10-13 16:11 ` Rob Clark 2020-10-15 6:55 ` Christoph Hellwig 0 siblings, 2 replies; 10+ messages in thread From: Robin Murphy @ 2020-10-13 13:42 UTC (permalink / raw) To: Christoph Hellwig, Jonathan Marek Cc: David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, iommu, Daniel Vetter, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul On 2020-10-07 07:25, Christoph Hellwig wrote: > On Tue, Oct 06, 2020 at 09:19:32AM -0400, Jonathan Marek wrote: >> One example why drm/msm can't use DMA API is multiple page table support >> (that is landing in 5.10), which is something that definitely couldn't work >> with DMA API. >> >> Another one is being able to choose the address for mappings, which AFAIK >> DMA API can't do (somewhat related to this: qcom hardware often has ranges >> of allowed addresses, which the dma_mask mechanism fails to represent, what >> I see is drivers using dma_mask as a "maximum address", and since addresses >> are allocated from the top it generally works) > > That sounds like a good enough rason to use the IOMMU API. I just > wanted to make sure this really makes sense. I still think this situation would be best handled with a variant of dma_ops_bypass that also guarantees to bypass SWIOTLB, and can be set automatically when attaching to an unmanaged IOMMU domain. That way the device driver can make DMA API calls in the appropriate places that do the right thing either way, and only needs logic to decide whether to use the returned DMA addresses directly or ignore them if it knows they're overridden by its own IOMMU mapping. Robin. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-13 13:42 ` Robin Murphy @ 2020-10-13 16:11 ` Rob Clark 2020-10-15 6:55 ` Christoph Hellwig 1 sibling, 0 replies; 10+ messages in thread From: Rob Clark @ 2020-10-13 16:11 UTC (permalink / raw) To: Robin Murphy Cc: Jonathan Marek, David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, Christoph Hellwig, list@263.net:IOMMU DRIVERS <iommu@lists.linux-foundation.org>, Joerg Roedel <joro@8bytes.org>, , Daniel Vetter, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul On Tue, Oct 13, 2020 at 6:42 AM Robin Murphy <robin.murphy@arm.com> wrote: > > On 2020-10-07 07:25, Christoph Hellwig wrote: > > On Tue, Oct 06, 2020 at 09:19:32AM -0400, Jonathan Marek wrote: > >> One example why drm/msm can't use DMA API is multiple page table support > >> (that is landing in 5.10), which is something that definitely couldn't work > >> with DMA API. > >> > >> Another one is being able to choose the address for mappings, which AFAIK > >> DMA API can't do (somewhat related to this: qcom hardware often has ranges > >> of allowed addresses, which the dma_mask mechanism fails to represent, what > >> I see is drivers using dma_mask as a "maximum address", and since addresses > >> are allocated from the top it generally works) > > > > That sounds like a good enough rason to use the IOMMU API. I just > > wanted to make sure this really makes sense. > > I still think this situation would be best handled with a variant of > dma_ops_bypass that also guarantees to bypass SWIOTLB, and can be set > automatically when attaching to an unmanaged IOMMU domain. That way the > device driver can make DMA API calls in the appropriate places that do > the right thing either way, and only needs logic to decide whether to > use the returned DMA addresses directly or ignore them if it knows > they're overridden by its own IOMMU mapping. > That would be pretty ideal from my PoV BR, -R _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-13 13:42 ` Robin Murphy 2020-10-13 16:11 ` Rob Clark @ 2020-10-15 6:55 ` Christoph Hellwig 2020-10-15 15:33 ` Daniel Vetter 1 sibling, 1 reply; 10+ messages in thread From: Christoph Hellwig @ 2020-10-15 6:55 UTC (permalink / raw) To: Robin Murphy Cc: Sean Paul, Jonathan Marek, David Airlie, open list:DRM DRIVER FOR MSM ADRENO GPU, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, Christoph Hellwig, iommu, Daniel Vetter, freedreno On Tue, Oct 13, 2020 at 02:42:38PM +0100, Robin Murphy wrote: > I still think this situation would be best handled with a variant of > dma_ops_bypass that also guarantees to bypass SWIOTLB, and can be set > automatically when attaching to an unmanaged IOMMU domain. dma_ops_bypass should mostly do the right thing as-is. swiotlb bouncing is triggered of two things: 1) the dma_mask. This is under control of the driver, and obviously if it is too small for a legit reason we can't just proceed 2) force_dma_unencrypted() - we'd need to do an opt-out here, either by a flag or by being smart and looking for an attached iommu on the device > That way the > device driver can make DMA API calls in the appropriate places that do the > right thing either way, and only needs logic to decide whether to use the > returned DMA addresses directly or ignore them if it knows they're > overridden by its own IOMMU mapping. I'd be happy to review patches for this. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-15 6:55 ` Christoph Hellwig @ 2020-10-15 15:33 ` Daniel Vetter 2020-10-15 15:43 ` Christoph Hellwig 0 siblings, 1 reply; 10+ messages in thread From: Daniel Vetter @ 2020-10-15 15:33 UTC (permalink / raw) To: Christoph Hellwig Cc: Sean Paul, Jonathan Marek, David Airlie, Robin Murphy, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, iommu, Daniel Vetter, open list:DRM DRIVER FOR MSM ADRENO GPU, freedreno On Thu, Oct 15, 2020 at 07:55:32AM +0100, Christoph Hellwig wrote: > On Tue, Oct 13, 2020 at 02:42:38PM +0100, Robin Murphy wrote: > > I still think this situation would be best handled with a variant of > > dma_ops_bypass that also guarantees to bypass SWIOTLB, and can be set > > automatically when attaching to an unmanaged IOMMU domain. > > dma_ops_bypass should mostly do the right thing as-is. swiotlb bouncing > is triggered of two things: > > 1) the dma_mask. This is under control of the driver, and obviously > if it is too small for a legit reason we can't just proceed Somewhat related, but is there a way to tell the dma-api to fail instead of falling back to swiotlb? In many case for gpu drivers it's much better if we fall back to dma_alloc_coherent and manage the copying ourselves instead of abstracting this away in the dma-api. Currently that's "solved" rather pessimistically by always allocating from dma_alloc_coherent if swiotlb could be in the picture (at least for ttm based drivers, i915 just falls over). -Daniel > 2) force_dma_unencrypted() - we'd need to do an opt-out here, either > by a flag or by being smart and looking for an attached iommu on > the device > > > That way the > > device driver can make DMA API calls in the appropriate places that do the > > right thing either way, and only needs logic to decide whether to use the > > returned DMA addresses directly or ignore them if it knows they're > > overridden by its own IOMMU mapping. > > I'd be happy to review patches for this. -- Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-15 15:33 ` Daniel Vetter @ 2020-10-15 15:43 ` Christoph Hellwig 2020-10-23 6:48 ` Christoph Hellwig 0 siblings, 1 reply; 10+ messages in thread From: Christoph Hellwig @ 2020-10-15 15:43 UTC (permalink / raw) To: Christoph Hellwig, Robin Murphy, Jonathan Marek, David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, iommu, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul On Thu, Oct 15, 2020 at 05:33:34PM +0200, Daniel Vetter wrote: > On Thu, Oct 15, 2020 at 07:55:32AM +0100, Christoph Hellwig wrote: > > On Tue, Oct 13, 2020 at 02:42:38PM +0100, Robin Murphy wrote: > > > I still think this situation would be best handled with a variant of > > > dma_ops_bypass that also guarantees to bypass SWIOTLB, and can be set > > > automatically when attaching to an unmanaged IOMMU domain. > > > > dma_ops_bypass should mostly do the right thing as-is. swiotlb bouncing > > is triggered of two things: > > > > 1) the dma_mask. This is under control of the driver, and obviously > > if it is too small for a legit reason we can't just proceed > > Somewhat related, but is there a way to tell the dma-api to fail instead > of falling back to swiotlb? In many case for gpu drivers it's much better > if we fall back to dma_alloc_coherent and manage the copying ourselves > instead of abstracting this away in the dma-api. Currently that's "solved" > rather pessimistically by always allocating from dma_alloc_coherent if > swiotlb could be in the picture (at least for ttm based drivers, i915 just > falls over). Is this for the alloc_pages plus manually map logic in various drivers? They should switch to the new dma_alloc_pages API that I'll send to Linus for 5.10 soon. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-15 15:43 ` Christoph Hellwig @ 2020-10-23 6:48 ` Christoph Hellwig 0 siblings, 0 replies; 10+ messages in thread From: Christoph Hellwig @ 2020-10-23 6:48 UTC (permalink / raw) To: Christoph Hellwig, Robin Murphy, Jonathan Marek, David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, iommu, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul On Thu, Oct 15, 2020 at 04:43:01PM +0100, Christoph Hellwig wrote: > > Somewhat related, but is there a way to tell the dma-api to fail instead > > of falling back to swiotlb? In many case for gpu drivers it's much better > > if we fall back to dma_alloc_coherent and manage the copying ourselves > > instead of abstracting this away in the dma-api. Currently that's "solved" > > rather pessimistically by always allocating from dma_alloc_coherent if > > swiotlb could be in the picture (at least for ttm based drivers, i915 just > > falls over). > > Is this for the alloc_pages plus manually map logic in various drivers? > > They should switch to the new dma_alloc_pages API that I'll send to > Linus for 5.10 soon. Daniel, can you clarify this? _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance 2020-10-06 7:23 ` [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance Christoph Hellwig 2020-10-06 13:19 ` Jonathan Marek @ 2020-10-08 8:27 ` Joerg Roedel 1 sibling, 0 replies; 10+ messages in thread From: Joerg Roedel @ 2020-10-08 8:27 UTC (permalink / raw) To: Christoph Hellwig Cc: Jonathan Marek, David Airlie, freedreno, open list, open list:DRM DRIVER FOR MSM ADRENO GPU, iommu, Daniel Vetter, open list:DRM DRIVER FOR MSM ADRENO GPU, Sean Paul, Robin Murphy On Tue, Oct 06, 2020 at 08:23:06AM +0100, Christoph Hellwig wrote: > If people want to use the "raw" IOMMU API with not cache coherent > devices we'll need a cache maintainance API that goes along with it. > It could either be formally part of the IOMMU API or be separate. The IOMMU-API does not care about the caching effects of DMA, is manages IO address spaces for devices. I also don't know how this would be going to be implemented, the IOMMU-API does not have the concept of handles for mapped ranges and does not care about CPU virtual addresses (which are needed for cache flushes) of the memory it maps into IO page-tables. So I think a cache management API should be separate from the IOMMU-API. Regards, Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-10-23 6:48 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <20201001002709.21361-1-jonathan@marek.ca> [not found] ` <20201001002709.21361-3-jonathan@marek.ca> [not found] ` <20201002075321.GA7547@infradead.org> [not found] ` <b22fb797-67b0-a912-1d23-2b47c9a9e674@marek.ca> [not found] ` <20201005082914.GA31702@infradead.org> [not found] ` <3e0b91be-e4a4-4ea5-7d58-6e71b8d51932@marek.ca> 2020-10-06 7:23 ` [PATCH 2/3] drm/msm: add DRM_MSM_GEM_SYNC_CACHE for non-coherent cache maintenance Christoph Hellwig 2020-10-06 13:19 ` Jonathan Marek 2020-10-07 6:25 ` Christoph Hellwig 2020-10-13 13:42 ` Robin Murphy 2020-10-13 16:11 ` Rob Clark 2020-10-15 6:55 ` Christoph Hellwig 2020-10-15 15:33 ` Daniel Vetter 2020-10-15 15:43 ` Christoph Hellwig 2020-10-23 6:48 ` Christoph Hellwig 2020-10-08 8:27 ` Joerg Roedel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).