From: Arnd Bergmann <arnd@arndb.de> To: Robin Murphy <robin.murphy@arm.com> Cc: Nikita Yushchenko <nikita.yoush@cogentembedded.com>, Will Deacon <will.deacon@arm.com>, linux-arm-kernel@lists.infradead.org, Catalin Marinas <catalin.marinas@arm.com>, linux-kernel@vger.kernel.org, linux-renesas-soc@vger.kernel.org, Simon Horman <horms@verge.net.au>, Bjorn Helgaas <bhelgaas@google.com>, artemi.ivanov@cogentembedded.com, fkan@apm.com, Christoph Hellwig <hch@lst.de> Subject: Re: [PATCH v2] arm64: do not set dma masks that device connection can't handle Date: Tue, 10 Jan 2017 14:42:23 +0100 [thread overview] Message-ID: <6116278.nQQUSuo3l4@wuerfel> (raw) In-Reply-To: <11daacde-5399-039f-80a3-01d7bd13e9e8@arm.com> On Tuesday, January 10, 2017 1:25:12 PM CET Robin Murphy wrote: > On 10/01/17 12:47, Nikita Yushchenko wrote: > >> The point here is that an IOMMU doesn't solve your issue, and the > >> IOMMU-backed DMA ops need the same treatment. In light of that, it really > >> feels to me like the DMA masks should be restricted in of_dma_configure > >> so that the parent mask is taken into account there, rather than hook > >> into each set of DMA ops to intercept set_dma_mask. We'd still need to > >> do something to stop dma_set_mask widening the mask if it was restricted > >> by of_dma_configure, but I think Robin (cc'd) was playing with that. > > > > What issue "IOMMU doesn't solve"? > > > > Issue I'm trying to address is - inconsistency within swiotlb > > dma_map_ops, where (1) any wide mask is silently accepted, but (2) then > > mask is used to decide if bounce buffers are needed or not. This > > inconsistency causes NVMe+R-Car cobmo not working (and breaking memory > > instead). > > The fundamental underlying problem is the "any wide mask is silently > accepted" part, and that applies equally to IOMMU ops as well. It's a much rarer problem for the IOMMU case though, because it only impacts devices that are restricted to addressing of less than 32-bits. If you have an IOMMU enabled, the dma-mapping interface does not care if the device can do wider than 32 bit addressing, as it will never hand out IOVAs above 0xffffffff. > > I just can't think out what similar issue iommu can have. > > Do you mean that in iommu case, mask also must not be set to whatever > > wider than initial value? Why? What is the use of mask in iommu case? Is > > there any real case when iommu can't address all memory existing in the > > system? > > There's a very subtle misunderstanding there - the DMA mask does not > describe the memory a device can address, it describes the range of > addresses the device is capable of generating. Yes, in the non-IOMMU > case they are equivalent, but once you put an IOMMU in between, the > problem is merely shifted from "what range of physical addresses can > this device access" to "what range of IOVAs is valid to give to this > device" - the fact that those IOVAs can map to any underlying physical > address only obviates the need for any bouncing at the memory end; it > doesn't remove the fact that the device has a hardware addressing > limitation which needs to be accommodated. > > The thread Will linked to describes that equivalent version of your > problem - the IOMMU gives the device 48-bit addresses which get > erroneously truncated because it doesn't know that only 42 bits are > actually wired up. That situation still requires the device's DMA mask > to correctly describe its addressing capability just as yours does. That problem should only impact virtual machines which have a guest bus address space covering more than 42 bits of physical RAM, whereas the problem we have with swiotlb is for the dma-mapping interface. > > With this direction, semantics of dma mask becomes even more > > questionable. I'd say dma_mask is candidate for removal (or to move to > > swiotlb's or iommu's local area) > > We still need a way for drivers to communicate a device's probed > addressing capability to SWIOTLB, so there's always going to have to be > *some* sort of public interface. Personally, the change in semantics I'd > like to see is to make dma_set_mask() only fail if DMA is entirely > disallowed - in the normal case it would always succeed, but the DMA API > implementation would be permitted to set a smaller mask than requested > (this is effectively what the x86 IOMMU ops do already). With swiotlb enabled, it only needs to fail if the mask does not contain the swiotlb bounce buffer area, either because the start of RAM is outside of the mask, or the bounce area has been allocated at the end of ZONE_DMA and the mask is smaller than ZONE_DMA. Arnd
WARNING: multiple messages have this Message-ID (diff)
From: arnd@arndb.de (Arnd Bergmann) To: linux-arm-kernel@lists.infradead.org Subject: [PATCH v2] arm64: do not set dma masks that device connection can't handle Date: Tue, 10 Jan 2017 14:42:23 +0100 [thread overview] Message-ID: <6116278.nQQUSuo3l4@wuerfel> (raw) In-Reply-To: <11daacde-5399-039f-80a3-01d7bd13e9e8@arm.com> On Tuesday, January 10, 2017 1:25:12 PM CET Robin Murphy wrote: > On 10/01/17 12:47, Nikita Yushchenko wrote: > >> The point here is that an IOMMU doesn't solve your issue, and the > >> IOMMU-backed DMA ops need the same treatment. In light of that, it really > >> feels to me like the DMA masks should be restricted in of_dma_configure > >> so that the parent mask is taken into account there, rather than hook > >> into each set of DMA ops to intercept set_dma_mask. We'd still need to > >> do something to stop dma_set_mask widening the mask if it was restricted > >> by of_dma_configure, but I think Robin (cc'd) was playing with that. > > > > What issue "IOMMU doesn't solve"? > > > > Issue I'm trying to address is - inconsistency within swiotlb > > dma_map_ops, where (1) any wide mask is silently accepted, but (2) then > > mask is used to decide if bounce buffers are needed or not. This > > inconsistency causes NVMe+R-Car cobmo not working (and breaking memory > > instead). > > The fundamental underlying problem is the "any wide mask is silently > accepted" part, and that applies equally to IOMMU ops as well. It's a much rarer problem for the IOMMU case though, because it only impacts devices that are restricted to addressing of less than 32-bits. If you have an IOMMU enabled, the dma-mapping interface does not care if the device can do wider than 32 bit addressing, as it will never hand out IOVAs above 0xffffffff. > > I just can't think out what similar issue iommu can have. > > Do you mean that in iommu case, mask also must not be set to whatever > > wider than initial value? Why? What is the use of mask in iommu case? Is > > there any real case when iommu can't address all memory existing in the > > system? > > There's a very subtle misunderstanding there - the DMA mask does not > describe the memory a device can address, it describes the range of > addresses the device is capable of generating. Yes, in the non-IOMMU > case they are equivalent, but once you put an IOMMU in between, the > problem is merely shifted from "what range of physical addresses can > this device access" to "what range of IOVAs is valid to give to this > device" - the fact that those IOVAs can map to any underlying physical > address only obviates the need for any bouncing at the memory end; it > doesn't remove the fact that the device has a hardware addressing > limitation which needs to be accommodated. > > The thread Will linked to describes that equivalent version of your > problem - the IOMMU gives the device 48-bit addresses which get > erroneously truncated because it doesn't know that only 42 bits are > actually wired up. That situation still requires the device's DMA mask > to correctly describe its addressing capability just as yours does. That problem should only impact virtual machines which have a guest bus address space covering more than 42 bits of physical RAM, whereas the problem we have with swiotlb is for the dma-mapping interface. > > With this direction, semantics of dma mask becomes even more > > questionable. I'd say dma_mask is candidate for removal (or to move to > > swiotlb's or iommu's local area) > > We still need a way for drivers to communicate a device's probed > addressing capability to SWIOTLB, so there's always going to have to be > *some* sort of public interface. Personally, the change in semantics I'd > like to see is to make dma_set_mask() only fail if DMA is entirely > disallowed - in the normal case it would always succeed, but the DMA API > implementation would be permitted to set a smaller mask than requested > (this is effectively what the x86 IOMMU ops do already). With swiotlb enabled, it only needs to fail if the mask does not contain the swiotlb bounce buffer area, either because the start of RAM is outside of the mask, or the bounce area has been allocated at the end of ZONE_DMA and the mask is smaller than ZONE_DMA. Arnd
next prev parent reply other threads:[~2017-01-10 15:00 UTC|newest] Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top 2017-01-09 7:30 [PATCH v2] arm64: do not set dma masks that device connection can't handle Nikita Yushchenko 2017-01-09 7:30 ` Nikita Yushchenko 2017-01-10 11:51 ` Will Deacon 2017-01-10 11:51 ` Will Deacon 2017-01-10 12:47 ` Nikita Yushchenko 2017-01-10 12:47 ` Nikita Yushchenko 2017-01-10 13:12 ` Arnd Bergmann 2017-01-10 13:12 ` Arnd Bergmann 2017-01-10 13:25 ` Robin Murphy 2017-01-10 13:25 ` Robin Murphy 2017-01-10 13:42 ` Arnd Bergmann [this message] 2017-01-10 13:42 ` Arnd Bergmann 2017-01-10 14:16 ` Robin Murphy 2017-01-10 14:16 ` Robin Murphy 2017-01-10 15:06 ` Arnd Bergmann 2017-01-10 15:06 ` Arnd Bergmann 2017-01-11 12:37 ` Nikita Yushchenko 2017-01-11 12:37 ` Nikita Yushchenko 2017-01-11 16:21 ` Arnd Bergmann 2017-01-11 16:21 ` Arnd Bergmann 2017-01-11 18:28 ` Robin Murphy 2017-01-11 18:28 ` Robin Murphy 2017-01-10 14:59 ` Christoph Hellwig 2017-01-10 14:59 ` Christoph Hellwig 2017-01-10 14:00 ` [PATCH] arm64: avoid increasing DMA masks above what hardware supports Nikita Yushchenko 2017-01-10 14:00 ` Nikita Yushchenko 2017-01-10 17:14 ` Robin Murphy 2017-01-10 17:14 ` Robin Murphy 2017-01-11 7:59 ` Nikita Yushchenko 2017-01-11 7:59 ` Nikita Yushchenko 2017-01-11 11:54 ` Robin Murphy 2017-01-11 11:54 ` Robin Murphy 2017-01-11 13:41 ` Nikita Yushchenko 2017-01-11 13:41 ` Nikita Yushchenko 2017-01-11 14:50 ` Robin Murphy 2017-01-11 14:50 ` Robin Murphy 2017-01-11 16:03 ` Nikita Yushchenko 2017-01-11 16:50 ` Robin Murphy 2017-01-11 16:50 ` Robin Murphy 2017-01-11 18:31 ` [PATCH 0/2] arm64: fix handling of DMA masks wider than bus supports Nikita Yushchenko 2017-01-11 18:31 ` Nikita Yushchenko 2017-01-11 18:31 ` [PATCH 1/2] dma-mapping: let arch know origin of dma range passed to arch_setup_dma_ops() Nikita Yushchenko 2017-01-11 18:31 ` Nikita Yushchenko 2017-01-11 21:08 ` Arnd Bergmann 2017-01-11 21:08 ` Arnd Bergmann 2017-01-12 5:52 ` Nikita Yushchenko 2017-01-12 5:52 ` Nikita Yushchenko 2017-01-12 6:33 ` Nikita Yushchenko 2017-01-12 6:33 ` Nikita Yushchenko 2017-01-12 13:28 ` Arnd Bergmann 2017-01-12 13:28 ` Arnd Bergmann 2017-01-12 13:39 ` Nikita Yushchenko 2017-01-12 13:39 ` Nikita Yushchenko 2017-01-12 12:16 ` Will Deacon 2017-01-12 12:16 ` Will Deacon 2017-01-12 13:25 ` Arnd Bergmann 2017-01-12 13:25 ` Arnd Bergmann 2017-01-12 13:43 ` Robin Murphy 2017-01-12 13:43 ` Robin Murphy 2017-01-13 10:40 ` kbuild test robot 2017-01-13 10:40 ` kbuild test robot 2017-01-11 18:31 ` [PATCH 2/2] arm64: avoid increasing DMA masks above what hardware supports Nikita Yushchenko 2017-01-11 18:31 ` Nikita Yushchenko 2017-01-11 21:11 ` Arnd Bergmann 2017-01-11 21:11 ` Arnd Bergmann 2017-01-12 5:53 ` Nikita Yushchenko 2017-01-12 5:53 ` Nikita Yushchenko 2017-01-13 10:16 ` kbuild test robot 2017-01-13 10:16 ` kbuild test robot 2017-01-10 14:01 ` [PATCH v2] arm64: do not set dma masks that device connection can't handle Nikita Yushchenko 2017-01-10 14:01 ` Nikita Yushchenko 2017-01-10 14:57 ` Christoph Hellwig 2017-01-10 14:57 ` Christoph Hellwig 2017-01-10 14:51 ` Christoph Hellwig 2017-01-10 14:51 ` Christoph Hellwig
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=6116278.nQQUSuo3l4@wuerfel \ --to=arnd@arndb.de \ --cc=artemi.ivanov@cogentembedded.com \ --cc=bhelgaas@google.com \ --cc=catalin.marinas@arm.com \ --cc=fkan@apm.com \ --cc=hch@lst.de \ --cc=horms@verge.net.au \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-renesas-soc@vger.kernel.org \ --cc=nikita.yoush@cogentembedded.com \ --cc=robin.murphy@arm.com \ --cc=will.deacon@arm.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.