From mboxrd@z Thu Jan 1 00:00:00 1970 From: Will Deacon Subject: Re: [PATCH] virtio: Try to untangle DMA coherency Date: Wed, 1 Feb 2017 14:57:11 +0000 Message-ID: <20170201145711.GC8177@arm.com> References: <8a6494f6409c20b4609cd6bdcdd751f68b5c0564.1485951731.git.robin.murphy@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <8a6494f6409c20b4609cd6bdcdd751f68b5c0564.1485951731.git.robin.murphy@arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Robin Murphy Cc: mark.rutland@arm.com, devicetree@vger.kernel.org, pawel.moll@arm.com, mst@redhat.com, virtualization@lists.linux-foundation.org, robh+dt@kernel.org, linux-arm-kernel@lists.infradead.org List-Id: devicetree@vger.kernel.org On Wed, Feb 01, 2017 at 12:25:57PM +0000, Robin Murphy wrote: > By forcing on DMA API usage for ARM systems, we have inadvertently > kicked open a hornets' nest in terms of cache-coherency. Namely that > unless the virtio device is explicitly described as capable of coherent > DMA by firmware, the DMA APIs on ARM and other DT-based platforms will > assume it is non-coherent. This turns out to cause a big problem for the > likes of QEMU and kvmtool, which generate virtio-mmio devices in their > guest DTs but neglect to add the often-overlooked "dma-coherent" > property; as a result, we end up with the guest making non-cacheable > accesses to the vring, the host doing so cacheably, both talking past > each other and things going horribly wrong. > > To prevent regressing those existing use cases relying on implicit > coherency, but still fixing the original problem of (coherent PCI) > legacy devices behind IOMMUs, take the more conservative approach of > only hitting the DMA API switch for coherent devices, where we can be > sure it is safe, and preserving the old non-DMA behaviour otherwise. > This does not affect devices setting the VIRTIO_F_IOMMU_PLATFORM flag, > which as before are still at the mercy of architecture code correctly > knowing their coherency, so explicitly call this out in the virtio-mmio > DT binding in the hope of heading off any further workarounds for future > firmware mishaps. > > Fixes: c7070619f340 ("vring: Force use of DMA API for ARM-based systems with legacy devices") > Signed-off-by: Robin Murphy > --- > Documentation/devicetree/bindings/virtio/mmio.txt | 3 +++ > drivers/virtio/virtio_ring.c | 11 ++++++++--- > 2 files changed, 11 insertions(+), 3 deletions(-) > > diff --git a/Documentation/devicetree/bindings/virtio/mmio.txt b/Documentation/devicetree/bindings/virtio/mmio.txt > index 5069c1b8e193..8f2a981e1010 100644 > --- a/Documentation/devicetree/bindings/virtio/mmio.txt > +++ b/Documentation/devicetree/bindings/virtio/mmio.txt > @@ -7,6 +7,8 @@ Required properties: > - compatible: "virtio,mmio" compatibility string > - reg: control registers base address and size including configuration space > - interrupts: interrupt generated by the device > +- dma-coherent: required if the device (or host emulation) accesses memory > + cache-coherently, absent otherwise So QEMU is getting with not providing this property at the moment and this patch continues to ensure that works coherently, which is the right thing to do. However, if QEMU ever emulates devices with VIRTIO_F_IOMMU_PLATFORM, then it will need to provide this property for coherent virtio-mmio devices upstream of the IOMMU otherwise things won't work. I think that's acceptable given that VIRTIO_F_IOMMU_PLATFORM has never done the right thing with respect to coherency if "dma-coherent" is not present, but it would be nice to call that out somewhere so that QEMU developers can be aware of this pitfall. Could we add something like: Linux implementation note: virtio-mmio devices that do not advertise the VIRTIO_F_IOMMU_PLATFORM feature are treated as cache-coherent irrespective of the "dma-coherent" property. If VIRTIO_F_IOMMU_PLATFORM is advertised, then the "dma-coherent" property must accurately reflect the coherency of the device. ? I know that the binding documentation needs to be OS agnostic, but I think it's useful to describe the Linux behaviour here given that QEMU is technically in violation of the binding after this patch is applied. Will From mboxrd@z Thu Jan 1 00:00:00 1970 From: will.deacon@arm.com (Will Deacon) Date: Wed, 1 Feb 2017 14:57:11 +0000 Subject: [PATCH] virtio: Try to untangle DMA coherency In-Reply-To: <8a6494f6409c20b4609cd6bdcdd751f68b5c0564.1485951731.git.robin.murphy@arm.com> References: <8a6494f6409c20b4609cd6bdcdd751f68b5c0564.1485951731.git.robin.murphy@arm.com> Message-ID: <20170201145711.GC8177@arm.com> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Wed, Feb 01, 2017 at 12:25:57PM +0000, Robin Murphy wrote: > By forcing on DMA API usage for ARM systems, we have inadvertently > kicked open a hornets' nest in terms of cache-coherency. Namely that > unless the virtio device is explicitly described as capable of coherent > DMA by firmware, the DMA APIs on ARM and other DT-based platforms will > assume it is non-coherent. This turns out to cause a big problem for the > likes of QEMU and kvmtool, which generate virtio-mmio devices in their > guest DTs but neglect to add the often-overlooked "dma-coherent" > property; as a result, we end up with the guest making non-cacheable > accesses to the vring, the host doing so cacheably, both talking past > each other and things going horribly wrong. > > To prevent regressing those existing use cases relying on implicit > coherency, but still fixing the original problem of (coherent PCI) > legacy devices behind IOMMUs, take the more conservative approach of > only hitting the DMA API switch for coherent devices, where we can be > sure it is safe, and preserving the old non-DMA behaviour otherwise. > This does not affect devices setting the VIRTIO_F_IOMMU_PLATFORM flag, > which as before are still at the mercy of architecture code correctly > knowing their coherency, so explicitly call this out in the virtio-mmio > DT binding in the hope of heading off any further workarounds for future > firmware mishaps. > > Fixes: c7070619f340 ("vring: Force use of DMA API for ARM-based systems with legacy devices") > Signed-off-by: Robin Murphy > --- > Documentation/devicetree/bindings/virtio/mmio.txt | 3 +++ > drivers/virtio/virtio_ring.c | 11 ++++++++--- > 2 files changed, 11 insertions(+), 3 deletions(-) > > diff --git a/Documentation/devicetree/bindings/virtio/mmio.txt b/Documentation/devicetree/bindings/virtio/mmio.txt > index 5069c1b8e193..8f2a981e1010 100644 > --- a/Documentation/devicetree/bindings/virtio/mmio.txt > +++ b/Documentation/devicetree/bindings/virtio/mmio.txt > @@ -7,6 +7,8 @@ Required properties: > - compatible: "virtio,mmio" compatibility string > - reg: control registers base address and size including configuration space > - interrupts: interrupt generated by the device > +- dma-coherent: required if the device (or host emulation) accesses memory > + cache-coherently, absent otherwise So QEMU is getting with not providing this property at the moment and this patch continues to ensure that works coherently, which is the right thing to do. However, if QEMU ever emulates devices with VIRTIO_F_IOMMU_PLATFORM, then it will need to provide this property for coherent virtio-mmio devices upstream of the IOMMU otherwise things won't work. I think that's acceptable given that VIRTIO_F_IOMMU_PLATFORM has never done the right thing with respect to coherency if "dma-coherent" is not present, but it would be nice to call that out somewhere so that QEMU developers can be aware of this pitfall. Could we add something like: Linux implementation note: virtio-mmio devices that do not advertise the VIRTIO_F_IOMMU_PLATFORM feature are treated as cache-coherent irrespective of the "dma-coherent" property. If VIRTIO_F_IOMMU_PLATFORM is advertised, then the "dma-coherent" property must accurately reflect the coherency of the device. ? I know that the binding documentation needs to be OS agnostic, but I think it's useful to describe the Linux behaviour here given that QEMU is technically in violation of the binding after this patch is applied. Will