From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751544AbbKIMPx (ORCPT ); Mon, 9 Nov 2015 07:15:53 -0500 Received: from mx1.redhat.com ([209.132.183.28]:41901 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750858AbbKIMPt (ORCPT ); Mon, 9 Nov 2015 07:15:49 -0500 Date: Mon, 9 Nov 2015 14:15:42 +0200 From: "Michael S. Tsirkin" To: Andy Lutomirski Cc: linux-kernel@vger.kernel.org, "David S. Miller" , sparclinux@vger.kernel.org, Joerg Roedel , Christian Borntraeger , Cornelia Huck , Sebastian Ott , Paolo Bonzini , Christoph Hellwig , benh@kernel.crashing.org, KVM , dwmw2@infradead.org, Martin Schwidefsky , linux-s390 , virtualization@lists.linux-foundation.org Subject: Re: [PATCH v4 0/6] virtio core DMA API conversion Message-ID: <20151109133624-mutt-send-email-mst@redhat.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Oct 29, 2015 at 06:09:45PM -0700, Andy Lutomirski wrote: > This switches virtio to use the DMA API unconditionally. I'm sure > it breaks things, but it seems to work on x86 using virtio-pci, with > and without Xen, and using both the modern 1.0 variant and the > legacy variant. > > This appears to work on native and Xen x86_64 using both modern and > legacy virtio-pci. It also appears to work on arm and arm64. > > It definitely won't work as-is on s390x, and I haven't been able to > test Christian's patches because I can't get virtio-ccw to work in > QEMU at all. I don't know what I'm doing wrong. > > It doesn't work on ppc64. Ben, consider yourself pinged to send me > a patch :) > > It doesn't work on sparc64. I didn't realize at Kernel Summit that > sparc64 has the same problem as ppc64. > > DaveM, for background, we're trying to fix virtio to use the DMA > API. That will require that every platform that uses virtio > supplies valid DMA operations on devices that use virtio_ring. > Unfortunately, QEMU historically ignores the IOMMU on virtio > devices. > > On x86, this isn't really a problem. x86 has a nice way for the > platform to describe which devices are behind an IOMMU, and QEMU > will be adjusted accordingly. The only thing that will break is a > recently-added experimental mode. Well that's not exactly true. I think we would like to make it possible to put virtio devices behind an IOMMU on x86, but if this means existing guests break, then many people won't be able to use this option: having to find out which kernel version your guest is running is a significant burden. So on the host side, we need to detect guests that don't program the IOMMU and make QEMU ignore it. I think we need to figure out a way to do this before we commit to the guest change. Additionally, IOMMU overhead is very high when running within the VM. So for uses such as VFIO, we'd like a way to make something like iommu-pt the default. > Ben's plan for powerpc is to add a quirk for existing virtio-pci > devices and to eventually update the devicetree stuff to allow QEMU > to tell the guest which devices use the IOMMU. > > AFAICT sparc has a similar problem to powerpc. DaveM, can you come > up with a straightforward way to get sparc's DMA API to work > correctly for virtio-pci devices? > > NB: Sadly, the platforms I've successfully tested on don't include any > big-endian platforms, so there could still be lurking endian problems. > > Changes from v3: > - More big-endian fixes. > - Added better virtio-ring APIs that handle allocation and use them in > virtio-mmio and virtio-pci. > - Switch to Michael's virtio-net patch. > > Changes from v2: > - Fix vring_mapping_error incorrect argument > > Changes from v1: > - Fix an endian conversion error causing a BUG to hit. > - Fix a DMA ordering issue (swiotlb=force works now). > - Minor cleanups. > > Andy Lutomirski (5): > virtio_ring: Support DMA APIs > virtio_pci: Use the DMA API > virtio: Add improved queue allocation API > virtio_mmio: Use the DMA API > virtio_pci: Use the DMA API > > Michael S. Tsirkin (1): > virtio-net: Stop doing DMA from the stack > > drivers/net/virtio_net.c | 34 ++-- > drivers/virtio/Kconfig | 2 +- > drivers/virtio/virtio_mmio.c | 67 ++----- > drivers/virtio/virtio_pci_common.h | 6 - > drivers/virtio/virtio_pci_legacy.c | 42 ++--- > drivers/virtio/virtio_pci_modern.c | 61 ++----- > drivers/virtio/virtio_ring.c | 348 ++++++++++++++++++++++++++++++------- > include/linux/virtio.h | 23 ++- > include/linux/virtio_ring.h | 35 ++++ > tools/virtio/linux/dma-mapping.h | 17 ++ > 10 files changed, 426 insertions(+), 209 deletions(-) > create mode 100644 tools/virtio/linux/dma-mapping.h > > -- > 2.4.3 From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [PATCH v4 0/6] virtio core DMA API conversion Date: Mon, 9 Nov 2015 14:15:42 +0200 Message-ID: <20151109133624-mutt-send-email-mst@redhat.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org List-Archive: List-Post: To: Andy Lutomirski Cc: linux-s390 , Joerg Roedel , KVM , benh@kernel.crashing.org, Sebastian Ott , virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Christian Borntraeger , sparclinux@vger.kernel.org, Paolo Bonzini , dwmw2@infradead.org, Christoph Hellwig , "David S. Miller" , Martin Schwidefsky List-ID: On Thu, Oct 29, 2015 at 06:09:45PM -0700, Andy Lutomirski wrote: > This switches virtio to use the DMA API unconditionally. I'm sure > it breaks things, but it seems to work on x86 using virtio-pci, with > and without Xen, and using both the modern 1.0 variant and the > legacy variant. > > This appears to work on native and Xen x86_64 using both modern and > legacy virtio-pci. It also appears to work on arm and arm64. > > It definitely won't work as-is on s390x, and I haven't been able to > test Christian's patches because I can't get virtio-ccw to work in > QEMU at all. I don't know what I'm doing wrong. > > It doesn't work on ppc64. Ben, consider yourself pinged to send me > a patch :) > > It doesn't work on sparc64. I didn't realize at Kernel Summit that > sparc64 has the same problem as ppc64. > > DaveM, for background, we're trying to fix virtio to use the DMA > API. That will require that every platform that uses virtio > supplies valid DMA operations on devices that use virtio_ring. > Unfortunately, QEMU historically ignores the IOMMU on virtio > devices. > > On x86, this isn't really a problem. x86 has a nice way for the > platform to describe which devices are behind an IOMMU, and QEMU > will be adjusted accordingly. The only thing that will break is a > recently-added experimental mode. Well that's not exactly true. I think we would like to make it possible to put virtio devices behind an IOMMU on x86, but if this means existing guests break, then many people won't be able to use this option: having to find out which kernel version your guest is running is a significant burden. So on the host side, we need to detect guests that don't program the IOMMU and make QEMU ignore it. I think we need to figure out a way to do this before we commit to the guest change. Additionally, IOMMU overhead is very high when running within the VM. So for uses such as VFIO, we'd like a way to make something like iommu-pt the default. > Ben's plan for powerpc is to add a quirk for existing virtio-pci > devices and to eventually update the devicetree stuff to allow QEMU > to tell the guest which devices use the IOMMU. > > AFAICT sparc has a similar problem to powerpc. DaveM, can you come > up with a straightforward way to get sparc's DMA API to work > correctly for virtio-pci devices? > > NB: Sadly, the platforms I've successfully tested on don't include any > big-endian platforms, so there could still be lurking endian problems. > > Changes from v3: > - More big-endian fixes. > - Added better virtio-ring APIs that handle allocation and use them in > virtio-mmio and virtio-pci. > - Switch to Michael's virtio-net patch. > > Changes from v2: > - Fix vring_mapping_error incorrect argument > > Changes from v1: > - Fix an endian conversion error causing a BUG to hit. > - Fix a DMA ordering issue (swiotlb=force works now). > - Minor cleanups. > > Andy Lutomirski (5): > virtio_ring: Support DMA APIs > virtio_pci: Use the DMA API > virtio: Add improved queue allocation API > virtio_mmio: Use the DMA API > virtio_pci: Use the DMA API > > Michael S. Tsirkin (1): > virtio-net: Stop doing DMA from the stack > > drivers/net/virtio_net.c | 34 ++-- > drivers/virtio/Kconfig | 2 +- > drivers/virtio/virtio_mmio.c | 67 ++----- > drivers/virtio/virtio_pci_common.h | 6 - > drivers/virtio/virtio_pci_legacy.c | 42 ++--- > drivers/virtio/virtio_pci_modern.c | 61 ++----- > drivers/virtio/virtio_ring.c | 348 ++++++++++++++++++++++++++++++------- > include/linux/virtio.h | 23 ++- > include/linux/virtio_ring.h | 35 ++++ > tools/virtio/linux/dma-mapping.h | 17 ++ > 10 files changed, 426 insertions(+), 209 deletions(-) > create mode 100644 tools/virtio/linux/dma-mapping.h > > -- > 2.4.3 From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Date: Mon, 09 Nov 2015 12:15:42 +0000 Subject: Re: [PATCH v4 0/6] virtio core DMA API conversion Message-Id: <20151109133624-mutt-send-email-mst@redhat.com> List-Id: References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Andy Lutomirski Cc: linux-s390 , Joerg Roedel , KVM , benh@kernel.crashing.org, Sebastian Ott , virtualization@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Christian Borntraeger , sparclinux@vger.kernel.org, Paolo Bonzini , dwmw2@infradead.org, Christoph Hellwig , "David S. Miller" , Martin Schwidefsky On Thu, Oct 29, 2015 at 06:09:45PM -0700, Andy Lutomirski wrote: > This switches virtio to use the DMA API unconditionally. I'm sure > it breaks things, but it seems to work on x86 using virtio-pci, with > and without Xen, and using both the modern 1.0 variant and the > legacy variant. > > This appears to work on native and Xen x86_64 using both modern and > legacy virtio-pci. It also appears to work on arm and arm64. > > It definitely won't work as-is on s390x, and I haven't been able to > test Christian's patches because I can't get virtio-ccw to work in > QEMU at all. I don't know what I'm doing wrong. > > It doesn't work on ppc64. Ben, consider yourself pinged to send me > a patch :) > > It doesn't work on sparc64. I didn't realize at Kernel Summit that > sparc64 has the same problem as ppc64. > > DaveM, for background, we're trying to fix virtio to use the DMA > API. That will require that every platform that uses virtio > supplies valid DMA operations on devices that use virtio_ring. > Unfortunately, QEMU historically ignores the IOMMU on virtio > devices. > > On x86, this isn't really a problem. x86 has a nice way for the > platform to describe which devices are behind an IOMMU, and QEMU > will be adjusted accordingly. The only thing that will break is a > recently-added experimental mode. Well that's not exactly true. I think we would like to make it possible to put virtio devices behind an IOMMU on x86, but if this means existing guests break, then many people won't be able to use this option: having to find out which kernel version your guest is running is a significant burden. So on the host side, we need to detect guests that don't program the IOMMU and make QEMU ignore it. I think we need to figure out a way to do this before we commit to the guest change. Additionally, IOMMU overhead is very high when running within the VM. So for uses such as VFIO, we'd like a way to make something like iommu-pt the default. > Ben's plan for powerpc is to add a quirk for existing virtio-pci > devices and to eventually update the devicetree stuff to allow QEMU > to tell the guest which devices use the IOMMU. > > AFAICT sparc has a similar problem to powerpc. DaveM, can you come > up with a straightforward way to get sparc's DMA API to work > correctly for virtio-pci devices? > > NB: Sadly, the platforms I've successfully tested on don't include any > big-endian platforms, so there could still be lurking endian problems. > > Changes from v3: > - More big-endian fixes. > - Added better virtio-ring APIs that handle allocation and use them in > virtio-mmio and virtio-pci. > - Switch to Michael's virtio-net patch. > > Changes from v2: > - Fix vring_mapping_error incorrect argument > > Changes from v1: > - Fix an endian conversion error causing a BUG to hit. > - Fix a DMA ordering issue (swiotlb=force works now). > - Minor cleanups. > > Andy Lutomirski (5): > virtio_ring: Support DMA APIs > virtio_pci: Use the DMA API > virtio: Add improved queue allocation API > virtio_mmio: Use the DMA API > virtio_pci: Use the DMA API > > Michael S. Tsirkin (1): > virtio-net: Stop doing DMA from the stack > > drivers/net/virtio_net.c | 34 ++-- > drivers/virtio/Kconfig | 2 +- > drivers/virtio/virtio_mmio.c | 67 ++----- > drivers/virtio/virtio_pci_common.h | 6 - > drivers/virtio/virtio_pci_legacy.c | 42 ++--- > drivers/virtio/virtio_pci_modern.c | 61 ++----- > drivers/virtio/virtio_ring.c | 348 ++++++++++++++++++++++++++++++------- > include/linux/virtio.h | 23 ++- > include/linux/virtio_ring.h | 35 ++++ > tools/virtio/linux/dma-mapping.h | 17 ++ > 10 files changed, 426 insertions(+), 209 deletions(-) > create mode 100644 tools/virtio/linux/dma-mapping.h > > -- > 2.4.3