From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking Date: Mon, 4 Jan 2016 15:41:04 -0500 Message-ID: <20160104204104.GB17427@char.us.oracle.com> References: <20151213212557.5410.48577.stgit@localhost.localdomain> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: tianyu.lan@intel.com, yang.zhang.wz@gmail.com, alex.williamson@redhat.com, kvm@vger.kernel.org, mst@redhat.com, linux-pci@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, qemu-devel@nongnu.org, agraf@suse.de, dgilbert@redhat.com To: Alexander Duyck Return-path: Content-Disposition: inline In-Reply-To: <20151213212557.5410.48577.stgit@localhost.localdomain> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org List-Id: kvm.vger.kernel.org On Sun, Dec 13, 2015 at 01:28:09PM -0800, Alexander Duyck wrote: > This patch set is meant to be the guest side code for a proof of concept > involving leaving pass-through devices in the guest during the warm-up > phase of guest live migration. In order to accomplish this I have added a What does that mean? 'warm-up-phase'? > new function called dma_mark_dirty that will mark the pages associated with > the DMA transaction as dirty in the case of either an unmap or a > sync_.*_for_cpu where the DMA direction is either DMA_FROM_DEVICE or > DMA_BIDIRECTIONAL. The pass-through device must still be removed before > the stop-and-copy phase, however allowing the device to be present should > significantly improve the performance of the guest during the warm-up > period. .. if the warm-up phase is short I presume? If the warm-up phase takes a long time (busy guest that is of 1TB size) it wouldn't help much as the tracking of these DMA's may be quite long? > > This current implementation is very preliminary and there are number of > items still missing. Specifically in order to make this a more complete > solution we need to support: > 1. Notifying hypervisor that drivers are dirtying DMA pages received .. And somehow giving the hypervisor the GPFN so it can retain the PFN in the VT-d as long as possible. > 2. Bypassing page dirtying when it is not needed. How would this work with with device doing DMA operations _after_ the migration? That is the driver submits and DMA READ.. migrates away, device is unplugged, VT-d context is torn down - device does the DMA READ gets an VT-d error... and what then? How should the device on the other host replay the DMA READ? > > The two mechanisms referenced above would likely require coordination with > QEMU and as such are open to discussion. I haven't attempted to address > them as I am not sure there is a consensus as of yet. My personal > preference would be to add a vendor-specific configuration block to the > emulated pci-bridge interfaces created by QEMU that would allow us to > essentially extend shpc to support guest live migration with pass-through > devices. shpc? > > The functionality in this patch set is currently disabled by default. To > enable it you can select "SWIOTLB page dirtying" from the "Processor type > and features" menu. > > --- > > Alexander Duyck (3): > swiotlb: Fold static unmap and sync calls into calling functions > xen/swiotlb: Fold static unmap and sync calls into calling functions > x86: Create dma_mark_dirty to dirty pages used for DMA by VM guest > > > arch/arm/include/asm/dma-mapping.h | 3 + > arch/arm64/include/asm/dma-mapping.h | 5 +- > arch/ia64/include/asm/dma.h | 1 > arch/mips/include/asm/dma-mapping.h | 1 > arch/powerpc/include/asm/swiotlb.h | 1 > arch/tile/include/asm/dma-mapping.h | 1 > arch/unicore32/include/asm/dma-mapping.h | 1 > arch/x86/Kconfig | 11 ++++ > arch/x86/include/asm/swiotlb.h | 26 ++++++++ > drivers/xen/swiotlb-xen.c | 92 +++++++++++++----------------- > lib/swiotlb.c | 83 ++++++++++++--------------- > 11 files changed, 123 insertions(+), 102 deletions(-) > > -- From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48082) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aGBwQ-0002jg-Jc for qemu-devel@nongnu.org; Mon, 04 Jan 2016 15:41:23 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aGBwM-0006VI-DK for qemu-devel@nongnu.org; Mon, 04 Jan 2016 15:41:22 -0500 Received: from aserp1040.oracle.com ([141.146.126.69]:22459) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aGBwM-0006V4-4R for qemu-devel@nongnu.org; Mon, 04 Jan 2016 15:41:18 -0500 Date: Mon, 4 Jan 2016 15:41:04 -0500 From: Konrad Rzeszutek Wilk Message-ID: <20160104204104.GB17427@char.us.oracle.com> References: <20151213212557.5410.48577.stgit@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20151213212557.5410.48577.stgit@localhost.localdomain> Subject: Re: [Qemu-devel] [RFC PATCH 0/3] x86: Add support for guest DMA dirty page tracking List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alexander Duyck Cc: tianyu.lan@intel.com, yang.zhang.wz@gmail.com, alex.williamson@redhat.com, kvm@vger.kernel.org, mst@redhat.com, linux-pci@vger.kernel.org, x86@kernel.org, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, qemu-devel@nongnu.org, agraf@suse.de, dgilbert@redhat.com On Sun, Dec 13, 2015 at 01:28:09PM -0800, Alexander Duyck wrote: > This patch set is meant to be the guest side code for a proof of concept > involving leaving pass-through devices in the guest during the warm-up > phase of guest live migration. In order to accomplish this I have added a What does that mean? 'warm-up-phase'? > new function called dma_mark_dirty that will mark the pages associated with > the DMA transaction as dirty in the case of either an unmap or a > sync_.*_for_cpu where the DMA direction is either DMA_FROM_DEVICE or > DMA_BIDIRECTIONAL. The pass-through device must still be removed before > the stop-and-copy phase, however allowing the device to be present should > significantly improve the performance of the guest during the warm-up > period. .. if the warm-up phase is short I presume? If the warm-up phase takes a long time (busy guest that is of 1TB size) it wouldn't help much as the tracking of these DMA's may be quite long? > > This current implementation is very preliminary and there are number of > items still missing. Specifically in order to make this a more complete > solution we need to support: > 1. Notifying hypervisor that drivers are dirtying DMA pages received .. And somehow giving the hypervisor the GPFN so it can retain the PFN in the VT-d as long as possible. > 2. Bypassing page dirtying when it is not needed. How would this work with with device doing DMA operations _after_ the migration? That is the driver submits and DMA READ.. migrates away, device is unplugged, VT-d context is torn down - device does the DMA READ gets an VT-d error... and what then? How should the device on the other host replay the DMA READ? > > The two mechanisms referenced above would likely require coordination with > QEMU and as such are open to discussion. I haven't attempted to address > them as I am not sure there is a consensus as of yet. My personal > preference would be to add a vendor-specific configuration block to the > emulated pci-bridge interfaces created by QEMU that would allow us to > essentially extend shpc to support guest live migration with pass-through > devices. shpc? > > The functionality in this patch set is currently disabled by default. To > enable it you can select "SWIOTLB page dirtying" from the "Processor type > and features" menu. > > --- > > Alexander Duyck (3): > swiotlb: Fold static unmap and sync calls into calling functions > xen/swiotlb: Fold static unmap and sync calls into calling functions > x86: Create dma_mark_dirty to dirty pages used for DMA by VM guest > > > arch/arm/include/asm/dma-mapping.h | 3 + > arch/arm64/include/asm/dma-mapping.h | 5 +- > arch/ia64/include/asm/dma.h | 1 > arch/mips/include/asm/dma-mapping.h | 1 > arch/powerpc/include/asm/swiotlb.h | 1 > arch/tile/include/asm/dma-mapping.h | 1 > arch/unicore32/include/asm/dma-mapping.h | 1 > arch/x86/Kconfig | 11 ++++ > arch/x86/include/asm/swiotlb.h | 26 ++++++++ > drivers/xen/swiotlb-xen.c | 92 +++++++++++++----------------- > lib/swiotlb.c | 83 ++++++++++++--------------- > 11 files changed, 123 insertions(+), 102 deletions(-) > > --