From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753746AbbLILTZ (ORCPT ); Wed, 9 Dec 2015 06:19:25 -0500 Received: from mga09.intel.com ([134.134.136.24]:35995 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751410AbbLILTW (ORCPT ); Wed, 9 Dec 2015 06:19:22 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,403,1444719600"; d="scan'208";a="703499742" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC To: "Michael S. Tsirkin" References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis@virtualopensystems.com" , Alex Williamson , "b.reynal@virtualopensystems.com" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm@vger.kernel.org" , Paolo Bonzini , "qemu-devel@nongnu.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , "Brandeburg, Jesse" , "Ronciak, John" , "linux-api@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Williams, Mitch A" , Netdev , "Nelson, Shannon" , Wei Yang , "zajec5@gmail.com" From: "Lan, Tianyu" Message-ID: <56680E33.6040204@intel.com> Date: Wed, 9 Dec 2015 19:19:15 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <20151209122831-mutt-send-email-mst@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: >> Hi Michael & Alexander: >> Thanks a lot for your comments and suggestions. > > It's nice that it's appreciated, but you then go on and ignore > all that I have written here: > https://www.mail-archive.com/kvm@vger.kernel.org/msg123826.html > No, I will reply it separately and according your suggestion to snip it into 3 thread. >> We still need to support Windows guest for migration and this is why our >> patches keep all changes in the driver since it's impossible to change >> Windows kernel. > > This is not a reasonable argument. It makes no sense to duplicate code > on Linux because you must duplicate code on Windows. Let's assume you > must do it in the driver on windows because windows has closed source > drivers. What does it matter? Linux can still do it as part of DMA API > and have it apply to all drivers. > Sure. Duplicated code should be encapsulated and make it able to reuse by other drivers. Just like you said the dummy write part. I meant the framework should not require to change Windows kernel code (such as PM core or PCI bus driver)and this will block implementation on the Windows. I think it's not problem to duplicate code in the Windows drivers. >> Following is my idea to do DMA tracking. >> >> Inject event to VF driver after memory iterate stage >> and before stop VCPU and then VF driver marks dirty all >> using DMA memory. The new allocated pages also need to >> be marked dirty before stopping VCPU. All dirty memory >> in this time slot will be migrated until stop-and-copy >> stage. We also need to make sure to disable VF via clearing the >> bus master enable bit for VF before migrating these memory. >> >> The dma page allocated by VF driver also needs to reserve space >> to do dummy write. > > I suggested ways to do it all in the hypervisor without driver hacks, or > hide it within DMA API without need to reserve extra space. Both > approaches seem much cleaner. > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Lan, Tianyu" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 9 Dec 2015 19:19:15 +0800 Message-ID: <56680E33.6040204@intel.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" , " To: "Michael S. Tsirkin" Return-path: In-Reply-To: <20151209122831-mutt-send-email-mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org List-Id: netdev.vger.kernel.org On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: >> Hi Michael & Alexander: >> Thanks a lot for your comments and suggestions. > > It's nice that it's appreciated, but you then go on and ignore > all that I have written here: > https://www.mail-archive.com/kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg123826.html > No, I will reply it separately and according your suggestion to snip it into 3 thread. >> We still need to support Windows guest for migration and this is why our >> patches keep all changes in the driver since it's impossible to change >> Windows kernel. > > This is not a reasonable argument. It makes no sense to duplicate code > on Linux because you must duplicate code on Windows. Let's assume you > must do it in the driver on windows because windows has closed source > drivers. What does it matter? Linux can still do it as part of DMA API > and have it apply to all drivers. > Sure. Duplicated code should be encapsulated and make it able to reuse by other drivers. Just like you said the dummy write part. I meant the framework should not require to change Windows kernel code (such as PM core or PCI bus driver)and this will block implementation on the Windows. I think it's not problem to duplicate code in the Windows drivers. >> Following is my idea to do DMA tracking. >> >> Inject event to VF driver after memory iterate stage >> and before stop VCPU and then VF driver marks dirty all >> using DMA memory. The new allocated pages also need to >> be marked dirty before stopping VCPU. All dirty memory >> in this time slot will be migrated until stop-and-copy >> stage. We also need to make sure to disable VF via clearing the >> bus master enable bit for VF before migrating these memory. >> >> The dma page allocated by VF driver also needs to reserve space >> to do dummy write. > > I suggested ways to do it all in the hypervisor without driver hacks, or > hide it within DMA API without need to reserve extra space. Both > approaches seem much cleaner. > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Lan, Tianyu" Subject: Re: [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC Date: Wed, 9 Dec 2015 19:19:15 +0800 Message-ID: <56680E33.6040204@intel.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20151209122831-mutt-send-email-mst-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> Sender: linux-api-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: "Michael S. Tsirkin" Cc: Alexander Duyck , "Dong, Eddie" , "a.motakis-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Alex Williamson , "b.reynal-lrHrjnjw1UfHK3s98zE1ajGjJy/sRE9J@public.gmane.org" , Bjorn Helgaas , "Wyborny, Carolyn" , "Skidmore, Donald C" , "Jani, Nrupal" , Alexander Graf , "kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , Paolo Bonzini , "qemu-devel-qX2TKyscuCcdnm+yROfE0A@public.gmane.org" , "Tantilov, Emil S" , Or Gerlitz , "Rustad, Mark D" , Eric Auger , intel-wired-lan , "Kirsher, Jeffrey T" List-Id: linux-api@vger.kernel.org On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: >> Hi Michael & Alexander: >> Thanks a lot for your comments and suggestions. > > It's nice that it's appreciated, but you then go on and ignore > all that I have written here: > https://www.mail-archive.com/kvm-u79uwXL29TY76Z2rM5mHXA@public.gmane.org/msg123826.html > No, I will reply it separately and according your suggestion to snip it into 3 thread. >> We still need to support Windows guest for migration and this is why our >> patches keep all changes in the driver since it's impossible to change >> Windows kernel. > > This is not a reasonable argument. It makes no sense to duplicate code > on Linux because you must duplicate code on Windows. Let's assume you > must do it in the driver on windows because windows has closed source > drivers. What does it matter? Linux can still do it as part of DMA API > and have it apply to all drivers. > Sure. Duplicated code should be encapsulated and make it able to reuse by other drivers. Just like you said the dummy write part. I meant the framework should not require to change Windows kernel code (such as PM core or PCI bus driver)and this will block implementation on the Windows. I think it's not problem to duplicate code in the Windows drivers. >> Following is my idea to do DMA tracking. >> >> Inject event to VF driver after memory iterate stage >> and before stop VCPU and then VF driver marks dirty all >> using DMA memory. The new allocated pages also need to >> be marked dirty before stopping VCPU. All dirty memory >> in this time slot will be migrated until stop-and-copy >> stage. We also need to make sure to disable VF via clearing the >> bus master enable bit for VF before migrating these memory. >> >> The dma page allocated by VF driver also needs to reserve space >> to do dummy write. > > I suggested ways to do it all in the hypervisor without driver hacks, or > hide it within DMA API without need to reserve extra space. Both > approaches seem much cleaner. > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:38178) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a6cmN-0006Wn-1x for qemu-devel@nongnu.org; Wed, 09 Dec 2015 06:19:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a6cmI-0007Ne-SO for qemu-devel@nongnu.org; Wed, 09 Dec 2015 06:19:27 -0500 Received: from mga14.intel.com ([192.55.52.115]:43990) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a6cmI-0007NM-I8 for qemu-devel@nongnu.org; Wed, 09 Dec 2015 06:19:22 -0500 References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> From: "Lan, Tianyu" Message-ID: <56680E33.6040204@intel.com> Date: Wed, 9 Dec 2015 19:19:15 +0800 MIME-Version: 1.0 In-Reply-To: <20151209122831-mutt-send-email-mst@redhat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael S. Tsirkin" Cc: Wei Yang , "Tantilov, Emil S" , "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" , Alexander Duyck , "Brandeburg, Jesse" , "Rustad, Mark D" , "Wyborny, Carolyn" , Eric Auger , "Skidmore, Donald C" , "zajec5@gmail.com" , Alexander Graf , intel-wired-lan , "Kirsher, Jeffrey T" , Or Gerlitz , "Williams, Mitch A" , "Jani, Nrupal" , Bjorn Helgaas , "a.motakis@virtualopensystems.com" , "b.reynal@virtualopensystems.com" , "linux-api@vger.kernel.org" , "Nelson, Shannon" , "Dong, Eddie" , Alex Williamson , "linux-kernel@vger.kernel.org" , "Ronciak, John" , Netdev , Paolo Bonzini On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: >> Hi Michael & Alexander: >> Thanks a lot for your comments and suggestions. > > It's nice that it's appreciated, but you then go on and ignore > all that I have written here: > https://www.mail-archive.com/kvm@vger.kernel.org/msg123826.html > No, I will reply it separately and according your suggestion to snip it into 3 thread. >> We still need to support Windows guest for migration and this is why our >> patches keep all changes in the driver since it's impossible to change >> Windows kernel. > > This is not a reasonable argument. It makes no sense to duplicate code > on Linux because you must duplicate code on Windows. Let's assume you > must do it in the driver on windows because windows has closed source > drivers. What does it matter? Linux can still do it as part of DMA API > and have it apply to all drivers. > Sure. Duplicated code should be encapsulated and make it able to reuse by other drivers. Just like you said the dummy write part. I meant the framework should not require to change Windows kernel code (such as PM core or PCI bus driver)and this will block implementation on the Windows. I think it's not problem to duplicate code in the Windows drivers. >> Following is my idea to do DMA tracking. >> >> Inject event to VF driver after memory iterate stage >> and before stop VCPU and then VF driver marks dirty all >> using DMA memory. The new allocated pages also need to >> be marked dirty before stopping VCPU. All dirty memory >> in this time slot will be migrated until stop-and-copy >> stage. We also need to make sure to disable VF via clearing the >> bus master enable bit for VF before migrating these memory. >> >> The dma page allocated by VF driver also needs to reserve space >> to do dummy write. > > I suggested ways to do it all in the hypervisor without driver hacks, or > hide it within DMA API without need to reserve extra space. Both > approaches seem much cleaner. > This sounds reasonable. We may discuss it detail in the separate thread. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lan, Tianyu Date: Wed, 9 Dec 2015 19:19:15 +0800 Subject: [Intel-wired-lan] [RFC PATCH V2 0/3] IXGBE/VFIO: Add live migration support for SRIOV NIC In-Reply-To: <20151209122831-mutt-send-email-mst@redhat.com> References: <565BF285.4040507@intel.com> <565DB6FF.1050602@intel.com> <20151201171140-mutt-send-email-mst@redhat.com> <20151201193026-mutt-send-email-mst@redhat.com> <20151202105955-mutt-send-email-mst@redhat.com> <5661C000.8070201@intel.com> <20151209122831-mutt-send-email-mst@redhat.com> Message-ID: <56680E33.6040204@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: intel-wired-lan@osuosl.org List-ID: On 12/9/2015 6:37 PM, Michael S. Tsirkin wrote: > On Sat, Dec 05, 2015 at 12:32:00AM +0800, Lan, Tianyu wrote: >> Hi Michael & Alexander: >> Thanks a lot for your comments and suggestions. > > It's nice that it's appreciated, but you then go on and ignore > all that I have written here: > https://www.mail-archive.com/kvm at vger.kernel.org/msg123826.html > No, I will reply it separately and according your suggestion to snip it into 3 thread. >> We still need to support Windows guest for migration and this is why our >> patches keep all changes in the driver since it's impossible to change >> Windows kernel. > > This is not a reasonable argument. It makes no sense to duplicate code > on Linux because you must duplicate code on Windows. Let's assume you > must do it in the driver on windows because windows has closed source > drivers. What does it matter? Linux can still do it as part of DMA API > and have it apply to all drivers. > Sure. Duplicated code should be encapsulated and make it able to reuse by other drivers. Just like you said the dummy write part. I meant the framework should not require to change Windows kernel code (such as PM core or PCI bus driver)and this will block implementation on the Windows. I think it's not problem to duplicate code in the Windows drivers. >> Following is my idea to do DMA tracking. >> >> Inject event to VF driver after memory iterate stage >> and before stop VCPU and then VF driver marks dirty all >> using DMA memory. The new allocated pages also need to >> be marked dirty before stopping VCPU. All dirty memory >> in this time slot will be migrated until stop-and-copy >> stage. We also need to make sure to disable VF via clearing the >> bus master enable bit for VF before migrating these memory. >> >> The dma page allocated by VF driver also needs to reserve space >> to do dummy write. > > I suggested ways to do it all in the hypervisor without driver hacks, or > hide it within DMA API without need to reserve extra space. Both > approaches seem much cleaner. > This sounds reasonable. We may discuss it detail in the separate thread.