[PATCH v1 1/1] vfio: Make migration support non experimental by default.

[PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Tarun Gupta]

VFIO migration support in QEMU is experimental as of now, which was done to
provide soak time and resolve concerns regarding bit-stream.
But, with the patches discussed in
https://www.mail-archive.com/qemu-devel@nongnu.org/msg784931.html , we have
corrected ordering of saving PCI config space and bit-stream.

So, this patch proposes to make vfio migration support in QEMU to be enabled
by default. Tested by successfully migrating mdev device.

Signed-off-by: Tarun Gupta <targupta@nvidia.com>
Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
---
 hw/vfio/pci.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index f74be78209..15e26f460b 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
     DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
                     VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
     DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
-                     vbasedev.enable_migration, false),
+                     vbasedev.enable_migration, true),
     DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
     DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
                      vbasedev.ram_block_discard_allowed, false),
-- 
2.27.0

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Daniel P. Berrangé]

On Mon, Mar 08, 2021 at 09:39:49PM +0530, Tarun Gupta wrote:
> VFIO migration support in QEMU is experimental as of now, which was done to
> provide soak time and resolve concerns regarding bit-stream.
> But, with the patches discussed in
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg784931.html , we have
> corrected ordering of saving PCI config space and bit-stream.
> 
> So, this patch proposes to make vfio migration support in QEMU to be enabled
> by default. Tested by successfully migrating mdev device.
> 
> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> ---
>  hw/vfio/pci.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index f74be78209..15e26f460b 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
>      DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
>                      VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>      DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
> -                     vbasedev.enable_migration, false),
> +                     vbasedev.enable_migration, true),

If it isn't experimental then why do we even need an experimental 'x-'
property at all ?

IOW, rather than changing this to "true", shouldn't we just be deleting
the x-enable-migration property entirely and have the code just do the
right thing.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Alex Williamson]

On Mon, 8 Mar 2021 16:36:34 +0000
Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Mon, Mar 08, 2021 at 09:39:49PM +0530, Tarun Gupta wrote:
> > VFIO migration support in QEMU is experimental as of now, which was done to
> > provide soak time and resolve concerns regarding bit-stream.
> > But, with the patches discussed in
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg784931.html , we have
> > corrected ordering of saving PCI config space and bit-stream.
> > 
> > So, this patch proposes to make vfio migration support in QEMU to be enabled
> > by default. Tested by successfully migrating mdev device.
> > 
> > Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> > ---
> >  hw/vfio/pci.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> > index f74be78209..15e26f460b 100644
> > --- a/hw/vfio/pci.c
> > +++ b/hw/vfio/pci.c
> > @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
> >      DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
> >                      VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
> >      DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
> > -                     vbasedev.enable_migration, false),
> > +                     vbasedev.enable_migration, true),  
> 
> If it isn't experimental then why do we even need an experimental 'x-'
> property at all ?
> 
> IOW, rather than changing this to "true", shouldn't we just be deleting
> the x-enable-migration property entirely and have the code just do the
> right thing.

I don't think it's necessarily invalid to have experimental disables
for supported features.  We actually have quite a few of those already.
Most of them are generally aimed at debugging, for example disabling
direct mappings or acceleration paths that might mask an access when
trying to trace the behavior of a device.  We might want to consider
changing the polarity of the option to avoid user confusion, ie.
x-disable-migration.  I'm not fully convinced that there might not be
valid non-experimental use cases for such an option, but it seems like
that should be supported at a base device class level rather than per
driver, so best to be left experimental here.  Thanks,

Alex

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Cornelia Huck]

On Mon, 8 Mar 2021 21:39:49 +0530
Tarun Gupta <targupta@nvidia.com> wrote:

> VFIO migration support in QEMU is experimental as of now, which was done to
> provide soak time and resolve concerns regarding bit-stream.
> But, with the patches discussed in
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg784931.html , we have
> corrected ordering of saving PCI config space and bit-stream.
> 
> So, this patch proposes to make vfio migration support in QEMU to be enabled
> by default. Tested by successfully migrating mdev device.
> 
> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> ---
>  hw/vfio/pci.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index f74be78209..15e26f460b 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
>      DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
>                      VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>      DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
> -                     vbasedev.enable_migration, false),
> +                     vbasedev.enable_migration, true),
>      DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
>      DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
>                       vbasedev.ram_block_discard_allowed, false),

I think we were still expecting some documentation for this feature,
and I don't recall any update there.

Also, I don't think we have any vendor driver implementation integrated
yet?

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Alex Williamson]

[Cc +Intel]

On Mon, 8 Mar 2021 21:39:49 +0530
Tarun Gupta <targupta@nvidia.com> wrote:

> VFIO migration support in QEMU is experimental as of now, which was done to
> provide soak time and resolve concerns regarding bit-stream.
> But, with the patches discussed in
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg784931.html , we have
> corrected ordering of saving PCI config space and bit-stream.
> 
> So, this patch proposes to make vfio migration support in QEMU to be enabled
> by default. Tested by successfully migrating mdev device.
> 
> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> ---
>  hw/vfio/pci.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index f74be78209..15e26f460b 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
>      DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
>                      VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>      DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
> -                     vbasedev.enable_migration, false),
> +                     vbasedev.enable_migration, true),
>      DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
>      DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
>                       vbasedev.ram_block_discard_allowed, false),

Looking back at the commit where this was added:

commit cf254988a50d4164c86a356c80b8d3ae0ccaa005
Author: Alex Williamson <alex.williamson@redhat.com>
Date:   Mon Nov 9 11:56:02 2020 -0700

    vfio: Make migration support experimental
    
    Support for migration of vfio devices is still in flux.  Developers
    are attempting to add support for new devices and new architectures,
    but none are yet readily available for validation.  We have concerns
    whether we're transferring device resources at the right point in the
    migration, whether we're guaranteeing that updates during pre-copy are
    migrated, and whether we can provide bit-stream compatibility should
    any of this change.  Even the question of whether devices should
    participate in dirty page tracking during pre-copy seems contentious.
    In short, migration support has not had enough soak time and it feels
    premature to mark it as supported.
    
    Create an experimental option such that we can continue to develop.
    
    [Retaining previous acks/reviews for a previously identical code
     change with different specifics in the commit log.]
    
    Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
    Acked-by: Cornelia Huck <cohuck@redhat.com>
    Signed-off-by: Alex Williamson <alex.williamson@redhat.com>


What has tangibly changed since then?  I think we have patches on-list
to address the known issue of PCI config space (MSI) ordering, which
related to enabling migration on ARM platforms.  Do we have
significantly more confidence in our ability to make compatible
enhancement to the migration bitstream?  This was a particularly
troublesome point for me if we have any hope of calling this
supportable.  As far as I know, there are still no open source vendor
drivers supporting migration for community testing.  We're also still
missing the documentation that was promised previously, as Connie noted.

Huawei and Intel, what's your confidence level and what can you share
regarding support for this implementation?  Thanks,

Alex

RE: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Tian, Kevin]

> From: Alex Williamson <alex.williamson@redhat.com>
> Sent: Tuesday, March 9, 2021 6:51 AM
> 
> [Cc +Intel]
> 
> On Mon, 8 Mar 2021 21:39:49 +0530
> Tarun Gupta <targupta@nvidia.com> wrote:
> 
> > VFIO migration support in QEMU is experimental as of now, which was
> done to
> > provide soak time and resolve concerns regarding bit-stream.
> > But, with the patches discussed in
> > https://www.mail-archive.com/qemu-
> devel@nongnu.org/msg784931.html , we have
> > corrected ordering of saving PCI config space and bit-stream.
> >
> > So, this patch proposes to make vfio migration support in QEMU to be
> enabled
> > by default. Tested by successfully migrating mdev device.
> >
> > Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> > ---
> >  hw/vfio/pci.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> > index f74be78209..15e26f460b 100644
> > --- a/hw/vfio/pci.c
> > +++ b/hw/vfio/pci.c
> > @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
> >      DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
> >                      VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
> >      DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
> > -                     vbasedev.enable_migration, false),
> > +                     vbasedev.enable_migration, true),
> >      DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap,
> false),
> >      DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
> >                       vbasedev.ram_block_discard_allowed, false),
> 
> Looking back at the commit where this was added:
> 
> commit cf254988a50d4164c86a356c80b8d3ae0ccaa005
> Author: Alex Williamson <alex.williamson@redhat.com>
> Date:   Mon Nov 9 11:56:02 2020 -0700
> 
>     vfio: Make migration support experimental
> 
>     Support for migration of vfio devices is still in flux.  Developers
>     are attempting to add support for new devices and new architectures,
>     but none are yet readily available for validation.  We have concerns
>     whether we're transferring device resources at the right point in the
>     migration, whether we're guaranteeing that updates during pre-copy are
>     migrated, and whether we can provide bit-stream compatibility should
>     any of this change.  Even the question of whether devices should
>     participate in dirty page tracking during pre-copy seems contentious.
>     In short, migration support has not had enough soak time and it feels
>     premature to mark it as supported.
> 
>     Create an experimental option such that we can continue to develop.
> 
>     [Retaining previous acks/reviews for a previously identical code
>      change with different specifics in the commit log.]
> 
>     Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>     Acked-by: Cornelia Huck <cohuck@redhat.com>
>     Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> 
> 
> What has tangibly changed since then?  I think we have patches on-list
> to address the known issue of PCI config space (MSI) ordering, which
> related to enabling migration on ARM platforms.  Do we have
> significantly more confidence in our ability to make compatible
> enhancement to the migration bitstream?  This was a particularly
> troublesome point for me if we have any hope of calling this
> supportable.  As far as I know, there are still no open source vendor
> drivers supporting migration for community testing.  We're also still
> missing the documentation that was promised previously, as Connie noted.
> 
> Huawei and Intel, what's your confidence level and what can you share
> regarding support for this implementation?  Thanks,
> 

Internally our GVT-g live migration support is still experimental, and due
to resource/priority adjustment the upstreaming plan for this feature is
currently on hold. Timing-wise I'd expect IDXD will be the 1st Intel driver
which formally supports live migration (after its core functionalities - mdev/
vSVA are upstreamed). Alternatively once the vfio-pci-core library work 
is completed I believe many interests will be also arose regarding to VF
live migration (e.g. NIC). But none of the options may come in short term... 

Thanks
Kevin

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Shenming Lu]

On 2021/3/9 6:51, Alex Williamson wrote:
> [Cc +Intel]
> 
> On Mon, 8 Mar 2021 21:39:49 +0530
> Tarun Gupta <targupta@nvidia.com> wrote:
> 
>> VFIO migration support in QEMU is experimental as of now, which was done to
>> provide soak time and resolve concerns regarding bit-stream.
>> But, with the patches discussed in
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg784931.html , we have
>> corrected ordering of saving PCI config space and bit-stream.
>>
>> So, this patch proposes to make vfio migration support in QEMU to be enabled
>> by default. Tested by successfully migrating mdev device.
>>
>> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
>> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
>> ---
>>  hw/vfio/pci.c | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
>> index f74be78209..15e26f460b 100644
>> --- a/hw/vfio/pci.c
>> +++ b/hw/vfio/pci.c
>> @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
>>      DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
>>                      VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>>      DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
>> -                     vbasedev.enable_migration, false),
>> +                     vbasedev.enable_migration, true),
>>      DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
>>      DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
>>                       vbasedev.ram_block_discard_allowed, false),
> 
> Looking back at the commit where this was added:
> 
> commit cf254988a50d4164c86a356c80b8d3ae0ccaa005
> Author: Alex Williamson <alex.williamson@redhat.com>
> Date:   Mon Nov 9 11:56:02 2020 -0700
> 
>     vfio: Make migration support experimental
>     
>     Support for migration of vfio devices is still in flux.  Developers
>     are attempting to add support for new devices and new architectures,
>     but none are yet readily available for validation.  We have concerns
>     whether we're transferring device resources at the right point in the
>     migration, whether we're guaranteeing that updates during pre-copy are
>     migrated, and whether we can provide bit-stream compatibility should
>     any of this change.  Even the question of whether devices should
>     participate in dirty page tracking during pre-copy seems contentious.
>     In short, migration support has not had enough soak time and it feels
>     premature to mark it as supported.
>     
>     Create an experimental option such that we can continue to develop.
>     
>     [Retaining previous acks/reviews for a previously identical code
>      change with different specifics in the commit log.]
>     
>     Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>     Acked-by: Cornelia Huck <cohuck@redhat.com>
>     Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
> 
> 
> What has tangibly changed since then?  I think we have patches on-list
> to address the known issue of PCI config space (MSI) ordering, which
> related to enabling migration on ARM platforms.  Do we have
> significantly more confidence in our ability to make compatible
> enhancement to the migration bitstream?  This was a particularly
> troublesome point for me if we have any hope of calling this
> supportable.  As far as I know, there are still no open source vendor
> drivers supporting migration for community testing.  We're also still
> missing the documentation that was promised previously, as Connie noted.
> 
> Huawei and Intel, what's your confidence level and what can you share
> regarding support for this implementation?  Thanks,

We have sent a number of patches regarding VFIO migration from our own test
(the support for this in our accelerator driver is still in experiment),
some of them are still on-list []...

[] https://lore.kernel.org/qemu-devel/20210310094106.2191-2-jiangkunkun@huawei.com/
   https://lore.kernel.org/linux-arm-kernel/20210126124444.27136-1-zhukeqian1@huawei.com/

Thanks,
Shenming

> 
> Alex
> 
> .
> 

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Claudio Fontana]

On 3/8/21 5:09 PM, Tarun Gupta wrote:
> VFIO migration support in QEMU is experimental as of now, which was done to
> provide soak time and resolve concerns regarding bit-stream.
> But, with the patches discussed in
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg784931.html , we have
> corrected ordering of saving PCI config space and bit-stream.
> 
> So, this patch proposes to make vfio migration support in QEMU to be enabled
> by default. Tested by successfully migrating mdev device.
> 
> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> ---
>  hw/vfio/pci.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index f74be78209..15e26f460b 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
>      DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
>                      VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>      DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
> -                     vbasedev.enable_migration, false),
> +                     vbasedev.enable_migration, true),
>      DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
>      DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
>                       vbasedev.ram_block_discard_allowed, false),
> 

Hello,

has plain snapshot been tested?
If I issue the HMP command "savevm", and then "loadvm", will things work fine?

Thanks,

CLaudio

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Kirti Wankhede]

On 7/10/2021 1:14 PM, Claudio Fontana wrote:
> On 3/8/21 5:09 PM, Tarun Gupta wrote:
>> VFIO migration support in QEMU is experimental as of now, which was done to
>> provide soak time and resolve concerns regarding bit-stream.
>> But, with the patches discussed in
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fqemu-devel%40nongnu.org%2Fmsg784931.html&data=04%7C01%7Ckwankhede%40nvidia.com%7C98194e8a856f4e6b611c08d943769ab5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637614998961553398%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=A2EY9LEqGE0BSrT25h2WtWonb5oi0O%2B6%2BQmvhVf8Wd4%3D&reserved=0 , we have
>> corrected ordering of saving PCI config space and bit-stream.
>>
>> So, this patch proposes to make vfio migration support in QEMU to be enabled
>> by default. Tested by successfully migrating mdev device.
>>
>> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
>> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
>> ---
>>   hw/vfio/pci.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
>> index f74be78209..15e26f460b 100644
>> --- a/hw/vfio/pci.c
>> +++ b/hw/vfio/pci.c
>> @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
>>       DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
>>                       VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>>       DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice,
>> -                     vbasedev.enable_migration, false),
>> +                     vbasedev.enable_migration, true),
>>       DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false),
>>       DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
>>                        vbasedev.ram_block_discard_allowed, false),
>>
> 
> Hello,
> 
> has plain snapshot been tested?

Yes.

> If I issue the HMP command "savevm", and then "loadvm", will things work fine?

Yes

Thanks,
Kirti

Re: [PATCH v1 1/1] vfio: Make migration support non experimental by default.

[Liang Yan]

On 7/14/21 6:19 AM, Kirti Wankhede wrote:
>
>
> On 7/10/2021 1:14 PM, Claudio Fontana wrote:
>> On 3/8/21 5:09 PM, Tarun Gupta wrote:
>>> VFIO migration support in QEMU is experimental as of now, which was
>>> done to
>>> provide soak time and resolve concerns regarding bit-stream.
>>> But, with the patches discussed in
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.mail-archive.com%2Fqemu-devel%40nongnu.org%2Fmsg784931.html&data=04%7C01%7Ckwankhede%40nvidia.com%7C98194e8a856f4e6b611c08d943769ab5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637614998961553398%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=A2EY9LEqGE0BSrT25h2WtWonb5oi0O%2B6%2BQmvhVf8Wd4%3D&reserved=0
>>> , we have
>>> corrected ordering of saving PCI config space and bit-stream.
>>>
>>> So, this patch proposes to make vfio migration support in QEMU to be
>>> enabled
>>> by default. Tested by successfully migrating mdev device.
>>>
>>> Signed-off-by: Tarun Gupta <targupta@nvidia.com>
>>> Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
>>> ---
>>>   hw/vfio/pci.c | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
>>> index f74be78209..15e26f460b 100644
>>> --- a/hw/vfio/pci.c
>>> +++ b/hw/vfio/pci.c
>>> @@ -3199,7 +3199,7 @@ static Property vfio_pci_dev_properties[] = {
>>>       DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features,
>>>                       VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false),
>>>       DEFINE_PROP_BOOL("", VFIOPCIDevice,
>>> -                     vbasedev.enable_migration, false),
>>> +                     vbasedev.enable_migration, true),
>>>       DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap,
>>> false),
>>>       DEFINE_PROP_BOOL("x-balloon-allowed", VFIOPCIDevice,
>>>                        vbasedev.ram_block_discard_allowed, false),
>>>
>>
>> Hello,
>>
>> has plain snapshot been tested?
>
> Yes.
>
>> If I issue the HMP command "savevm", and then "loadvm", will things
>> work fine?
>
> Yes
>

Hello Kirti,

I enabled x-enable-migration and did some hack on failover_pair_id,
finally made  "virsh save/restore" and "savevm/loadvm"work through.
However, it seems vGPU did not get involved in the real migration
process, the qemu trace file confirmed it, there is no vfio section for
savevm_section_start at all.

I am using kernel 5.8 and latest qemu, vGPU 12.2 with one V100. I am
wondering if there is a version compatible requirement or need extra
setup. Could you share your test setup here? Thanks in advance.

Regards,

Liang



> Thanks,
> Kirti
>