[Qemu-devel] [QEMU PATCH v5 0/6] migration: ensure hotplug and migration work together

[Qemu-devel] [QEMU PATCH v5 0/6] migration: ensure hotplug and migration work together

[Jianjun Duan]

Hi all,
   The previous patches seem to get buried deep somewhere. I am sending the lated rebased version. Comments are welcome.

v5: - Rebased to David's ppc-for-2.8. Previous versions are:

v4: - Introduce a way to set customized instance_id in SaveStateEntry. Use it
      to set instance_id for DRC using its unique index to address David 
      Gibson's concern.
    - Rename VMS_CSTM to VMS_LINKED based on Paolo Bonzini's suggestions.
    - Clean up qjson stuff in put_qtailq. 
    - Add trace for put_qtailq and get_qtailq based on David Gilbert's 
      suggestion.

    - Based on David's ppc-for-2.7. 

v3: - Simplify overall design followng discussion with Paolo. No longer need
      metadata to migrate QTAILQ.
    - Extend VMStateInfo instead of adding similar fields to VMStateField.
    - Clean up macros in qemu/queue.h.
(link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg05695.html)

v2: - Introduce a general approach to migrate QTAILQ in qemu/queue.h.
    - Migrate signalled field in the DRC state.
    - Put the newly added migrating fields in subsections so that backward 
      migration is not broken.  
    - Set detach_cb field right after migration so that a migrated hot-unplug
      event could finish its course.
(link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg04188.html)

v1: - Inital version.
(link: https://lists.nongnu.org/archive/html/qemu-devel/2016-04/msg02601.html)

To make guest device (PCI, CPU and memory) hotplug work together 
with guest migration, spapr drc state needs be transmitted in
migration. This patch defines the VMStateDescription struct for
spapr drc state to enable it.

To fix the potential racing between hotplug events on guest and 
guest migration, ccs_list and pending_events of spapr state need be 
transmitted in migration. This patch also takes care of it.

Jianjun Duan (6):
  migration: alternative way to set instance_id in SaveStateEntry
  migration: spapr_drc: defined VMStateDescription struct
  migration: extend VMStateInfo
  migration: migrate QTAILQ
  migration: spapr: migrate ccs_list in spapr state
  migration: spapr: migrate pending_events of spapr state

 hw/net/vmxnet3.c            |  18 +++--
 hw/nvram/eeprom93xx.c       |   6 +-
 hw/nvram/fw_cfg.c           |   6 +-
 hw/pci/msix.c               |   6 +-
 hw/pci/pci.c                |  12 ++--
 hw/pci/shpc.c               |   5 +-
 hw/ppc/spapr.c              |  67 ++++++++++++++++++
 hw/ppc/spapr_drc.c          |  69 +++++++++++++++++++
 hw/ppc/spapr_events.c       |  22 +++---
 hw/ppc/spapr_pci.c          |  22 ++++++
 hw/scsi/scsi-bus.c          |   6 +-
 hw/timer/twl92230.c         |   6 +-
 hw/usb/redirect.c           |  18 +++--
 hw/virtio/virtio-pci.c      |   6 +-
 hw/virtio/virtio.c          |   6 +-
 include/hw/ppc/spapr.h      |   3 +-
 include/hw/ppc/spapr_drc.h  |   9 +++
 include/hw/qdev-core.h      |   6 ++
 include/migration/vmstate.h |  36 ++++++++--
 include/qemu/queue.h        |  32 +++++++++
 migration/savevm.c          |  25 +++++--
 migration/trace-events      |   4 ++
 migration/vmstate.c         | 161 ++++++++++++++++++++++++++++++++++----------
 target-alpha/machine.c      |   5 +-
 target-arm/machine.c        |  12 ++--
 target-i386/machine.c       |  21 ++++--
 target-mips/machine.c       |  10 +--
 target-ppc/machine.c        |  10 +--
 target-sparc/machine.c      |   5 +-
 29 files changed, 505 insertions(+), 109 deletions(-)

-- 
1.9.1

[Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[Jianjun Duan]

In QOM(QEMU Object Model) migrated objects are identified with instance_id
which is calculated automatically using their path in the QOM composition
tree. For some objects, this path could change from source to target in
migration. To migrate such objects, we need to make sure the instance_id does
not change from source to target. We add a hook in DeviceClass to do customized
instance_id calculation in such cases.

As a result, in these cases compat will not be set in the concerned
SaveStateEntry. This will prevent the inconsistent idstr to be sent over in
migration. We could have set alias_id in a similar way. But that will be
overloading the purpose of alias_id.

The first application will be setting instance_id for DRC using its unique
index. Doing this makes the instance_id of DRC to be consistent across migration
and supports flexible management of DRC objects in migration.

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
---
 include/hw/qdev-core.h |  6 ++++++
 migration/savevm.c     | 20 ++++++++++++++++++--
 2 files changed, 24 insertions(+), 2 deletions(-)

diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
index 2c97347..a012e8e 100644
--- a/include/hw/qdev-core.h
+++ b/include/hw/qdev-core.h
@@ -139,6 +139,12 @@ typedef struct DeviceClass {
     qdev_initfn init; /* TODO remove, once users are converted to realize */
     qdev_event exit; /* TODO remove, once users are converted to unrealize */
     const char *bus_type;
+
+    /* When this field is set, qemu will use it to get an unique instance_id
+     * instead of calculating an auto idstr and instanc_id for the relevant
+     * SaveStateEntry
+     */
+    int (*dev_get_instance_id)(DeviceState *dev);
 } DeviceClass;
 
 typedef struct NamedGPIOList NamedGPIOList;
diff --git a/migration/savevm.c b/migration/savevm.c
index 33a2911..ef5c3d1 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -495,6 +495,11 @@ int register_savevm_live(DeviceState *dev,
                          void *opaque)
 {
     SaveStateEntry *se;
+    /* when it is a device and it provides a way to get instance_id,
+     * we will use it and skip setting idstr and compat.
+     */
+    bool flag = (dev != NULL) &&
+                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);
 
     se = g_new0(SaveStateEntry, 1);
     se->version_id = version_id;
@@ -507,7 +512,7 @@ int register_savevm_live(DeviceState *dev,
         se->is_ram = 1;
     }
 
-    if (dev) {
+    if (dev && !flag) {
         char *id = qdev_get_dev_path(dev);
         if (id) {
             pstrcpy(se->idstr, sizeof(se->idstr), id);
@@ -523,6 +528,9 @@ int register_savevm_live(DeviceState *dev,
     }
     pstrcat(se->idstr, sizeof(se->idstr), idstr);
 
+    if (flag) {
+        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
+    }
     if (instance_id == -1) {
         se->instance_id = calculate_new_instance_id(se->idstr);
     } else {
@@ -580,6 +588,11 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
                                    int required_for_version)
 {
     SaveStateEntry *se;
+    /* when it is a device and it provides a way to get instance_id,
+     * we will use it and skip setting idstr and compat.
+     */
+    bool flag = (dev != NULL) &&
+                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);
 
     /* If this triggers, alias support can be dropped for the vmsd. */
     assert(alias_id == -1 || required_for_version >= vmsd->minimum_version_id);
@@ -591,7 +604,7 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
     se->vmsd = vmsd;
     se->alias_id = alias_id;
 
-    if (dev) {
+    if (dev && !flag) {
         char *id = qdev_get_dev_path(dev);
         if (id) {
             pstrcpy(se->idstr, sizeof(se->idstr), id);
@@ -607,6 +620,9 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
     }
     pstrcat(se->idstr, sizeof(se->idstr), vmsd->name);
 
+    if (flag) {
+        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
+    }
     if (instance_id == -1) {
         se->instance_id = calculate_new_instance_id(se->idstr);
     } else {
-- 
1.9.1

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[Dr. David Alan Gilbert]

* Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> In QOM(QEMU Object Model) migrated objects are identified with instance_id
> which is calculated automatically using their path in the QOM composition
> tree. For some objects, this path could change from source to target in
> migration. To migrate such objects, we need to make sure the instance_id does
> not change from source to target. We add a hook in DeviceClass to do customized
> instance_id calculation in such cases.

Can you explain a bit about why the path changes from source to destination;
the path here should be a feature of the guest state not the host, and so I
don't understand why it changes.

> As a result, in these cases compat will not be set in the concerned
> SaveStateEntry. This will prevent the inconsistent idstr to be sent over in
> migration. We could have set alias_id in a similar way. But that will be
> overloading the purpose of alias_id.
> 
> The first application will be setting instance_id for DRC using its unique
> index. Doing this makes the instance_id of DRC to be consistent across migration
> and supports flexible management of DRC objects in migration.

Is there a reason to use a custom instance_id rather than a custom idstr?

> 
> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> ---
>  include/hw/qdev-core.h |  6 ++++++
>  migration/savevm.c     | 20 ++++++++++++++++++--
>  2 files changed, 24 insertions(+), 2 deletions(-)
> 
> diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
> index 2c97347..a012e8e 100644
> --- a/include/hw/qdev-core.h
> +++ b/include/hw/qdev-core.h
> @@ -139,6 +139,12 @@ typedef struct DeviceClass {
>      qdev_initfn init; /* TODO remove, once users are converted to realize */
>      qdev_event exit; /* TODO remove, once users are converted to unrealize */
>      const char *bus_type;
> +
> +    /* When this field is set, qemu will use it to get an unique instance_id
> +     * instead of calculating an auto idstr and instanc_id for the relevant
> +     * SaveStateEntry
> +     */
> +    int (*dev_get_instance_id)(DeviceState *dev);
>  } DeviceClass;
>  
>  typedef struct NamedGPIOList NamedGPIOList;
> diff --git a/migration/savevm.c b/migration/savevm.c
> index 33a2911..ef5c3d1 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -495,6 +495,11 @@ int register_savevm_live(DeviceState *dev,
>                           void *opaque)
>  {
>      SaveStateEntry *se;
> +    /* when it is a device and it provides a way to get instance_id,
> +     * we will use it and skip setting idstr and compat.
> +     */
> +    bool flag = (dev != NULL) &&
> +                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);

We must be able to get a more descriptive name than 'flag'; how about
 'has_custom_id'

>      se = g_new0(SaveStateEntry, 1);
>      se->version_id = version_id;
> @@ -507,7 +512,7 @@ int register_savevm_live(DeviceState *dev,
>          se->is_ram = 1;
>      }
>  
> -    if (dev) {
> +    if (dev && !flag) {
>          char *id = qdev_get_dev_path(dev);
>          if (id) {
>              pstrcpy(se->idstr, sizeof(se->idstr), id);
> @@ -523,6 +528,9 @@ int register_savevm_live(DeviceState *dev,
>      }
>      pstrcat(se->idstr, sizeof(se->idstr), idstr);
>  
> +    if (flag) {
> +        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
> +    }
>      if (instance_id == -1) {
>          se->instance_id = calculate_new_instance_id(se->idstr);
>      } else {
> @@ -580,6 +588,11 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>                                     int required_for_version)
>  {
>      SaveStateEntry *se;
> +    /* when it is a device and it provides a way to get instance_id,
> +     * we will use it and skip setting idstr and compat.
> +     */
> +    bool flag = (dev != NULL) &&
> +                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);
>  
>      /* If this triggers, alias support can be dropped for the vmsd. */
>      assert(alias_id == -1 || required_for_version >= vmsd->minimum_version_id);
> @@ -591,7 +604,7 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>      se->vmsd = vmsd;
>      se->alias_id = alias_id;
>  
> -    if (dev) {
> +    if (dev && !flag) {
>          char *id = qdev_get_dev_path(dev);
>          if (id) {
>              pstrcpy(se->idstr, sizeof(se->idstr), id);
> @@ -607,6 +620,9 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>      }
>      pstrcat(se->idstr, sizeof(se->idstr), vmsd->name);
>  
> +    if (flag) {
> +        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
> +    }
>      if (instance_id == -1) {
>          se->instance_id = calculate_new_instance_id(se->idstr);
>      } else {
> -- 
> 1.9.1

Dave
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[Jianjun Duan]

Please see comments below:

On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>> In QOM(QEMU Object Model) migrated objects are identified with instance_id
>> which is calculated automatically using their path in the QOM composition
>> tree. For some objects, this path could change from source to target in
>> migration. To migrate such objects, we need to make sure the instance_id does
>> not change from source to target. We add a hook in DeviceClass to do customized
>> instance_id calculation in such cases.
> 
> Can you explain a bit about why the path changes from source to destination;
> the path here should be a feature of the guest state not the host, and so I
> don't understand why it changes.
Please see the discussion with David in the previous versions:
http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html


>> As a result, in these cases compat will not be set in the concerned
>> SaveStateEntry. This will prevent the inconsistent idstr to be sent over in
>> migration. We could have set alias_id in a similar way. But that will be
>> overloading the purpose of alias_id.
>>
>> The first application will be setting instance_id for DRC using its unique
>> index. Doing this makes the instance_id of DRC to be consistent across migration
>> and supports flexible management of DRC objects in migration.
> 
> Is there a reason to use a custom instance_id rather than a custom idstr

It can be done either way. But it is easier to deal with a integer than
a string.


>>
>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>> ---
>>  include/hw/qdev-core.h |  6 ++++++
>>  migration/savevm.c     | 20 ++++++++++++++++++--
>>  2 files changed, 24 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
>> index 2c97347..a012e8e 100644
>> --- a/include/hw/qdev-core.h
>> +++ b/include/hw/qdev-core.h
>> @@ -139,6 +139,12 @@ typedef struct DeviceClass {
>>      qdev_initfn init; /* TODO remove, once users are converted to realize */
>>      qdev_event exit; /* TODO remove, once users are converted to unrealize */
>>      const char *bus_type;
>> +
>> +    /* When this field is set, qemu will use it to get an unique instance_id
>> +     * instead of calculating an auto idstr and instanc_id for the relevant
>> +     * SaveStateEntry
>> +     */
>> +    int (*dev_get_instance_id)(DeviceState *dev);
>>  } DeviceClass;
>>  
>>  typedef struct NamedGPIOList NamedGPIOList;
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index 33a2911..ef5c3d1 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -495,6 +495,11 @@ int register_savevm_live(DeviceState *dev,
>>                           void *opaque)
>>  {
>>      SaveStateEntry *se;
>> +    /* when it is a device and it provides a way to get instance_id,
>> +     * we will use it and skip setting idstr and compat.
>> +     */
>> +    bool flag = (dev != NULL) &&
>> +                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);
> 
> We must be able to get a more descriptive name than 'flag'; how about
>  'has_custom_id'
> 
>>      se = g_new0(SaveStateEntry, 1);
>>      se->version_id = version_id;
>> @@ -507,7 +512,7 @@ int register_savevm_live(DeviceState *dev,
>>          se->is_ram = 1;
>>      }
>>  
>> -    if (dev) {
>> +    if (dev && !flag) {
>>          char *id = qdev_get_dev_path(dev);
>>          if (id) {
>>              pstrcpy(se->idstr, sizeof(se->idstr), id);
>> @@ -523,6 +528,9 @@ int register_savevm_live(DeviceState *dev,
>>      }
>>      pstrcat(se->idstr, sizeof(se->idstr), idstr);
>>  
>> +    if (flag) {
>> +        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
>> +    }
>>      if (instance_id == -1) {
>>          se->instance_id = calculate_new_instance_id(se->idstr);
>>      } else {
>> @@ -580,6 +588,11 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>>                                     int required_for_version)
>>  {
>>      SaveStateEntry *se;
>> +    /* when it is a device and it provides a way to get instance_id,
>> +     * we will use it and skip setting idstr and compat.
>> +     */
>> +    bool flag = (dev != NULL) &&
>> +                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);
>>  
>>      /* If this triggers, alias support can be dropped for the vmsd. */
>>      assert(alias_id == -1 || required_for_version >= vmsd->minimum_version_id);
>> @@ -591,7 +604,7 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>>      se->vmsd = vmsd;
>>      se->alias_id = alias_id;
>>  
>> -    if (dev) {
>> +    if (dev && !flag) {
>>          char *id = qdev_get_dev_path(dev);
>>          if (id) {
>>              pstrcpy(se->idstr, sizeof(se->idstr), id);
>> @@ -607,6 +620,9 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>>      }
>>      pstrcat(se->idstr, sizeof(se->idstr), vmsd->name);
>>  
>> +    if (flag) {
>> +        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
>> +    }
>>      if (instance_id == -1) {
>>          se->instance_id = calculate_new_instance_id(se->idstr);
>>      } else {
>> -- 
>> 1.9.1
> 
> Dave
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 


Thanks,
Jianjun

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2911 bytes --]

On Wed, Oct 05, 2016 at 09:44:57AM -0700, Jianjun Duan wrote:
> Please see comments below:
> 
> On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> >> In QOM(QEMU Object Model) migrated objects are identified with instance_id
> >> which is calculated automatically using their path in the QOM composition
> >> tree. For some objects, this path could change from source to target in
> >> migration. To migrate such objects, we need to make sure the instance_id does
> >> not change from source to target. We add a hook in DeviceClass to do customized
> >> instance_id calculation in such cases.
> > 
> > Can you explain a bit about why the path changes from source to destination;
> > the path here should be a feature of the guest state not the host, and so I
> > don't understand why it changes.
> Please see the discussion with David in the previous versions:
> http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html

Um.. your description above really isn't an accurate summary of that
discussion.

The point is not that the qom path will vary from source to
destination for some arbitrary reason, but rather that we anticipate
future changes in the QOM structure.  Specifically we're considering
eliminating the DRC objects, and folding their (limited) state into an
array in the parent object (either the machine or a PCI host bridge).

That would change the qom paths, and hence the auto-generated instance
ids, which would break migration between qemu versions before and
after the restructure.

I'm not sure that changing the instance ids is enough though, anyway,
since we're talking about eliminating the object entirely, the
class/type information in the migration stream also wouldn't match.

Dave, if you have ideas on how to deal with that, I'd love to hear
them

> 
> >> As a result, in these cases compat will not be set in the concerned
> >> SaveStateEntry. This will prevent the inconsistent idstr to be sent over in
> >> migration. We could have set alias_id in a similar way. But that will be
> >> overloading the purpose of alias_id.
> >>
> >> The first application will be setting instance_id for DRC using its unique
> >> index. Doing this makes the instance_id of DRC to be consistent across migration
> >> and supports flexible management of DRC objects in migration.
> > 
> > Is there a reason to use a custom instance_id rather than a custom idstr
> 
> It can be done either way. But it is easier to deal with a integer than
> a string.

A bit, but I don't think that's a good enough reason to introduce a
second mechanism for overriding instance id allocations.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[Dr. David Alan Gilbert]

* David Gibson (david@gibson.dropbear.id.au) wrote:
> On Wed, Oct 05, 2016 at 09:44:57AM -0700, Jianjun Duan wrote:
> > Please see comments below:
> > 
> > On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > >> In QOM(QEMU Object Model) migrated objects are identified with instance_id
> > >> which is calculated automatically using their path in the QOM composition
> > >> tree. For some objects, this path could change from source to target in
> > >> migration. To migrate such objects, we need to make sure the instance_id does
> > >> not change from source to target. We add a hook in DeviceClass to do customized
> > >> instance_id calculation in such cases.
> > > 
> > > Can you explain a bit about why the path changes from source to destination;
> > > the path here should be a feature of the guest state not the host, and so I
> > > don't understand why it changes.
> > Please see the discussion with David in the previous versions:
> > http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html
> 
> Um.. your description above really isn't an accurate summary of that
> discussion.
> 
> The point is not that the qom path will vary from source to
> destination for some arbitrary reason, but rather that we anticipate
> future changes in the QOM structure.  Specifically we're considering
> eliminating the DRC objects, and folding their (limited) state into an
> array in the parent object (either the machine or a PCI host bridge).
> 
> That would change the qom paths, and hence the auto-generated instance
> ids, which would break migration between qemu versions before and
> after the restructure.
> 
> I'm not sure that changing the instance ids is enough though, anyway,
> since we're talking about eliminating the object entirely, the
> class/type information in the migration stream also wouldn't match.
> 
> Dave, if you have ideas on how to deal with that, I'd love to hear
> them

Eliminating the object entirely would be tricky to deal with;
allowing the structure to change would work if instead of a custom instance_id
you had a custom idstr.

However, the question then becomes why is the structure changing so much;
ideally things in the migration stream should represent some tangible object
that corresponds to something real, but (again ideally) the contents
of the stream should reflect the state of those objects not the current
implementation - so if you want to change the implementation the stream
doesn't change.  Is it your implementation, or the understanding of what
the objects actually represent that's in flux?

Dave

> 
> > 
> > >> As a result, in these cases compat will not be set in the concerned
> > >> SaveStateEntry. This will prevent the inconsistent idstr to be sent over in
> > >> migration. We could have set alias_id in a similar way. But that will be
> > >> overloading the purpose of alias_id.
> > >>
> > >> The first application will be setting instance_id for DRC using its unique
> > >> index. Doing this makes the instance_id of DRC to be consistent across migration
> > >> and supports flexible management of DRC objects in migration.
> > > 
> > > Is there a reason to use a custom instance_id rather than a custom idstr
> > 
> > It can be done either way. But it is easier to deal with a integer than
> > a string.
> 
> A bit, but I don't think that's a good enough reason to introduce a
> second mechanism for overriding instance id allocations.
> 
> -- 
> David Gibson			| I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
> 				| _way_ _around_!
> http://www.ozlabs.org/~dgibson


--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5715 bytes --]

On Fri, Oct 07, 2016 at 09:07:49AM +0100, Dr. David Alan Gilbert wrote:
> * David Gibson (david@gibson.dropbear.id.au) wrote:
> > On Wed, Oct 05, 2016 at 09:44:57AM -0700, Jianjun Duan wrote:
> > > Please see comments below:
> > > 
> > > On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> > > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > > >> In QOM(QEMU Object Model) migrated objects are identified with instance_id
> > > >> which is calculated automatically using their path in the QOM composition
> > > >> tree. For some objects, this path could change from source to target in
> > > >> migration. To migrate such objects, we need to make sure the instance_id does
> > > >> not change from source to target. We add a hook in DeviceClass to do customized
> > > >> instance_id calculation in such cases.
> > > > 
> > > > Can you explain a bit about why the path changes from source to destination;
> > > > the path here should be a feature of the guest state not the host, and so I
> > > > don't understand why it changes.
> > > Please see the discussion with David in the previous versions:
> > > http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html
> > 
> > Um.. your description above really isn't an accurate summary of that
> > discussion.
> > 
> > The point is not that the qom path will vary from source to
> > destination for some arbitrary reason, but rather that we anticipate
> > future changes in the QOM structure.  Specifically we're considering
> > eliminating the DRC objects, and folding their (limited) state into an
> > array in the parent object (either the machine or a PCI host bridge).
> > 
> > That would change the qom paths, and hence the auto-generated instance
> > ids, which would break migration between qemu versions before and
> > after the restructure.
> > 
> > I'm not sure that changing the instance ids is enough though, anyway,
> > since we're talking about eliminating the object entirely, the
> > class/type information in the migration stream also wouldn't match.
> > 
> > Dave, if you have ideas on how to deal with that, I'd love to hear
> > them
> 
> Eliminating the object entirely would be tricky to deal with;
> allowing the structure to change would work if instead of a custom instance_id
> you had a custom idstr.

Sorry, two misunderstandings here.

  * When I said "structure" I meant in the high-level sense of what
    objects exist and are responsible for what things, not in the
    'struct WhateverState' sense.

  * In fact right now eliminating the objects would be ok, since they
    have no migration state (which causes the problems this series is
    trying to address).  But applying this series which adds migration
    state would make it difficult to eliminate the objects in future.
    That's an awkward constraint given that we've already got some
    hints that these objects are not a good idea.

> However, the question then becomes why is the structure changing so much;
> ideally things in the migration stream should represent some tangible object
> that corresponds to something real, but (again ideally) the contents
> of the stream should reflect the state of those objects not the current
> implementation - so if you want to change the implementation the stream
> doesn't change.  Is it your implementation, or the understanding of what
> the objects actually represent that's in flux?

So, the objects in question are DRCs - "Dynamic Re-configuration
Connector"; silly IBM talk for "a port into which something can be
hotplugged", bascially.  These aren't "real" devices, but rather a
firmware/hypervisor abstraction which are used to describe a hotplug
point.  Each DRC allows either one CPU core, one PCI device, or one
LMB (256MiB chunk of RAM) to be hotplugged (or removed).  The PCI DRCs
are "owned" by the PCI host bridge to which the device would be
connected, the CPU and memory DRCs are owned by the machine.

The state variables which Jianjun Duan is adding to migration are
values defined in the PAPR (hypervisor interface) spec, and so are
tangible enough to be sensible to migrate.  At the moment, each LMB is
represented by a discrete QOM object, but I've been thinking for a
while that this may be a mistake.  In particular it's a problem for
the LMB DRCs - because each LMB is only 256MiB of memory, we end up
with thousands, maybe tens of thousands of DRC objects on a guest with
with large maxmem (even if initial memory is small).  The QOM
infrastructure doesn't deal terribly well that many object child
properties.

The construction of those DRCs used to be an N^3 algorithm, which as
you'd imagine caused horrible startup times with large maxmem (minutes
to hours above ~512GiB).  We fixed it to be only N^2, so now the
startup delays are merely bad (at least once you get to ~2TiB+
maxmem).

We could fix that by changing QOM to use a hash or tree for object
children, instead of a plain list, but I'm not sure if it makes sense
to do that with only this use case.

So, for some time, I've been considering rewriting the DRC stuff to be
implemented via QOM interface on the "owner" object, rather than
separate objects for every single connector.

So the difficulty is that if we add the state migration to the DRC
objects as they stand, we prevent ourselves from doing a refactor
that's probably a good idea.  But no-one's yet had time to actually
thrash out such a refactor. :/

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[Michael Roth]

Quoting David Gibson (2016-10-10 00:31:20)
> On Fri, Oct 07, 2016 at 09:07:49AM +0100, Dr. David Alan Gilbert wrote:
> > * David Gibson (david@gibson.dropbear.id.au) wrote:
> > > On Wed, Oct 05, 2016 at 09:44:57AM -0700, Jianjun Duan wrote:
> > > > Please see comments below:
> > > > 
> > > > On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> > > > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > > > >> In QOM(QEMU Object Model) migrated objects are identified with instance_id
> > > > >> which is calculated automatically using their path in the QOM composition
> > > > >> tree. For some objects, this path could change from source to target in
> > > > >> migration. To migrate such objects, we need to make sure the instance_id does
> > > > >> not change from source to target. We add a hook in DeviceClass to do customized
> > > > >> instance_id calculation in such cases.
> > > > > 
> > > > > Can you explain a bit about why the path changes from source to destination;
> > > > > the path here should be a feature of the guest state not the host, and so I
> > > > > don't understand why it changes.
> > > > Please see the discussion with David in the previous versions:
> > > > http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html
> > > 
> > > Um.. your description above really isn't an accurate summary of that
> > > discussion.
> > > 
> > > The point is not that the qom path will vary from source to
> > > destination for some arbitrary reason, but rather that we anticipate
> > > future changes in the QOM structure.  Specifically we're considering
> > > eliminating the DRC objects, and folding their (limited) state into an
> > > array in the parent object (either the machine or a PCI host bridge).
> > > 
> > > That would change the qom paths, and hence the auto-generated instance
> > > ids, which would break migration between qemu versions before and
> > > after the restructure.
> > > 
> > > I'm not sure that changing the instance ids is enough though, anyway,
> > > since we're talking about eliminating the object entirely, the
> > > class/type information in the migration stream also wouldn't match.
> > > 
> > > Dave, if you have ideas on how to deal with that, I'd love to hear
> > > them
> > 
> > Eliminating the object entirely would be tricky to deal with;
> > allowing the structure to change would work if instead of a custom instance_id
> > you had a custom idstr.
> 
> Sorry, two misunderstandings here.
> 
>   * When I said "structure" I meant in the high-level sense of what
>     objects exist and are responsible for what things, not in the
>     'struct WhateverState' sense.
> 
>   * In fact right now eliminating the objects would be ok, since they
>     have no migration state (which causes the problems this series is
>     trying to address).  But applying this series which adds migration
>     state would make it difficult to eliminate the objects in future.
>     That's an awkward constraint given that we've already got some
>     hints that these objects are not a good idea.
> 
> > However, the question then becomes why is the structure changing so much;
> > ideally things in the migration stream should represent some tangible object
> > that corresponds to something real, but (again ideally) the contents
> > of the stream should reflect the state of those objects not the current
> > implementation - so if you want to change the implementation the stream
> > doesn't change.  Is it your implementation, or the understanding of what
> > the objects actually represent that's in flux?
> 
> So, the objects in question are DRCs - "Dynamic Re-configuration
> Connector"; silly IBM talk for "a port into which something can be
> hotplugged", bascially.  These aren't "real" devices, but rather a
> firmware/hypervisor abstraction which are used to describe a hotplug
> point.  Each DRC allows either one CPU core, one PCI device, or one
> LMB (256MiB chunk of RAM) to be hotplugged (or removed).  The PCI DRCs
> are "owned" by the PCI host bridge to which the device would be
> connected, the CPU and memory DRCs are owned by the machine.
> 
> The state variables which Jianjun Duan is adding to migration are
> values defined in the PAPR (hypervisor interface) spec, and so are
> tangible enough to be sensible to migrate.  At the moment, each LMB is
> represented by a discrete QOM object, but I've been thinking for a
> while that this may be a mistake.  In particular it's a problem for
> the LMB DRCs - because each LMB is only 256MiB of memory, we end up
> with thousands, maybe tens of thousands of DRC objects on a guest with
> with large maxmem (even if initial memory is small).  The QOM
> infrastructure doesn't deal terribly well that many object child
> properties.
> 
> The construction of those DRCs used to be an N^3 algorithm, which as
> you'd imagine caused horrible startup times with large maxmem (minutes
> to hours above ~512GiB).  We fixed it to be only N^2, so now the
> startup delays are merely bad (at least once you get to ~2TiB+
> maxmem).
> 
> We could fix that by changing QOM to use a hash or tree for object
> children, instead of a plain list, but I'm not sure if it makes sense
> to do that with only this use case.

Is that still as much an issue as of this patch?

commit b604a854e843505007c59d68112c654556102a20
Author: Pavel Fedin <p.fedin@samsung.com>
Date:   Tue Oct 13 13:37:45 2015 +0100

    qom: Replace object property list with GHashTable
    
    ARM GICv3 systems with large number of CPUs create lots of IRQ pins. Since
    every pin is represented as a property, number of these properties becomes
    very large. Every property add first makes sure there's no duplicates.
    Traversing the list becomes very slow, therefore QEMU initialization takes
    significant time (several seconds for e. g. 16 CPUs).
    
    This patch replaces list with GHashTable, making lookup very fast. The only
    drawback is that object_child_foreach() and object_child_foreach_recursive()
    cannot add or remove properties during traversal, since GHashTableIter does
    not have modify-safe version. However, the code seems not to modify objects
    via these functions.

> 
> So, for some time, I've been considering rewriting the DRC stuff to be
> implemented via QOM interface on the "owner" object, rather than
> separate objects for every single connector.

At a high-level I think I'd prefer improving QOM over working around it's
performance issues. It seems like those limits are bound to be pushed
eventually given the wide range of hardware it's meant to support
in theory.

But identifying a DRC state in the migration stream with a stable value based
on DRC index makes sense either way. If I understand this patch it seems
like we have that: if a custom instance_id getter is available, we avoid
using the QOM path for idstr and use the default of vmsd->name, then use
the instance_id from the getter. If we dropped DRCs as objects, we'd need
a mechanism other than DeviceClass->dev_get_instance_id() to provide this
custom instance_id, but otherwise so long as the state was the same, and
the VMSD was the same, it seems like it should be doable in theory.

> 
> So the difficulty is that if we add the state migration to the DRC
> objects as they stand, we prevent ourselves from doing a refactor
> that's probably a good idea.  But no-one's yet had time to actually
> thrash out such a refactor. :/
> 
> -- 
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 8574 bytes --]

On Tue, Oct 11, 2016 at 11:17:35AM -0500, Michael Roth wrote:
> Quoting David Gibson (2016-10-10 00:31:20)
> > On Fri, Oct 07, 2016 at 09:07:49AM +0100, Dr. David Alan Gilbert wrote:
> > > * David Gibson (david@gibson.dropbear.id.au) wrote:
> > > > On Wed, Oct 05, 2016 at 09:44:57AM -0700, Jianjun Duan wrote:
> > > > > Please see comments below:
> > > > > 
> > > > > On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> > > > > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > > > > >> In QOM(QEMU Object Model) migrated objects are identified with instance_id
> > > > > >> which is calculated automatically using their path in the QOM composition
> > > > > >> tree. For some objects, this path could change from source to target in
> > > > > >> migration. To migrate such objects, we need to make sure the instance_id does
> > > > > >> not change from source to target. We add a hook in DeviceClass to do customized
> > > > > >> instance_id calculation in such cases.
> > > > > > 
> > > > > > Can you explain a bit about why the path changes from source to destination;
> > > > > > the path here should be a feature of the guest state not the host, and so I
> > > > > > don't understand why it changes.
> > > > > Please see the discussion with David in the previous versions:
> > > > > http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html
> > > > 
> > > > Um.. your description above really isn't an accurate summary of that
> > > > discussion.
> > > > 
> > > > The point is not that the qom path will vary from source to
> > > > destination for some arbitrary reason, but rather that we anticipate
> > > > future changes in the QOM structure.  Specifically we're considering
> > > > eliminating the DRC objects, and folding their (limited) state into an
> > > > array in the parent object (either the machine or a PCI host bridge).
> > > > 
> > > > That would change the qom paths, and hence the auto-generated instance
> > > > ids, which would break migration between qemu versions before and
> > > > after the restructure.
> > > > 
> > > > I'm not sure that changing the instance ids is enough though, anyway,
> > > > since we're talking about eliminating the object entirely, the
> > > > class/type information in the migration stream also wouldn't match.
> > > > 
> > > > Dave, if you have ideas on how to deal with that, I'd love to hear
> > > > them
> > > 
> > > Eliminating the object entirely would be tricky to deal with;
> > > allowing the structure to change would work if instead of a custom instance_id
> > > you had a custom idstr.
> > 
> > Sorry, two misunderstandings here.
> > 
> >   * When I said "structure" I meant in the high-level sense of what
> >     objects exist and are responsible for what things, not in the
> >     'struct WhateverState' sense.
> > 
> >   * In fact right now eliminating the objects would be ok, since they
> >     have no migration state (which causes the problems this series is
> >     trying to address).  But applying this series which adds migration
> >     state would make it difficult to eliminate the objects in future.
> >     That's an awkward constraint given that we've already got some
> >     hints that these objects are not a good idea.
> > 
> > > However, the question then becomes why is the structure changing so much;
> > > ideally things in the migration stream should represent some tangible object
> > > that corresponds to something real, but (again ideally) the contents
> > > of the stream should reflect the state of those objects not the current
> > > implementation - so if you want to change the implementation the stream
> > > doesn't change.  Is it your implementation, or the understanding of what
> > > the objects actually represent that's in flux?
> > 
> > So, the objects in question are DRCs - "Dynamic Re-configuration
> > Connector"; silly IBM talk for "a port into which something can be
> > hotplugged", bascially.  These aren't "real" devices, but rather a
> > firmware/hypervisor abstraction which are used to describe a hotplug
> > point.  Each DRC allows either one CPU core, one PCI device, or one
> > LMB (256MiB chunk of RAM) to be hotplugged (or removed).  The PCI DRCs
> > are "owned" by the PCI host bridge to which the device would be
> > connected, the CPU and memory DRCs are owned by the machine.
> > 
> > The state variables which Jianjun Duan is adding to migration are
> > values defined in the PAPR (hypervisor interface) spec, and so are
> > tangible enough to be sensible to migrate.  At the moment, each LMB is
> > represented by a discrete QOM object, but I've been thinking for a
> > while that this may be a mistake.  In particular it's a problem for
> > the LMB DRCs - because each LMB is only 256MiB of memory, we end up
> > with thousands, maybe tens of thousands of DRC objects on a guest with
> > with large maxmem (even if initial memory is small).  The QOM
> > infrastructure doesn't deal terribly well that many object child
> > properties.
> > 
> > The construction of those DRCs used to be an N^3 algorithm, which as
> > you'd imagine caused horrible startup times with large maxmem (minutes
> > to hours above ~512GiB).  We fixed it to be only N^2, so now the
> > startup delays are merely bad (at least once you get to ~2TiB+
> > maxmem).
> > 
> > We could fix that by changing QOM to use a hash or tree for object
> > children, instead of a plain list, but I'm not sure if it makes sense
> > to do that with only this use case.
> 
> Is that still as much an issue as of this patch?
> 
> commit b604a854e843505007c59d68112c654556102a20
> Author: Pavel Fedin <p.fedin@samsung.com>
> Date:   Tue Oct 13 13:37:45 2015 +0100
> 
>     qom: Replace object property list with GHashTable
>     
>     ARM GICv3 systems with large number of CPUs create lots of IRQ pins. Since
>     every pin is represented as a property, number of these properties becomes
>     very large. Every property add first makes sure there's no duplicates.
>     Traversing the list becomes very slow, therefore QEMU initialization takes
>     significant time (several seconds for e. g. 16 CPUs).
>     
>     This patch replaces list with GHashTable, making lookup very fast. The only
>     drawback is that object_child_foreach() and object_child_foreach_recursive()
>     cannot add or remove properties during traversal, since GHashTableIter does
>     not have modify-safe version. However, the code seems not to modify objects
>     via these functions.

Huh.. good point.  I'd completely forgotten that change had already
happened.

> > So, for some time, I've been considering rewriting the DRC stuff to be
> > implemented via QOM interface on the "owner" object, rather than
> > separate objects for every single connector.
> 
> At a high-level I think I'd prefer improving QOM over working around it's
> performance issues. It seems like those limits are bound to be pushed
> eventually given the wide range of hardware it's meant to support
> in theory.

Yeah, given we already have that hash, it probably makes sense to keep
the DRCs as "real" objects.

> But identifying a DRC state in the migration stream with a stable value based
> on DRC index makes sense either way. If I understand this patch it seems
> like we have that: if a custom instance_id getter is available, we avoid
> using the QOM path for idstr and use the default of vmsd->name, then use
> the instance_id from the getter. If we dropped DRCs as objects, we'd need
> a mechanism other than DeviceClass->dev_get_instance_id() to provide this
> custom instance_id, but otherwise so long as the state was the same, and
> the VMSD was the same, it seems like it should be doable in theory.

So, as noted in other comments, I'm not sure we actually need a new
mechanism for overriding the instance id.  But regardless of the
mechanism once we have a stable id based on the DRC index, yes it
should be ok to migrate the state variables of each DRC this way.

> 
> > 
> > So the difficulty is that if we add the state migration to the DRC
> > objects as they stand, we prevent ourselves from doing a refactor
> > that's probably a good idea.  But no-one's yet had time to actually
> > thrash out such a refactor. :/
> > 
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[Michael Roth]

Quoting David Gibson (2016-10-06 21:54:42)
> On Wed, Oct 05, 2016 at 09:44:57AM -0700, Jianjun Duan wrote:
> > Please see comments below:
> > 
> > On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > >> In QOM(QEMU Object Model) migrated objects are identified with instance_id
> > >> which is calculated automatically using their path in the QOM composition
> > >> tree. For some objects, this path could change from source to target in
> > >> migration. To migrate such objects, we need to make sure the instance_id does
> > >> not change from source to target. We add a hook in DeviceClass to do customized
> > >> instance_id calculation in such cases.
> > > 
> > > Can you explain a bit about why the path changes from source to destination;
> > > the path here should be a feature of the guest state not the host, and so I
> > > don't understand why it changes.
> > Please see the discussion with David in the previous versions:
> > http://lists.nongnu.org/archive/html/qemu-ppc/2016-06/msg00062.html
> 
> Um.. your description above really isn't an accurate summary of that
> discussion.
> 
> The point is not that the qom path will vary from source to
> destination for some arbitrary reason, but rather that we anticipate
> future changes in the QOM structure.  Specifically we're considering
> eliminating the DRC objects, and folding their (limited) state into an
> array in the parent object (either the machine or a PCI host bridge).
> 
> That would change the qom paths, and hence the auto-generated instance
> ids, which would break migration between qemu versions before and
> after the restructure.
> 
> I'm not sure that changing the instance ids is enough though, anyway,
> since we're talking about eliminating the object entirely, the
> class/type information in the migration stream also wouldn't match.
> 
> Dave, if you have ideas on how to deal with that, I'd love to hear
> them
> 
> > 
> > >> As a result, in these cases compat will not be set in the concerned
> > >> SaveStateEntry. This will prevent the inconsistent idstr to be sent over in
> > >> migration. We could have set alias_id in a similar way. But that will be
> > >> overloading the purpose of alias_id.
> > >>
> > >> The first application will be setting instance_id for DRC using its unique
> > >> index. Doing this makes the instance_id of DRC to be consistent across migration
> > >> and supports flexible management of DRC objects in migration.
> > > 
> > > Is there a reason to use a custom instance_id rather than a custom idstr
> > 
> > It can be done either way. But it is easier to deal with a integer than
> > a string.
> 
> A bit, but I don't think that's a good enough reason to introduce a
> second mechanism for overriding instance id allocations.

I'm looking at picking up this patch and the next for DRC migration
since it turns out it's needed to fix memory unplug after migration.

I think the gist of it is that vmstate_register() doesn't provide
a way to specify a custom idstr, and instead relies on vmsd->name
 (or qom-path for Devices), only register_savevm_live() does. Both
however provide a way of specifying a custom instance_id. The only
real issue is that in the case of Devices we end up ignoring the
instance_id. So I propose using the existing interface to set the
instance_id, and relegating all knowledge of
DeviceClass->dev_get_instance_id() to qdev (and other callers
as-needed). Maybe something like this?

diff --git a/hw/core/qdev.c b/hw/core/qdev.c
index d835e62..7323152 100644
--- a/hw/core/qdev.c
+++ b/hw/core/qdev.c
@@ -933,7 +933,11 @@ static void device_set_realized(Object *obj, bool value, Error **errp)
         }
 
         if (qdev_get_vmsd(dev)) {
-            vmstate_register_with_alias_id(dev, -1, qdev_get_vmsd(dev), dev,
+            int instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id
+                ? DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev)
+                : -1;
+
+            vmstate_register_with_alias_id(dev, instance_id, qdev_get_vmsd(dev), dev,
                                            dev->instance_id_alias,
                                            dev->alias_required_for_version);
         }
diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
index 2c97347..a012e8e 100644
--- a/include/hw/qdev-core.h
+++ b/include/hw/qdev-core.h
@@ -139,6 +139,12 @@ typedef struct DeviceClass {
     qdev_initfn init; /* TODO remove, once users are converted to realize */
     qdev_event exit; /* TODO remove, once users are converted to unrealize */
     const char *bus_type;
+
+    /* When this field is set, qemu will use it to get an unique instance_id
+     * instead of calculating an auto idstr and instanc_id for the relevant
+     * SaveStateEntry
+     */
+    int (*dev_get_instance_id)(DeviceState *dev);
 } DeviceClass;
 
 typedef struct NamedGPIOList NamedGPIOList;
diff --git a/migration/savevm.c b/migration/savevm.c
index 0363372..a95fff9 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -556,7 +556,7 @@ int register_savevm_live(DeviceState *dev,
         se->is_ram = 1;
     }
 
-    if (dev) {
+    if (dev && instance_id == -1) {
         char *id = qdev_get_dev_path(dev);
         if (id) {
             pstrcpy(se->idstr, sizeof(se->idstr), id);
@@ -640,7 +640,7 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
     se->vmsd = vmsd;
     se->alias_id = alias_id;
 
-    if (dev) {
+    if (dev && instance_id == -1) {
         char *id = qdev_get_dev_path(dev);
         if (id) {
             pstrcpy(se->idstr, sizeof(se->idstr), id);

> 
> -- 
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 1/6] migration: alternative way to set instance_id in SaveStateEntry

[Jianjun Duan]

On 10/05/2016 03:12 AM, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>> In QOM(QEMU Object Model) migrated objects are identified with instance_id
>> which is calculated automatically using their path in the QOM composition
>> tree. For some objects, this path could change from source to target in
>> migration. To migrate such objects, we need to make sure the instance_id does
>> not change from source to target. We add a hook in DeviceClass to do customized
>> instance_id calculation in such cases.
> 
> Can you explain a bit about why the path changes from source to destination;
> the path here should be a feature of the guest state not the host, and so I
> don't understand why it changes.
> 
>> As a result, in these cases compat will not be set in the concerned
>> SaveStateEntry. This will prevent the inconsistent idstr to be sent over in
>> migration. We could have set alias_id in a similar way. But that will be
>> overloading the purpose of alias_id.
>>
>> The first application will be setting instance_id for DRC using its unique
>> index. Doing this makes the instance_id of DRC to be consistent across migration
>> and supports flexible management of DRC objects in migration.
> 
> Is there a reason to use a custom instance_id rather than a custom idstr?
> 
>>
>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>> ---
>>  include/hw/qdev-core.h |  6 ++++++
>>  migration/savevm.c     | 20 ++++++++++++++++++--
>>  2 files changed, 24 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h
>> index 2c97347..a012e8e 100644
>> --- a/include/hw/qdev-core.h
>> +++ b/include/hw/qdev-core.h
>> @@ -139,6 +139,12 @@ typedef struct DeviceClass {
>>      qdev_initfn init; /* TODO remove, once users are converted to realize */
>>      qdev_event exit; /* TODO remove, once users are converted to unrealize */
>>      const char *bus_type;
>> +
>> +    /* When this field is set, qemu will use it to get an unique instance_id
>> +     * instead of calculating an auto idstr and instanc_id for the relevant
>> +     * SaveStateEntry
>> +     */
>> +    int (*dev_get_instance_id)(DeviceState *dev);
>>  } DeviceClass;
>>  
>>  typedef struct NamedGPIOList NamedGPIOList;
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index 33a2911..ef5c3d1 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -495,6 +495,11 @@ int register_savevm_live(DeviceState *dev,
>>                           void *opaque)
>>  {
>>      SaveStateEntry *se;
>> +    /* when it is a device and it provides a way to get instance_id,
>> +     * we will use it and skip setting idstr and compat.
>> +     */
>> +    bool flag = (dev != NULL) &&
>> +                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);
> 
> We must be able to get a more descriptive name than 'flag'; how about
>  'has_custom_id'
Sure. Will wait for more comments and address them together.

>>      se = g_new0(SaveStateEntry, 1);
>>      se->version_id = version_id;
>> @@ -507,7 +512,7 @@ int register_savevm_live(DeviceState *dev,
>>          se->is_ram = 1;
>>      }
>>  
>> -    if (dev) {
>> +    if (dev && !flag) {
>>          char *id = qdev_get_dev_path(dev);
>>          if (id) {
>>              pstrcpy(se->idstr, sizeof(se->idstr), id);
>> @@ -523,6 +528,9 @@ int register_savevm_live(DeviceState *dev,
>>      }
>>      pstrcat(se->idstr, sizeof(se->idstr), idstr);
>>  
>> +    if (flag) {
>> +        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
>> +    }
>>      if (instance_id == -1) {
>>          se->instance_id = calculate_new_instance_id(se->idstr);
>>      } else {
>> @@ -580,6 +588,11 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>>                                     int required_for_version)
>>  {
>>      SaveStateEntry *se;
>> +    /* when it is a device and it provides a way to get instance_id,
>> +     * we will use it and skip setting idstr and compat.
>> +     */
>> +    bool flag = (dev != NULL) &&
>> +                (DEVICE_GET_CLASS(dev)->dev_get_instance_id != NULL);
>>  
>>      /* If this triggers, alias support can be dropped for the vmsd. */
>>      assert(alias_id == -1 || required_for_version >= vmsd->minimum_version_id);
>> @@ -591,7 +604,7 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>>      se->vmsd = vmsd;
>>      se->alias_id = alias_id;
>>  
>> -    if (dev) {
>> +    if (dev && !flag) {
>>          char *id = qdev_get_dev_path(dev);
>>          if (id) {
>>              pstrcpy(se->idstr, sizeof(se->idstr), id);
>> @@ -607,6 +620,9 @@ int vmstate_register_with_alias_id(DeviceState *dev, int instance_id,
>>      }
>>      pstrcat(se->idstr, sizeof(se->idstr), vmsd->name);
>>  
>> +    if (flag) {
>> +        instance_id = DEVICE_GET_CLASS(dev)->dev_get_instance_id(dev);
>> +    }
>>      if (instance_id == -1) {
>>          se->instance_id = calculate_new_instance_id(se->idstr);
>>      } else {
>> -- 
>> 1.9.1
> 
> Dave
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
Thanks,
Jianjun

[Qemu-devel] [QEMU PATCH v5 2/6] migration: spapr_drc: defined VMStateDescription struct

[Jianjun Duan]

To manage hotplug/unplug of dynamic resources such as PCI cards,
memory, and CPU on sPAPR guests, a firmware abstraction known as
a Dynamic Resource Connector (DRC) is used to assign a particular
dynamic resource to the guest, and provide an interface for the
guest to manage configuration/removal of the resource associated
with it.

To migrate the hotplugged resources in migration, the
associated DRC state need be migrated. To migrate the DRC state,
we defined the VMStateDescription struct for spapr_drc to enable
the transmission of spapr_drc state in migration.

Not all the elements in the DRC state are migrated. Only those
ones modifiable or needed by guest actions or device add/remove
operation are migrated. From the perspective of device
hotplugging, if we hotplug a device on the source, we need to
"coldplug" it on the target. The states across two hosts for the
same device are not the same. Ideally we want the states be same
after migration so that the device would function as hotplugged
on the target. For example we can unplug it. The minimum DRC
state we need to transfer should cover all the pieces changed by
hotplugging. Out of the elements of the DRC state, isolation_state,
allocation_sate, and configured are involved in the DR state
transition diagram from PAPR+ 2.7, 13.4. configured and signalled
are needed in attaching and detaching devices. indicator_state
provides users with hardware state information. These 6 elements
are migrated.

detach_cb in the DRC state is a function pointer that cannot be
migrated. We set it right after DRC state is migrated so that
a migrated hot-unplug event could finish its work.

The instance_id is used to identify objects in migration. We set
instance_id of DRC using the unique index so that it is the same
across migration.

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
---
 hw/ppc/spapr_drc.c         | 69 ++++++++++++++++++++++++++++++++++++++++++++++
 hw/ppc/spapr_pci.c         | 22 +++++++++++++++
 include/hw/ppc/spapr_drc.h |  9 ++++++
 3 files changed, 100 insertions(+)

diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c
index 6e54fd4..369ec02 100644
--- a/hw/ppc/spapr_drc.c
+++ b/hw/ppc/spapr_drc.c
@@ -615,6 +615,71 @@ static void spapr_dr_connector_instance_init(Object *obj)
                         NULL, NULL, NULL, NULL);
 }
 
+static bool spapr_drc_needed(void *opaque)
+{
+    sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque;
+    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
+    bool rc = false;
+    sPAPRDREntitySense value;
+
+    drck->entity_sense(drc, &value);
+    /* If no dev is plugged in there is no need to migrate the DRC state */
+    if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) {
+        return false;
+    }
+    /*
+     * If there is dev plugged in, we need to migrate the DRC state when
+     * it is different from cold-plugged state
+     */
+    switch(drc->type) {
+    /* for PCI type */
+    case SPAPR_DR_CONNECTOR_TYPE_PCI:
+        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) &&
+               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) &&
+               drc->configured && drc->signalled && !drc->awaiting_release);
+        break;
+    /* for LMB type */
+    case SPAPR_DR_CONNECTOR_TYPE_LMB:
+        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) &&
+               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) &&
+               drc->configured && drc->signalled && !drc->awaiting_release);
+        break;
+    default:
+        ;
+    }
+
+    return rc;
+}
+
+/* detach_cb needs be set since it is not migrated */
+static void postmigrate_set_detach_cb(sPAPRDRConnector *drc,
+                                      spapr_drc_detach_cb *detach_cb)
+{
+    drc->detach_cb = detach_cb;
+}
+
+/* return the unique drc index as instance_id for qom interfaces*/
+static int get_instance_id(DeviceState *dev)
+{
+    return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev)));
+}
+
+static const VMStateDescription vmstate_spapr_drc = {
+    .name = "spapr_drc",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = spapr_drc_needed,
+    .fields  = (VMStateField []) {
+        VMSTATE_UINT32(isolation_state, sPAPRDRConnector),
+        VMSTATE_UINT32(allocation_state, sPAPRDRConnector),
+        VMSTATE_UINT32(indicator_state, sPAPRDRConnector),
+        VMSTATE_BOOL(configured, sPAPRDRConnector),
+        VMSTATE_BOOL(awaiting_release, sPAPRDRConnector),
+        VMSTATE_BOOL(signalled, sPAPRDRConnector),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
 {
     DeviceClass *dk = DEVICE_CLASS(k);
@@ -623,6 +688,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
     dk->reset = reset;
     dk->realize = realize;
     dk->unrealize = unrealize;
+    dk->vmsd = &vmstate_spapr_drc;
+    dk->dev_get_instance_id = get_instance_id;
     drck->set_isolation_state = set_isolation_state;
     drck->set_indicator_state = set_indicator_state;
     drck->set_allocation_state = set_allocation_state;
@@ -636,6 +703,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
     drck->detach = detach;
     drck->release_pending = release_pending;
     drck->set_signalled = set_signalled;
+    drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb;
+
     /*
      * Reason: it crashes FIXME find and document the real reason
      */
diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
index 4f00865..080471c 100644
--- a/hw/ppc/spapr_pci.c
+++ b/hw/ppc/spapr_pci.c
@@ -1594,11 +1594,33 @@ static void spapr_pci_pre_save(void *opaque)
     }
 }
 
+/*
+ * detach_cb in the DRC state is a function pointer that cannot be
+ * migrated. We set it right after migration so that a migrated
+ * hot-unplug event could finish its work.
+ */
+static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev,
+                                 void *opaque)
+{
+    sPAPRPHBState *sphb = opaque;
+    sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev);
+    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
+    drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb);
+}
+
 static int spapr_pci_post_load(void *opaque, int version_id)
 {
     sPAPRPHBState *sphb = opaque;
     gpointer key, value;
     int i;
+    PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus;
+    unsigned int bus_no = 0;
+
+    /* Set detach_cb for the drc unconditionally after migration */
+    if (bus) {
+        pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb,
+                            &bus_no);
+    }
 
     for (i = 0; i < sphb->msi_devs_num; ++i) {
         key = g_memdup(&sphb->msi_devs[i].key,
diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h
index fa531d5..17589c8 100644
--- a/include/hw/ppc/spapr_drc.h
+++ b/include/hw/ppc/spapr_drc.h
@@ -192,6 +192,15 @@ typedef struct sPAPRDRConnectorClass {
                    void *detach_cb_opaque, Error **errp);
     bool (*release_pending)(sPAPRDRConnector *drc);
     void (*set_signalled)(sPAPRDRConnector *drc);
+
+    /*
+     * QEMU interface for setting detach_cb after migration.
+     * detach_cb in the DRC state is a function pointer that cannot be
+     * migrated. We set it right after migration so that a migrated
+     * hot-unplug event could finish its work.
+     */
+    void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc,
+                                      spapr_drc_detach_cb *detach_cb);
 } sPAPRDRConnectorClass;
 
 sPAPRDRConnector *spapr_dr_connector_new(Object *owner,
-- 
1.9.1

Re: [Qemu-devel] [QEMU PATCH v5 2/6] migration: spapr_drc: defined VMStateDescription struct

[Dr. David Alan Gilbert]

* Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> To manage hotplug/unplug of dynamic resources such as PCI cards,
> memory, and CPU on sPAPR guests, a firmware abstraction known as
> a Dynamic Resource Connector (DRC) is used to assign a particular
> dynamic resource to the guest, and provide an interface for the
> guest to manage configuration/removal of the resource associated
> with it.
> 
> To migrate the hotplugged resources in migration, the
> associated DRC state need be migrated. To migrate the DRC state,
> we defined the VMStateDescription struct for spapr_drc to enable
> the transmission of spapr_drc state in migration.
> 
> Not all the elements in the DRC state are migrated. Only those
> ones modifiable or needed by guest actions or device add/remove
> operation are migrated. From the perspective of device
> hotplugging, if we hotplug a device on the source, we need to
> "coldplug" it on the target. The states across two hosts for the
> same device are not the same. Ideally we want the states be same
> after migration so that the device would function as hotplugged
> on the target. For example we can unplug it. The minimum DRC
> state we need to transfer should cover all the pieces changed by
> hotplugging. Out of the elements of the DRC state, isolation_state,
> allocation_sate, and configured are involved in the DR state
> transition diagram from PAPR+ 2.7, 13.4. configured and signalled
> are needed in attaching and detaching devices. indicator_state
> provides users with hardware state information. These 6 elements
> are migrated.
> 
> detach_cb in the DRC state is a function pointer that cannot be
> migrated. We set it right after DRC state is migrated so that
> a migrated hot-unplug event could finish its work.

Be careful with that; it'll get tricky if you have a bunch of different
possible callbacks.   If you want to explicitly migrate it then you'd
have to have an enum of different functions that could be called rather
than storing the pointer explicitly.

> The instance_id is used to identify objects in migration. We set
> instance_id of DRC using the unique index so that it is the same
> across migration.
> 
> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>

I think this is OK from a migration point of view; I'll leave
it to someone else to check the Power side of things.

> ---
>  hw/ppc/spapr_drc.c         | 69 ++++++++++++++++++++++++++++++++++++++++++++++
>  hw/ppc/spapr_pci.c         | 22 +++++++++++++++
>  include/hw/ppc/spapr_drc.h |  9 ++++++
>  3 files changed, 100 insertions(+)
> 
> diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c
> index 6e54fd4..369ec02 100644
> --- a/hw/ppc/spapr_drc.c
> +++ b/hw/ppc/spapr_drc.c
> @@ -615,6 +615,71 @@ static void spapr_dr_connector_instance_init(Object *obj)
>                          NULL, NULL, NULL, NULL);
>  }
>  
> +static bool spapr_drc_needed(void *opaque)
> +{
> +    sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque;
> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> +    bool rc = false;
> +    sPAPRDREntitySense value;
> +
> +    drck->entity_sense(drc, &value);
> +    /* If no dev is plugged in there is no need to migrate the DRC state */
> +    if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) {
> +        return false;
> +    }
> +    /*
> +     * If there is dev plugged in, we need to migrate the DRC state when
> +     * it is different from cold-plugged state
> +     */
> +    switch(drc->type) {
> +    /* for PCI type */
> +    case SPAPR_DR_CONNECTOR_TYPE_PCI:
> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) &&
> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) &&
> +               drc->configured && drc->signalled && !drc->awaiting_release);
> +        break;
> +    /* for LMB type */
> +    case SPAPR_DR_CONNECTOR_TYPE_LMB:
> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) &&
> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) &&
> +               drc->configured && drc->signalled && !drc->awaiting_release);
> +        break;
> +    default:
> +        ;
> +    }
> +
> +    return rc;
> +}
> +
> +/* detach_cb needs be set since it is not migrated */
> +static void postmigrate_set_detach_cb(sPAPRDRConnector *drc,
> +                                      spapr_drc_detach_cb *detach_cb)
> +{
> +    drc->detach_cb = detach_cb;
> +}
> +
> +/* return the unique drc index as instance_id for qom interfaces*/
> +static int get_instance_id(DeviceState *dev)
> +{
> +    return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev)));
> +}
> +
> +static const VMStateDescription vmstate_spapr_drc = {
> +    .name = "spapr_drc",
> +    .version_id = 1,
> +    .minimum_version_id = 1,
> +    .needed = spapr_drc_needed,
> +    .fields  = (VMStateField []) {
> +        VMSTATE_UINT32(isolation_state, sPAPRDRConnector),
> +        VMSTATE_UINT32(allocation_state, sPAPRDRConnector),
> +        VMSTATE_UINT32(indicator_state, sPAPRDRConnector),
> +        VMSTATE_BOOL(configured, sPAPRDRConnector),
> +        VMSTATE_BOOL(awaiting_release, sPAPRDRConnector),
> +        VMSTATE_BOOL(signalled, sPAPRDRConnector),
> +        VMSTATE_END_OF_LIST()
> +    }
> +};
> +
>  static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>  {
>      DeviceClass *dk = DEVICE_CLASS(k);
> @@ -623,6 +688,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>      dk->reset = reset;
>      dk->realize = realize;
>      dk->unrealize = unrealize;
> +    dk->vmsd = &vmstate_spapr_drc;
> +    dk->dev_get_instance_id = get_instance_id;
>      drck->set_isolation_state = set_isolation_state;
>      drck->set_indicator_state = set_indicator_state;
>      drck->set_allocation_state = set_allocation_state;
> @@ -636,6 +703,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>      drck->detach = detach;
>      drck->release_pending = release_pending;
>      drck->set_signalled = set_signalled;
> +    drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb;
> +
>      /*
>       * Reason: it crashes FIXME find and document the real reason
>       */
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index 4f00865..080471c 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -1594,11 +1594,33 @@ static void spapr_pci_pre_save(void *opaque)
>      }
>  }
>  
> +/*
> + * detach_cb in the DRC state is a function pointer that cannot be
> + * migrated. We set it right after migration so that a migrated
> + * hot-unplug event could finish its work.
> + */
> +static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev,
> +                                 void *opaque)
> +{
> +    sPAPRPHBState *sphb = opaque;
> +    sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev);
> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> +    drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb);
> +}
> +
>  static int spapr_pci_post_load(void *opaque, int version_id)
>  {
>      sPAPRPHBState *sphb = opaque;
>      gpointer key, value;
>      int i;
> +    PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus;
> +    unsigned int bus_no = 0;
> +
> +    /* Set detach_cb for the drc unconditionally after migration */
> +    if (bus) {
> +        pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb,
> +                            &bus_no);
> +    }
>  
>      for (i = 0; i < sphb->msi_devs_num; ++i) {
>          key = g_memdup(&sphb->msi_devs[i].key,
> diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h
> index fa531d5..17589c8 100644
> --- a/include/hw/ppc/spapr_drc.h
> +++ b/include/hw/ppc/spapr_drc.h
> @@ -192,6 +192,15 @@ typedef struct sPAPRDRConnectorClass {
>                     void *detach_cb_opaque, Error **errp);
>      bool (*release_pending)(sPAPRDRConnector *drc);
>      void (*set_signalled)(sPAPRDRConnector *drc);
> +
> +    /*
> +     * QEMU interface for setting detach_cb after migration.
> +     * detach_cb in the DRC state is a function pointer that cannot be
> +     * migrated. We set it right after migration so that a migrated
> +     * hot-unplug event could finish its work.
> +     */
> +    void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc,
> +                                      spapr_drc_detach_cb *detach_cb);
>  } sPAPRDRConnectorClass;
>  
>  sPAPRDRConnector *spapr_dr_connector_new(Object *owner,
> -- 
> 1.9.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [QEMU PATCH v5 2/6] migration: spapr_drc: defined VMStateDescription struct

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 9679 bytes --]

On Wed, Oct 05, 2016 at 12:38:53PM +0100, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > To manage hotplug/unplug of dynamic resources such as PCI cards,
> > memory, and CPU on sPAPR guests, a firmware abstraction known as
> > a Dynamic Resource Connector (DRC) is used to assign a particular
> > dynamic resource to the guest, and provide an interface for the
> > guest to manage configuration/removal of the resource associated
> > with it.
> > 
> > To migrate the hotplugged resources in migration, the
> > associated DRC state need be migrated. To migrate the DRC state,
> > we defined the VMStateDescription struct for spapr_drc to enable
> > the transmission of spapr_drc state in migration.
> > 
> > Not all the elements in the DRC state are migrated. Only those
> > ones modifiable or needed by guest actions or device add/remove
> > operation are migrated. From the perspective of device
> > hotplugging, if we hotplug a device on the source, we need to
> > "coldplug" it on the target. The states across two hosts for the
> > same device are not the same. Ideally we want the states be same
> > after migration so that the device would function as hotplugged
> > on the target. For example we can unplug it. The minimum DRC
> > state we need to transfer should cover all the pieces changed by
> > hotplugging. Out of the elements of the DRC state, isolation_state,
> > allocation_sate, and configured are involved in the DR state
> > transition diagram from PAPR+ 2.7, 13.4. configured and signalled
> > are needed in attaching and detaching devices. indicator_state
> > provides users with hardware state information. These 6 elements
> > are migrated.
> > 
> > detach_cb in the DRC state is a function pointer that cannot be
> > migrated. We set it right after DRC state is migrated so that
> > a migrated hot-unplug event could finish its work.
> 
> Be careful with that; it'll get tricky if you have a bunch of different
> possible callbacks.   If you want to explicitly migrate it then you'd
> have to have an enum of different functions that could be called rather
> than storing the pointer explicitly.

Quite.  Looking at the code, I'm quite baffled as to how detach_cb and
detach_cb_opaque get populated in the first place anyway.  All the
assignments I can see seem to be assigning the same variable to itself
(loaded in the caller, passed to another function, stored back to the
same place in the callee).

> 
> > The instance_id is used to identify objects in migration. We set
> > instance_id of DRC using the unique index so that it is the same
> > across migration.
> > 
> > Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> 
> I think this is OK from a migration point of view; I'll leave
> it to someone else to check the Power side of things.
> 
> > ---
> >  hw/ppc/spapr_drc.c         | 69 ++++++++++++++++++++++++++++++++++++++++++++++
> >  hw/ppc/spapr_pci.c         | 22 +++++++++++++++
> >  include/hw/ppc/spapr_drc.h |  9 ++++++
> >  3 files changed, 100 insertions(+)
> > 
> > diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c
> > index 6e54fd4..369ec02 100644
> > --- a/hw/ppc/spapr_drc.c
> > +++ b/hw/ppc/spapr_drc.c
> > @@ -615,6 +615,71 @@ static void spapr_dr_connector_instance_init(Object *obj)
> >                          NULL, NULL, NULL, NULL);
> >  }
> >  
> > +static bool spapr_drc_needed(void *opaque)
> > +{
> > +    sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque;
> > +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > +    bool rc = false;
> > +    sPAPRDREntitySense value;
> > +
> > +    drck->entity_sense(drc, &value);
> > +    /* If no dev is plugged in there is no need to migrate the DRC state */
> > +    if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) {
> > +        return false;
> > +    }
> > +    /*
> > +     * If there is dev plugged in, we need to migrate the DRC state when
> > +     * it is different from cold-plugged state
> > +     */
> > +    switch(drc->type) {
> > +    /* for PCI type */
> > +    case SPAPR_DR_CONNECTOR_TYPE_PCI:
> > +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) &&
> > +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) &&
> > +               drc->configured && drc->signalled && !drc->awaiting_release);
> > +        break;
> > +    /* for LMB type */
> > +    case SPAPR_DR_CONNECTOR_TYPE_LMB:
> > +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) &&
> > +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) &&
> > +               drc->configured && drc->signalled && !drc->awaiting_release);
> > +        break;
> > +    default:
> > +        ;
> > +    }
> > +
> > +    return rc;
> > +}
> > +
> > +/* detach_cb needs be set since it is not migrated */
> > +static void postmigrate_set_detach_cb(sPAPRDRConnector *drc,
> > +                                      spapr_drc_detach_cb *detach_cb)
> > +{
> > +    drc->detach_cb = detach_cb;
> > +}
> > +
> > +/* return the unique drc index as instance_id for qom interfaces*/
> > +static int get_instance_id(DeviceState *dev)
> > +{
> > +    return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev)));
> > +}
> > +
> > +static const VMStateDescription vmstate_spapr_drc = {
> > +    .name = "spapr_drc",
> > +    .version_id = 1,
> > +    .minimum_version_id = 1,
> > +    .needed = spapr_drc_needed,
> > +    .fields  = (VMStateField []) {
> > +        VMSTATE_UINT32(isolation_state, sPAPRDRConnector),
> > +        VMSTATE_UINT32(allocation_state, sPAPRDRConnector),
> > +        VMSTATE_UINT32(indicator_state, sPAPRDRConnector),
> > +        VMSTATE_BOOL(configured, sPAPRDRConnector),
> > +        VMSTATE_BOOL(awaiting_release, sPAPRDRConnector),
> > +        VMSTATE_BOOL(signalled, sPAPRDRConnector),
> > +        VMSTATE_END_OF_LIST()
> > +    }
> > +};
> > +
> >  static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
> >  {
> >      DeviceClass *dk = DEVICE_CLASS(k);
> > @@ -623,6 +688,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
> >      dk->reset = reset;
> >      dk->realize = realize;
> >      dk->unrealize = unrealize;
> > +    dk->vmsd = &vmstate_spapr_drc;
> > +    dk->dev_get_instance_id = get_instance_id;
> >      drck->set_isolation_state = set_isolation_state;
> >      drck->set_indicator_state = set_indicator_state;
> >      drck->set_allocation_state = set_allocation_state;
> > @@ -636,6 +703,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
> >      drck->detach = detach;
> >      drck->release_pending = release_pending;
> >      drck->set_signalled = set_signalled;
> > +    drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb;
> > +
> >      /*
> >       * Reason: it crashes FIXME find and document the real reason
> >       */
> > diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> > index 4f00865..080471c 100644
> > --- a/hw/ppc/spapr_pci.c
> > +++ b/hw/ppc/spapr_pci.c
> > @@ -1594,11 +1594,33 @@ static void spapr_pci_pre_save(void *opaque)
> >      }
> >  }
> >  
> > +/*
> > + * detach_cb in the DRC state is a function pointer that cannot be
> > + * migrated. We set it right after migration so that a migrated
> > + * hot-unplug event could finish its work.
> > + */
> > +static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev,
> > +                                 void *opaque)
> > +{
> > +    sPAPRPHBState *sphb = opaque;
> > +    sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev);
> > +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> > +    drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb);
> > +}
> > +
> >  static int spapr_pci_post_load(void *opaque, int version_id)
> >  {
> >      sPAPRPHBState *sphb = opaque;
> >      gpointer key, value;
> >      int i;
> > +    PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus;
> > +    unsigned int bus_no = 0;
> > +
> > +    /* Set detach_cb for the drc unconditionally after migration */
> > +    if (bus) {
> > +        pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb,
> > +                            &bus_no);
> > +    }
> >  
> >      for (i = 0; i < sphb->msi_devs_num; ++i) {
> >          key = g_memdup(&sphb->msi_devs[i].key,
> > diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h
> > index fa531d5..17589c8 100644
> > --- a/include/hw/ppc/spapr_drc.h
> > +++ b/include/hw/ppc/spapr_drc.h
> > @@ -192,6 +192,15 @@ typedef struct sPAPRDRConnectorClass {
> >                     void *detach_cb_opaque, Error **errp);
> >      bool (*release_pending)(sPAPRDRConnector *drc);
> >      void (*set_signalled)(sPAPRDRConnector *drc);
> > +
> > +    /*
> > +     * QEMU interface for setting detach_cb after migration.
> > +     * detach_cb in the DRC state is a function pointer that cannot be
> > +     * migrated. We set it right after migration so that a migrated
> > +     * hot-unplug event could finish its work.
> > +     */
> > +    void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc,
> > +                                      spapr_drc_detach_cb *detach_cb);
> >  } sPAPRDRConnectorClass;
> >  
> >  sPAPRDRConnector *spapr_dr_connector_new(Object *owner,

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 2/6] migration: spapr_drc: defined VMStateDescription struct

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 9199 bytes --]

On Mon, Oct 03, 2016 at 11:24:53AM -0700, Jianjun Duan wrote:
> To manage hotplug/unplug of dynamic resources such as PCI cards,
> memory, and CPU on sPAPR guests, a firmware abstraction known as
> a Dynamic Resource Connector (DRC) is used to assign a particular
> dynamic resource to the guest, and provide an interface for the
> guest to manage configuration/removal of the resource associated
> with it.
> 
> To migrate the hotplugged resources in migration, the
> associated DRC state need be migrated. To migrate the DRC state,
> we defined the VMStateDescription struct for spapr_drc to enable
> the transmission of spapr_drc state in migration.
> 
> Not all the elements in the DRC state are migrated. Only those
> ones modifiable or needed by guest actions or device add/remove
> operation are migrated. From the perspective of device
> hotplugging, if we hotplug a device on the source, we need to
> "coldplug" it on the target. The states across two hosts for the
> same device are not the same. Ideally we want the states be same
> after migration so that the device would function as hotplugged
> on the target. For example we can unplug it. The minimum DRC
> state we need to transfer should cover all the pieces changed by
> hotplugging. Out of the elements of the DRC state, isolation_state,
> allocation_sate, and configured are involved in the DR state
> transition diagram from PAPR+ 2.7, 13.4. configured and signalled
> are needed in attaching and detaching devices. indicator_state
> provides users with hardware state information. These 6 elements
> are migrated.

Hmm.. are you saying that the DRC state of a coldplugged device (after
we've fully booted) is different from the DRC state of a hotplugged
device (after all the hotplug operations have fully completed)?

If that's correct that sounds like a general bug in the DRC state
management, not something only related to migration.

Looking at the code, though, that doesn't really seem to be what it's
doing.

> detach_cb in the DRC state is a function pointer that cannot be
> migrated. We set it right after DRC state is migrated so that
> a migrated hot-unplug event could finish its work.
> 
> The instance_id is used to identify objects in migration. We set
> instance_id of DRC using the unique index so that it is the same
> across migration.
> 
> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> ---
>  hw/ppc/spapr_drc.c         | 69 ++++++++++++++++++++++++++++++++++++++++++++++
>  hw/ppc/spapr_pci.c         | 22 +++++++++++++++
>  include/hw/ppc/spapr_drc.h |  9 ++++++
>  3 files changed, 100 insertions(+)
> 
> diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c
> index 6e54fd4..369ec02 100644
> --- a/hw/ppc/spapr_drc.c
> +++ b/hw/ppc/spapr_drc.c
> @@ -615,6 +615,71 @@ static void spapr_dr_connector_instance_init(Object *obj)
>                          NULL, NULL, NULL, NULL);
>  }
>  
> +static bool spapr_drc_needed(void *opaque)
> +{
> +    sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque;
> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> +    bool rc = false;
> +    sPAPRDREntitySense value;
> +
> +    drck->entity_sense(drc, &value);
> +    /* If no dev is plugged in there is no need to migrate the DRC state */
> +    if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) {
> +        return false;
> +    }
> +    /*
> +     * If there is dev plugged in, we need to migrate the DRC state when
> +     * it is different from cold-plugged state
> +     */
> +    switch(drc->type) {
> +    /* for PCI type */
> +    case SPAPR_DR_CONNECTOR_TYPE_PCI:
> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) &&
> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) &&
> +               drc->configured && drc->signalled && !drc->awaiting_release);
> +        break;
> +    /* for LMB type */
> +    case SPAPR_DR_CONNECTOR_TYPE_LMB:
> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) &&
> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) &&
> +               drc->configured && drc->signalled && !drc->awaiting_release);
> +        break;

What about CPU type?z

> +    default:
> +        ;
> +    }
> +
> +    return rc;
> +}
> +
> +/* detach_cb needs be set since it is not migrated */
> +static void postmigrate_set_detach_cb(sPAPRDRConnector *drc,
> +                                      spapr_drc_detach_cb *detach_cb)
> +{
> +    drc->detach_cb = detach_cb;
> +}
> +
> +/* return the unique drc index as instance_id for qom interfaces*/
> +static int get_instance_id(DeviceState *dev)
> +{
> +    return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev)));
> +}
> +
> +static const VMStateDescription vmstate_spapr_drc = {
> +    .name = "spapr_drc",
> +    .version_id = 1,
> +    .minimum_version_id = 1,
> +    .needed = spapr_drc_needed,
> +    .fields  = (VMStateField []) {
> +        VMSTATE_UINT32(isolation_state, sPAPRDRConnector),
> +        VMSTATE_UINT32(allocation_state, sPAPRDRConnector),
> +        VMSTATE_UINT32(indicator_state, sPAPRDRConnector),
> +        VMSTATE_BOOL(configured, sPAPRDRConnector),
> +        VMSTATE_BOOL(awaiting_release, sPAPRDRConnector),
> +        VMSTATE_BOOL(signalled, sPAPRDRConnector),
> +        VMSTATE_END_OF_LIST()
> +    }
> +};
> +
>  static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>  {
>      DeviceClass *dk = DEVICE_CLASS(k);
> @@ -623,6 +688,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>      dk->reset = reset;
>      dk->realize = realize;
>      dk->unrealize = unrealize;
> +    dk->vmsd = &vmstate_spapr_drc;
> +    dk->dev_get_instance_id = get_instance_id;
>      drck->set_isolation_state = set_isolation_state;
>      drck->set_indicator_state = set_indicator_state;
>      drck->set_allocation_state = set_allocation_state;
> @@ -636,6 +703,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>      drck->detach = detach;
>      drck->release_pending = release_pending;
>      drck->set_signalled = set_signalled;
> +    drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb;
> +
>      /*
>       * Reason: it crashes FIXME find and document the real reason
>       */
> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> index 4f00865..080471c 100644
> --- a/hw/ppc/spapr_pci.c
> +++ b/hw/ppc/spapr_pci.c
> @@ -1594,11 +1594,33 @@ static void spapr_pci_pre_save(void *opaque)
>      }
>  }
>  
> +/*
> + * detach_cb in the DRC state is a function pointer that cannot be
> + * migrated. We set it right after migration so that a migrated
> + * hot-unplug event could finish its work.
> + */
> +static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev,
> +                                 void *opaque)
> +{
> +    sPAPRPHBState *sphb = opaque;
> +    sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev);
> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> +    drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb);

Why is spapr_phb_remove_pci_device_cb the right callback rather than
something else?  This doesn't seem sensible.

More to the point, you're not restoring detach_cb_opaque, which means
the detach_cb callback won't work properly anyway.

> +}
> +
>  static int spapr_pci_post_load(void *opaque, int version_id)
>  {
>      sPAPRPHBState *sphb = opaque;
>      gpointer key, value;
>      int i;
> +    PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus;
> +    unsigned int bus_no = 0;
> +
> +    /* Set detach_cb for the drc unconditionally after migration */
> +    if (bus) {
> +        pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb,
> +                            &bus_no);
> +    }
>  
>      for (i = 0; i < sphb->msi_devs_num; ++i) {
>          key = g_memdup(&sphb->msi_devs[i].key,
> diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h
> index fa531d5..17589c8 100644
> --- a/include/hw/ppc/spapr_drc.h
> +++ b/include/hw/ppc/spapr_drc.h
> @@ -192,6 +192,15 @@ typedef struct sPAPRDRConnectorClass {
>                     void *detach_cb_opaque, Error **errp);
>      bool (*release_pending)(sPAPRDRConnector *drc);
>      void (*set_signalled)(sPAPRDRConnector *drc);
> +
> +    /*
> +     * QEMU interface for setting detach_cb after migration.
> +     * detach_cb in the DRC state is a function pointer that cannot be
> +     * migrated. We set it right after migration so that a migrated
> +     * hot-unplug event could finish its work.
> +     */
> +    void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc,
> +                                      spapr_drc_detach_cb *detach_cb);
>  } sPAPRDRConnectorClass;
>  
>  sPAPRDRConnector *spapr_dr_connector_new(Object *owner,

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 2/6] migration: spapr_drc: defined VMStateDescription struct

[Jianjun Duan]

On 10/06/2016 08:12 PM, David Gibson wrote:
> On Mon, Oct 03, 2016 at 11:24:53AM -0700, Jianjun Duan wrote:
>> To manage hotplug/unplug of dynamic resources such as PCI cards,
>> memory, and CPU on sPAPR guests, a firmware abstraction known as
>> a Dynamic Resource Connector (DRC) is used to assign a particular
>> dynamic resource to the guest, and provide an interface for the
>> guest to manage configuration/removal of the resource associated
>> with it.
>>
>> To migrate the hotplugged resources in migration, the
>> associated DRC state need be migrated. To migrate the DRC state,
>> we defined the VMStateDescription struct for spapr_drc to enable
>> the transmission of spapr_drc state in migration.
>>
>> Not all the elements in the DRC state are migrated. Only those
>> ones modifiable or needed by guest actions or device add/remove
>> operation are migrated. From the perspective of device
>> hotplugging, if we hotplug a device on the source, we need to
>> "coldplug" it on the target. The states across two hosts for the
>> same device are not the same. Ideally we want the states be same
>> after migration so that the device would function as hotplugged
>> on the target. For example we can unplug it. The minimum DRC
>> state we need to transfer should cover all the pieces changed by
>> hotplugging. Out of the elements of the DRC state, isolation_state,
>> allocation_sate, and configured are involved in the DR state
>> transition diagram from PAPR+ 2.7, 13.4. configured and signalled
>> are needed in attaching and detaching devices. indicator_state
>> provides users with hardware state information. These 6 elements
>> are migrated.
> 
> Hmm.. are you saying that the DRC state of a coldplugged device (after
> we've fully booted) is different from the DRC state of a hotplugged
> device (after all the hotplug operations have fully completed)?
> 
> If that's correct that sounds like a general bug in the DRC state
> management, not something only related to migration.
> 
> Looking at the code, though, that doesn't really seem to be what it's
> doing.
> 

After hotplugging a device, changes may be made to its DRC state on the
source. But on target the state is fresh. The possible changes are shown
in spapr_drc_needed function.
>> detach_cb in the DRC state is a function pointer that cannot be
>> migrated. We set it right after DRC state is migrated so that
>> a migrated hot-unplug event could finish its work.
>>
>> The instance_id is used to identify objects in migration. We set
>> instance_id of DRC using the unique index so that it is the same
>> across migration.
>>
>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>> ---
>>  hw/ppc/spapr_drc.c         | 69 ++++++++++++++++++++++++++++++++++++++++++++++
>>  hw/ppc/spapr_pci.c         | 22 +++++++++++++++
>>  include/hw/ppc/spapr_drc.h |  9 ++++++
>>  3 files changed, 100 insertions(+)
>>
>> diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c
>> index 6e54fd4..369ec02 100644
>> --- a/hw/ppc/spapr_drc.c
>> +++ b/hw/ppc/spapr_drc.c
>> @@ -615,6 +615,71 @@ static void spapr_dr_connector_instance_init(Object *obj)
>>                          NULL, NULL, NULL, NULL);
>>  }
>>  
>> +static bool spapr_drc_needed(void *opaque)
>> +{
>> +    sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque;
>> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
>> +    bool rc = false;
>> +    sPAPRDREntitySense value;
>> +
>> +    drck->entity_sense(drc, &value);
>> +    /* If no dev is plugged in there is no need to migrate the DRC state */
>> +    if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) {
>> +        return false;
>> +    }
>> +    /*
>> +     * If there is dev plugged in, we need to migrate the DRC state when
>> +     * it is different from cold-plugged state
>> +     */
>> +    switch(drc->type) {
>> +    /* for PCI type */
>> +    case SPAPR_DR_CONNECTOR_TYPE_PCI:
>> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) &&
>> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) &&
>> +               drc->configured && drc->signalled && !drc->awaiting_release);
>> +        break;
>> +    /* for LMB type */
>> +    case SPAPR_DR_CONNECTOR_TYPE_LMB:
>> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) &&
>> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) &&
>> +               drc->configured && drc->signalled && !drc->awaiting_release);
>> +        break;
> 
> What about CPU type?z

For CPU Type, those states don't change from source to host.

>> +    default:
>> +        ;
>> +    }
>> +
>> +    return rc;
>> +}
>> +
>> +/* detach_cb needs be set since it is not migrated */
>> +static void postmigrate_set_detach_cb(sPAPRDRConnector *drc,
>> +                                      spapr_drc_detach_cb *detach_cb)
>> +{
>> +    drc->detach_cb = detach_cb;
>> +}
>> +
>> +/* return the unique drc index as instance_id for qom interfaces*/
>> +static int get_instance_id(DeviceState *dev)
>> +{
>> +    return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev)));
>> +}
>> +
>> +static const VMStateDescription vmstate_spapr_drc = {
>> +    .name = "spapr_drc",
>> +    .version_id = 1,
>> +    .minimum_version_id = 1,
>> +    .needed = spapr_drc_needed,
>> +    .fields  = (VMStateField []) {
>> +        VMSTATE_UINT32(isolation_state, sPAPRDRConnector),
>> +        VMSTATE_UINT32(allocation_state, sPAPRDRConnector),
>> +        VMSTATE_UINT32(indicator_state, sPAPRDRConnector),
>> +        VMSTATE_BOOL(configured, sPAPRDRConnector),
>> +        VMSTATE_BOOL(awaiting_release, sPAPRDRConnector),
>> +        VMSTATE_BOOL(signalled, sPAPRDRConnector),
>> +        VMSTATE_END_OF_LIST()
>> +    }
>> +};
>> +
>>  static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>>  {
>>      DeviceClass *dk = DEVICE_CLASS(k);
>> @@ -623,6 +688,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>>      dk->reset = reset;
>>      dk->realize = realize;
>>      dk->unrealize = unrealize;
>> +    dk->vmsd = &vmstate_spapr_drc;
>> +    dk->dev_get_instance_id = get_instance_id;
>>      drck->set_isolation_state = set_isolation_state;
>>      drck->set_indicator_state = set_indicator_state;
>>      drck->set_allocation_state = set_allocation_state;
>> @@ -636,6 +703,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>>      drck->detach = detach;
>>      drck->release_pending = release_pending;
>>      drck->set_signalled = set_signalled;
>> +    drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb;
>> +
>>      /*
>>       * Reason: it crashes FIXME find and document the real reason
>>       */
>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>> index 4f00865..080471c 100644
>> --- a/hw/ppc/spapr_pci.c
>> +++ b/hw/ppc/spapr_pci.c
>> @@ -1594,11 +1594,33 @@ static void spapr_pci_pre_save(void *opaque)
>>      }
>>  }
>>  
>> +/*
>> + * detach_cb in the DRC state is a function pointer that cannot be
>> + * migrated. We set it right after migration so that a migrated
>> + * hot-unplug event could finish its work.
>> + */
>> +static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev,
>> +                                 void *opaque)
>> +{
>> +    sPAPRPHBState *sphb = opaque;
>> +    sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev);
>> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
>> +    drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb);
> 
> Why is spapr_phb_remove_pci_device_cb the right callback rather than
> something else?  This doesn't seem sensible.
> 
> More to the point, you're not restoring detach_cb_opaque, which means
> the detach_cb callback won't work properly anyway.
> 
If set_isolation_state/set_allocation_state is called on target without
spapr_phb_remove_pci_device being called earlier, drc->detach_cb is
null. Its value is spapr_phb_remove_pci_device_cb and is set in
spapr_phb_remove_pci_device.

You are right about detach_cb. It needs be restored.

>> +}
>> +
>>  static int spapr_pci_post_load(void *opaque, int version_id)
>>  {
>>      sPAPRPHBState *sphb = opaque;
>>      gpointer key, value;
>>      int i;
>> +    PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus;
>> +    unsigned int bus_no = 0;
>> +
>> +    /* Set detach_cb for the drc unconditionally after migration */
>> +    if (bus) {
>> +        pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb,
>> +                            &bus_no);
>> +    }
>>  
>>      for (i = 0; i < sphb->msi_devs_num; ++i) {
>>          key = g_memdup(&sphb->msi_devs[i].key,
>> diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h
>> index fa531d5..17589c8 100644
>> --- a/include/hw/ppc/spapr_drc.h
>> +++ b/include/hw/ppc/spapr_drc.h
>> @@ -192,6 +192,15 @@ typedef struct sPAPRDRConnectorClass {
>>                     void *detach_cb_opaque, Error **errp);
>>      bool (*release_pending)(sPAPRDRConnector *drc);
>>      void (*set_signalled)(sPAPRDRConnector *drc);
>> +
>> +    /*
>> +     * QEMU interface for setting detach_cb after migration.
>> +     * detach_cb in the DRC state is a function pointer that cannot be
>> +     * migrated. We set it right after migration so that a migrated
>> +     * hot-unplug event could finish its work.
>> +     */
>> +    void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc,
>> +                                      spapr_drc_detach_cb *detach_cb);
>>  } sPAPRDRConnectorClass;
>>  
>>  sPAPRDRConnector *spapr_dr_connector_new(Object *owner,
> 

Thanks,
Jianjun

Re: [Qemu-devel] [QEMU PATCH v5 2/6] migration: spapr_drc: defined VMStateDescription struct

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 11368 bytes --]

On Fri, Oct 07, 2016 at 10:17:45AM -0700, Jianjun Duan wrote:
> 
> 
> On 10/06/2016 08:12 PM, David Gibson wrote:
> > On Mon, Oct 03, 2016 at 11:24:53AM -0700, Jianjun Duan wrote:
> >> To manage hotplug/unplug of dynamic resources such as PCI cards,
> >> memory, and CPU on sPAPR guests, a firmware abstraction known as
> >> a Dynamic Resource Connector (DRC) is used to assign a particular
> >> dynamic resource to the guest, and provide an interface for the
> >> guest to manage configuration/removal of the resource associated
> >> with it.
> >>
> >> To migrate the hotplugged resources in migration, the
> >> associated DRC state need be migrated. To migrate the DRC state,
> >> we defined the VMStateDescription struct for spapr_drc to enable
> >> the transmission of spapr_drc state in migration.
> >>
> >> Not all the elements in the DRC state are migrated. Only those
> >> ones modifiable or needed by guest actions or device add/remove
> >> operation are migrated. From the perspective of device
> >> hotplugging, if we hotplug a device on the source, we need to
> >> "coldplug" it on the target. The states across two hosts for the
> >> same device are not the same. Ideally we want the states be same
> >> after migration so that the device would function as hotplugged
> >> on the target. For example we can unplug it. The minimum DRC
> >> state we need to transfer should cover all the pieces changed by
> >> hotplugging. Out of the elements of the DRC state, isolation_state,
> >> allocation_sate, and configured are involved in the DR state
> >> transition diagram from PAPR+ 2.7, 13.4. configured and signalled
> >> are needed in attaching and detaching devices. indicator_state
> >> provides users with hardware state information. These 6 elements
> >> are migrated.
> > 
> > Hmm.. are you saying that the DRC state of a coldplugged device (after
> > we've fully booted) is different from the DRC state of a hotplugged
> > device (after all the hotplug operations have fully completed)?
> > 
> > If that's correct that sounds like a general bug in the DRC state
> > management, not something only related to migration.
> > 
> > Looking at the code, though, that doesn't really seem to be what it's
> > doing.
> > 
> 
> After hotplugging a device, changes may be made to its DRC state on the
> source. But on target the state is fresh. The possible changes are shown
> in spapr_drc_needed function.

Ok.  If you can somehow include the content of the paragraph above in
your commit messages that would make the rationale clearer.

> >> detach_cb in the DRC state is a function pointer that cannot be
> >> migrated. We set it right after DRC state is migrated so that
> >> a migrated hot-unplug event could finish its work.
> >>
> >> The instance_id is used to identify objects in migration. We set
> >> instance_id of DRC using the unique index so that it is the same
> >> across migration.
> >>
> >> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> >> ---
> >>  hw/ppc/spapr_drc.c         | 69 ++++++++++++++++++++++++++++++++++++++++++++++
> >>  hw/ppc/spapr_pci.c         | 22 +++++++++++++++
> >>  include/hw/ppc/spapr_drc.h |  9 ++++++
> >>  3 files changed, 100 insertions(+)
> >>
> >> diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c
> >> index 6e54fd4..369ec02 100644
> >> --- a/hw/ppc/spapr_drc.c
> >> +++ b/hw/ppc/spapr_drc.c
> >> @@ -615,6 +615,71 @@ static void spapr_dr_connector_instance_init(Object *obj)
> >>                          NULL, NULL, NULL, NULL);
> >>  }
> >>  
> >> +static bool spapr_drc_needed(void *opaque)
> >> +{
> >> +    sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque;
> >> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> >> +    bool rc = false;
> >> +    sPAPRDREntitySense value;
> >> +
> >> +    drck->entity_sense(drc, &value);
> >> +    /* If no dev is plugged in there is no need to migrate the DRC state */
> >> +    if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) {
> >> +        return false;
> >> +    }
> >> +    /*
> >> +     * If there is dev plugged in, we need to migrate the DRC state when
> >> +     * it is different from cold-plugged state
> >> +     */
> >> +    switch(drc->type) {
> >> +    /* for PCI type */
> >> +    case SPAPR_DR_CONNECTOR_TYPE_PCI:
> >> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) &&
> >> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) &&
> >> +               drc->configured && drc->signalled && !drc->awaiting_release);
> >> +        break;
> >> +    /* for LMB type */
> >> +    case SPAPR_DR_CONNECTOR_TYPE_LMB:
> >> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) &&
> >> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) &&
> >> +               drc->configured && drc->signalled && !drc->awaiting_release);
> >> +        break;
> > 
> > What about CPU type?z
> 
> For CPU Type, those states don't change from source to host.

Is that a property of the CPU type specifically, or is that just
because current guests don't mess with the CPU DRC states the way they
do with other devices?

> >> +    default:
> >> +        ;
> >> +    }
> >> +
> >> +    return rc;
> >> +}
> >> +
> >> +/* detach_cb needs be set since it is not migrated */
> >> +static void postmigrate_set_detach_cb(sPAPRDRConnector *drc,
> >> +                                      spapr_drc_detach_cb *detach_cb)
> >> +{
> >> +    drc->detach_cb = detach_cb;
> >> +}
> >> +
> >> +/* return the unique drc index as instance_id for qom interfaces*/
> >> +static int get_instance_id(DeviceState *dev)
> >> +{
> >> +    return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev)));
> >> +}
> >> +
> >> +static const VMStateDescription vmstate_spapr_drc = {
> >> +    .name = "spapr_drc",
> >> +    .version_id = 1,
> >> +    .minimum_version_id = 1,
> >> +    .needed = spapr_drc_needed,
> >> +    .fields  = (VMStateField []) {
> >> +        VMSTATE_UINT32(isolation_state, sPAPRDRConnector),
> >> +        VMSTATE_UINT32(allocation_state, sPAPRDRConnector),
> >> +        VMSTATE_UINT32(indicator_state, sPAPRDRConnector),
> >> +        VMSTATE_BOOL(configured, sPAPRDRConnector),
> >> +        VMSTATE_BOOL(awaiting_release, sPAPRDRConnector),
> >> +        VMSTATE_BOOL(signalled, sPAPRDRConnector),
> >> +        VMSTATE_END_OF_LIST()
> >> +    }
> >> +};
> >> +
> >>  static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
> >>  {
> >>      DeviceClass *dk = DEVICE_CLASS(k);
> >> @@ -623,6 +688,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
> >>      dk->reset = reset;
> >>      dk->realize = realize;
> >>      dk->unrealize = unrealize;
> >> +    dk->vmsd = &vmstate_spapr_drc;
> >> +    dk->dev_get_instance_id = get_instance_id;
> >>      drck->set_isolation_state = set_isolation_state;
> >>      drck->set_indicator_state = set_indicator_state;
> >>      drck->set_allocation_state = set_allocation_state;
> >> @@ -636,6 +703,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
> >>      drck->detach = detach;
> >>      drck->release_pending = release_pending;
> >>      drck->set_signalled = set_signalled;
> >> +    drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb;
> >> +
> >>      /*
> >>       * Reason: it crashes FIXME find and document the real reason
> >>       */
> >> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
> >> index 4f00865..080471c 100644
> >> --- a/hw/ppc/spapr_pci.c
> >> +++ b/hw/ppc/spapr_pci.c
> >> @@ -1594,11 +1594,33 @@ static void spapr_pci_pre_save(void *opaque)
> >>      }
> >>  }
> >>  
> >> +/*
> >> + * detach_cb in the DRC state is a function pointer that cannot be
> >> + * migrated. We set it right after migration so that a migrated
> >> + * hot-unplug event could finish its work.
> >> + */
> >> +static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev,
> >> +                                 void *opaque)
> >> +{
> >> +    sPAPRPHBState *sphb = opaque;
> >> +    sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev);
> >> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
> >> +    drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb);
> > 
> > Why is spapr_phb_remove_pci_device_cb the right callback rather than
> > something else?  This doesn't seem sensible.
> > 
> > More to the point, you're not restoring detach_cb_opaque, which means
> > the detach_cb callback won't work properly anyway.
> > 
> If set_isolation_state/set_allocation_state is called on target without
> spapr_phb_remove_pci_device being called earlier, drc->detach_cb is
> null. Its value is spapr_phb_remove_pci_device_cb and is set in
> spapr_phb_remove_pci_device.
> 
> You are right about detach_cb. It needs be restored.

Right.

It sounds like detach_cb is doing double duty - it is pointing to what
the right detach function is, and also acting as a flag to say if a
detach operation is necessary.  This doesn't play well with migration.

In order to do sane migration, I suspect we need to split this into
different components - the callback value can maybe move to the class
instead of the DRC instance (or just use a switch on drc_type).  The
flag can just become a boolean which we can migrate.

> >> +}
> >> +
> >>  static int spapr_pci_post_load(void *opaque, int version_id)
> >>  {
> >>      sPAPRPHBState *sphb = opaque;
> >>      gpointer key, value;
> >>      int i;
> >> +    PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus;
> >> +    unsigned int bus_no = 0;
> >> +
> >> +    /* Set detach_cb for the drc unconditionally after migration */
> >> +    if (bus) {
> >> +        pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb,
> >> +                            &bus_no);
> >> +    }
> >>  
> >>      for (i = 0; i < sphb->msi_devs_num; ++i) {
> >>          key = g_memdup(&sphb->msi_devs[i].key,
> >> diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h
> >> index fa531d5..17589c8 100644
> >> --- a/include/hw/ppc/spapr_drc.h
> >> +++ b/include/hw/ppc/spapr_drc.h
> >> @@ -192,6 +192,15 @@ typedef struct sPAPRDRConnectorClass {
> >>                     void *detach_cb_opaque, Error **errp);
> >>      bool (*release_pending)(sPAPRDRConnector *drc);
> >>      void (*set_signalled)(sPAPRDRConnector *drc);
> >> +
> >> +    /*
> >> +     * QEMU interface for setting detach_cb after migration.
> >> +     * detach_cb in the DRC state is a function pointer that cannot be
> >> +     * migrated. We set it right after migration so that a migrated
> >> +     * hot-unplug event could finish its work.
> >> +     */
> >> +    void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc,
> >> +                                      spapr_drc_detach_cb *detach_cb);
> >>  } sPAPRDRConnectorClass;
> >>  
> >>  sPAPRDRConnector *spapr_dr_connector_new(Object *owner,
> > 
> 
> Thanks,
> Jianjun
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 2/6] migration: spapr_drc: defined VMStateDescription struct

[Jianjun Duan]

On 10/09/2016 10:09 PM, David Gibson wrote:
> On Fri, Oct 07, 2016 at 10:17:45AM -0700, Jianjun Duan wrote:
>>
>>
>> On 10/06/2016 08:12 PM, David Gibson wrote:
>>> On Mon, Oct 03, 2016 at 11:24:53AM -0700, Jianjun Duan wrote:
>>>> To manage hotplug/unplug of dynamic resources such as PCI cards,
>>>> memory, and CPU on sPAPR guests, a firmware abstraction known as
>>>> a Dynamic Resource Connector (DRC) is used to assign a particular
>>>> dynamic resource to the guest, and provide an interface for the
>>>> guest to manage configuration/removal of the resource associated
>>>> with it.
>>>>
>>>> To migrate the hotplugged resources in migration, the
>>>> associated DRC state need be migrated. To migrate the DRC state,
>>>> we defined the VMStateDescription struct for spapr_drc to enable
>>>> the transmission of spapr_drc state in migration.
>>>>
>>>> Not all the elements in the DRC state are migrated. Only those
>>>> ones modifiable or needed by guest actions or device add/remove
>>>> operation are migrated. From the perspective of device
>>>> hotplugging, if we hotplug a device on the source, we need to
>>>> "coldplug" it on the target. The states across two hosts for the
>>>> same device are not the same. Ideally we want the states be same
>>>> after migration so that the device would function as hotplugged
>>>> on the target. For example we can unplug it. The minimum DRC
>>>> state we need to transfer should cover all the pieces changed by
>>>> hotplugging. Out of the elements of the DRC state, isolation_state,
>>>> allocation_sate, and configured are involved in the DR state
>>>> transition diagram from PAPR+ 2.7, 13.4. configured and signalled
>>>> are needed in attaching and detaching devices. indicator_state
>>>> provides users with hardware state information. These 6 elements
>>>> are migrated.
>>>
>>> Hmm.. are you saying that the DRC state of a coldplugged device (after
>>> we've fully booted) is different from the DRC state of a hotplugged
>>> device (after all the hotplug operations have fully completed)?
>>>
>>> If that's correct that sounds like a general bug in the DRC state
>>> management, not something only related to migration.
>>>
>>> Looking at the code, though, that doesn't really seem to be what it's
>>> doing.
>>>
>>
>> After hotplugging a device, changes may be made to its DRC state on the
>> source. But on target the state is fresh. The possible changes are shown
>> in spapr_drc_needed function.
> 
> Ok.  If you can somehow include the content of the paragraph above in
> your commit messages that would make the rationale clearer.
> 
Will do.
>>>> detach_cb in the DRC state is a function pointer that cannot be
>>>> migrated. We set it right after DRC state is migrated so that
>>>> a migrated hot-unplug event could finish its work.
>>>>
>>>> The instance_id is used to identify objects in migration. We set
>>>> instance_id of DRC using the unique index so that it is the same
>>>> across migration.
>>>>
>>>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>>>> ---
>>>>  hw/ppc/spapr_drc.c         | 69 ++++++++++++++++++++++++++++++++++++++++++++++
>>>>  hw/ppc/spapr_pci.c         | 22 +++++++++++++++
>>>>  include/hw/ppc/spapr_drc.h |  9 ++++++
>>>>  3 files changed, 100 insertions(+)
>>>>
>>>> diff --git a/hw/ppc/spapr_drc.c b/hw/ppc/spapr_drc.c
>>>> index 6e54fd4..369ec02 100644
>>>> --- a/hw/ppc/spapr_drc.c
>>>> +++ b/hw/ppc/spapr_drc.c
>>>> @@ -615,6 +615,71 @@ static void spapr_dr_connector_instance_init(Object *obj)
>>>>                          NULL, NULL, NULL, NULL);
>>>>  }
>>>>  
>>>> +static bool spapr_drc_needed(void *opaque)
>>>> +{
>>>> +    sPAPRDRConnector *drc = (sPAPRDRConnector *)opaque;
>>>> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
>>>> +    bool rc = false;
>>>> +    sPAPRDREntitySense value;
>>>> +
>>>> +    drck->entity_sense(drc, &value);
>>>> +    /* If no dev is plugged in there is no need to migrate the DRC state */
>>>> +    if (value != SPAPR_DR_ENTITY_SENSE_PRESENT) {
>>>> +        return false;
>>>> +    }
>>>> +    /*
>>>> +     * If there is dev plugged in, we need to migrate the DRC state when
>>>> +     * it is different from cold-plugged state
>>>> +     */
>>>> +    switch(drc->type) {
>>>> +    /* for PCI type */
>>>> +    case SPAPR_DR_CONNECTOR_TYPE_PCI:
>>>> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_UNISOLATED) &&
>>>> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_USABLE) &&
>>>> +               drc->configured && drc->signalled && !drc->awaiting_release);
>>>> +        break;
>>>> +    /* for LMB type */
>>>> +    case SPAPR_DR_CONNECTOR_TYPE_LMB:
>>>> +        rc = !((drc->isolation_state == SPAPR_DR_ISOLATION_STATE_ISOLATED) &&
>>>> +               (drc->allocation_state == SPAPR_DR_ALLOCATION_STATE_UNUSABLE) &&
>>>> +               drc->configured && drc->signalled && !drc->awaiting_release);
>>>> +        break;
>>>
>>> What about CPU type?z
>>
>> For CPU Type, those states don't change from source to host.
> 
> Is that a property of the CPU type specifically, or is that just
> because current guests don't mess with the CPU DRC states the way they
> do with other devices?
> 
Need to double check. From the experiments I had done, I didn't see any
changes.
>>>> +    default:
>>>> +        ;
>>>> +    }
>>>> +
>>>> +    return rc;
>>>> +}
>>>> +
>>>> +/* detach_cb needs be set since it is not migrated */
>>>> +static void postmigrate_set_detach_cb(sPAPRDRConnector *drc,
>>>> +                                      spapr_drc_detach_cb *detach_cb)
>>>> +{
>>>> +    drc->detach_cb = detach_cb;
>>>> +}
>>>> +
>>>> +/* return the unique drc index as instance_id for qom interfaces*/
>>>> +static int get_instance_id(DeviceState *dev)
>>>> +{
>>>> +    return (int)get_index(SPAPR_DR_CONNECTOR(OBJECT(dev)));
>>>> +}
>>>> +
>>>> +static const VMStateDescription vmstate_spapr_drc = {
>>>> +    .name = "spapr_drc",
>>>> +    .version_id = 1,
>>>> +    .minimum_version_id = 1,
>>>> +    .needed = spapr_drc_needed,
>>>> +    .fields  = (VMStateField []) {
>>>> +        VMSTATE_UINT32(isolation_state, sPAPRDRConnector),
>>>> +        VMSTATE_UINT32(allocation_state, sPAPRDRConnector),
>>>> +        VMSTATE_UINT32(indicator_state, sPAPRDRConnector),
>>>> +        VMSTATE_BOOL(configured, sPAPRDRConnector),
>>>> +        VMSTATE_BOOL(awaiting_release, sPAPRDRConnector),
>>>> +        VMSTATE_BOOL(signalled, sPAPRDRConnector),
>>>> +        VMSTATE_END_OF_LIST()
>>>> +    }
>>>> +};
>>>> +
>>>>  static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>>>>  {
>>>>      DeviceClass *dk = DEVICE_CLASS(k);
>>>> @@ -623,6 +688,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>>>>      dk->reset = reset;
>>>>      dk->realize = realize;
>>>>      dk->unrealize = unrealize;
>>>> +    dk->vmsd = &vmstate_spapr_drc;
>>>> +    dk->dev_get_instance_id = get_instance_id;
>>>>      drck->set_isolation_state = set_isolation_state;
>>>>      drck->set_indicator_state = set_indicator_state;
>>>>      drck->set_allocation_state = set_allocation_state;
>>>> @@ -636,6 +703,8 @@ static void spapr_dr_connector_class_init(ObjectClass *k, void *data)
>>>>      drck->detach = detach;
>>>>      drck->release_pending = release_pending;
>>>>      drck->set_signalled = set_signalled;
>>>> +    drck->postmigrate_set_detach_cb = postmigrate_set_detach_cb;
>>>> +
>>>>      /*
>>>>       * Reason: it crashes FIXME find and document the real reason
>>>>       */
>>>> diff --git a/hw/ppc/spapr_pci.c b/hw/ppc/spapr_pci.c
>>>> index 4f00865..080471c 100644
>>>> --- a/hw/ppc/spapr_pci.c
>>>> +++ b/hw/ppc/spapr_pci.c
>>>> @@ -1594,11 +1594,33 @@ static void spapr_pci_pre_save(void *opaque)
>>>>      }
>>>>  }
>>>>  
>>>> +/*
>>>> + * detach_cb in the DRC state is a function pointer that cannot be
>>>> + * migrated. We set it right after migration so that a migrated
>>>> + * hot-unplug event could finish its work.
>>>> + */
>>>> +static void spapr_pci_set_detach_cb(PCIBus *bus, PCIDevice *pdev,
>>>> +                                 void *opaque)
>>>> +{
>>>> +    sPAPRPHBState *sphb = opaque;
>>>> +    sPAPRDRConnector *drc = spapr_phb_get_pci_drc(sphb, pdev);
>>>> +    sPAPRDRConnectorClass *drck = SPAPR_DR_CONNECTOR_GET_CLASS(drc);
>>>> +    drck->postmigrate_set_detach_cb(drc, spapr_phb_remove_pci_device_cb);
>>>
>>> Why is spapr_phb_remove_pci_device_cb the right callback rather than
>>> something else?  This doesn't seem sensible.
>>>
>>> More to the point, you're not restoring detach_cb_opaque, which means
>>> the detach_cb callback won't work properly anyway.
>>>
>> If set_isolation_state/set_allocation_state is called on target without
>> spapr_phb_remove_pci_device being called earlier, drc->detach_cb is
>> null. Its value is spapr_phb_remove_pci_device_cb and is set in
>> spapr_phb_remove_pci_device.
>>
>> You are right about detach_cb. It needs be restored.
> 
> Right.
> 
> It sounds like detach_cb is doing double duty - it is pointing to what
> the right detach function is, and also acting as a flag to say if a
> detach operation is necessary.  This doesn't play well with migration.
> 
> In order to do sane migration, I suspect we need to split this into
> different components - the callback value can maybe move to the class
> instead of the DRC instance (or just use a switch on drc_type).  The
> flag can just become a boolean which we can migrate.
> 

Maybe we can set all these callbacks when DRC is inited?


Thanks,
Jianjun
>>>> +}
>>>> +
>>>>  static int spapr_pci_post_load(void *opaque, int version_id)
>>>>  {
>>>>      sPAPRPHBState *sphb = opaque;
>>>>      gpointer key, value;
>>>>      int i;
>>>> +    PCIBus *bus = PCI_HOST_BRIDGE(sphb)->bus;
>>>> +    unsigned int bus_no = 0;
>>>> +
>>>> +    /* Set detach_cb for the drc unconditionally after migration */
>>>> +    if (bus) {
>>>> +        pci_for_each_device(bus, pci_bus_num(bus), spapr_pci_set_detach_cb,
>>>> +                            &bus_no);
>>>> +    }
>>>>  
>>>>      for (i = 0; i < sphb->msi_devs_num; ++i) {
>>>>          key = g_memdup(&sphb->msi_devs[i].key,
>>>> diff --git a/include/hw/ppc/spapr_drc.h b/include/hw/ppc/spapr_drc.h
>>>> index fa531d5..17589c8 100644
>>>> --- a/include/hw/ppc/spapr_drc.h
>>>> +++ b/include/hw/ppc/spapr_drc.h
>>>> @@ -192,6 +192,15 @@ typedef struct sPAPRDRConnectorClass {
>>>>                     void *detach_cb_opaque, Error **errp);
>>>>      bool (*release_pending)(sPAPRDRConnector *drc);
>>>>      void (*set_signalled)(sPAPRDRConnector *drc);
>>>> +
>>>> +    /*
>>>> +     * QEMU interface for setting detach_cb after migration.
>>>> +     * detach_cb in the DRC state is a function pointer that cannot be
>>>> +     * migrated. We set it right after migration so that a migrated
>>>> +     * hot-unplug event could finish its work.
>>>> +     */
>>>> +    void (*postmigrate_set_detach_cb)(sPAPRDRConnector *drc,
>>>> +                                      spapr_drc_detach_cb *detach_cb);
>>>>  } sPAPRDRConnectorClass;
>>>>  
>>>>  sPAPRDRConnector *spapr_dr_connector_new(Object *owner,
>>>
>>
>> Thanks,
>> Jianjun
>>
> 

[Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Jianjun Duan]

Current migration code cannot handle some data structures such as
QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
in VMStateInfo so that customized handling is supported.

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
---
 hw/net/vmxnet3.c            | 18 ++++++---
 hw/nvram/eeprom93xx.c       |  6 ++-
 hw/nvram/fw_cfg.c           |  6 ++-
 hw/pci/msix.c               |  6 ++-
 hw/pci/pci.c                | 12 ++++--
 hw/pci/shpc.c               |  5 ++-
 hw/scsi/scsi-bus.c          |  6 ++-
 hw/timer/twl92230.c         |  6 ++-
 hw/usb/redirect.c           | 18 ++++++---
 hw/virtio/virtio-pci.c      |  6 ++-
 hw/virtio/virtio.c          |  6 ++-
 include/migration/vmstate.h | 10 +++--
 migration/savevm.c          |  5 ++-
 migration/vmstate.c         | 95 ++++++++++++++++++++++++++++-----------------
 target-alpha/machine.c      |  5 ++-
 target-arm/machine.c        | 12 ++++--
 target-i386/machine.c       | 21 ++++++----
 target-mips/machine.c       | 10 +++--
 target-ppc/machine.c        | 10 +++--
 target-sparc/machine.c      |  5 ++-
 20 files changed, 171 insertions(+), 97 deletions(-)

diff --git a/hw/net/vmxnet3.c b/hw/net/vmxnet3.c
index 90f6943..943a960 100644
--- a/hw/net/vmxnet3.c
+++ b/hw/net/vmxnet3.c
@@ -2450,7 +2450,8 @@ static void vmxnet3_put_tx_stats_to_file(QEMUFile *f,
     qemu_put_be64(f, tx_stat->pktsTxDiscard);
 }
 
-static int vmxnet3_get_txq_descr(QEMUFile *f, void *pv, size_t size)
+static int vmxnet3_get_txq_descr(QEMUFile *f, void *pv, size_t size,
+    VMStateField *field)
 {
     Vmxnet3TxqDescr *r = pv;
 
@@ -2464,7 +2465,8 @@ static int vmxnet3_get_txq_descr(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void vmxnet3_put_txq_descr(QEMUFile *f, void *pv, size_t size)
+static void vmxnet3_put_txq_descr(QEMUFile *f, void *pv, size_t size,
+    VMStateField *field, QJSON *vmdesc)
 {
     Vmxnet3TxqDescr *r = pv;
 
@@ -2511,7 +2513,8 @@ static void vmxnet3_put_rx_stats_to_file(QEMUFile *f,
     qemu_put_be64(f, rx_stat->pktsRxError);
 }
 
-static int vmxnet3_get_rxq_descr(QEMUFile *f, void *pv, size_t size)
+static int vmxnet3_get_rxq_descr(QEMUFile *f, void *pv, size_t size,
+    VMStateField *field)
 {
     Vmxnet3RxqDescr *r = pv;
     int i;
@@ -2529,7 +2532,8 @@ static int vmxnet3_get_rxq_descr(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void vmxnet3_put_rxq_descr(QEMUFile *f, void *pv, size_t size)
+static void vmxnet3_put_rxq_descr(QEMUFile *f, void *pv, size_t size,
+    VMStateField *field, QJSON *vmdesc)
 {
     Vmxnet3RxqDescr *r = pv;
     int i;
@@ -2574,7 +2578,8 @@ static const VMStateInfo rxq_descr_info = {
     .put = vmxnet3_put_rxq_descr
 };
 
-static int vmxnet3_get_int_state(QEMUFile *f, void *pv, size_t size)
+static int vmxnet3_get_int_state(QEMUFile *f, void *pv, size_t size,
+    VMStateField *field)
 {
     Vmxnet3IntState *r = pv;
 
@@ -2585,7 +2590,8 @@ static int vmxnet3_get_int_state(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void vmxnet3_put_int_state(QEMUFile *f, void *pv, size_t size)
+static void vmxnet3_put_int_state(QEMUFile *f, void *pv, size_t size,
+    VMStateField *field, QJSON *vmdesc)
 {
     Vmxnet3IntState *r = pv;
 
diff --git a/hw/nvram/eeprom93xx.c b/hw/nvram/eeprom93xx.c
index 2c16fc2..76d5f41 100644
--- a/hw/nvram/eeprom93xx.c
+++ b/hw/nvram/eeprom93xx.c
@@ -94,14 +94,16 @@ struct _eeprom_t {
    This is a Big hack, but it is how the old state did it.
  */
 
-static int get_uint16_from_uint8(QEMUFile *f, void *pv, size_t size)
+static int get_uint16_from_uint8(QEMUFile *f, void *pv, size_t size,
+                                 VMStateField *field)
 {
     uint16_t *v = pv;
     *v = qemu_get_ubyte(f);
     return 0;
 }
 
-static void put_unused(QEMUFile *f, void *pv, size_t size)
+static void put_unused(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                       QJSON *vmdesc)
 {
     fprintf(stderr, "uint16_from_uint8 is used only for backwards compatibility.\n");
     fprintf(stderr, "Never should be used to write a new state.\n");
diff --git a/hw/nvram/fw_cfg.c b/hw/nvram/fw_cfg.c
index 92aa563..a8a4a7a 100644
--- a/hw/nvram/fw_cfg.c
+++ b/hw/nvram/fw_cfg.c
@@ -524,14 +524,16 @@ static void fw_cfg_reset(DeviceState *d)
    Or we broke compatibility in the state, or we can't use struct tm
  */
 
-static int get_uint32_as_uint16(QEMUFile *f, void *pv, size_t size)
+static int get_uint32_as_uint16(QEMUFile *f, void *pv, size_t size,
+                                VMStateField *field)
 {
     uint32_t *v = pv;
     *v = qemu_get_be16(f);
     return 0;
 }
 
-static void put_unused(QEMUFile *f, void *pv, size_t size)
+static void put_unused(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                       QJSON *vmdesc)
 {
     fprintf(stderr, "uint32_as_uint16 is only used for backward compatibility.\n");
     fprintf(stderr, "This functions shouldn't be called.\n");
diff --git a/hw/pci/msix.c b/hw/pci/msix.c
index 0ec1cb1..69e7a50 100644
--- a/hw/pci/msix.c
+++ b/hw/pci/msix.c
@@ -587,12 +587,14 @@ void msix_unset_vector_notifiers(PCIDevice *dev)
     dev->msix_vector_poll_notifier = NULL;
 }
 
-static void put_msix_state(QEMUFile *f, void *pv, size_t size)
+static void put_msix_state(QEMUFile *f, void *pv, size_t size,
+                           VMStateField *field, QJSON *vmdesc)
 {
     msix_save(pv, f);
 }
 
-static int get_msix_state(QEMUFile *f, void *pv, size_t size)
+static int get_msix_state(QEMUFile *f, void *pv, size_t size,
+                          VMStateField *field)
 {
     msix_load(pv, f);
     return 0;
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 24fae16..08c4547 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -445,7 +445,8 @@ int pci_bus_numa_node(PCIBus *bus)
     return PCI_BUS_GET_CLASS(bus)->numa_node(bus);
 }
 
-static int get_pci_config_device(QEMUFile *f, void *pv, size_t size)
+static int get_pci_config_device(QEMUFile *f, void *pv, size_t size,
+                                 VMStateField *field)
 {
     PCIDevice *s = container_of(pv, PCIDevice, config);
     PCIDeviceClass *pc = PCI_DEVICE_GET_CLASS(s);
@@ -484,7 +485,8 @@ static int get_pci_config_device(QEMUFile *f, void *pv, size_t size)
 }
 
 /* just put buffer */
-static void put_pci_config_device(QEMUFile *f, void *pv, size_t size)
+static void put_pci_config_device(QEMUFile *f, void *pv, size_t size,
+                                  VMStateField *field, QJSON *vmdesc)
 {
     const uint8_t **v = pv;
     assert(size == pci_config_size(container_of(pv, PCIDevice, config)));
@@ -497,7 +499,8 @@ static VMStateInfo vmstate_info_pci_config = {
     .put  = put_pci_config_device,
 };
 
-static int get_pci_irq_state(QEMUFile *f, void *pv, size_t size)
+static int get_pci_irq_state(QEMUFile *f, void *pv, size_t size,
+                             VMStateField *field)
 {
     PCIDevice *s = container_of(pv, PCIDevice, irq_state);
     uint32_t irq_state[PCI_NUM_PINS];
@@ -518,7 +521,8 @@ static int get_pci_irq_state(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_pci_irq_state(QEMUFile *f, void *pv, size_t size)
+static void put_pci_irq_state(QEMUFile *f, void *pv, size_t size,
+                              VMStateField *field, QJSON *vmdesc)
 {
     int i;
     PCIDevice *s = container_of(pv, PCIDevice, irq_state);
diff --git a/hw/pci/shpc.c b/hw/pci/shpc.c
index 3dcd472..9f82aa6 100644
--- a/hw/pci/shpc.c
+++ b/hw/pci/shpc.c
@@ -695,13 +695,14 @@ void shpc_cap_write_config(PCIDevice *d, uint32_t addr, uint32_t val, int l)
     shpc_cap_update_dword(d);
 }
 
-static void shpc_save(QEMUFile *f, void *pv, size_t size)
+static void shpc_save(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                QJSON *vmdesc)
 {
     PCIDevice *d = container_of(pv, PCIDevice, shpc);
     qemu_put_buffer(f, d->shpc->config, SHPC_SIZEOF(d));
 }
 
-static int shpc_load(QEMUFile *f, void *pv, size_t size)
+static int shpc_load(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     PCIDevice *d = container_of(pv, PCIDevice, shpc);
     int ret = qemu_get_buffer(f, d->shpc->config, SHPC_SIZEOF(d));
diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c
index 297216d..f40c10b 100644
--- a/hw/scsi/scsi-bus.c
+++ b/hw/scsi/scsi-bus.c
@@ -1945,7 +1945,8 @@ SCSIDevice *scsi_device_find(SCSIBus *bus, int channel, int id, int lun)
 
 /* SCSI request list.  For simplicity, pv points to the whole device */
 
-static void put_scsi_requests(QEMUFile *f, void *pv, size_t size)
+static void put_scsi_requests(QEMUFile *f, void *pv, size_t size,
+                              VMStateField *field, QJSON *vmdesc)
 {
     SCSIDevice *s = pv;
     SCSIBus *bus = DO_UPCAST(SCSIBus, qbus, s->qdev.parent_bus);
@@ -1970,7 +1971,8 @@ static void put_scsi_requests(QEMUFile *f, void *pv, size_t size)
     qemu_put_sbyte(f, 0);
 }
 
-static int get_scsi_requests(QEMUFile *f, void *pv, size_t size)
+static int get_scsi_requests(QEMUFile *f, void *pv, size_t size,
+                             VMStateField *field)
 {
     SCSIDevice *s = pv;
     SCSIBus *bus = DO_UPCAST(SCSIBus, qbus, s->qdev.parent_bus);
diff --git a/hw/timer/twl92230.c b/hw/timer/twl92230.c
index 7ba4e9a..95eb7f3 100644
--- a/hw/timer/twl92230.c
+++ b/hw/timer/twl92230.c
@@ -747,14 +747,16 @@ static int menelaus_rx(I2CSlave *i2c)
    Or we broke compatibility in the state, or we can't use struct tm
  */
 
-static int get_int32_as_uint16(QEMUFile *f, void *pv, size_t size)
+static int get_int32_as_uint16(QEMUFile *f, void *pv, size_t size,
+                               VMStateField *field)
 {
     int *v = pv;
     *v = qemu_get_be16(f);
     return 0;
 }
 
-static void put_int32_as_uint16(QEMUFile *f, void *pv, size_t size)
+static void put_int32_as_uint16(QEMUFile *f, void *pv, size_t size,
+                                VMStateField *field, QJSON *vmdesc)
 {
     int *v = pv;
     qemu_put_be16(f, *v);
diff --git a/hw/usb/redirect.c b/hw/usb/redirect.c
index 444672a..2ca4b46 100644
--- a/hw/usb/redirect.c
+++ b/hw/usb/redirect.c
@@ -2154,7 +2154,8 @@ static int usbredir_post_load(void *priv, int version_id)
 }
 
 /* For usbredirparser migration */
-static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
+static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused,
+                                void *opaque, QJSON *vmdesc)
 {
     USBRedirDevice *dev = priv;
     uint8_t *data;
@@ -2174,7 +2175,8 @@ static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
     free(data);
 }
 
-static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused)
+static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused,
+                               void *opaque)
 {
     USBRedirDevice *dev = priv;
     uint8_t *data;
@@ -2217,7 +2219,8 @@ static const VMStateInfo usbredir_parser_vmstate_info = {
 
 
 /* For buffered packets (iso/irq) queue migration */
-static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
+static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused,
+                               VMStateField *field, QJSON *vmdesc)
 {
     struct endp_data *endp = priv;
     USBRedirDevice *dev = endp->dev;
@@ -2237,7 +2240,8 @@ static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
     assert(i == endp->bufpq_size);
 }
 
-static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused)
+static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused,
+                              VMStateField *field)
 {
     struct endp_data *endp = priv;
     USBRedirDevice *dev = endp->dev;
@@ -2340,7 +2344,8 @@ static const VMStateDescription usbredir_ep_vmstate = {
 
 
 /* For PacketIdQueue migration */
-static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
+static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused,
+                                     VMStateField *field, QJSON *vmdesc)
 {
     struct PacketIdQueue *q = priv;
     USBRedirDevice *dev = q->dev;
@@ -2356,7 +2361,8 @@ static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
     assert(remain == 0);
 }
 
-static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused)
+static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused,
+                                    VMStateField *field)
 {
     struct PacketIdQueue *q = priv;
     USBRedirDevice *dev = q->dev;
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 2d60a00..38a7abd 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -108,7 +108,8 @@ static bool virtio_pci_has_extra_state(DeviceState *d)
     return proxy->flags & VIRTIO_PCI_FLAG_MIGRATE_EXTRA;
 }
 
-static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
+static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
+                                       VMStateField *field)
 {
     VirtIOPCIProxy *proxy = pv;
     int i;
@@ -137,7 +138,8 @@ static void virtio_pci_save_modern_queue_state(VirtIOPCIQueue *vq,
     qemu_put_be32(f, vq->used[1]);
 }
 
-static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
+static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
+                                        VMStateField *field, QJSON *vmdesc)
 {
     VirtIOPCIProxy *proxy = pv;
     int i;
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 18ce333..ade2683 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -1467,7 +1467,8 @@ static const VMStateDescription vmstate_virtio_ringsize = {
     }
 };
 
-static int get_extra_state(QEMUFile *f, void *pv, size_t size)
+static int get_extra_state(QEMUFile *f, void *pv, size_t size,
+                           VMStateField *field)
 {
     VirtIODevice *vdev = pv;
     BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
@@ -1480,7 +1481,8 @@ static int get_extra_state(QEMUFile *f, void *pv, size_t size)
     }
 }
 
-static void put_extra_state(QEMUFile *f, void *pv, size_t size)
+static void put_extra_state(QEMUFile *f, void *pv, size_t size,
+                            VMStateField *field, QJSON *vmdesc)
 {
     VirtIODevice *vdev = pv;
     BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 1638ee5..459dd4a 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -81,11 +81,13 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque);
 
 typedef struct VMStateInfo VMStateInfo;
 typedef struct VMStateDescription VMStateDescription;
+typedef struct VMStateField VMStateField;
 
 struct VMStateInfo {
     const char *name;
-    int (*get)(QEMUFile *f, void *pv, size_t size);
-    void (*put)(QEMUFile *f, void *pv, size_t size);
+    int (*get)(QEMUFile *f, void *pv, size_t size, VMStateField *field);
+    void (*put)(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                QJSON *vmdesc);
 };
 
 enum VMStateFlags {
@@ -186,7 +188,7 @@ enum VMStateFlags {
     VMS_MULTIPLY_ELEMENTS = 0x4000,
 };
 
-typedef struct {
+struct VMStateField {
     const char *name;
     size_t offset;
     size_t size;
@@ -199,7 +201,7 @@ typedef struct {
     const VMStateDescription *vmsd;
     int version_id;
     bool (*field_exists)(void *opaque, int version_id);
-} VMStateField;
+};
 
 struct VMStateDescription {
     const char *name;
diff --git a/migration/savevm.c b/migration/savevm.c
index ef5c3d1..13b53ad 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -220,14 +220,15 @@ void timer_get(QEMUFile *f, QEMUTimer *ts)
  * Not in vmstate.c to not add qemu-timer.c as dependency to vmstate.c
  */
 
-static int get_timer(QEMUFile *f, void *pv, size_t size)
+static int get_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     QEMUTimer *v = pv;
     timer_get(f, v);
     return 0;
 }
 
-static void put_timer(QEMUFile *f, void *pv, size_t size)
+static void put_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                QJSON *vmdesc)
 {
     QEMUTimer *v = pv;
     timer_put(f, v);
diff --git a/migration/vmstate.c b/migration/vmstate.c
index fc29acf..66802cb 100644
--- a/migration/vmstate.c
+++ b/migration/vmstate.c
@@ -122,7 +122,7 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
                     ret = vmstate_load_state(f, field->vmsd, addr,
                                              field->vmsd->version_id);
                 } else {
-                    ret = field->info->get(f, addr, size);
+                    ret = field->info->get(f, addr, size, NULL);
 
                 }
                 if (ret >= 0) {
@@ -328,7 +328,7 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
                 if (field->flags & VMS_STRUCT) {
                     vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
                 } else {
-                    field->info->put(f, addr, size);
+                    field->info->put(f, addr, size, NULL, NULL);
                 }
 
                 written_bytes = qemu_ftell_fast(f) - old_offset;
@@ -461,14 +461,15 @@ static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
 
 /* bool */
 
-static int get_bool(QEMUFile *f, void *pv, size_t size)
+static int get_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     bool *v = pv;
     *v = qemu_get_byte(f);
     return 0;
 }
 
-static void put_bool(QEMUFile *f, void *pv, size_t size)
+static void put_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                     QJSON *vmdesc)
 {
     bool *v = pv;
     qemu_put_byte(f, *v);
@@ -482,14 +483,15 @@ const VMStateInfo vmstate_info_bool = {
 
 /* 8 bit int */
 
-static int get_int8(QEMUFile *f, void *pv, size_t size)
+static int get_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     int8_t *v = pv;
     qemu_get_s8s(f, v);
     return 0;
 }
 
-static void put_int8(QEMUFile *f, void *pv, size_t size)
+static void put_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                     QJSON *vmdesc)
 {
     int8_t *v = pv;
     qemu_put_s8s(f, v);
@@ -503,14 +505,15 @@ const VMStateInfo vmstate_info_int8 = {
 
 /* 16 bit int */
 
-static int get_int16(QEMUFile *f, void *pv, size_t size)
+static int get_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     int16_t *v = pv;
     qemu_get_sbe16s(f, v);
     return 0;
 }
 
-static void put_int16(QEMUFile *f, void *pv, size_t size)
+static void put_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                      QJSON *vmdesc)
 {
     int16_t *v = pv;
     qemu_put_sbe16s(f, v);
@@ -524,14 +527,15 @@ const VMStateInfo vmstate_info_int16 = {
 
 /* 32 bit int */
 
-static int get_int32(QEMUFile *f, void *pv, size_t size)
+static int get_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     int32_t *v = pv;
     qemu_get_sbe32s(f, v);
     return 0;
 }
 
-static void put_int32(QEMUFile *f, void *pv, size_t size)
+static void put_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                      QJSON *vmdesc)
 {
     int32_t *v = pv;
     qemu_put_sbe32s(f, v);
@@ -546,7 +550,8 @@ const VMStateInfo vmstate_info_int32 = {
 /* 32 bit int. See that the received value is the same than the one
    in the field */
 
-static int get_int32_equal(QEMUFile *f, void *pv, size_t size)
+static int get_int32_equal(QEMUFile *f, void *pv, size_t size,
+                           VMStateField *field)
 {
     int32_t *v = pv;
     int32_t v2;
@@ -568,7 +573,7 @@ const VMStateInfo vmstate_info_int32_equal = {
  * and less than or equal to the one in the field.
  */
 
-static int get_int32_le(QEMUFile *f, void *pv, size_t size)
+static int get_int32_le(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     int32_t *cur = pv;
     int32_t loaded;
@@ -589,14 +594,15 @@ const VMStateInfo vmstate_info_int32_le = {
 
 /* 64 bit int */
 
-static int get_int64(QEMUFile *f, void *pv, size_t size)
+static int get_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     int64_t *v = pv;
     qemu_get_sbe64s(f, v);
     return 0;
 }
 
-static void put_int64(QEMUFile *f, void *pv, size_t size)
+static void put_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                      QJSON *vmdesc)
 {
     int64_t *v = pv;
     qemu_put_sbe64s(f, v);
@@ -610,14 +616,15 @@ const VMStateInfo vmstate_info_int64 = {
 
 /* 8 bit unsigned int */
 
-static int get_uint8(QEMUFile *f, void *pv, size_t size)
+static int get_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     uint8_t *v = pv;
     qemu_get_8s(f, v);
     return 0;
 }
 
-static void put_uint8(QEMUFile *f, void *pv, size_t size)
+static void put_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                      QJSON *vmdesc)
 {
     uint8_t *v = pv;
     qemu_put_8s(f, v);
@@ -631,14 +638,15 @@ const VMStateInfo vmstate_info_uint8 = {
 
 /* 16 bit unsigned int */
 
-static int get_uint16(QEMUFile *f, void *pv, size_t size)
+static int get_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     uint16_t *v = pv;
     qemu_get_be16s(f, v);
     return 0;
 }
 
-static void put_uint16(QEMUFile *f, void *pv, size_t size)
+static void put_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                       QJSON *vmdesc)
 {
     uint16_t *v = pv;
     qemu_put_be16s(f, v);
@@ -652,14 +660,15 @@ const VMStateInfo vmstate_info_uint16 = {
 
 /* 32 bit unsigned int */
 
-static int get_uint32(QEMUFile *f, void *pv, size_t size)
+static int get_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     uint32_t *v = pv;
     qemu_get_be32s(f, v);
     return 0;
 }
 
-static void put_uint32(QEMUFile *f, void *pv, size_t size)
+static void put_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                       QJSON *vmdesc)
 {
     uint32_t *v = pv;
     qemu_put_be32s(f, v);
@@ -674,7 +683,8 @@ const VMStateInfo vmstate_info_uint32 = {
 /* 32 bit uint. See that the received value is the same than the one
    in the field */
 
-static int get_uint32_equal(QEMUFile *f, void *pv, size_t size)
+static int get_uint32_equal(QEMUFile *f, void *pv, size_t size,
+                            VMStateField *field)
 {
     uint32_t *v = pv;
     uint32_t v2;
@@ -694,14 +704,15 @@ const VMStateInfo vmstate_info_uint32_equal = {
 
 /* 64 bit unsigned int */
 
-static int get_uint64(QEMUFile *f, void *pv, size_t size)
+static int get_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     uint64_t *v = pv;
     qemu_get_be64s(f, v);
     return 0;
 }
 
-static void put_uint64(QEMUFile *f, void *pv, size_t size)
+static void put_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                       QJSON *vmdesc)
 {
     uint64_t *v = pv;
     qemu_put_be64s(f, v);
@@ -716,7 +727,8 @@ const VMStateInfo vmstate_info_uint64 = {
 /* 64 bit unsigned int. See that the received value is the same than the one
    in the field */
 
-static int get_uint64_equal(QEMUFile *f, void *pv, size_t size)
+static int get_uint64_equal(QEMUFile *f, void *pv, size_t size,
+                            VMStateField *field)
 {
     uint64_t *v = pv;
     uint64_t v2;
@@ -737,7 +749,8 @@ const VMStateInfo vmstate_info_uint64_equal = {
 /* 8 bit int. See that the received value is the same than the one
    in the field */
 
-static int get_uint8_equal(QEMUFile *f, void *pv, size_t size)
+static int get_uint8_equal(QEMUFile *f, void *pv, size_t size,
+                           VMStateField *field)
 {
     uint8_t *v = pv;
     uint8_t v2;
@@ -758,7 +771,8 @@ const VMStateInfo vmstate_info_uint8_equal = {
 /* 16 bit unsigned int int. See that the received value is the same than the one
    in the field */
 
-static int get_uint16_equal(QEMUFile *f, void *pv, size_t size)
+static int get_uint16_equal(QEMUFile *f, void *pv, size_t size,
+                            VMStateField *field)
 {
     uint16_t *v = pv;
     uint16_t v2;
@@ -778,7 +792,8 @@ const VMStateInfo vmstate_info_uint16_equal = {
 
 /* floating point */
 
-static int get_float64(QEMUFile *f, void *pv, size_t size)
+static int get_float64(QEMUFile *f, void *pv, size_t size,
+                       VMStateField *field)
 {
     float64 *v = pv;
 
@@ -786,7 +801,8 @@ static int get_float64(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_float64(QEMUFile *f, void *pv, size_t size)
+static void put_float64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                        QJSON *vmdesc)
 {
     uint64_t *v = pv;
 
@@ -801,7 +817,8 @@ const VMStateInfo vmstate_info_float64 = {
 
 /* CPU_DoubleU type */
 
-static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
+static int get_cpudouble(QEMUFile *f, void *pv, size_t size,
+                         VMStateField *field)
 {
     CPU_DoubleU *v = pv;
     qemu_get_be32s(f, &v->l.upper);
@@ -809,7 +826,8 @@ static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_cpudouble(QEMUFile *f, void *pv, size_t size)
+static void put_cpudouble(QEMUFile *f, void *pv, size_t size,
+                          VMStateField *field, QJSON *vmdesc)
 {
     CPU_DoubleU *v = pv;
     qemu_put_be32s(f, &v->l.upper);
@@ -824,14 +842,16 @@ const VMStateInfo vmstate_info_cpudouble = {
 
 /* uint8_t buffers */
 
-static int get_buffer(QEMUFile *f, void *pv, size_t size)
+static int get_buffer(QEMUFile *f, void *pv, size_t size,
+                      VMStateField *field)
 {
     uint8_t *v = pv;
     qemu_get_buffer(f, v, size);
     return 0;
 }
 
-static void put_buffer(QEMUFile *f, void *pv, size_t size)
+static void put_buffer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                       QJSON *vmdesc)
 {
     uint8_t *v = pv;
     qemu_put_buffer(f, v, size);
@@ -846,7 +866,8 @@ const VMStateInfo vmstate_info_buffer = {
 /* unused buffers: space that was used for some fields that are
    not useful anymore */
 
-static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
+static int get_unused_buffer(QEMUFile *f, void *pv, size_t size,
+                             VMStateField *field)
 {
     uint8_t buf[1024];
     int block_len;
@@ -859,7 +880,8 @@ static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
    return 0;
 }
 
-static void put_unused_buffer(QEMUFile *f, void *pv, size_t size)
+static void put_unused_buffer(QEMUFile *f, void *pv, size_t size,
+                              VMStateField *field, QJSON *vmdesc)
 {
     static const uint8_t buf[1024];
     int block_len;
@@ -884,7 +906,7 @@ const VMStateInfo vmstate_info_unused_buffer = {
  */
 /* This is the number of 64 bit words sent over the wire */
 #define BITS_TO_U64S(nr) DIV_ROUND_UP(nr, 64)
-static int get_bitmap(QEMUFile *f, void *pv, size_t size)
+static int get_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     unsigned long *bmp = pv;
     int i, idx = 0;
@@ -898,7 +920,8 @@ static int get_bitmap(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_bitmap(QEMUFile *f, void *pv, size_t size)
+static void put_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                QJSON *vmdesc)
 {
     unsigned long *bmp = pv;
     int i, idx = 0;
diff --git a/target-alpha/machine.c b/target-alpha/machine.c
index 710b783..48e3278 100644
--- a/target-alpha/machine.c
+++ b/target-alpha/machine.c
@@ -5,14 +5,15 @@
 #include "hw/boards.h"
 #include "migration/cpu.h"
 
-static int get_fpcr(QEMUFile *f, void *opaque, size_t size)
+static int get_fpcr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
 {
     CPUAlphaState *env = opaque;
     cpu_alpha_store_fpcr(env, qemu_get_be64(f));
     return 0;
 }
 
-static void put_fpcr(QEMUFile *f, void *opaque, size_t size)
+static void put_fpcr(QEMUFile *f, void *opaque, size_t size,
+                     VMStateField *field, QJSON *vmdesc)
 {
     CPUAlphaState *env = opaque;
     qemu_put_be64(f, cpu_alpha_load_fpcr(env));
diff --git a/target-arm/machine.c b/target-arm/machine.c
index 7a6ca31..41a0de9 100644
--- a/target-arm/machine.c
+++ b/target-arm/machine.c
@@ -17,7 +17,8 @@ static bool vfp_needed(void *opaque)
     return arm_feature(env, ARM_FEATURE_VFP);
 }
 
-static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
+static int get_fpscr(QEMUFile *f, void *opaque, size_t size,
+                     VMStateField *field)
 {
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
@@ -27,7 +28,8 @@ static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
     return 0;
 }
 
-static void put_fpscr(QEMUFile *f, void *opaque, size_t size)
+static void put_fpscr(QEMUFile *f, void *opaque, size_t size,
+                      VMStateField *field, QJSON *vmdesc)
 {
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
@@ -163,7 +165,8 @@ static const VMStateDescription vmstate_pmsav7 = {
     }
 };
 
-static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
+static int get_cpsr(QEMUFile *f, void *opaque, size_t size,
+                    VMStateField *field)
 {
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
@@ -180,7 +183,8 @@ static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
     return 0;
 }
 
-static void put_cpsr(QEMUFile *f, void *opaque, size_t size)
+static void put_cpsr(QEMUFile *f, void *opaque, size_t size,
+                     VMStateField *field, QJSON *vmdesc)
 {
     ARMCPU *cpu = opaque;
     CPUARMState *env = &cpu->env;
diff --git a/target-i386/machine.c b/target-i386/machine.c
index 71c0e4d..1df19e2 100644
--- a/target-i386/machine.c
+++ b/target-i386/machine.c
@@ -139,7 +139,8 @@ static const VMStateDescription vmstate_mtrr_var = {
 #define VMSTATE_MTRR_VARS(_field, _state, _n, _v)                    \
     VMSTATE_STRUCT_ARRAY(_field, _state, _n, _v, vmstate_mtrr_var, MTRRVar)
 
-static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size)
+static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size,
+                            VMStateField *field, QJSON *vmdesc)
 {
     fprintf(stderr, "call put_fpreg() with invalid arguments\n");
     exit(0);
@@ -167,7 +168,8 @@ static void fp64_to_fp80(union x86_longdouble *p, uint64_t temp)
     p->exp = e;
 }
 
-static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
+static int get_fpreg(QEMUFile *f, void *opaque, size_t size,
+                     VMStateField *field)
 {
     FPReg *fp_reg = opaque;
     uint64_t mant;
@@ -179,7 +181,8 @@ static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
     return 0;
 }
 
-static void put_fpreg(QEMUFile *f, void *opaque, size_t size)
+static void put_fpreg(QEMUFile *f, void *opaque, size_t size,
+                      VMStateField *field, QJSON *vmdesc)
 {
     FPReg *fp_reg = opaque;
     uint64_t mant;
@@ -197,7 +200,8 @@ static const VMStateInfo vmstate_fpreg = {
     .put  = put_fpreg,
 };
 
-static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size)
+static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size,
+                           VMStateField *field)
 {
     union x86_longdouble *p = opaque;
     uint64_t mant;
@@ -214,7 +218,8 @@ static const VMStateInfo vmstate_fpreg_1_mmx = {
     .put  = put_fpreg_error,
 };
 
-static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size)
+static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size,
+                              VMStateField *field)
 {
     union x86_longdouble *p = opaque;
     uint64_t mant;
@@ -276,14 +281,16 @@ static bool less_than_7(void *opaque, int version_id)
     return version_id < 7;
 }
 
-static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
+static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
+                                VMStateField *field)
 {
     uint64_t *v = pv;
     *v = qemu_get_be32(f);
     return 0;
 }
 
-static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
+static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
+                                 VMStateField *field, QJSON *vmdesc)
 {
     uint64_t *v = pv;
     qemu_put_be32(f, *v);
diff --git a/target-mips/machine.c b/target-mips/machine.c
index a27f2f1..179084c 100644
--- a/target-mips/machine.c
+++ b/target-mips/machine.c
@@ -20,7 +20,7 @@ static int cpu_post_load(void *opaque, int version_id)
 
 /* FPU state */
 
-static int get_fpr(QEMUFile *f, void *pv, size_t size)
+static int get_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     int i;
     fpr_t *v = pv;
@@ -31,7 +31,8 @@ static int get_fpr(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_fpr(QEMUFile *f, void *pv, size_t size)
+static void put_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                    QJSON *vmdesc)
 {
     int i;
     fpr_t *v = pv;
@@ -125,7 +126,7 @@ const VMStateDescription vmstate_mvp = {
 
 /* TLB state */
 
-static int get_tlb(QEMUFile *f, void *pv, size_t size)
+static int get_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     r4k_tlb_t *v = pv;
     uint16_t flags;
@@ -152,7 +153,8 @@ static int get_tlb(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_tlb(QEMUFile *f, void *pv, size_t size)
+static void put_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                    QJSON *vmdesc)
 {
     r4k_tlb_t *v = pv;
 
diff --git a/target-ppc/machine.c b/target-ppc/machine.c
index 4820f22..0e1822c 100644
--- a/target-ppc/machine.c
+++ b/target-ppc/machine.c
@@ -106,7 +106,7 @@ static int cpu_load_old(QEMUFile *f, void *opaque, int version_id)
     return 0;
 }
 
-static int get_avr(QEMUFile *f, void *pv, size_t size)
+static int get_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     ppc_avr_t *v = pv;
 
@@ -116,7 +116,8 @@ static int get_avr(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_avr(QEMUFile *f, void *pv, size_t size)
+static void put_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                    QJSON *vmdesc)
 {
     ppc_avr_t *v = pv;
 
@@ -324,7 +325,7 @@ static const VMStateDescription vmstate_sr = {
 };
 
 #ifdef TARGET_PPC64
-static int get_slbe(QEMUFile *f, void *pv, size_t size)
+static int get_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field)
 {
     ppc_slb_t *v = pv;
 
@@ -334,7 +335,8 @@ static int get_slbe(QEMUFile *f, void *pv, size_t size)
     return 0;
 }
 
-static void put_slbe(QEMUFile *f, void *pv, size_t size)
+static void put_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field,
+                     QJSON *vmdesc)
 {
     ppc_slb_t *v = pv;
 
diff --git a/target-sparc/machine.c b/target-sparc/machine.c
index 59c92f7..3194e03 100644
--- a/target-sparc/machine.c
+++ b/target-sparc/machine.c
@@ -59,7 +59,7 @@ static const VMStateDescription vmstate_tlb_entry = {
 };
 #endif
 
-static int get_psr(QEMUFile *f, void *opaque, size_t size)
+static int get_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
 {
     SPARCCPU *cpu = opaque;
     CPUSPARCState *env = &cpu->env;
@@ -72,7 +72,8 @@ static int get_psr(QEMUFile *f, void *opaque, size_t size)
     return 0;
 }
 
-static void put_psr(QEMUFile *f, void *opaque, size_t size)
+static void put_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field,
+                QJSON *vmdesc)
 {
     SPARCCPU *cpu = opaque;
     CPUSPARCState *env = &cpu->env;
-- 
1.9.1

Re: [Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Dr. David Alan Gilbert]

* Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> Current migration code cannot handle some data structures such as
> QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
> in VMStateInfo so that customized handling is supported.
> 
> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> ---
>  hw/net/vmxnet3.c            | 18 ++++++---
>  hw/nvram/eeprom93xx.c       |  6 ++-
>  hw/nvram/fw_cfg.c           |  6 ++-
>  hw/pci/msix.c               |  6 ++-
>  hw/pci/pci.c                | 12 ++++--
>  hw/pci/shpc.c               |  5 ++-
>  hw/scsi/scsi-bus.c          |  6 ++-
>  hw/timer/twl92230.c         |  6 ++-
>  hw/usb/redirect.c           | 18 ++++++---
>  hw/virtio/virtio-pci.c      |  6 ++-
>  hw/virtio/virtio.c          |  6 ++-
>  include/migration/vmstate.h | 10 +++--
>  migration/savevm.c          |  5 ++-
>  migration/vmstate.c         | 95 ++++++++++++++++++++++++++++-----------------
>  target-alpha/machine.c      |  5 ++-
>  target-arm/machine.c        | 12 ++++--
>  target-i386/machine.c       | 21 ++++++----
>  target-mips/machine.c       | 10 +++--
>  target-ppc/machine.c        | 10 +++--
>  target-sparc/machine.c      |  5 ++-
>  20 files changed, 171 insertions(+), 97 deletions(-)
> 

<snip>

> diff --git a/hw/usb/redirect.c b/hw/usb/redirect.c
> index 444672a..2ca4b46 100644
> --- a/hw/usb/redirect.c
> +++ b/hw/usb/redirect.c
> @@ -2154,7 +2154,8 @@ static int usbredir_post_load(void *priv, int version_id)
>  }
>  
>  /* For usbredirparser migration */
> -static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
> +static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused,
> +                                void *opaque, QJSON *vmdesc)
>  {
>      USBRedirDevice *dev = priv;
>      uint8_t *data;
> @@ -2174,7 +2175,8 @@ static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
>      free(data);
>  }
>  
> -static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused)
> +static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused,
> +                               void *opaque)

Neither of these built for me; I had to change those to VMStateField rather than void *;

also is this series tested ontop of Halil's patches - because without them
I'm finding I also had to fix up most of the other virtio devices.

Dave


>      USBRedirDevice *dev = priv;
>      uint8_t *data;
> @@ -2217,7 +2219,8 @@ static const VMStateInfo usbredir_parser_vmstate_info = {
>  
>  
>  /* For buffered packets (iso/irq) queue migration */
> -static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
> +static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused,
> +                               VMStateField *field, QJSON *vmdesc)
>  {
>      struct endp_data *endp = priv;
>      USBRedirDevice *dev = endp->dev;
> @@ -2237,7 +2240,8 @@ static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
>      assert(i == endp->bufpq_size);
>  }
>  
> -static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused)
> +static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused,
> +                              VMStateField *field)
>  {
>      struct endp_data *endp = priv;
>      USBRedirDevice *dev = endp->dev;
> @@ -2340,7 +2344,8 @@ static const VMStateDescription usbredir_ep_vmstate = {
>  
>  
>  /* For PacketIdQueue migration */
> -static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> +static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused,
> +                                     VMStateField *field, QJSON *vmdesc)
>  {
>      struct PacketIdQueue *q = priv;
>      USBRedirDevice *dev = q->dev;
> @@ -2356,7 +2361,8 @@ static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
>      assert(remain == 0);
>  }
>  
> -static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> +static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused,
> +                                    VMStateField *field)
>  {
>      struct PacketIdQueue *q = priv;
>      USBRedirDevice *dev = q->dev;
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index 2d60a00..38a7abd 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -108,7 +108,8 @@ static bool virtio_pci_has_extra_state(DeviceState *d)
>      return proxy->flags & VIRTIO_PCI_FLAG_MIGRATE_EXTRA;
>  }
>  
> -static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
> +static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
> +                                       VMStateField *field)
>  {
>      VirtIOPCIProxy *proxy = pv;
>      int i;
> @@ -137,7 +138,8 @@ static void virtio_pci_save_modern_queue_state(VirtIOPCIQueue *vq,
>      qemu_put_be32(f, vq->used[1]);
>  }
>  
> -static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
> +static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
> +                                        VMStateField *field, QJSON *vmdesc)
>  {
>      VirtIOPCIProxy *proxy = pv;
>      int i;
> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> index 18ce333..ade2683 100644
> --- a/hw/virtio/virtio.c
> +++ b/hw/virtio/virtio.c
> @@ -1467,7 +1467,8 @@ static const VMStateDescription vmstate_virtio_ringsize = {
>      }
>  };
>  
> -static int get_extra_state(QEMUFile *f, void *pv, size_t size)
> +static int get_extra_state(QEMUFile *f, void *pv, size_t size,
> +                           VMStateField *field)
>  {
>      VirtIODevice *vdev = pv;
>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
> @@ -1480,7 +1481,8 @@ static int get_extra_state(QEMUFile *f, void *pv, size_t size)
>      }
>  }
>  
> -static void put_extra_state(QEMUFile *f, void *pv, size_t size)
> +static void put_extra_state(QEMUFile *f, void *pv, size_t size,
> +                            VMStateField *field, QJSON *vmdesc)
>  {
>      VirtIODevice *vdev = pv;
>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> index 1638ee5..459dd4a 100644
> --- a/include/migration/vmstate.h
> +++ b/include/migration/vmstate.h
> @@ -81,11 +81,13 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque);
>  
>  typedef struct VMStateInfo VMStateInfo;
>  typedef struct VMStateDescription VMStateDescription;
> +typedef struct VMStateField VMStateField;
>  
>  struct VMStateInfo {
>      const char *name;
> -    int (*get)(QEMUFile *f, void *pv, size_t size);
> -    void (*put)(QEMUFile *f, void *pv, size_t size);
> +    int (*get)(QEMUFile *f, void *pv, size_t size, VMStateField *field);
> +    void (*put)(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                QJSON *vmdesc);
>  };
>  
>  enum VMStateFlags {
> @@ -186,7 +188,7 @@ enum VMStateFlags {
>      VMS_MULTIPLY_ELEMENTS = 0x4000,
>  };
>  
> -typedef struct {
> +struct VMStateField {
>      const char *name;
>      size_t offset;
>      size_t size;
> @@ -199,7 +201,7 @@ typedef struct {
>      const VMStateDescription *vmsd;
>      int version_id;
>      bool (*field_exists)(void *opaque, int version_id);
> -} VMStateField;
> +};
>  
>  struct VMStateDescription {
>      const char *name;
> diff --git a/migration/savevm.c b/migration/savevm.c
> index ef5c3d1..13b53ad 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -220,14 +220,15 @@ void timer_get(QEMUFile *f, QEMUTimer *ts)
>   * Not in vmstate.c to not add qemu-timer.c as dependency to vmstate.c
>   */
>  
> -static int get_timer(QEMUFile *f, void *pv, size_t size)
> +static int get_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      QEMUTimer *v = pv;
>      timer_get(f, v);
>      return 0;
>  }
>  
> -static void put_timer(QEMUFile *f, void *pv, size_t size)
> +static void put_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                QJSON *vmdesc)
>  {
>      QEMUTimer *v = pv;
>      timer_put(f, v);
> diff --git a/migration/vmstate.c b/migration/vmstate.c
> index fc29acf..66802cb 100644
> --- a/migration/vmstate.c
> +++ b/migration/vmstate.c
> @@ -122,7 +122,7 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
>                      ret = vmstate_load_state(f, field->vmsd, addr,
>                                               field->vmsd->version_id);
>                  } else {
> -                    ret = field->info->get(f, addr, size);
> +                    ret = field->info->get(f, addr, size, NULL);
>  
>                  }
>                  if (ret >= 0) {
> @@ -328,7 +328,7 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>                  if (field->flags & VMS_STRUCT) {
>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>                  } else {
> -                    field->info->put(f, addr, size);
> +                    field->info->put(f, addr, size, NULL, NULL);
>                  }
>  
>                  written_bytes = qemu_ftell_fast(f) - old_offset;
> @@ -461,14 +461,15 @@ static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
>  
>  /* bool */
>  
> -static int get_bool(QEMUFile *f, void *pv, size_t size)
> +static int get_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      bool *v = pv;
>      *v = qemu_get_byte(f);
>      return 0;
>  }
>  
> -static void put_bool(QEMUFile *f, void *pv, size_t size)
> +static void put_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                     QJSON *vmdesc)
>  {
>      bool *v = pv;
>      qemu_put_byte(f, *v);
> @@ -482,14 +483,15 @@ const VMStateInfo vmstate_info_bool = {
>  
>  /* 8 bit int */
>  
> -static int get_int8(QEMUFile *f, void *pv, size_t size)
> +static int get_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      int8_t *v = pv;
>      qemu_get_s8s(f, v);
>      return 0;
>  }
>  
> -static void put_int8(QEMUFile *f, void *pv, size_t size)
> +static void put_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                     QJSON *vmdesc)
>  {
>      int8_t *v = pv;
>      qemu_put_s8s(f, v);
> @@ -503,14 +505,15 @@ const VMStateInfo vmstate_info_int8 = {
>  
>  /* 16 bit int */
>  
> -static int get_int16(QEMUFile *f, void *pv, size_t size)
> +static int get_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      int16_t *v = pv;
>      qemu_get_sbe16s(f, v);
>      return 0;
>  }
>  
> -static void put_int16(QEMUFile *f, void *pv, size_t size)
> +static void put_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                      QJSON *vmdesc)
>  {
>      int16_t *v = pv;
>      qemu_put_sbe16s(f, v);
> @@ -524,14 +527,15 @@ const VMStateInfo vmstate_info_int16 = {
>  
>  /* 32 bit int */
>  
> -static int get_int32(QEMUFile *f, void *pv, size_t size)
> +static int get_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      int32_t *v = pv;
>      qemu_get_sbe32s(f, v);
>      return 0;
>  }
>  
> -static void put_int32(QEMUFile *f, void *pv, size_t size)
> +static void put_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                      QJSON *vmdesc)
>  {
>      int32_t *v = pv;
>      qemu_put_sbe32s(f, v);
> @@ -546,7 +550,8 @@ const VMStateInfo vmstate_info_int32 = {
>  /* 32 bit int. See that the received value is the same than the one
>     in the field */
>  
> -static int get_int32_equal(QEMUFile *f, void *pv, size_t size)
> +static int get_int32_equal(QEMUFile *f, void *pv, size_t size,
> +                           VMStateField *field)
>  {
>      int32_t *v = pv;
>      int32_t v2;
> @@ -568,7 +573,7 @@ const VMStateInfo vmstate_info_int32_equal = {
>   * and less than or equal to the one in the field.
>   */
>  
> -static int get_int32_le(QEMUFile *f, void *pv, size_t size)
> +static int get_int32_le(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      int32_t *cur = pv;
>      int32_t loaded;
> @@ -589,14 +594,15 @@ const VMStateInfo vmstate_info_int32_le = {
>  
>  /* 64 bit int */
>  
> -static int get_int64(QEMUFile *f, void *pv, size_t size)
> +static int get_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      int64_t *v = pv;
>      qemu_get_sbe64s(f, v);
>      return 0;
>  }
>  
> -static void put_int64(QEMUFile *f, void *pv, size_t size)
> +static void put_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                      QJSON *vmdesc)
>  {
>      int64_t *v = pv;
>      qemu_put_sbe64s(f, v);
> @@ -610,14 +616,15 @@ const VMStateInfo vmstate_info_int64 = {
>  
>  /* 8 bit unsigned int */
>  
> -static int get_uint8(QEMUFile *f, void *pv, size_t size)
> +static int get_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      uint8_t *v = pv;
>      qemu_get_8s(f, v);
>      return 0;
>  }
>  
> -static void put_uint8(QEMUFile *f, void *pv, size_t size)
> +static void put_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                      QJSON *vmdesc)
>  {
>      uint8_t *v = pv;
>      qemu_put_8s(f, v);
> @@ -631,14 +638,15 @@ const VMStateInfo vmstate_info_uint8 = {
>  
>  /* 16 bit unsigned int */
>  
> -static int get_uint16(QEMUFile *f, void *pv, size_t size)
> +static int get_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      uint16_t *v = pv;
>      qemu_get_be16s(f, v);
>      return 0;
>  }
>  
> -static void put_uint16(QEMUFile *f, void *pv, size_t size)
> +static void put_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                       QJSON *vmdesc)
>  {
>      uint16_t *v = pv;
>      qemu_put_be16s(f, v);
> @@ -652,14 +660,15 @@ const VMStateInfo vmstate_info_uint16 = {
>  
>  /* 32 bit unsigned int */
>  
> -static int get_uint32(QEMUFile *f, void *pv, size_t size)
> +static int get_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      uint32_t *v = pv;
>      qemu_get_be32s(f, v);
>      return 0;
>  }
>  
> -static void put_uint32(QEMUFile *f, void *pv, size_t size)
> +static void put_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                       QJSON *vmdesc)
>  {
>      uint32_t *v = pv;
>      qemu_put_be32s(f, v);
> @@ -674,7 +683,8 @@ const VMStateInfo vmstate_info_uint32 = {
>  /* 32 bit uint. See that the received value is the same than the one
>     in the field */
>  
> -static int get_uint32_equal(QEMUFile *f, void *pv, size_t size)
> +static int get_uint32_equal(QEMUFile *f, void *pv, size_t size,
> +                            VMStateField *field)
>  {
>      uint32_t *v = pv;
>      uint32_t v2;
> @@ -694,14 +704,15 @@ const VMStateInfo vmstate_info_uint32_equal = {
>  
>  /* 64 bit unsigned int */
>  
> -static int get_uint64(QEMUFile *f, void *pv, size_t size)
> +static int get_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      uint64_t *v = pv;
>      qemu_get_be64s(f, v);
>      return 0;
>  }
>  
> -static void put_uint64(QEMUFile *f, void *pv, size_t size)
> +static void put_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                       QJSON *vmdesc)
>  {
>      uint64_t *v = pv;
>      qemu_put_be64s(f, v);
> @@ -716,7 +727,8 @@ const VMStateInfo vmstate_info_uint64 = {
>  /* 64 bit unsigned int. See that the received value is the same than the one
>     in the field */
>  
> -static int get_uint64_equal(QEMUFile *f, void *pv, size_t size)
> +static int get_uint64_equal(QEMUFile *f, void *pv, size_t size,
> +                            VMStateField *field)
>  {
>      uint64_t *v = pv;
>      uint64_t v2;
> @@ -737,7 +749,8 @@ const VMStateInfo vmstate_info_uint64_equal = {
>  /* 8 bit int. See that the received value is the same than the one
>     in the field */
>  
> -static int get_uint8_equal(QEMUFile *f, void *pv, size_t size)
> +static int get_uint8_equal(QEMUFile *f, void *pv, size_t size,
> +                           VMStateField *field)
>  {
>      uint8_t *v = pv;
>      uint8_t v2;
> @@ -758,7 +771,8 @@ const VMStateInfo vmstate_info_uint8_equal = {
>  /* 16 bit unsigned int int. See that the received value is the same than the one
>     in the field */
>  
> -static int get_uint16_equal(QEMUFile *f, void *pv, size_t size)
> +static int get_uint16_equal(QEMUFile *f, void *pv, size_t size,
> +                            VMStateField *field)
>  {
>      uint16_t *v = pv;
>      uint16_t v2;
> @@ -778,7 +792,8 @@ const VMStateInfo vmstate_info_uint16_equal = {
>  
>  /* floating point */
>  
> -static int get_float64(QEMUFile *f, void *pv, size_t size)
> +static int get_float64(QEMUFile *f, void *pv, size_t size,
> +                       VMStateField *field)
>  {
>      float64 *v = pv;
>  
> @@ -786,7 +801,8 @@ static int get_float64(QEMUFile *f, void *pv, size_t size)
>      return 0;
>  }
>  
> -static void put_float64(QEMUFile *f, void *pv, size_t size)
> +static void put_float64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                        QJSON *vmdesc)
>  {
>      uint64_t *v = pv;
>  
> @@ -801,7 +817,8 @@ const VMStateInfo vmstate_info_float64 = {
>  
>  /* CPU_DoubleU type */
>  
> -static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
> +static int get_cpudouble(QEMUFile *f, void *pv, size_t size,
> +                         VMStateField *field)
>  {
>      CPU_DoubleU *v = pv;
>      qemu_get_be32s(f, &v->l.upper);
> @@ -809,7 +826,8 @@ static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
>      return 0;
>  }
>  
> -static void put_cpudouble(QEMUFile *f, void *pv, size_t size)
> +static void put_cpudouble(QEMUFile *f, void *pv, size_t size,
> +                          VMStateField *field, QJSON *vmdesc)
>  {
>      CPU_DoubleU *v = pv;
>      qemu_put_be32s(f, &v->l.upper);
> @@ -824,14 +842,16 @@ const VMStateInfo vmstate_info_cpudouble = {
>  
>  /* uint8_t buffers */
>  
> -static int get_buffer(QEMUFile *f, void *pv, size_t size)
> +static int get_buffer(QEMUFile *f, void *pv, size_t size,
> +                      VMStateField *field)
>  {
>      uint8_t *v = pv;
>      qemu_get_buffer(f, v, size);
>      return 0;
>  }
>  
> -static void put_buffer(QEMUFile *f, void *pv, size_t size)
> +static void put_buffer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                       QJSON *vmdesc)
>  {
>      uint8_t *v = pv;
>      qemu_put_buffer(f, v, size);
> @@ -846,7 +866,8 @@ const VMStateInfo vmstate_info_buffer = {
>  /* unused buffers: space that was used for some fields that are
>     not useful anymore */
>  
> -static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
> +static int get_unused_buffer(QEMUFile *f, void *pv, size_t size,
> +                             VMStateField *field)
>  {
>      uint8_t buf[1024];
>      int block_len;
> @@ -859,7 +880,8 @@ static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
>     return 0;
>  }
>  
> -static void put_unused_buffer(QEMUFile *f, void *pv, size_t size)
> +static void put_unused_buffer(QEMUFile *f, void *pv, size_t size,
> +                              VMStateField *field, QJSON *vmdesc)
>  {
>      static const uint8_t buf[1024];
>      int block_len;
> @@ -884,7 +906,7 @@ const VMStateInfo vmstate_info_unused_buffer = {
>   */
>  /* This is the number of 64 bit words sent over the wire */
>  #define BITS_TO_U64S(nr) DIV_ROUND_UP(nr, 64)
> -static int get_bitmap(QEMUFile *f, void *pv, size_t size)
> +static int get_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      unsigned long *bmp = pv;
>      int i, idx = 0;
> @@ -898,7 +920,8 @@ static int get_bitmap(QEMUFile *f, void *pv, size_t size)
>      return 0;
>  }
>  
> -static void put_bitmap(QEMUFile *f, void *pv, size_t size)
> +static void put_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                QJSON *vmdesc)
>  {
>      unsigned long *bmp = pv;
>      int i, idx = 0;
> diff --git a/target-alpha/machine.c b/target-alpha/machine.c
> index 710b783..48e3278 100644
> --- a/target-alpha/machine.c
> +++ b/target-alpha/machine.c
> @@ -5,14 +5,15 @@
>  #include "hw/boards.h"
>  #include "migration/cpu.h"
>  
> -static int get_fpcr(QEMUFile *f, void *opaque, size_t size)
> +static int get_fpcr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
>  {
>      CPUAlphaState *env = opaque;
>      cpu_alpha_store_fpcr(env, qemu_get_be64(f));
>      return 0;
>  }
>  
> -static void put_fpcr(QEMUFile *f, void *opaque, size_t size)
> +static void put_fpcr(QEMUFile *f, void *opaque, size_t size,
> +                     VMStateField *field, QJSON *vmdesc)
>  {
>      CPUAlphaState *env = opaque;
>      qemu_put_be64(f, cpu_alpha_load_fpcr(env));
> diff --git a/target-arm/machine.c b/target-arm/machine.c
> index 7a6ca31..41a0de9 100644
> --- a/target-arm/machine.c
> +++ b/target-arm/machine.c
> @@ -17,7 +17,8 @@ static bool vfp_needed(void *opaque)
>      return arm_feature(env, ARM_FEATURE_VFP);
>  }
>  
> -static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
> +static int get_fpscr(QEMUFile *f, void *opaque, size_t size,
> +                     VMStateField *field)
>  {
>      ARMCPU *cpu = opaque;
>      CPUARMState *env = &cpu->env;
> @@ -27,7 +28,8 @@ static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
>      return 0;
>  }
>  
> -static void put_fpscr(QEMUFile *f, void *opaque, size_t size)
> +static void put_fpscr(QEMUFile *f, void *opaque, size_t size,
> +                      VMStateField *field, QJSON *vmdesc)
>  {
>      ARMCPU *cpu = opaque;
>      CPUARMState *env = &cpu->env;
> @@ -163,7 +165,8 @@ static const VMStateDescription vmstate_pmsav7 = {
>      }
>  };
>  
> -static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
> +static int get_cpsr(QEMUFile *f, void *opaque, size_t size,
> +                    VMStateField *field)
>  {
>      ARMCPU *cpu = opaque;
>      CPUARMState *env = &cpu->env;
> @@ -180,7 +183,8 @@ static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
>      return 0;
>  }
>  
> -static void put_cpsr(QEMUFile *f, void *opaque, size_t size)
> +static void put_cpsr(QEMUFile *f, void *opaque, size_t size,
> +                     VMStateField *field, QJSON *vmdesc)
>  {
>      ARMCPU *cpu = opaque;
>      CPUARMState *env = &cpu->env;
> diff --git a/target-i386/machine.c b/target-i386/machine.c
> index 71c0e4d..1df19e2 100644
> --- a/target-i386/machine.c
> +++ b/target-i386/machine.c
> @@ -139,7 +139,8 @@ static const VMStateDescription vmstate_mtrr_var = {
>  #define VMSTATE_MTRR_VARS(_field, _state, _n, _v)                    \
>      VMSTATE_STRUCT_ARRAY(_field, _state, _n, _v, vmstate_mtrr_var, MTRRVar)
>  
> -static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size)
> +static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size,
> +                            VMStateField *field, QJSON *vmdesc)
>  {
>      fprintf(stderr, "call put_fpreg() with invalid arguments\n");
>      exit(0);
> @@ -167,7 +168,8 @@ static void fp64_to_fp80(union x86_longdouble *p, uint64_t temp)
>      p->exp = e;
>  }
>  
> -static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
> +static int get_fpreg(QEMUFile *f, void *opaque, size_t size,
> +                     VMStateField *field)
>  {
>      FPReg *fp_reg = opaque;
>      uint64_t mant;
> @@ -179,7 +181,8 @@ static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
>      return 0;
>  }
>  
> -static void put_fpreg(QEMUFile *f, void *opaque, size_t size)
> +static void put_fpreg(QEMUFile *f, void *opaque, size_t size,
> +                      VMStateField *field, QJSON *vmdesc)
>  {
>      FPReg *fp_reg = opaque;
>      uint64_t mant;
> @@ -197,7 +200,8 @@ static const VMStateInfo vmstate_fpreg = {
>      .put  = put_fpreg,
>  };
>  
> -static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size)
> +static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size,
> +                           VMStateField *field)
>  {
>      union x86_longdouble *p = opaque;
>      uint64_t mant;
> @@ -214,7 +218,8 @@ static const VMStateInfo vmstate_fpreg_1_mmx = {
>      .put  = put_fpreg_error,
>  };
>  
> -static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size)
> +static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size,
> +                              VMStateField *field)
>  {
>      union x86_longdouble *p = opaque;
>      uint64_t mant;
> @@ -276,14 +281,16 @@ static bool less_than_7(void *opaque, int version_id)
>      return version_id < 7;
>  }
>  
> -static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
> +static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
> +                                VMStateField *field)
>  {
>      uint64_t *v = pv;
>      *v = qemu_get_be32(f);
>      return 0;
>  }
>  
> -static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
> +static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
> +                                 VMStateField *field, QJSON *vmdesc)
>  {
>      uint64_t *v = pv;
>      qemu_put_be32(f, *v);
> diff --git a/target-mips/machine.c b/target-mips/machine.c
> index a27f2f1..179084c 100644
> --- a/target-mips/machine.c
> +++ b/target-mips/machine.c
> @@ -20,7 +20,7 @@ static int cpu_post_load(void *opaque, int version_id)
>  
>  /* FPU state */
>  
> -static int get_fpr(QEMUFile *f, void *pv, size_t size)
> +static int get_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      int i;
>      fpr_t *v = pv;
> @@ -31,7 +31,8 @@ static int get_fpr(QEMUFile *f, void *pv, size_t size)
>      return 0;
>  }
>  
> -static void put_fpr(QEMUFile *f, void *pv, size_t size)
> +static void put_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                    QJSON *vmdesc)
>  {
>      int i;
>      fpr_t *v = pv;
> @@ -125,7 +126,7 @@ const VMStateDescription vmstate_mvp = {
>  
>  /* TLB state */
>  
> -static int get_tlb(QEMUFile *f, void *pv, size_t size)
> +static int get_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      r4k_tlb_t *v = pv;
>      uint16_t flags;
> @@ -152,7 +153,8 @@ static int get_tlb(QEMUFile *f, void *pv, size_t size)
>      return 0;
>  }
>  
> -static void put_tlb(QEMUFile *f, void *pv, size_t size)
> +static void put_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                    QJSON *vmdesc)
>  {
>      r4k_tlb_t *v = pv;
>  
> diff --git a/target-ppc/machine.c b/target-ppc/machine.c
> index 4820f22..0e1822c 100644
> --- a/target-ppc/machine.c
> +++ b/target-ppc/machine.c
> @@ -106,7 +106,7 @@ static int cpu_load_old(QEMUFile *f, void *opaque, int version_id)
>      return 0;
>  }
>  
> -static int get_avr(QEMUFile *f, void *pv, size_t size)
> +static int get_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      ppc_avr_t *v = pv;
>  
> @@ -116,7 +116,8 @@ static int get_avr(QEMUFile *f, void *pv, size_t size)
>      return 0;
>  }
>  
> -static void put_avr(QEMUFile *f, void *pv, size_t size)
> +static void put_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                    QJSON *vmdesc)
>  {
>      ppc_avr_t *v = pv;
>  
> @@ -324,7 +325,7 @@ static const VMStateDescription vmstate_sr = {
>  };
>  
>  #ifdef TARGET_PPC64
> -static int get_slbe(QEMUFile *f, void *pv, size_t size)
> +static int get_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>  {
>      ppc_slb_t *v = pv;
>  
> @@ -334,7 +335,8 @@ static int get_slbe(QEMUFile *f, void *pv, size_t size)
>      return 0;
>  }
>  
> -static void put_slbe(QEMUFile *f, void *pv, size_t size)
> +static void put_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> +                     QJSON *vmdesc)
>  {
>      ppc_slb_t *v = pv;
>  
> diff --git a/target-sparc/machine.c b/target-sparc/machine.c
> index 59c92f7..3194e03 100644
> --- a/target-sparc/machine.c
> +++ b/target-sparc/machine.c
> @@ -59,7 +59,7 @@ static const VMStateDescription vmstate_tlb_entry = {
>  };
>  #endif
>  
> -static int get_psr(QEMUFile *f, void *opaque, size_t size)
> +static int get_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
>  {
>      SPARCCPU *cpu = opaque;
>      CPUSPARCState *env = &cpu->env;
> @@ -72,7 +72,8 @@ static int get_psr(QEMUFile *f, void *opaque, size_t size)
>      return 0;
>  }
>  
> -static void put_psr(QEMUFile *f, void *opaque, size_t size)
> +static void put_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field,
> +                QJSON *vmdesc)
>  {
>      SPARCCPU *cpu = opaque;
>      CPUSPARCState *env = &cpu->env;
> -- 
> 1.9.1
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Jianjun Duan]

On 10/07/2016 05:08 AM, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>> Current migration code cannot handle some data structures such as
>> QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
>> in VMStateInfo so that customized handling is supported.
>>
>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>> ---
>>  hw/net/vmxnet3.c            | 18 ++++++---
>>  hw/nvram/eeprom93xx.c       |  6 ++-
>>  hw/nvram/fw_cfg.c           |  6 ++-
>>  hw/pci/msix.c               |  6 ++-
>>  hw/pci/pci.c                | 12 ++++--
>>  hw/pci/shpc.c               |  5 ++-
>>  hw/scsi/scsi-bus.c          |  6 ++-
>>  hw/timer/twl92230.c         |  6 ++-
>>  hw/usb/redirect.c           | 18 ++++++---
>>  hw/virtio/virtio-pci.c      |  6 ++-
>>  hw/virtio/virtio.c          |  6 ++-
>>  include/migration/vmstate.h | 10 +++--
>>  migration/savevm.c          |  5 ++-
>>  migration/vmstate.c         | 95 ++++++++++++++++++++++++++++-----------------
>>  target-alpha/machine.c      |  5 ++-
>>  target-arm/machine.c        | 12 ++++--
>>  target-i386/machine.c       | 21 ++++++----
>>  target-mips/machine.c       | 10 +++--
>>  target-ppc/machine.c        | 10 +++--
>>  target-sparc/machine.c      |  5 ++-
>>  20 files changed, 171 insertions(+), 97 deletions(-)
>>
> 
> <snip>
> 
>> diff --git a/hw/usb/redirect.c b/hw/usb/redirect.c
>> index 444672a..2ca4b46 100644
>> --- a/hw/usb/redirect.c
>> +++ b/hw/usb/redirect.c
>> @@ -2154,7 +2154,8 @@ static int usbredir_post_load(void *priv, int version_id)
>>  }
>>  
>>  /* For usbredirparser migration */
>> -static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
>> +static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused,
>> +                                void *opaque, QJSON *vmdesc)
>>  {
>>      USBRedirDevice *dev = priv;
>>      uint8_t *data;
>> @@ -2174,7 +2175,8 @@ static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
>>      free(data);
>>  }
>>  
>> -static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused)
>> +static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused,
>> +                               void *opaque)
> 
> Neither of these built for me; I had to change those to VMStateField rather than void *;
> 
> also is this series tested ontop of Halil's patches - because without them
> I'm finding I also had to fix up most of the other virtio devices.
> 
> Dave

I built it on top of ppc-for-2.8 without problems. Is Hail's patch in
ppc-for-2.8 yet?

Thanks,
Jianjun

> 
> 
>>      USBRedirDevice *dev = priv;
>>      uint8_t *data;
>> @@ -2217,7 +2219,8 @@ static const VMStateInfo usbredir_parser_vmstate_info = {
>>  
>>  
>>  /* For buffered packets (iso/irq) queue migration */
>> -static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
>> +static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused,
>> +                               VMStateField *field, QJSON *vmdesc)
>>  {
>>      struct endp_data *endp = priv;
>>      USBRedirDevice *dev = endp->dev;
>> @@ -2237,7 +2240,8 @@ static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
>>      assert(i == endp->bufpq_size);
>>  }
>>  
>> -static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused)
>> +static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused,
>> +                              VMStateField *field)
>>  {
>>      struct endp_data *endp = priv;
>>      USBRedirDevice *dev = endp->dev;
>> @@ -2340,7 +2344,8 @@ static const VMStateDescription usbredir_ep_vmstate = {
>>  
>>  
>>  /* For PacketIdQueue migration */
>> -static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
>> +static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused,
>> +                                     VMStateField *field, QJSON *vmdesc)
>>  {
>>      struct PacketIdQueue *q = priv;
>>      USBRedirDevice *dev = q->dev;
>> @@ -2356,7 +2361,8 @@ static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
>>      assert(remain == 0);
>>  }
>>  
>> -static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused)
>> +static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused,
>> +                                    VMStateField *field)
>>  {
>>      struct PacketIdQueue *q = priv;
>>      USBRedirDevice *dev = q->dev;
>> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
>> index 2d60a00..38a7abd 100644
>> --- a/hw/virtio/virtio-pci.c
>> +++ b/hw/virtio/virtio-pci.c
>> @@ -108,7 +108,8 @@ static bool virtio_pci_has_extra_state(DeviceState *d)
>>      return proxy->flags & VIRTIO_PCI_FLAG_MIGRATE_EXTRA;
>>  }
>>  
>> -static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
>> +static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
>> +                                       VMStateField *field)
>>  {
>>      VirtIOPCIProxy *proxy = pv;
>>      int i;
>> @@ -137,7 +138,8 @@ static void virtio_pci_save_modern_queue_state(VirtIOPCIQueue *vq,
>>      qemu_put_be32(f, vq->used[1]);
>>  }
>>  
>> -static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
>> +static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
>> +                                        VMStateField *field, QJSON *vmdesc)
>>  {
>>      VirtIOPCIProxy *proxy = pv;
>>      int i;
>> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
>> index 18ce333..ade2683 100644
>> --- a/hw/virtio/virtio.c
>> +++ b/hw/virtio/virtio.c
>> @@ -1467,7 +1467,8 @@ static const VMStateDescription vmstate_virtio_ringsize = {
>>      }
>>  };
>>  
>> -static int get_extra_state(QEMUFile *f, void *pv, size_t size)
>> +static int get_extra_state(QEMUFile *f, void *pv, size_t size,
>> +                           VMStateField *field)
>>  {
>>      VirtIODevice *vdev = pv;
>>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
>> @@ -1480,7 +1481,8 @@ static int get_extra_state(QEMUFile *f, void *pv, size_t size)
>>      }
>>  }
>>  
>> -static void put_extra_state(QEMUFile *f, void *pv, size_t size)
>> +static void put_extra_state(QEMUFile *f, void *pv, size_t size,
>> +                            VMStateField *field, QJSON *vmdesc)
>>  {
>>      VirtIODevice *vdev = pv;
>>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
>> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
>> index 1638ee5..459dd4a 100644
>> --- a/include/migration/vmstate.h
>> +++ b/include/migration/vmstate.h
>> @@ -81,11 +81,13 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque);
>>  
>>  typedef struct VMStateInfo VMStateInfo;
>>  typedef struct VMStateDescription VMStateDescription;
>> +typedef struct VMStateField VMStateField;
>>  
>>  struct VMStateInfo {
>>      const char *name;
>> -    int (*get)(QEMUFile *f, void *pv, size_t size);
>> -    void (*put)(QEMUFile *f, void *pv, size_t size);
>> +    int (*get)(QEMUFile *f, void *pv, size_t size, VMStateField *field);
>> +    void (*put)(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                QJSON *vmdesc);
>>  };
>>  
>>  enum VMStateFlags {
>> @@ -186,7 +188,7 @@ enum VMStateFlags {
>>      VMS_MULTIPLY_ELEMENTS = 0x4000,
>>  };
>>  
>> -typedef struct {
>> +struct VMStateField {
>>      const char *name;
>>      size_t offset;
>>      size_t size;
>> @@ -199,7 +201,7 @@ typedef struct {
>>      const VMStateDescription *vmsd;
>>      int version_id;
>>      bool (*field_exists)(void *opaque, int version_id);
>> -} VMStateField;
>> +};
>>  
>>  struct VMStateDescription {
>>      const char *name;
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index ef5c3d1..13b53ad 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -220,14 +220,15 @@ void timer_get(QEMUFile *f, QEMUTimer *ts)
>>   * Not in vmstate.c to not add qemu-timer.c as dependency to vmstate.c
>>   */
>>  
>> -static int get_timer(QEMUFile *f, void *pv, size_t size)
>> +static int get_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      QEMUTimer *v = pv;
>>      timer_get(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_timer(QEMUFile *f, void *pv, size_t size)
>> +static void put_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                QJSON *vmdesc)
>>  {
>>      QEMUTimer *v = pv;
>>      timer_put(f, v);
>> diff --git a/migration/vmstate.c b/migration/vmstate.c
>> index fc29acf..66802cb 100644
>> --- a/migration/vmstate.c
>> +++ b/migration/vmstate.c
>> @@ -122,7 +122,7 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
>>                      ret = vmstate_load_state(f, field->vmsd, addr,
>>                                               field->vmsd->version_id);
>>                  } else {
>> -                    ret = field->info->get(f, addr, size);
>> +                    ret = field->info->get(f, addr, size, NULL);
>>  
>>                  }
>>                  if (ret >= 0) {
>> @@ -328,7 +328,7 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>                  if (field->flags & VMS_STRUCT) {
>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>>                  } else {
>> -                    field->info->put(f, addr, size);
>> +                    field->info->put(f, addr, size, NULL, NULL);
>>                  }
>>  
>>                  written_bytes = qemu_ftell_fast(f) - old_offset;
>> @@ -461,14 +461,15 @@ static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
>>  
>>  /* bool */
>>  
>> -static int get_bool(QEMUFile *f, void *pv, size_t size)
>> +static int get_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      bool *v = pv;
>>      *v = qemu_get_byte(f);
>>      return 0;
>>  }
>>  
>> -static void put_bool(QEMUFile *f, void *pv, size_t size)
>> +static void put_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                     QJSON *vmdesc)
>>  {
>>      bool *v = pv;
>>      qemu_put_byte(f, *v);
>> @@ -482,14 +483,15 @@ const VMStateInfo vmstate_info_bool = {
>>  
>>  /* 8 bit int */
>>  
>> -static int get_int8(QEMUFile *f, void *pv, size_t size)
>> +static int get_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      int8_t *v = pv;
>>      qemu_get_s8s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_int8(QEMUFile *f, void *pv, size_t size)
>> +static void put_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                     QJSON *vmdesc)
>>  {
>>      int8_t *v = pv;
>>      qemu_put_s8s(f, v);
>> @@ -503,14 +505,15 @@ const VMStateInfo vmstate_info_int8 = {
>>  
>>  /* 16 bit int */
>>  
>> -static int get_int16(QEMUFile *f, void *pv, size_t size)
>> +static int get_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      int16_t *v = pv;
>>      qemu_get_sbe16s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_int16(QEMUFile *f, void *pv, size_t size)
>> +static void put_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                      QJSON *vmdesc)
>>  {
>>      int16_t *v = pv;
>>      qemu_put_sbe16s(f, v);
>> @@ -524,14 +527,15 @@ const VMStateInfo vmstate_info_int16 = {
>>  
>>  /* 32 bit int */
>>  
>> -static int get_int32(QEMUFile *f, void *pv, size_t size)
>> +static int get_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      int32_t *v = pv;
>>      qemu_get_sbe32s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_int32(QEMUFile *f, void *pv, size_t size)
>> +static void put_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                      QJSON *vmdesc)
>>  {
>>      int32_t *v = pv;
>>      qemu_put_sbe32s(f, v);
>> @@ -546,7 +550,8 @@ const VMStateInfo vmstate_info_int32 = {
>>  /* 32 bit int. See that the received value is the same than the one
>>     in the field */
>>  
>> -static int get_int32_equal(QEMUFile *f, void *pv, size_t size)
>> +static int get_int32_equal(QEMUFile *f, void *pv, size_t size,
>> +                           VMStateField *field)
>>  {
>>      int32_t *v = pv;
>>      int32_t v2;
>> @@ -568,7 +573,7 @@ const VMStateInfo vmstate_info_int32_equal = {
>>   * and less than or equal to the one in the field.
>>   */
>>  
>> -static int get_int32_le(QEMUFile *f, void *pv, size_t size)
>> +static int get_int32_le(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      int32_t *cur = pv;
>>      int32_t loaded;
>> @@ -589,14 +594,15 @@ const VMStateInfo vmstate_info_int32_le = {
>>  
>>  /* 64 bit int */
>>  
>> -static int get_int64(QEMUFile *f, void *pv, size_t size)
>> +static int get_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      int64_t *v = pv;
>>      qemu_get_sbe64s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_int64(QEMUFile *f, void *pv, size_t size)
>> +static void put_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                      QJSON *vmdesc)
>>  {
>>      int64_t *v = pv;
>>      qemu_put_sbe64s(f, v);
>> @@ -610,14 +616,15 @@ const VMStateInfo vmstate_info_int64 = {
>>  
>>  /* 8 bit unsigned int */
>>  
>> -static int get_uint8(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      uint8_t *v = pv;
>>      qemu_get_8s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_uint8(QEMUFile *f, void *pv, size_t size)
>> +static void put_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                      QJSON *vmdesc)
>>  {
>>      uint8_t *v = pv;
>>      qemu_put_8s(f, v);
>> @@ -631,14 +638,15 @@ const VMStateInfo vmstate_info_uint8 = {
>>  
>>  /* 16 bit unsigned int */
>>  
>> -static int get_uint16(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      uint16_t *v = pv;
>>      qemu_get_be16s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_uint16(QEMUFile *f, void *pv, size_t size)
>> +static void put_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                       QJSON *vmdesc)
>>  {
>>      uint16_t *v = pv;
>>      qemu_put_be16s(f, v);
>> @@ -652,14 +660,15 @@ const VMStateInfo vmstate_info_uint16 = {
>>  
>>  /* 32 bit unsigned int */
>>  
>> -static int get_uint32(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      uint32_t *v = pv;
>>      qemu_get_be32s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_uint32(QEMUFile *f, void *pv, size_t size)
>> +static void put_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                       QJSON *vmdesc)
>>  {
>>      uint32_t *v = pv;
>>      qemu_put_be32s(f, v);
>> @@ -674,7 +683,8 @@ const VMStateInfo vmstate_info_uint32 = {
>>  /* 32 bit uint. See that the received value is the same than the one
>>     in the field */
>>  
>> -static int get_uint32_equal(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint32_equal(QEMUFile *f, void *pv, size_t size,
>> +                            VMStateField *field)
>>  {
>>      uint32_t *v = pv;
>>      uint32_t v2;
>> @@ -694,14 +704,15 @@ const VMStateInfo vmstate_info_uint32_equal = {
>>  
>>  /* 64 bit unsigned int */
>>  
>> -static int get_uint64(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      uint64_t *v = pv;
>>      qemu_get_be64s(f, v);
>>      return 0;
>>  }
>>  
>> -static void put_uint64(QEMUFile *f, void *pv, size_t size)
>> +static void put_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                       QJSON *vmdesc)
>>  {
>>      uint64_t *v = pv;
>>      qemu_put_be64s(f, v);
>> @@ -716,7 +727,8 @@ const VMStateInfo vmstate_info_uint64 = {
>>  /* 64 bit unsigned int. See that the received value is the same than the one
>>     in the field */
>>  
>> -static int get_uint64_equal(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint64_equal(QEMUFile *f, void *pv, size_t size,
>> +                            VMStateField *field)
>>  {
>>      uint64_t *v = pv;
>>      uint64_t v2;
>> @@ -737,7 +749,8 @@ const VMStateInfo vmstate_info_uint64_equal = {
>>  /* 8 bit int. See that the received value is the same than the one
>>     in the field */
>>  
>> -static int get_uint8_equal(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint8_equal(QEMUFile *f, void *pv, size_t size,
>> +                           VMStateField *field)
>>  {
>>      uint8_t *v = pv;
>>      uint8_t v2;
>> @@ -758,7 +771,8 @@ const VMStateInfo vmstate_info_uint8_equal = {
>>  /* 16 bit unsigned int int. See that the received value is the same than the one
>>     in the field */
>>  
>> -static int get_uint16_equal(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint16_equal(QEMUFile *f, void *pv, size_t size,
>> +                            VMStateField *field)
>>  {
>>      uint16_t *v = pv;
>>      uint16_t v2;
>> @@ -778,7 +792,8 @@ const VMStateInfo vmstate_info_uint16_equal = {
>>  
>>  /* floating point */
>>  
>> -static int get_float64(QEMUFile *f, void *pv, size_t size)
>> +static int get_float64(QEMUFile *f, void *pv, size_t size,
>> +                       VMStateField *field)
>>  {
>>      float64 *v = pv;
>>  
>> @@ -786,7 +801,8 @@ static int get_float64(QEMUFile *f, void *pv, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_float64(QEMUFile *f, void *pv, size_t size)
>> +static void put_float64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                        QJSON *vmdesc)
>>  {
>>      uint64_t *v = pv;
>>  
>> @@ -801,7 +817,8 @@ const VMStateInfo vmstate_info_float64 = {
>>  
>>  /* CPU_DoubleU type */
>>  
>> -static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
>> +static int get_cpudouble(QEMUFile *f, void *pv, size_t size,
>> +                         VMStateField *field)
>>  {
>>      CPU_DoubleU *v = pv;
>>      qemu_get_be32s(f, &v->l.upper);
>> @@ -809,7 +826,8 @@ static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_cpudouble(QEMUFile *f, void *pv, size_t size)
>> +static void put_cpudouble(QEMUFile *f, void *pv, size_t size,
>> +                          VMStateField *field, QJSON *vmdesc)
>>  {
>>      CPU_DoubleU *v = pv;
>>      qemu_put_be32s(f, &v->l.upper);
>> @@ -824,14 +842,16 @@ const VMStateInfo vmstate_info_cpudouble = {
>>  
>>  /* uint8_t buffers */
>>  
>> -static int get_buffer(QEMUFile *f, void *pv, size_t size)
>> +static int get_buffer(QEMUFile *f, void *pv, size_t size,
>> +                      VMStateField *field)
>>  {
>>      uint8_t *v = pv;
>>      qemu_get_buffer(f, v, size);
>>      return 0;
>>  }
>>  
>> -static void put_buffer(QEMUFile *f, void *pv, size_t size)
>> +static void put_buffer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                       QJSON *vmdesc)
>>  {
>>      uint8_t *v = pv;
>>      qemu_put_buffer(f, v, size);
>> @@ -846,7 +866,8 @@ const VMStateInfo vmstate_info_buffer = {
>>  /* unused buffers: space that was used for some fields that are
>>     not useful anymore */
>>  
>> -static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
>> +static int get_unused_buffer(QEMUFile *f, void *pv, size_t size,
>> +                             VMStateField *field)
>>  {
>>      uint8_t buf[1024];
>>      int block_len;
>> @@ -859,7 +880,8 @@ static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
>>     return 0;
>>  }
>>  
>> -static void put_unused_buffer(QEMUFile *f, void *pv, size_t size)
>> +static void put_unused_buffer(QEMUFile *f, void *pv, size_t size,
>> +                              VMStateField *field, QJSON *vmdesc)
>>  {
>>      static const uint8_t buf[1024];
>>      int block_len;
>> @@ -884,7 +906,7 @@ const VMStateInfo vmstate_info_unused_buffer = {
>>   */
>>  /* This is the number of 64 bit words sent over the wire */
>>  #define BITS_TO_U64S(nr) DIV_ROUND_UP(nr, 64)
>> -static int get_bitmap(QEMUFile *f, void *pv, size_t size)
>> +static int get_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      unsigned long *bmp = pv;
>>      int i, idx = 0;
>> @@ -898,7 +920,8 @@ static int get_bitmap(QEMUFile *f, void *pv, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_bitmap(QEMUFile *f, void *pv, size_t size)
>> +static void put_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                QJSON *vmdesc)
>>  {
>>      unsigned long *bmp = pv;
>>      int i, idx = 0;
>> diff --git a/target-alpha/machine.c b/target-alpha/machine.c
>> index 710b783..48e3278 100644
>> --- a/target-alpha/machine.c
>> +++ b/target-alpha/machine.c
>> @@ -5,14 +5,15 @@
>>  #include "hw/boards.h"
>>  #include "migration/cpu.h"
>>  
>> -static int get_fpcr(QEMUFile *f, void *opaque, size_t size)
>> +static int get_fpcr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
>>  {
>>      CPUAlphaState *env = opaque;
>>      cpu_alpha_store_fpcr(env, qemu_get_be64(f));
>>      return 0;
>>  }
>>  
>> -static void put_fpcr(QEMUFile *f, void *opaque, size_t size)
>> +static void put_fpcr(QEMUFile *f, void *opaque, size_t size,
>> +                     VMStateField *field, QJSON *vmdesc)
>>  {
>>      CPUAlphaState *env = opaque;
>>      qemu_put_be64(f, cpu_alpha_load_fpcr(env));
>> diff --git a/target-arm/machine.c b/target-arm/machine.c
>> index 7a6ca31..41a0de9 100644
>> --- a/target-arm/machine.c
>> +++ b/target-arm/machine.c
>> @@ -17,7 +17,8 @@ static bool vfp_needed(void *opaque)
>>      return arm_feature(env, ARM_FEATURE_VFP);
>>  }
>>  
>> -static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
>> +static int get_fpscr(QEMUFile *f, void *opaque, size_t size,
>> +                     VMStateField *field)
>>  {
>>      ARMCPU *cpu = opaque;
>>      CPUARMState *env = &cpu->env;
>> @@ -27,7 +28,8 @@ static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_fpscr(QEMUFile *f, void *opaque, size_t size)
>> +static void put_fpscr(QEMUFile *f, void *opaque, size_t size,
>> +                      VMStateField *field, QJSON *vmdesc)
>>  {
>>      ARMCPU *cpu = opaque;
>>      CPUARMState *env = &cpu->env;
>> @@ -163,7 +165,8 @@ static const VMStateDescription vmstate_pmsav7 = {
>>      }
>>  };
>>  
>> -static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
>> +static int get_cpsr(QEMUFile *f, void *opaque, size_t size,
>> +                    VMStateField *field)
>>  {
>>      ARMCPU *cpu = opaque;
>>      CPUARMState *env = &cpu->env;
>> @@ -180,7 +183,8 @@ static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_cpsr(QEMUFile *f, void *opaque, size_t size)
>> +static void put_cpsr(QEMUFile *f, void *opaque, size_t size,
>> +                     VMStateField *field, QJSON *vmdesc)
>>  {
>>      ARMCPU *cpu = opaque;
>>      CPUARMState *env = &cpu->env;
>> diff --git a/target-i386/machine.c b/target-i386/machine.c
>> index 71c0e4d..1df19e2 100644
>> --- a/target-i386/machine.c
>> +++ b/target-i386/machine.c
>> @@ -139,7 +139,8 @@ static const VMStateDescription vmstate_mtrr_var = {
>>  #define VMSTATE_MTRR_VARS(_field, _state, _n, _v)                    \
>>      VMSTATE_STRUCT_ARRAY(_field, _state, _n, _v, vmstate_mtrr_var, MTRRVar)
>>  
>> -static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size)
>> +static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size,
>> +                            VMStateField *field, QJSON *vmdesc)
>>  {
>>      fprintf(stderr, "call put_fpreg() with invalid arguments\n");
>>      exit(0);
>> @@ -167,7 +168,8 @@ static void fp64_to_fp80(union x86_longdouble *p, uint64_t temp)
>>      p->exp = e;
>>  }
>>  
>> -static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
>> +static int get_fpreg(QEMUFile *f, void *opaque, size_t size,
>> +                     VMStateField *field)
>>  {
>>      FPReg *fp_reg = opaque;
>>      uint64_t mant;
>> @@ -179,7 +181,8 @@ static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_fpreg(QEMUFile *f, void *opaque, size_t size)
>> +static void put_fpreg(QEMUFile *f, void *opaque, size_t size,
>> +                      VMStateField *field, QJSON *vmdesc)
>>  {
>>      FPReg *fp_reg = opaque;
>>      uint64_t mant;
>> @@ -197,7 +200,8 @@ static const VMStateInfo vmstate_fpreg = {
>>      .put  = put_fpreg,
>>  };
>>  
>> -static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size)
>> +static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size,
>> +                           VMStateField *field)
>>  {
>>      union x86_longdouble *p = opaque;
>>      uint64_t mant;
>> @@ -214,7 +218,8 @@ static const VMStateInfo vmstate_fpreg_1_mmx = {
>>      .put  = put_fpreg_error,
>>  };
>>  
>> -static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size)
>> +static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size,
>> +                              VMStateField *field)
>>  {
>>      union x86_longdouble *p = opaque;
>>      uint64_t mant;
>> @@ -276,14 +281,16 @@ static bool less_than_7(void *opaque, int version_id)
>>      return version_id < 7;
>>  }
>>  
>> -static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
>> +static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
>> +                                VMStateField *field)
>>  {
>>      uint64_t *v = pv;
>>      *v = qemu_get_be32(f);
>>      return 0;
>>  }
>>  
>> -static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
>> +static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
>> +                                 VMStateField *field, QJSON *vmdesc)
>>  {
>>      uint64_t *v = pv;
>>      qemu_put_be32(f, *v);
>> diff --git a/target-mips/machine.c b/target-mips/machine.c
>> index a27f2f1..179084c 100644
>> --- a/target-mips/machine.c
>> +++ b/target-mips/machine.c
>> @@ -20,7 +20,7 @@ static int cpu_post_load(void *opaque, int version_id)
>>  
>>  /* FPU state */
>>  
>> -static int get_fpr(QEMUFile *f, void *pv, size_t size)
>> +static int get_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      int i;
>>      fpr_t *v = pv;
>> @@ -31,7 +31,8 @@ static int get_fpr(QEMUFile *f, void *pv, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_fpr(QEMUFile *f, void *pv, size_t size)
>> +static void put_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                    QJSON *vmdesc)
>>  {
>>      int i;
>>      fpr_t *v = pv;
>> @@ -125,7 +126,7 @@ const VMStateDescription vmstate_mvp = {
>>  
>>  /* TLB state */
>>  
>> -static int get_tlb(QEMUFile *f, void *pv, size_t size)
>> +static int get_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      r4k_tlb_t *v = pv;
>>      uint16_t flags;
>> @@ -152,7 +153,8 @@ static int get_tlb(QEMUFile *f, void *pv, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_tlb(QEMUFile *f, void *pv, size_t size)
>> +static void put_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                    QJSON *vmdesc)
>>  {
>>      r4k_tlb_t *v = pv;
>>  
>> diff --git a/target-ppc/machine.c b/target-ppc/machine.c
>> index 4820f22..0e1822c 100644
>> --- a/target-ppc/machine.c
>> +++ b/target-ppc/machine.c
>> @@ -106,7 +106,7 @@ static int cpu_load_old(QEMUFile *f, void *opaque, int version_id)
>>      return 0;
>>  }
>>  
>> -static int get_avr(QEMUFile *f, void *pv, size_t size)
>> +static int get_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      ppc_avr_t *v = pv;
>>  
>> @@ -116,7 +116,8 @@ static int get_avr(QEMUFile *f, void *pv, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_avr(QEMUFile *f, void *pv, size_t size)
>> +static void put_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                    QJSON *vmdesc)
>>  {
>>      ppc_avr_t *v = pv;
>>  
>> @@ -324,7 +325,7 @@ static const VMStateDescription vmstate_sr = {
>>  };
>>  
>>  #ifdef TARGET_PPC64
>> -static int get_slbe(QEMUFile *f, void *pv, size_t size)
>> +static int get_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field)
>>  {
>>      ppc_slb_t *v = pv;
>>  
>> @@ -334,7 +335,8 @@ static int get_slbe(QEMUFile *f, void *pv, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_slbe(QEMUFile *f, void *pv, size_t size)
>> +static void put_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field,
>> +                     QJSON *vmdesc)
>>  {
>>      ppc_slb_t *v = pv;
>>  
>> diff --git a/target-sparc/machine.c b/target-sparc/machine.c
>> index 59c92f7..3194e03 100644
>> --- a/target-sparc/machine.c
>> +++ b/target-sparc/machine.c
>> @@ -59,7 +59,7 @@ static const VMStateDescription vmstate_tlb_entry = {
>>  };
>>  #endif
>>  
>> -static int get_psr(QEMUFile *f, void *opaque, size_t size)
>> +static int get_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
>>  {
>>      SPARCCPU *cpu = opaque;
>>      CPUSPARCState *env = &cpu->env;
>> @@ -72,7 +72,8 @@ static int get_psr(QEMUFile *f, void *opaque, size_t size)
>>      return 0;
>>  }
>>  
>> -static void put_psr(QEMUFile *f, void *opaque, size_t size)
>> +static void put_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field,
>> +                QJSON *vmdesc)
>>  {
>>      SPARCCPU *cpu = opaque;
>>      CPUSPARCState *env = &cpu->env;
>> -- 
>> 1.9.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Dr. David Alan Gilbert]

* Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> 
> 
> On 10/07/2016 05:08 AM, Dr. David Alan Gilbert wrote:
> > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> >> Current migration code cannot handle some data structures such as
> >> QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
> >> in VMStateInfo so that customized handling is supported.
> >>
> >> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> >> ---
> >>  hw/net/vmxnet3.c            | 18 ++++++---
> >>  hw/nvram/eeprom93xx.c       |  6 ++-
> >>  hw/nvram/fw_cfg.c           |  6 ++-
> >>  hw/pci/msix.c               |  6 ++-
> >>  hw/pci/pci.c                | 12 ++++--
> >>  hw/pci/shpc.c               |  5 ++-
> >>  hw/scsi/scsi-bus.c          |  6 ++-
> >>  hw/timer/twl92230.c         |  6 ++-
> >>  hw/usb/redirect.c           | 18 ++++++---
> >>  hw/virtio/virtio-pci.c      |  6 ++-
> >>  hw/virtio/virtio.c          |  6 ++-
> >>  include/migration/vmstate.h | 10 +++--
> >>  migration/savevm.c          |  5 ++-
> >>  migration/vmstate.c         | 95 ++++++++++++++++++++++++++++-----------------
> >>  target-alpha/machine.c      |  5 ++-
> >>  target-arm/machine.c        | 12 ++++--
> >>  target-i386/machine.c       | 21 ++++++----
> >>  target-mips/machine.c       | 10 +++--
> >>  target-ppc/machine.c        | 10 +++--
> >>  target-sparc/machine.c      |  5 ++-
> >>  20 files changed, 171 insertions(+), 97 deletions(-)
> >>
> > 
> > <snip>
> > 
> >> diff --git a/hw/usb/redirect.c b/hw/usb/redirect.c
> >> index 444672a..2ca4b46 100644
> >> --- a/hw/usb/redirect.c
> >> +++ b/hw/usb/redirect.c
> >> @@ -2154,7 +2154,8 @@ static int usbredir_post_load(void *priv, int version_id)
> >>  }
> >>  
> >>  /* For usbredirparser migration */
> >> -static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
> >> +static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused,
> >> +                                void *opaque, QJSON *vmdesc)
> >>  {
> >>      USBRedirDevice *dev = priv;
> >>      uint8_t *data;
> >> @@ -2174,7 +2175,8 @@ static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
> >>      free(data);
> >>  }
> >>  
> >> -static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused)
> >> +static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused,
> >> +                               void *opaque)
> > 
> > Neither of these built for me; I had to change those to VMStateField rather than void *;
> > 
> > also is this series tested ontop of Halil's patches - because without them
> > I'm finding I also had to fix up most of the other virtio devices.
> > 
> > Dave
> 
> I built it on top of ppc-for-2.8 without problems. Is Hail's patch in
> ppc-for-2.8 yet?

I don't know about ppc-for-2.8;  patches for inclusion should work on the current
head unless stated otherwise.
Also, make sure you have the usbredir libraries installed and as much else as possible
to make sure you cover all the .get/.put functions - they're all over!

Dave

> Thanks,
> Jianjun
> 
> > 
> > 
> >>      USBRedirDevice *dev = priv;
> >>      uint8_t *data;
> >> @@ -2217,7 +2219,8 @@ static const VMStateInfo usbredir_parser_vmstate_info = {
> >>  
> >>  
> >>  /* For buffered packets (iso/irq) queue migration */
> >> -static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
> >> +static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused,
> >> +                               VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      struct endp_data *endp = priv;
> >>      USBRedirDevice *dev = endp->dev;
> >> @@ -2237,7 +2240,8 @@ static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
> >>      assert(i == endp->bufpq_size);
> >>  }
> >>  
> >> -static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused)
> >> +static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused,
> >> +                              VMStateField *field)
> >>  {
> >>      struct endp_data *endp = priv;
> >>      USBRedirDevice *dev = endp->dev;
> >> @@ -2340,7 +2344,8 @@ static const VMStateDescription usbredir_ep_vmstate = {
> >>  
> >>  
> >>  /* For PacketIdQueue migration */
> >> -static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> >> +static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused,
> >> +                                     VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      struct PacketIdQueue *q = priv;
> >>      USBRedirDevice *dev = q->dev;
> >> @@ -2356,7 +2361,8 @@ static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> >>      assert(remain == 0);
> >>  }
> >>  
> >> -static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> >> +static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused,
> >> +                                    VMStateField *field)
> >>  {
> >>      struct PacketIdQueue *q = priv;
> >>      USBRedirDevice *dev = q->dev;
> >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> >> index 2d60a00..38a7abd 100644
> >> --- a/hw/virtio/virtio-pci.c
> >> +++ b/hw/virtio/virtio-pci.c
> >> @@ -108,7 +108,8 @@ static bool virtio_pci_has_extra_state(DeviceState *d)
> >>      return proxy->flags & VIRTIO_PCI_FLAG_MIGRATE_EXTRA;
> >>  }
> >>  
> >> -static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
> >> +static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
> >> +                                       VMStateField *field)
> >>  {
> >>      VirtIOPCIProxy *proxy = pv;
> >>      int i;
> >> @@ -137,7 +138,8 @@ static void virtio_pci_save_modern_queue_state(VirtIOPCIQueue *vq,
> >>      qemu_put_be32(f, vq->used[1]);
> >>  }
> >>  
> >> -static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
> >> +static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
> >> +                                        VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      VirtIOPCIProxy *proxy = pv;
> >>      int i;
> >> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> >> index 18ce333..ade2683 100644
> >> --- a/hw/virtio/virtio.c
> >> +++ b/hw/virtio/virtio.c
> >> @@ -1467,7 +1467,8 @@ static const VMStateDescription vmstate_virtio_ringsize = {
> >>      }
> >>  };
> >>  
> >> -static int get_extra_state(QEMUFile *f, void *pv, size_t size)
> >> +static int get_extra_state(QEMUFile *f, void *pv, size_t size,
> >> +                           VMStateField *field)
> >>  {
> >>      VirtIODevice *vdev = pv;
> >>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
> >> @@ -1480,7 +1481,8 @@ static int get_extra_state(QEMUFile *f, void *pv, size_t size)
> >>      }
> >>  }
> >>  
> >> -static void put_extra_state(QEMUFile *f, void *pv, size_t size)
> >> +static void put_extra_state(QEMUFile *f, void *pv, size_t size,
> >> +                            VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      VirtIODevice *vdev = pv;
> >>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
> >> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> >> index 1638ee5..459dd4a 100644
> >> --- a/include/migration/vmstate.h
> >> +++ b/include/migration/vmstate.h
> >> @@ -81,11 +81,13 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque);
> >>  
> >>  typedef struct VMStateInfo VMStateInfo;
> >>  typedef struct VMStateDescription VMStateDescription;
> >> +typedef struct VMStateField VMStateField;
> >>  
> >>  struct VMStateInfo {
> >>      const char *name;
> >> -    int (*get)(QEMUFile *f, void *pv, size_t size);
> >> -    void (*put)(QEMUFile *f, void *pv, size_t size);
> >> +    int (*get)(QEMUFile *f, void *pv, size_t size, VMStateField *field);
> >> +    void (*put)(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                QJSON *vmdesc);
> >>  };
> >>  
> >>  enum VMStateFlags {
> >> @@ -186,7 +188,7 @@ enum VMStateFlags {
> >>      VMS_MULTIPLY_ELEMENTS = 0x4000,
> >>  };
> >>  
> >> -typedef struct {
> >> +struct VMStateField {
> >>      const char *name;
> >>      size_t offset;
> >>      size_t size;
> >> @@ -199,7 +201,7 @@ typedef struct {
> >>      const VMStateDescription *vmsd;
> >>      int version_id;
> >>      bool (*field_exists)(void *opaque, int version_id);
> >> -} VMStateField;
> >> +};
> >>  
> >>  struct VMStateDescription {
> >>      const char *name;
> >> diff --git a/migration/savevm.c b/migration/savevm.c
> >> index ef5c3d1..13b53ad 100644
> >> --- a/migration/savevm.c
> >> +++ b/migration/savevm.c
> >> @@ -220,14 +220,15 @@ void timer_get(QEMUFile *f, QEMUTimer *ts)
> >>   * Not in vmstate.c to not add qemu-timer.c as dependency to vmstate.c
> >>   */
> >>  
> >> -static int get_timer(QEMUFile *f, void *pv, size_t size)
> >> +static int get_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      QEMUTimer *v = pv;
> >>      timer_get(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_timer(QEMUFile *f, void *pv, size_t size)
> >> +static void put_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                QJSON *vmdesc)
> >>  {
> >>      QEMUTimer *v = pv;
> >>      timer_put(f, v);
> >> diff --git a/migration/vmstate.c b/migration/vmstate.c
> >> index fc29acf..66802cb 100644
> >> --- a/migration/vmstate.c
> >> +++ b/migration/vmstate.c
> >> @@ -122,7 +122,7 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
> >>                      ret = vmstate_load_state(f, field->vmsd, addr,
> >>                                               field->vmsd->version_id);
> >>                  } else {
> >> -                    ret = field->info->get(f, addr, size);
> >> +                    ret = field->info->get(f, addr, size, NULL);
> >>  
> >>                  }
> >>                  if (ret >= 0) {
> >> @@ -328,7 +328,7 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
> >>                  if (field->flags & VMS_STRUCT) {
> >>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
> >>                  } else {
> >> -                    field->info->put(f, addr, size);
> >> +                    field->info->put(f, addr, size, NULL, NULL);
> >>                  }
> >>  
> >>                  written_bytes = qemu_ftell_fast(f) - old_offset;
> >> @@ -461,14 +461,15 @@ static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
> >>  
> >>  /* bool */
> >>  
> >> -static int get_bool(QEMUFile *f, void *pv, size_t size)
> >> +static int get_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      bool *v = pv;
> >>      *v = qemu_get_byte(f);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_bool(QEMUFile *f, void *pv, size_t size)
> >> +static void put_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                     QJSON *vmdesc)
> >>  {
> >>      bool *v = pv;
> >>      qemu_put_byte(f, *v);
> >> @@ -482,14 +483,15 @@ const VMStateInfo vmstate_info_bool = {
> >>  
> >>  /* 8 bit int */
> >>  
> >> -static int get_int8(QEMUFile *f, void *pv, size_t size)
> >> +static int get_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      int8_t *v = pv;
> >>      qemu_get_s8s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_int8(QEMUFile *f, void *pv, size_t size)
> >> +static void put_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                     QJSON *vmdesc)
> >>  {
> >>      int8_t *v = pv;
> >>      qemu_put_s8s(f, v);
> >> @@ -503,14 +505,15 @@ const VMStateInfo vmstate_info_int8 = {
> >>  
> >>  /* 16 bit int */
> >>  
> >> -static int get_int16(QEMUFile *f, void *pv, size_t size)
> >> +static int get_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      int16_t *v = pv;
> >>      qemu_get_sbe16s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_int16(QEMUFile *f, void *pv, size_t size)
> >> +static void put_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                      QJSON *vmdesc)
> >>  {
> >>      int16_t *v = pv;
> >>      qemu_put_sbe16s(f, v);
> >> @@ -524,14 +527,15 @@ const VMStateInfo vmstate_info_int16 = {
> >>  
> >>  /* 32 bit int */
> >>  
> >> -static int get_int32(QEMUFile *f, void *pv, size_t size)
> >> +static int get_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      int32_t *v = pv;
> >>      qemu_get_sbe32s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_int32(QEMUFile *f, void *pv, size_t size)
> >> +static void put_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                      QJSON *vmdesc)
> >>  {
> >>      int32_t *v = pv;
> >>      qemu_put_sbe32s(f, v);
> >> @@ -546,7 +550,8 @@ const VMStateInfo vmstate_info_int32 = {
> >>  /* 32 bit int. See that the received value is the same than the one
> >>     in the field */
> >>  
> >> -static int get_int32_equal(QEMUFile *f, void *pv, size_t size)
> >> +static int get_int32_equal(QEMUFile *f, void *pv, size_t size,
> >> +                           VMStateField *field)
> >>  {
> >>      int32_t *v = pv;
> >>      int32_t v2;
> >> @@ -568,7 +573,7 @@ const VMStateInfo vmstate_info_int32_equal = {
> >>   * and less than or equal to the one in the field.
> >>   */
> >>  
> >> -static int get_int32_le(QEMUFile *f, void *pv, size_t size)
> >> +static int get_int32_le(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      int32_t *cur = pv;
> >>      int32_t loaded;
> >> @@ -589,14 +594,15 @@ const VMStateInfo vmstate_info_int32_le = {
> >>  
> >>  /* 64 bit int */
> >>  
> >> -static int get_int64(QEMUFile *f, void *pv, size_t size)
> >> +static int get_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      int64_t *v = pv;
> >>      qemu_get_sbe64s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_int64(QEMUFile *f, void *pv, size_t size)
> >> +static void put_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                      QJSON *vmdesc)
> >>  {
> >>      int64_t *v = pv;
> >>      qemu_put_sbe64s(f, v);
> >> @@ -610,14 +616,15 @@ const VMStateInfo vmstate_info_int64 = {
> >>  
> >>  /* 8 bit unsigned int */
> >>  
> >> -static int get_uint8(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      uint8_t *v = pv;
> >>      qemu_get_8s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_uint8(QEMUFile *f, void *pv, size_t size)
> >> +static void put_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                      QJSON *vmdesc)
> >>  {
> >>      uint8_t *v = pv;
> >>      qemu_put_8s(f, v);
> >> @@ -631,14 +638,15 @@ const VMStateInfo vmstate_info_uint8 = {
> >>  
> >>  /* 16 bit unsigned int */
> >>  
> >> -static int get_uint16(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      uint16_t *v = pv;
> >>      qemu_get_be16s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_uint16(QEMUFile *f, void *pv, size_t size)
> >> +static void put_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                       QJSON *vmdesc)
> >>  {
> >>      uint16_t *v = pv;
> >>      qemu_put_be16s(f, v);
> >> @@ -652,14 +660,15 @@ const VMStateInfo vmstate_info_uint16 = {
> >>  
> >>  /* 32 bit unsigned int */
> >>  
> >> -static int get_uint32(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      uint32_t *v = pv;
> >>      qemu_get_be32s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_uint32(QEMUFile *f, void *pv, size_t size)
> >> +static void put_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                       QJSON *vmdesc)
> >>  {
> >>      uint32_t *v = pv;
> >>      qemu_put_be32s(f, v);
> >> @@ -674,7 +683,8 @@ const VMStateInfo vmstate_info_uint32 = {
> >>  /* 32 bit uint. See that the received value is the same than the one
> >>     in the field */
> >>  
> >> -static int get_uint32_equal(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint32_equal(QEMUFile *f, void *pv, size_t size,
> >> +                            VMStateField *field)
> >>  {
> >>      uint32_t *v = pv;
> >>      uint32_t v2;
> >> @@ -694,14 +704,15 @@ const VMStateInfo vmstate_info_uint32_equal = {
> >>  
> >>  /* 64 bit unsigned int */
> >>  
> >> -static int get_uint64(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      uint64_t *v = pv;
> >>      qemu_get_be64s(f, v);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_uint64(QEMUFile *f, void *pv, size_t size)
> >> +static void put_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                       QJSON *vmdesc)
> >>  {
> >>      uint64_t *v = pv;
> >>      qemu_put_be64s(f, v);
> >> @@ -716,7 +727,8 @@ const VMStateInfo vmstate_info_uint64 = {
> >>  /* 64 bit unsigned int. See that the received value is the same than the one
> >>     in the field */
> >>  
> >> -static int get_uint64_equal(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint64_equal(QEMUFile *f, void *pv, size_t size,
> >> +                            VMStateField *field)
> >>  {
> >>      uint64_t *v = pv;
> >>      uint64_t v2;
> >> @@ -737,7 +749,8 @@ const VMStateInfo vmstate_info_uint64_equal = {
> >>  /* 8 bit int. See that the received value is the same than the one
> >>     in the field */
> >>  
> >> -static int get_uint8_equal(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint8_equal(QEMUFile *f, void *pv, size_t size,
> >> +                           VMStateField *field)
> >>  {
> >>      uint8_t *v = pv;
> >>      uint8_t v2;
> >> @@ -758,7 +771,8 @@ const VMStateInfo vmstate_info_uint8_equal = {
> >>  /* 16 bit unsigned int int. See that the received value is the same than the one
> >>     in the field */
> >>  
> >> -static int get_uint16_equal(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint16_equal(QEMUFile *f, void *pv, size_t size,
> >> +                            VMStateField *field)
> >>  {
> >>      uint16_t *v = pv;
> >>      uint16_t v2;
> >> @@ -778,7 +792,8 @@ const VMStateInfo vmstate_info_uint16_equal = {
> >>  
> >>  /* floating point */
> >>  
> >> -static int get_float64(QEMUFile *f, void *pv, size_t size)
> >> +static int get_float64(QEMUFile *f, void *pv, size_t size,
> >> +                       VMStateField *field)
> >>  {
> >>      float64 *v = pv;
> >>  
> >> @@ -786,7 +801,8 @@ static int get_float64(QEMUFile *f, void *pv, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_float64(QEMUFile *f, void *pv, size_t size)
> >> +static void put_float64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                        QJSON *vmdesc)
> >>  {
> >>      uint64_t *v = pv;
> >>  
> >> @@ -801,7 +817,8 @@ const VMStateInfo vmstate_info_float64 = {
> >>  
> >>  /* CPU_DoubleU type */
> >>  
> >> -static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
> >> +static int get_cpudouble(QEMUFile *f, void *pv, size_t size,
> >> +                         VMStateField *field)
> >>  {
> >>      CPU_DoubleU *v = pv;
> >>      qemu_get_be32s(f, &v->l.upper);
> >> @@ -809,7 +826,8 @@ static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_cpudouble(QEMUFile *f, void *pv, size_t size)
> >> +static void put_cpudouble(QEMUFile *f, void *pv, size_t size,
> >> +                          VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      CPU_DoubleU *v = pv;
> >>      qemu_put_be32s(f, &v->l.upper);
> >> @@ -824,14 +842,16 @@ const VMStateInfo vmstate_info_cpudouble = {
> >>  
> >>  /* uint8_t buffers */
> >>  
> >> -static int get_buffer(QEMUFile *f, void *pv, size_t size)
> >> +static int get_buffer(QEMUFile *f, void *pv, size_t size,
> >> +                      VMStateField *field)
> >>  {
> >>      uint8_t *v = pv;
> >>      qemu_get_buffer(f, v, size);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_buffer(QEMUFile *f, void *pv, size_t size)
> >> +static void put_buffer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                       QJSON *vmdesc)
> >>  {
> >>      uint8_t *v = pv;
> >>      qemu_put_buffer(f, v, size);
> >> @@ -846,7 +866,8 @@ const VMStateInfo vmstate_info_buffer = {
> >>  /* unused buffers: space that was used for some fields that are
> >>     not useful anymore */
> >>  
> >> -static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
> >> +static int get_unused_buffer(QEMUFile *f, void *pv, size_t size,
> >> +                             VMStateField *field)
> >>  {
> >>      uint8_t buf[1024];
> >>      int block_len;
> >> @@ -859,7 +880,8 @@ static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
> >>     return 0;
> >>  }
> >>  
> >> -static void put_unused_buffer(QEMUFile *f, void *pv, size_t size)
> >> +static void put_unused_buffer(QEMUFile *f, void *pv, size_t size,
> >> +                              VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      static const uint8_t buf[1024];
> >>      int block_len;
> >> @@ -884,7 +906,7 @@ const VMStateInfo vmstate_info_unused_buffer = {
> >>   */
> >>  /* This is the number of 64 bit words sent over the wire */
> >>  #define BITS_TO_U64S(nr) DIV_ROUND_UP(nr, 64)
> >> -static int get_bitmap(QEMUFile *f, void *pv, size_t size)
> >> +static int get_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      unsigned long *bmp = pv;
> >>      int i, idx = 0;
> >> @@ -898,7 +920,8 @@ static int get_bitmap(QEMUFile *f, void *pv, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_bitmap(QEMUFile *f, void *pv, size_t size)
> >> +static void put_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                QJSON *vmdesc)
> >>  {
> >>      unsigned long *bmp = pv;
> >>      int i, idx = 0;
> >> diff --git a/target-alpha/machine.c b/target-alpha/machine.c
> >> index 710b783..48e3278 100644
> >> --- a/target-alpha/machine.c
> >> +++ b/target-alpha/machine.c
> >> @@ -5,14 +5,15 @@
> >>  #include "hw/boards.h"
> >>  #include "migration/cpu.h"
> >>  
> >> -static int get_fpcr(QEMUFile *f, void *opaque, size_t size)
> >> +static int get_fpcr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
> >>  {
> >>      CPUAlphaState *env = opaque;
> >>      cpu_alpha_store_fpcr(env, qemu_get_be64(f));
> >>      return 0;
> >>  }
> >>  
> >> -static void put_fpcr(QEMUFile *f, void *opaque, size_t size)
> >> +static void put_fpcr(QEMUFile *f, void *opaque, size_t size,
> >> +                     VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      CPUAlphaState *env = opaque;
> >>      qemu_put_be64(f, cpu_alpha_load_fpcr(env));
> >> diff --git a/target-arm/machine.c b/target-arm/machine.c
> >> index 7a6ca31..41a0de9 100644
> >> --- a/target-arm/machine.c
> >> +++ b/target-arm/machine.c
> >> @@ -17,7 +17,8 @@ static bool vfp_needed(void *opaque)
> >>      return arm_feature(env, ARM_FEATURE_VFP);
> >>  }
> >>  
> >> -static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
> >> +static int get_fpscr(QEMUFile *f, void *opaque, size_t size,
> >> +                     VMStateField *field)
> >>  {
> >>      ARMCPU *cpu = opaque;
> >>      CPUARMState *env = &cpu->env;
> >> @@ -27,7 +28,8 @@ static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_fpscr(QEMUFile *f, void *opaque, size_t size)
> >> +static void put_fpscr(QEMUFile *f, void *opaque, size_t size,
> >> +                      VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      ARMCPU *cpu = opaque;
> >>      CPUARMState *env = &cpu->env;
> >> @@ -163,7 +165,8 @@ static const VMStateDescription vmstate_pmsav7 = {
> >>      }
> >>  };
> >>  
> >> -static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
> >> +static int get_cpsr(QEMUFile *f, void *opaque, size_t size,
> >> +                    VMStateField *field)
> >>  {
> >>      ARMCPU *cpu = opaque;
> >>      CPUARMState *env = &cpu->env;
> >> @@ -180,7 +183,8 @@ static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_cpsr(QEMUFile *f, void *opaque, size_t size)
> >> +static void put_cpsr(QEMUFile *f, void *opaque, size_t size,
> >> +                     VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      ARMCPU *cpu = opaque;
> >>      CPUARMState *env = &cpu->env;
> >> diff --git a/target-i386/machine.c b/target-i386/machine.c
> >> index 71c0e4d..1df19e2 100644
> >> --- a/target-i386/machine.c
> >> +++ b/target-i386/machine.c
> >> @@ -139,7 +139,8 @@ static const VMStateDescription vmstate_mtrr_var = {
> >>  #define VMSTATE_MTRR_VARS(_field, _state, _n, _v)                    \
> >>      VMSTATE_STRUCT_ARRAY(_field, _state, _n, _v, vmstate_mtrr_var, MTRRVar)
> >>  
> >> -static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size)
> >> +static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size,
> >> +                            VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      fprintf(stderr, "call put_fpreg() with invalid arguments\n");
> >>      exit(0);
> >> @@ -167,7 +168,8 @@ static void fp64_to_fp80(union x86_longdouble *p, uint64_t temp)
> >>      p->exp = e;
> >>  }
> >>  
> >> -static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
> >> +static int get_fpreg(QEMUFile *f, void *opaque, size_t size,
> >> +                     VMStateField *field)
> >>  {
> >>      FPReg *fp_reg = opaque;
> >>      uint64_t mant;
> >> @@ -179,7 +181,8 @@ static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_fpreg(QEMUFile *f, void *opaque, size_t size)
> >> +static void put_fpreg(QEMUFile *f, void *opaque, size_t size,
> >> +                      VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      FPReg *fp_reg = opaque;
> >>      uint64_t mant;
> >> @@ -197,7 +200,8 @@ static const VMStateInfo vmstate_fpreg = {
> >>      .put  = put_fpreg,
> >>  };
> >>  
> >> -static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size)
> >> +static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size,
> >> +                           VMStateField *field)
> >>  {
> >>      union x86_longdouble *p = opaque;
> >>      uint64_t mant;
> >> @@ -214,7 +218,8 @@ static const VMStateInfo vmstate_fpreg_1_mmx = {
> >>      .put  = put_fpreg_error,
> >>  };
> >>  
> >> -static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size)
> >> +static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size,
> >> +                              VMStateField *field)
> >>  {
> >>      union x86_longdouble *p = opaque;
> >>      uint64_t mant;
> >> @@ -276,14 +281,16 @@ static bool less_than_7(void *opaque, int version_id)
> >>      return version_id < 7;
> >>  }
> >>  
> >> -static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
> >> +static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
> >> +                                VMStateField *field)
> >>  {
> >>      uint64_t *v = pv;
> >>      *v = qemu_get_be32(f);
> >>      return 0;
> >>  }
> >>  
> >> -static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
> >> +static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
> >> +                                 VMStateField *field, QJSON *vmdesc)
> >>  {
> >>      uint64_t *v = pv;
> >>      qemu_put_be32(f, *v);
> >> diff --git a/target-mips/machine.c b/target-mips/machine.c
> >> index a27f2f1..179084c 100644
> >> --- a/target-mips/machine.c
> >> +++ b/target-mips/machine.c
> >> @@ -20,7 +20,7 @@ static int cpu_post_load(void *opaque, int version_id)
> >>  
> >>  /* FPU state */
> >>  
> >> -static int get_fpr(QEMUFile *f, void *pv, size_t size)
> >> +static int get_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      int i;
> >>      fpr_t *v = pv;
> >> @@ -31,7 +31,8 @@ static int get_fpr(QEMUFile *f, void *pv, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_fpr(QEMUFile *f, void *pv, size_t size)
> >> +static void put_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                    QJSON *vmdesc)
> >>  {
> >>      int i;
> >>      fpr_t *v = pv;
> >> @@ -125,7 +126,7 @@ const VMStateDescription vmstate_mvp = {
> >>  
> >>  /* TLB state */
> >>  
> >> -static int get_tlb(QEMUFile *f, void *pv, size_t size)
> >> +static int get_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      r4k_tlb_t *v = pv;
> >>      uint16_t flags;
> >> @@ -152,7 +153,8 @@ static int get_tlb(QEMUFile *f, void *pv, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_tlb(QEMUFile *f, void *pv, size_t size)
> >> +static void put_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                    QJSON *vmdesc)
> >>  {
> >>      r4k_tlb_t *v = pv;
> >>  
> >> diff --git a/target-ppc/machine.c b/target-ppc/machine.c
> >> index 4820f22..0e1822c 100644
> >> --- a/target-ppc/machine.c
> >> +++ b/target-ppc/machine.c
> >> @@ -106,7 +106,7 @@ static int cpu_load_old(QEMUFile *f, void *opaque, int version_id)
> >>      return 0;
> >>  }
> >>  
> >> -static int get_avr(QEMUFile *f, void *pv, size_t size)
> >> +static int get_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      ppc_avr_t *v = pv;
> >>  
> >> @@ -116,7 +116,8 @@ static int get_avr(QEMUFile *f, void *pv, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_avr(QEMUFile *f, void *pv, size_t size)
> >> +static void put_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                    QJSON *vmdesc)
> >>  {
> >>      ppc_avr_t *v = pv;
> >>  
> >> @@ -324,7 +325,7 @@ static const VMStateDescription vmstate_sr = {
> >>  };
> >>  
> >>  #ifdef TARGET_PPC64
> >> -static int get_slbe(QEMUFile *f, void *pv, size_t size)
> >> +static int get_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> >>  {
> >>      ppc_slb_t *v = pv;
> >>  
> >> @@ -334,7 +335,8 @@ static int get_slbe(QEMUFile *f, void *pv, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_slbe(QEMUFile *f, void *pv, size_t size)
> >> +static void put_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> >> +                     QJSON *vmdesc)
> >>  {
> >>      ppc_slb_t *v = pv;
> >>  
> >> diff --git a/target-sparc/machine.c b/target-sparc/machine.c
> >> index 59c92f7..3194e03 100644
> >> --- a/target-sparc/machine.c
> >> +++ b/target-sparc/machine.c
> >> @@ -59,7 +59,7 @@ static const VMStateDescription vmstate_tlb_entry = {
> >>  };
> >>  #endif
> >>  
> >> -static int get_psr(QEMUFile *f, void *opaque, size_t size)
> >> +static int get_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
> >>  {
> >>      SPARCCPU *cpu = opaque;
> >>      CPUSPARCState *env = &cpu->env;
> >> @@ -72,7 +72,8 @@ static int get_psr(QEMUFile *f, void *opaque, size_t size)
> >>      return 0;
> >>  }
> >>  
> >> -static void put_psr(QEMUFile *f, void *opaque, size_t size)
> >> +static void put_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field,
> >> +                QJSON *vmdesc)
> >>  {
> >>      SPARCCPU *cpu = opaque;
> >>      CPUSPARCState *env = &cpu->env;
> >> -- 
> >> 1.9.1
> >>
> >>
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 34876 bytes --]

On Fri, Oct 07, 2016 at 07:42:12PM +0100, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > 
> > 
> > On 10/07/2016 05:08 AM, Dr. David Alan Gilbert wrote:
> > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > >> Current migration code cannot handle some data structures such as
> > >> QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
> > >> in VMStateInfo so that customized handling is supported.
> > >>
> > >> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> > >> ---
> > >>  hw/net/vmxnet3.c            | 18 ++++++---
> > >>  hw/nvram/eeprom93xx.c       |  6 ++-
> > >>  hw/nvram/fw_cfg.c           |  6 ++-
> > >>  hw/pci/msix.c               |  6 ++-
> > >>  hw/pci/pci.c                | 12 ++++--
> > >>  hw/pci/shpc.c               |  5 ++-
> > >>  hw/scsi/scsi-bus.c          |  6 ++-
> > >>  hw/timer/twl92230.c         |  6 ++-
> > >>  hw/usb/redirect.c           | 18 ++++++---
> > >>  hw/virtio/virtio-pci.c      |  6 ++-
> > >>  hw/virtio/virtio.c          |  6 ++-
> > >>  include/migration/vmstate.h | 10 +++--
> > >>  migration/savevm.c          |  5 ++-
> > >>  migration/vmstate.c         | 95 ++++++++++++++++++++++++++++-----------------
> > >>  target-alpha/machine.c      |  5 ++-
> > >>  target-arm/machine.c        | 12 ++++--
> > >>  target-i386/machine.c       | 21 ++++++----
> > >>  target-mips/machine.c       | 10 +++--
> > >>  target-ppc/machine.c        | 10 +++--
> > >>  target-sparc/machine.c      |  5 ++-
> > >>  20 files changed, 171 insertions(+), 97 deletions(-)
> > >>
> > > 
> > > <snip>
> > > 
> > >> diff --git a/hw/usb/redirect.c b/hw/usb/redirect.c
> > >> index 444672a..2ca4b46 100644
> > >> --- a/hw/usb/redirect.c
> > >> +++ b/hw/usb/redirect.c
> > >> @@ -2154,7 +2154,8 @@ static int usbredir_post_load(void *priv, int version_id)
> > >>  }
> > >>  
> > >>  /* For usbredirparser migration */
> > >> -static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
> > >> +static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused,
> > >> +                                void *opaque, QJSON *vmdesc)
> > >>  {
> > >>      USBRedirDevice *dev = priv;
> > >>      uint8_t *data;
> > >> @@ -2174,7 +2175,8 @@ static void usbredir_put_parser(QEMUFile *f, void *priv, size_t unused)
> > >>      free(data);
> > >>  }
> > >>  
> > >> -static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused)
> > >> +static int usbredir_get_parser(QEMUFile *f, void *priv, size_t unused,
> > >> +                               void *opaque)
> > > 
> > > Neither of these built for me; I had to change those to VMStateField rather than void *;
> > > 
> > > also is this series tested ontop of Halil's patches - because without them
> > > I'm finding I also had to fix up most of the other virtio devices.
> > > 
> > > Dave
> > 
> > I built it on top of ppc-for-2.8 without problems. Is Hail's patch in
> > ppc-for-2.8 yet?

If it's not in master, and not ppc specific, it won't be in ppc-for-2.8.

> I don't know about ppc-for-2.8;  patches for inclusion should work on the current
> head unless stated otherwise.

Right.  These started as ppc specific patches, for which being on top
of ppc-for-2.8 would be the right choice (ppc-for-2.8 regularly
rebases on master).  However, they now include a bunch of generic
migration changes, it should be done directly against master.  This
may require splitting the series into generic and ppc specific
portions.

> Also, make sure you have the usbredir libraries installed and as much else as possible
> to make sure you cover all the .get/.put functions - they're all over!
> 
> Dave
> 
> > Thanks,
> > Jianjun
> > 
> > > 
> > > 
> > >>      USBRedirDevice *dev = priv;
> > >>      uint8_t *data;
> > >> @@ -2217,7 +2219,8 @@ static const VMStateInfo usbredir_parser_vmstate_info = {
> > >>  
> > >>  
> > >>  /* For buffered packets (iso/irq) queue migration */
> > >> -static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
> > >> +static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused,
> > >> +                               VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      struct endp_data *endp = priv;
> > >>      USBRedirDevice *dev = endp->dev;
> > >> @@ -2237,7 +2240,8 @@ static void usbredir_put_bufpq(QEMUFile *f, void *priv, size_t unused)
> > >>      assert(i == endp->bufpq_size);
> > >>  }
> > >>  
> > >> -static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused)
> > >> +static int usbredir_get_bufpq(QEMUFile *f, void *priv, size_t unused,
> > >> +                              VMStateField *field)
> > >>  {
> > >>      struct endp_data *endp = priv;
> > >>      USBRedirDevice *dev = endp->dev;
> > >> @@ -2340,7 +2344,8 @@ static const VMStateDescription usbredir_ep_vmstate = {
> > >>  
> > >>  
> > >>  /* For PacketIdQueue migration */
> > >> -static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> > >> +static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused,
> > >> +                                     VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      struct PacketIdQueue *q = priv;
> > >>      USBRedirDevice *dev = q->dev;
> > >> @@ -2356,7 +2361,8 @@ static void usbredir_put_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> > >>      assert(remain == 0);
> > >>  }
> > >>  
> > >> -static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused)
> > >> +static int usbredir_get_packet_id_q(QEMUFile *f, void *priv, size_t unused,
> > >> +                                    VMStateField *field)
> > >>  {
> > >>      struct PacketIdQueue *q = priv;
> > >>      USBRedirDevice *dev = q->dev;
> > >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> > >> index 2d60a00..38a7abd 100644
> > >> --- a/hw/virtio/virtio-pci.c
> > >> +++ b/hw/virtio/virtio-pci.c
> > >> @@ -108,7 +108,8 @@ static bool virtio_pci_has_extra_state(DeviceState *d)
> > >>      return proxy->flags & VIRTIO_PCI_FLAG_MIGRATE_EXTRA;
> > >>  }
> > >>  
> > >> -static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
> > >> +                                       VMStateField *field)
> > >>  {
> > >>      VirtIOPCIProxy *proxy = pv;
> > >>      int i;
> > >> @@ -137,7 +138,8 @@ static void virtio_pci_save_modern_queue_state(VirtIOPCIQueue *vq,
> > >>      qemu_put_be32(f, vq->used[1]);
> > >>  }
> > >>  
> > >> -static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_virtio_pci_modern_state(QEMUFile *f, void *pv, size_t size,
> > >> +                                        VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      VirtIOPCIProxy *proxy = pv;
> > >>      int i;
> > >> diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
> > >> index 18ce333..ade2683 100644
> > >> --- a/hw/virtio/virtio.c
> > >> +++ b/hw/virtio/virtio.c
> > >> @@ -1467,7 +1467,8 @@ static const VMStateDescription vmstate_virtio_ringsize = {
> > >>      }
> > >>  };
> > >>  
> > >> -static int get_extra_state(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_extra_state(QEMUFile *f, void *pv, size_t size,
> > >> +                           VMStateField *field)
> > >>  {
> > >>      VirtIODevice *vdev = pv;
> > >>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
> > >> @@ -1480,7 +1481,8 @@ static int get_extra_state(QEMUFile *f, void *pv, size_t size)
> > >>      }
> > >>  }
> > >>  
> > >> -static void put_extra_state(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_extra_state(QEMUFile *f, void *pv, size_t size,
> > >> +                            VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      VirtIODevice *vdev = pv;
> > >>      BusState *qbus = qdev_get_parent_bus(DEVICE(vdev));
> > >> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> > >> index 1638ee5..459dd4a 100644
> > >> --- a/include/migration/vmstate.h
> > >> +++ b/include/migration/vmstate.h
> > >> @@ -81,11 +81,13 @@ void unregister_savevm(DeviceState *dev, const char *idstr, void *opaque);
> > >>  
> > >>  typedef struct VMStateInfo VMStateInfo;
> > >>  typedef struct VMStateDescription VMStateDescription;
> > >> +typedef struct VMStateField VMStateField;
> > >>  
> > >>  struct VMStateInfo {
> > >>      const char *name;
> > >> -    int (*get)(QEMUFile *f, void *pv, size_t size);
> > >> -    void (*put)(QEMUFile *f, void *pv, size_t size);
> > >> +    int (*get)(QEMUFile *f, void *pv, size_t size, VMStateField *field);
> > >> +    void (*put)(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                QJSON *vmdesc);
> > >>  };
> > >>  
> > >>  enum VMStateFlags {
> > >> @@ -186,7 +188,7 @@ enum VMStateFlags {
> > >>      VMS_MULTIPLY_ELEMENTS = 0x4000,
> > >>  };
> > >>  
> > >> -typedef struct {
> > >> +struct VMStateField {
> > >>      const char *name;
> > >>      size_t offset;
> > >>      size_t size;
> > >> @@ -199,7 +201,7 @@ typedef struct {
> > >>      const VMStateDescription *vmsd;
> > >>      int version_id;
> > >>      bool (*field_exists)(void *opaque, int version_id);
> > >> -} VMStateField;
> > >> +};
> > >>  
> > >>  struct VMStateDescription {
> > >>      const char *name;
> > >> diff --git a/migration/savevm.c b/migration/savevm.c
> > >> index ef5c3d1..13b53ad 100644
> > >> --- a/migration/savevm.c
> > >> +++ b/migration/savevm.c
> > >> @@ -220,14 +220,15 @@ void timer_get(QEMUFile *f, QEMUTimer *ts)
> > >>   * Not in vmstate.c to not add qemu-timer.c as dependency to vmstate.c
> > >>   */
> > >>  
> > >> -static int get_timer(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      QEMUTimer *v = pv;
> > >>      timer_get(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_timer(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_timer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                QJSON *vmdesc)
> > >>  {
> > >>      QEMUTimer *v = pv;
> > >>      timer_put(f, v);
> > >> diff --git a/migration/vmstate.c b/migration/vmstate.c
> > >> index fc29acf..66802cb 100644
> > >> --- a/migration/vmstate.c
> > >> +++ b/migration/vmstate.c
> > >> @@ -122,7 +122,7 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
> > >>                      ret = vmstate_load_state(f, field->vmsd, addr,
> > >>                                               field->vmsd->version_id);
> > >>                  } else {
> > >> -                    ret = field->info->get(f, addr, size);
> > >> +                    ret = field->info->get(f, addr, size, NULL);
> > >>  
> > >>                  }
> > >>                  if (ret >= 0) {
> > >> @@ -328,7 +328,7 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
> > >>                  if (field->flags & VMS_STRUCT) {
> > >>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
> > >>                  } else {
> > >> -                    field->info->put(f, addr, size);
> > >> +                    field->info->put(f, addr, size, NULL, NULL);
> > >>                  }
> > >>  
> > >>                  written_bytes = qemu_ftell_fast(f) - old_offset;
> > >> @@ -461,14 +461,15 @@ static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
> > >>  
> > >>  /* bool */
> > >>  
> > >> -static int get_bool(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      bool *v = pv;
> > >>      *v = qemu_get_byte(f);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_bool(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_bool(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                     QJSON *vmdesc)
> > >>  {
> > >>      bool *v = pv;
> > >>      qemu_put_byte(f, *v);
> > >> @@ -482,14 +483,15 @@ const VMStateInfo vmstate_info_bool = {
> > >>  
> > >>  /* 8 bit int */
> > >>  
> > >> -static int get_int8(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      int8_t *v = pv;
> > >>      qemu_get_s8s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_int8(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_int8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                     QJSON *vmdesc)
> > >>  {
> > >>      int8_t *v = pv;
> > >>      qemu_put_s8s(f, v);
> > >> @@ -503,14 +505,15 @@ const VMStateInfo vmstate_info_int8 = {
> > >>  
> > >>  /* 16 bit int */
> > >>  
> > >> -static int get_int16(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      int16_t *v = pv;
> > >>      qemu_get_sbe16s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_int16(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_int16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                      QJSON *vmdesc)
> > >>  {
> > >>      int16_t *v = pv;
> > >>      qemu_put_sbe16s(f, v);
> > >> @@ -524,14 +527,15 @@ const VMStateInfo vmstate_info_int16 = {
> > >>  
> > >>  /* 32 bit int */
> > >>  
> > >> -static int get_int32(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      int32_t *v = pv;
> > >>      qemu_get_sbe32s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_int32(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_int32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                      QJSON *vmdesc)
> > >>  {
> > >>      int32_t *v = pv;
> > >>      qemu_put_sbe32s(f, v);
> > >> @@ -546,7 +550,8 @@ const VMStateInfo vmstate_info_int32 = {
> > >>  /* 32 bit int. See that the received value is the same than the one
> > >>     in the field */
> > >>  
> > >> -static int get_int32_equal(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_int32_equal(QEMUFile *f, void *pv, size_t size,
> > >> +                           VMStateField *field)
> > >>  {
> > >>      int32_t *v = pv;
> > >>      int32_t v2;
> > >> @@ -568,7 +573,7 @@ const VMStateInfo vmstate_info_int32_equal = {
> > >>   * and less than or equal to the one in the field.
> > >>   */
> > >>  
> > >> -static int get_int32_le(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_int32_le(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      int32_t *cur = pv;
> > >>      int32_t loaded;
> > >> @@ -589,14 +594,15 @@ const VMStateInfo vmstate_info_int32_le = {
> > >>  
> > >>  /* 64 bit int */
> > >>  
> > >> -static int get_int64(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      int64_t *v = pv;
> > >>      qemu_get_sbe64s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_int64(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_int64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                      QJSON *vmdesc)
> > >>  {
> > >>      int64_t *v = pv;
> > >>      qemu_put_sbe64s(f, v);
> > >> @@ -610,14 +616,15 @@ const VMStateInfo vmstate_info_int64 = {
> > >>  
> > >>  /* 8 bit unsigned int */
> > >>  
> > >> -static int get_uint8(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      uint8_t *v = pv;
> > >>      qemu_get_8s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_uint8(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_uint8(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                      QJSON *vmdesc)
> > >>  {
> > >>      uint8_t *v = pv;
> > >>      qemu_put_8s(f, v);
> > >> @@ -631,14 +638,15 @@ const VMStateInfo vmstate_info_uint8 = {
> > >>  
> > >>  /* 16 bit unsigned int */
> > >>  
> > >> -static int get_uint16(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      uint16_t *v = pv;
> > >>      qemu_get_be16s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_uint16(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_uint16(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                       QJSON *vmdesc)
> > >>  {
> > >>      uint16_t *v = pv;
> > >>      qemu_put_be16s(f, v);
> > >> @@ -652,14 +660,15 @@ const VMStateInfo vmstate_info_uint16 = {
> > >>  
> > >>  /* 32 bit unsigned int */
> > >>  
> > >> -static int get_uint32(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      uint32_t *v = pv;
> > >>      qemu_get_be32s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_uint32(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_uint32(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                       QJSON *vmdesc)
> > >>  {
> > >>      uint32_t *v = pv;
> > >>      qemu_put_be32s(f, v);
> > >> @@ -674,7 +683,8 @@ const VMStateInfo vmstate_info_uint32 = {
> > >>  /* 32 bit uint. See that the received value is the same than the one
> > >>     in the field */
> > >>  
> > >> -static int get_uint32_equal(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint32_equal(QEMUFile *f, void *pv, size_t size,
> > >> +                            VMStateField *field)
> > >>  {
> > >>      uint32_t *v = pv;
> > >>      uint32_t v2;
> > >> @@ -694,14 +704,15 @@ const VMStateInfo vmstate_info_uint32_equal = {
> > >>  
> > >>  /* 64 bit unsigned int */
> > >>  
> > >> -static int get_uint64(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      uint64_t *v = pv;
> > >>      qemu_get_be64s(f, v);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_uint64(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_uint64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                       QJSON *vmdesc)
> > >>  {
> > >>      uint64_t *v = pv;
> > >>      qemu_put_be64s(f, v);
> > >> @@ -716,7 +727,8 @@ const VMStateInfo vmstate_info_uint64 = {
> > >>  /* 64 bit unsigned int. See that the received value is the same than the one
> > >>     in the field */
> > >>  
> > >> -static int get_uint64_equal(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint64_equal(QEMUFile *f, void *pv, size_t size,
> > >> +                            VMStateField *field)
> > >>  {
> > >>      uint64_t *v = pv;
> > >>      uint64_t v2;
> > >> @@ -737,7 +749,8 @@ const VMStateInfo vmstate_info_uint64_equal = {
> > >>  /* 8 bit int. See that the received value is the same than the one
> > >>     in the field */
> > >>  
> > >> -static int get_uint8_equal(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint8_equal(QEMUFile *f, void *pv, size_t size,
> > >> +                           VMStateField *field)
> > >>  {
> > >>      uint8_t *v = pv;
> > >>      uint8_t v2;
> > >> @@ -758,7 +771,8 @@ const VMStateInfo vmstate_info_uint8_equal = {
> > >>  /* 16 bit unsigned int int. See that the received value is the same than the one
> > >>     in the field */
> > >>  
> > >> -static int get_uint16_equal(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint16_equal(QEMUFile *f, void *pv, size_t size,
> > >> +                            VMStateField *field)
> > >>  {
> > >>      uint16_t *v = pv;
> > >>      uint16_t v2;
> > >> @@ -778,7 +792,8 @@ const VMStateInfo vmstate_info_uint16_equal = {
> > >>  
> > >>  /* floating point */
> > >>  
> > >> -static int get_float64(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_float64(QEMUFile *f, void *pv, size_t size,
> > >> +                       VMStateField *field)
> > >>  {
> > >>      float64 *v = pv;
> > >>  
> > >> @@ -786,7 +801,8 @@ static int get_float64(QEMUFile *f, void *pv, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_float64(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_float64(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                        QJSON *vmdesc)
> > >>  {
> > >>      uint64_t *v = pv;
> > >>  
> > >> @@ -801,7 +817,8 @@ const VMStateInfo vmstate_info_float64 = {
> > >>  
> > >>  /* CPU_DoubleU type */
> > >>  
> > >> -static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_cpudouble(QEMUFile *f, void *pv, size_t size,
> > >> +                         VMStateField *field)
> > >>  {
> > >>      CPU_DoubleU *v = pv;
> > >>      qemu_get_be32s(f, &v->l.upper);
> > >> @@ -809,7 +826,8 @@ static int get_cpudouble(QEMUFile *f, void *pv, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_cpudouble(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_cpudouble(QEMUFile *f, void *pv, size_t size,
> > >> +                          VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      CPU_DoubleU *v = pv;
> > >>      qemu_put_be32s(f, &v->l.upper);
> > >> @@ -824,14 +842,16 @@ const VMStateInfo vmstate_info_cpudouble = {
> > >>  
> > >>  /* uint8_t buffers */
> > >>  
> > >> -static int get_buffer(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_buffer(QEMUFile *f, void *pv, size_t size,
> > >> +                      VMStateField *field)
> > >>  {
> > >>      uint8_t *v = pv;
> > >>      qemu_get_buffer(f, v, size);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_buffer(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_buffer(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                       QJSON *vmdesc)
> > >>  {
> > >>      uint8_t *v = pv;
> > >>      qemu_put_buffer(f, v, size);
> > >> @@ -846,7 +866,8 @@ const VMStateInfo vmstate_info_buffer = {
> > >>  /* unused buffers: space that was used for some fields that are
> > >>     not useful anymore */
> > >>  
> > >> -static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_unused_buffer(QEMUFile *f, void *pv, size_t size,
> > >> +                             VMStateField *field)
> > >>  {
> > >>      uint8_t buf[1024];
> > >>      int block_len;
> > >> @@ -859,7 +880,8 @@ static int get_unused_buffer(QEMUFile *f, void *pv, size_t size)
> > >>     return 0;
> > >>  }
> > >>  
> > >> -static void put_unused_buffer(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_unused_buffer(QEMUFile *f, void *pv, size_t size,
> > >> +                              VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      static const uint8_t buf[1024];
> > >>      int block_len;
> > >> @@ -884,7 +906,7 @@ const VMStateInfo vmstate_info_unused_buffer = {
> > >>   */
> > >>  /* This is the number of 64 bit words sent over the wire */
> > >>  #define BITS_TO_U64S(nr) DIV_ROUND_UP(nr, 64)
> > >> -static int get_bitmap(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      unsigned long *bmp = pv;
> > >>      int i, idx = 0;
> > >> @@ -898,7 +920,8 @@ static int get_bitmap(QEMUFile *f, void *pv, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_bitmap(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_bitmap(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                QJSON *vmdesc)
> > >>  {
> > >>      unsigned long *bmp = pv;
> > >>      int i, idx = 0;
> > >> diff --git a/target-alpha/machine.c b/target-alpha/machine.c
> > >> index 710b783..48e3278 100644
> > >> --- a/target-alpha/machine.c
> > >> +++ b/target-alpha/machine.c
> > >> @@ -5,14 +5,15 @@
> > >>  #include "hw/boards.h"
> > >>  #include "migration/cpu.h"
> > >>  
> > >> -static int get_fpcr(QEMUFile *f, void *opaque, size_t size)
> > >> +static int get_fpcr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
> > >>  {
> > >>      CPUAlphaState *env = opaque;
> > >>      cpu_alpha_store_fpcr(env, qemu_get_be64(f));
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_fpcr(QEMUFile *f, void *opaque, size_t size)
> > >> +static void put_fpcr(QEMUFile *f, void *opaque, size_t size,
> > >> +                     VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      CPUAlphaState *env = opaque;
> > >>      qemu_put_be64(f, cpu_alpha_load_fpcr(env));
> > >> diff --git a/target-arm/machine.c b/target-arm/machine.c
> > >> index 7a6ca31..41a0de9 100644
> > >> --- a/target-arm/machine.c
> > >> +++ b/target-arm/machine.c
> > >> @@ -17,7 +17,8 @@ static bool vfp_needed(void *opaque)
> > >>      return arm_feature(env, ARM_FEATURE_VFP);
> > >>  }
> > >>  
> > >> -static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
> > >> +static int get_fpscr(QEMUFile *f, void *opaque, size_t size,
> > >> +                     VMStateField *field)
> > >>  {
> > >>      ARMCPU *cpu = opaque;
> > >>      CPUARMState *env = &cpu->env;
> > >> @@ -27,7 +28,8 @@ static int get_fpscr(QEMUFile *f, void *opaque, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_fpscr(QEMUFile *f, void *opaque, size_t size)
> > >> +static void put_fpscr(QEMUFile *f, void *opaque, size_t size,
> > >> +                      VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      ARMCPU *cpu = opaque;
> > >>      CPUARMState *env = &cpu->env;
> > >> @@ -163,7 +165,8 @@ static const VMStateDescription vmstate_pmsav7 = {
> > >>      }
> > >>  };
> > >>  
> > >> -static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
> > >> +static int get_cpsr(QEMUFile *f, void *opaque, size_t size,
> > >> +                    VMStateField *field)
> > >>  {
> > >>      ARMCPU *cpu = opaque;
> > >>      CPUARMState *env = &cpu->env;
> > >> @@ -180,7 +183,8 @@ static int get_cpsr(QEMUFile *f, void *opaque, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_cpsr(QEMUFile *f, void *opaque, size_t size)
> > >> +static void put_cpsr(QEMUFile *f, void *opaque, size_t size,
> > >> +                     VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      ARMCPU *cpu = opaque;
> > >>      CPUARMState *env = &cpu->env;
> > >> diff --git a/target-i386/machine.c b/target-i386/machine.c
> > >> index 71c0e4d..1df19e2 100644
> > >> --- a/target-i386/machine.c
> > >> +++ b/target-i386/machine.c
> > >> @@ -139,7 +139,8 @@ static const VMStateDescription vmstate_mtrr_var = {
> > >>  #define VMSTATE_MTRR_VARS(_field, _state, _n, _v)                    \
> > >>      VMSTATE_STRUCT_ARRAY(_field, _state, _n, _v, vmstate_mtrr_var, MTRRVar)
> > >>  
> > >> -static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size)
> > >> +static void put_fpreg_error(QEMUFile *f, void *opaque, size_t size,
> > >> +                            VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      fprintf(stderr, "call put_fpreg() with invalid arguments\n");
> > >>      exit(0);
> > >> @@ -167,7 +168,8 @@ static void fp64_to_fp80(union x86_longdouble *p, uint64_t temp)
> > >>      p->exp = e;
> > >>  }
> > >>  
> > >> -static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
> > >> +static int get_fpreg(QEMUFile *f, void *opaque, size_t size,
> > >> +                     VMStateField *field)
> > >>  {
> > >>      FPReg *fp_reg = opaque;
> > >>      uint64_t mant;
> > >> @@ -179,7 +181,8 @@ static int get_fpreg(QEMUFile *f, void *opaque, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_fpreg(QEMUFile *f, void *opaque, size_t size)
> > >> +static void put_fpreg(QEMUFile *f, void *opaque, size_t size,
> > >> +                      VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      FPReg *fp_reg = opaque;
> > >>      uint64_t mant;
> > >> @@ -197,7 +200,8 @@ static const VMStateInfo vmstate_fpreg = {
> > >>      .put  = put_fpreg,
> > >>  };
> > >>  
> > >> -static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size)
> > >> +static int get_fpreg_1_mmx(QEMUFile *f, void *opaque, size_t size,
> > >> +                           VMStateField *field)
> > >>  {
> > >>      union x86_longdouble *p = opaque;
> > >>      uint64_t mant;
> > >> @@ -214,7 +218,8 @@ static const VMStateInfo vmstate_fpreg_1_mmx = {
> > >>      .put  = put_fpreg_error,
> > >>  };
> > >>  
> > >> -static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size)
> > >> +static int get_fpreg_1_no_mmx(QEMUFile *f, void *opaque, size_t size,
> > >> +                              VMStateField *field)
> > >>  {
> > >>      union x86_longdouble *p = opaque;
> > >>      uint64_t mant;
> > >> @@ -276,14 +281,16 @@ static bool less_than_7(void *opaque, int version_id)
> > >>      return version_id < 7;
> > >>  }
> > >>  
> > >> -static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
> > >> +                                VMStateField *field)
> > >>  {
> > >>      uint64_t *v = pv;
> > >>      *v = qemu_get_be32(f);
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_uint64_as_uint32(QEMUFile *f, void *pv, size_t size,
> > >> +                                 VMStateField *field, QJSON *vmdesc)
> > >>  {
> > >>      uint64_t *v = pv;
> > >>      qemu_put_be32(f, *v);
> > >> diff --git a/target-mips/machine.c b/target-mips/machine.c
> > >> index a27f2f1..179084c 100644
> > >> --- a/target-mips/machine.c
> > >> +++ b/target-mips/machine.c
> > >> @@ -20,7 +20,7 @@ static int cpu_post_load(void *opaque, int version_id)
> > >>  
> > >>  /* FPU state */
> > >>  
> > >> -static int get_fpr(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      int i;
> > >>      fpr_t *v = pv;
> > >> @@ -31,7 +31,8 @@ static int get_fpr(QEMUFile *f, void *pv, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_fpr(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_fpr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                    QJSON *vmdesc)
> > >>  {
> > >>      int i;
> > >>      fpr_t *v = pv;
> > >> @@ -125,7 +126,7 @@ const VMStateDescription vmstate_mvp = {
> > >>  
> > >>  /* TLB state */
> > >>  
> > >> -static int get_tlb(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      r4k_tlb_t *v = pv;
> > >>      uint16_t flags;
> > >> @@ -152,7 +153,8 @@ static int get_tlb(QEMUFile *f, void *pv, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_tlb(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_tlb(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                    QJSON *vmdesc)
> > >>  {
> > >>      r4k_tlb_t *v = pv;
> > >>  
> > >> diff --git a/target-ppc/machine.c b/target-ppc/machine.c
> > >> index 4820f22..0e1822c 100644
> > >> --- a/target-ppc/machine.c
> > >> +++ b/target-ppc/machine.c
> > >> @@ -106,7 +106,7 @@ static int cpu_load_old(QEMUFile *f, void *opaque, int version_id)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static int get_avr(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      ppc_avr_t *v = pv;
> > >>  
> > >> @@ -116,7 +116,8 @@ static int get_avr(QEMUFile *f, void *pv, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_avr(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_avr(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                    QJSON *vmdesc)
> > >>  {
> > >>      ppc_avr_t *v = pv;
> > >>  
> > >> @@ -324,7 +325,7 @@ static const VMStateDescription vmstate_sr = {
> > >>  };
> > >>  
> > >>  #ifdef TARGET_PPC64
> > >> -static int get_slbe(QEMUFile *f, void *pv, size_t size)
> > >> +static int get_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field)
> > >>  {
> > >>      ppc_slb_t *v = pv;
> > >>  
> > >> @@ -334,7 +335,8 @@ static int get_slbe(QEMUFile *f, void *pv, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_slbe(QEMUFile *f, void *pv, size_t size)
> > >> +static void put_slbe(QEMUFile *f, void *pv, size_t size, VMStateField *field,
> > >> +                     QJSON *vmdesc)
> > >>  {
> > >>      ppc_slb_t *v = pv;
> > >>  
> > >> diff --git a/target-sparc/machine.c b/target-sparc/machine.c
> > >> index 59c92f7..3194e03 100644
> > >> --- a/target-sparc/machine.c
> > >> +++ b/target-sparc/machine.c
> > >> @@ -59,7 +59,7 @@ static const VMStateDescription vmstate_tlb_entry = {
> > >>  };
> > >>  #endif
> > >>  
> > >> -static int get_psr(QEMUFile *f, void *opaque, size_t size)
> > >> +static int get_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field)
> > >>  {
> > >>      SPARCCPU *cpu = opaque;
> > >>      CPUSPARCState *env = &cpu->env;
> > >> @@ -72,7 +72,8 @@ static int get_psr(QEMUFile *f, void *opaque, size_t size)
> > >>      return 0;
> > >>  }
> > >>  
> > >> -static void put_psr(QEMUFile *f, void *opaque, size_t size)
> > >> +static void put_psr(QEMUFile *f, void *opaque, size_t size, VMStateField *field,
> > >> +                QJSON *vmdesc)
> > >>  {
> > >>      SPARCCPU *cpu = opaque;
> > >>      CPUSPARCState *env = &cpu->env;
> > >>
> > >>
> > 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Halil Pasic]

[-- Attachment #1: Type: text/plain, Size: 3423 bytes --]



On 10/03/2016 08:24 PM, Jianjun Duan wrote:
> Current migration code cannot handle some data structures such as
> QTAILQ in qemu/queue.h. Here we extend the signatures of put/get
> in VMStateInfo so that customized handling is supported.
> 
> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> ---
>  hw/net/vmxnet3.c            | 18 ++++++---
>  hw/nvram/eeprom93xx.c       |  6 ++-
>  hw/nvram/fw_cfg.c           |  6 ++-
>  hw/pci/msix.c               |  6 ++-
>  hw/pci/pci.c                | 12 ++++--
>  hw/pci/shpc.c               |  5 ++-
>  hw/scsi/scsi-bus.c          |  6 ++-
>  hw/timer/twl92230.c         |  6 ++-
>  hw/usb/redirect.c           | 18 ++++++---
>  hw/virtio/virtio-pci.c      |  6 ++-
>  hw/virtio/virtio.c          |  6 ++-
>  include/migration/vmstate.h | 10 +++--
>  migration/savevm.c          |  5 ++-
>  migration/vmstate.c         | 95 ++++++++++++++++++++++++++++-----------------
>  target-alpha/machine.c      |  5 ++-
>  target-arm/machine.c        | 12 ++++--
>  target-i386/machine.c       | 21 ++++++----
>  target-mips/machine.c       | 10 +++--
>  target-ppc/machine.c        | 10 +++--
>  target-sparc/machine.c      |  5 ++-
>  20 files changed, 171 insertions(+), 97 deletions(-)

Hi Jianjun!

I'm not happy with the intrusive nature of this patch. We end up
with a bunch of unused parameters. 

Have you considered something like:

typedef struct {
[..]
    union{
        const VMStateInfo *info;
        const VMStateLinked *linked;
    };
    enum VMStateFlags flags;
[..]
} VMStateField;

/**
*  Handle linked datastructures. VMStateField.liked s to be used
* in conjunction with VMStateField.vmsd which describes a node of
* the datastucture without the pointers representing the links.
* The links are embedded in the node starting at VMStateField.start.
* The on wire representation of the information contained in links
* and the head element if the responsibility of a particular VMStateField
* instance.  VMStateLinked is responsible for saving/loading the whole 
* sub-tree whose root is the field in question including the allocation of
* memory for the nodes. The presence of VMStateField.linked is indicated
* by the VMS_LINKED flag in VMStateField.flags.
*/
struct VMStateLinked {                                       
    const char *name;                                                           
    void (*save)(QEMUFile *f, void *pv, VMStateField *field, QJSON *vmdesc);
    int (*load)(QEMUFile *f, void *pv, VMStateField *field);                                     
    /* Maybe: size_t offset_links; */                                                    
};

IMHO this would:
* allow us to keep the good old MVStateInfo objects unmodified and
  the semantic of VMStateInfo unchanged
* make clear that VMStateLinked does not care about the calculated size
  and provide a new anchor for documentation
* if overloading the semantic of VMStateField.start is not desired we
  could put it into  VMStateLinked, if needed we could also put more
  stuff in there
* have clearer separation between special handling for (linked/certain)
  data structures and the usual scenario with the DAG.

I would also suggest unit tests in test/test-vmstate.c. Maybe with
that the vmstate migration of QTAILQ could be factored out as a separate
patch series.

Cheers,
Halil



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Paolo Bonzini]

On 12/10/2016 13:59, Halil Pasic wrote:
> IMHO this would:
> * allow us to keep the good old MVStateInfo objects unmodified and
>   the semantic of VMStateInfo unchanged
> * make clear that VMStateLinked does not care about the calculated size
>   and provide a new anchor for documentation
> * if overloading the semantic of VMStateField.start is not desired we
>   could put it into  VMStateLinked, if needed we could also put more
>   stuff in there
> * have clearer separation between special handling for (linked/certain)
>   data structures and the usual scenario with the DAG.

No, I disagree.  We shouldn't be worried about making intrusive changes
to all invocations or declarations, if that leads to a simpler API.

I agree that overloading .start can be a bit ugly but you can choose to
overload .num_offset instead, which is already better.

> I would also suggest unit tests in test/test-vmstate.c. Maybe with
> that the vmstate migration of QTAILQ could be factored out as a separate
> patch series.

Yes, definitely.

Paolo

Re: [Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Halil Pasic]

On 10/12/2016 02:07 PM, Paolo Bonzini wrote:
> 
> On 12/10/2016 13:59, Halil Pasic wrote:
>> > IMHO this would:
>> > * allow us to keep the good old MVStateInfo objects unmodified and
>> >   the semantic of VMStateInfo unchanged
>> > * make clear that VMStateLinked does not care about the calculated size
>> >   and provide a new anchor for documentation
>> > * if overloading the semantic of VMStateField.start is not desired we
>> >   could put it into  VMStateLinked, if needed we could also put more
>> >   stuff in there
>> > * have clearer separation between special handling for (linked/certain)
>> >   data structures and the usual scenario with the DAG.
> No, I disagree.  We shouldn't be worried about making intrusive changes
> to all invocations or declarations, if that leads to a simpler API.

Paolo I agree on a theoretical level. It's just I do not see why this
particular change makes the API simpler. In my opinion this complicates
things because now all VMStateInfo's can do funky stuff. Having additional
state you can poke is rarely a simplification. Same goes for lots
of arguments especially if some of them are barely ever used. These
additional parameters contribute nothing for the large majority
of the cases (except maybe some head scratching when reading
the code).

No strong opinion here, it's just that I don't understand. I think one
trait of a simple API is that it is easy to document. Unfortunately
the documentation is quite sparse and in the patch the signature
change goes undocumented.

Well as I said, just an idea, motivated by how I understood he role of
VMStateInfo form reading the code.

Cheers,
Halil

Re: [Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Dr. David Alan Gilbert]

* Halil Pasic (pasic@linux.vnet.ibm.com) wrote:
> 
> 
> On 10/12/2016 02:07 PM, Paolo Bonzini wrote:
> > 
> > On 12/10/2016 13:59, Halil Pasic wrote:
> >> > IMHO this would:
> >> > * allow us to keep the good old MVStateInfo objects unmodified and
> >> >   the semantic of VMStateInfo unchanged
> >> > * make clear that VMStateLinked does not care about the calculated size
> >> >   and provide a new anchor for documentation
> >> > * if overloading the semantic of VMStateField.start is not desired we
> >> >   could put it into  VMStateLinked, if needed we could also put more
> >> >   stuff in there
> >> > * have clearer separation between special handling for (linked/certain)
> >> >   data structures and the usual scenario with the DAG.
> > No, I disagree.  We shouldn't be worried about making intrusive changes
> > to all invocations or declarations, if that leads to a simpler API.
> 
> Paolo I agree on a theoretical level. It's just I do not see why this
> particular change makes the API simpler. In my opinion this complicates
> things because now all VMStateInfo's can do funky stuff. Having additional
> state you can poke is rarely a simplification. Same goes for lots
> of arguments especially if some of them are barely ever used. These
> additional parameters contribute nothing for the large majority
> of the cases (except maybe some head scratching when reading
> the code).

I think it might depend how many VMStateInfo's we have.
My ideal rule would be there are no .get/.put implementations outside
of migration/ and we can trust that they would never do anything silly (right?);
so the extra parameters aren't going to be misused too badly.

However, we're probably quite a way from pulling all of the weirder
.get/.put implementations back in.

Dave

> No strong opinion here, it's just that I don't understand. I think one
> trait of a simple API is that it is easy to document. Unfortunately
> the documentation is quite sparse and in the patch the signature
> change goes undocumented.
> 
> Well as I said, just an idea, motivated by how I understood he role of
> VMStateInfo form reading the code.
> 
> Cheers,
> Halil
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Halil Pasic]

[-- Attachment #1: Type: text/plain, Size: 2364 bytes --]



On 10/12/2016 04:59 PM, Dr. David Alan Gilbert wrote:
>> Paolo I agree on a theoretical level. It's just I do not see why this
>> > particular change makes the API simpler. In my opinion this complicates
>> > things because now all VMStateInfo's can do funky stuff. Having additional
>> > state you can poke is rarely a simplification. Same goes for lots
>> > of arguments especially if some of them are barely ever used. These
>> > additional parameters contribute nothing for the large majority
>> > of the cases (except maybe some head scratching when reading
>> > the code).
> I think it might depend how many VMStateInfo's we have.
> My ideal rule would be there are no .get/.put implementations outside
> of migration/ and we can trust that they would never do anything silly (right?);

Your statement about ideally no .get/.put implementations outside
of migration/ is consistent with my initial understanding of VMStateInfo:
It's there to take care of the marshaling between the on wire representation
and the in memory representation of a single and preferably primitive
vmstate field (not consisting of further fields). Complex stuff like
arrays, structs, indirection via pointers and possibly allocation is
preferably handled elsewhere. So VMStateInfo is the basic building stones,
and the only place which should write to/read from the stream (in
ideal vmstate).

So in a perfect vmstate world complete type of VMStateInfo is not part of the
API (you do not care about how it's done outside vmstate/), but only the
(possibly pointers to) VMStateInfo's supported by the vmstate API.

Of course this is not realistic, at least at the moment.

On the other hand if VMStateInfo is meant for complete customization,
as Jianjun has put it, then it obviously has to be a full fledged member
of the API, and I think then your ideal rule makes no sense to me.

I also do think we will always need something for handling special
cases because we need to maintain compatibility -- see virtio migration
for example.

> so the extra parameters aren't going to be misused too badly.
> 

What would you consider bad misuse? I do not see this as a big concern
at the moment.

Cheers,
Halil

> However, we're probably quite a way from pulling all of the weirder
> .get/.put implementations back in.
> 
> Dave
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Dr. David Alan Gilbert]

* Halil Pasic (pasic@linux.vnet.ibm.com) wrote:
> 
> 
> On 10/12/2016 04:59 PM, Dr. David Alan Gilbert wrote:
> >> Paolo I agree on a theoretical level. It's just I do not see why this
> >> > particular change makes the API simpler. In my opinion this complicates
> >> > things because now all VMStateInfo's can do funky stuff. Having additional
> >> > state you can poke is rarely a simplification. Same goes for lots
> >> > of arguments especially if some of them are barely ever used. These
> >> > additional parameters contribute nothing for the large majority
> >> > of the cases (except maybe some head scratching when reading
> >> > the code).
> > I think it might depend how many VMStateInfo's we have.
> > My ideal rule would be there are no .get/.put implementations outside
> > of migration/ and we can trust that they would never do anything silly (right?);
> 
> Your statement about ideally no .get/.put implementations outside
> of migration/ is consistent with my initial understanding of VMStateInfo:
> It's there to take care of the marshaling between the on wire representation
> and the in memory representation of a single and preferably primitive
> vmstate field (not consisting of further fields). Complex stuff like
> arrays, structs, indirection via pointers and possibly allocation is
> preferably handled elsewhere. So VMStateInfo is the basic building stones,
> and the only place which should write to/read from the stream (in
> ideal vmstate).
> 
> So in a perfect vmstate world complete type of VMStateInfo is not part of the
> API (you do not care about how it's done outside vmstate/), but only the
> (possibly pointers to) VMStateInfo's supported by the vmstate API.
> 
> Of course this is not realistic, at least at the moment.
> 
> On the other hand if VMStateInfo is meant for complete customization,
> as Jianjun has put it, then it obviously has to be a full fledged member
> of the API, and I think then your ideal rule makes no sense to me.
> 
> I also do think we will always need something for handling special
> cases because we need to maintain compatibility -- see virtio migration
> for example.
> 
> > so the extra parameters aren't going to be misused too badly.
> > 
> 
> What would you consider bad misuse? I do not see this as a big concern
> at the moment.

I don't know; but the only justification for needing the VMS_LINKED flag
was that only those functions that were marked with VMS_LINKED would
be passed the new field.

Dave

> Cheers,
> Halil
> 
> > However, we're probably quite a way from pulling all of the weirder
> > .get/.put implementations back in.
> > 
> > Dave
> > 
> 



--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Jianjun Duan]

On 10/12/2016 05:07 AM, Paolo Bonzini wrote:
> 
> 
> On 12/10/2016 13:59, Halil Pasic wrote:
>> IMHO this would:
>> * allow us to keep the good old MVStateInfo objects unmodified and
>>   the semantic of VMStateInfo unchanged
>> * make clear that VMStateLinked does not care about the calculated size
>>   and provide a new anchor for documentation
>> * if overloading the semantic of VMStateField.start is not desired we
>>   could put it into  VMStateLinked, if needed we could also put more
>>   stuff in there
>> * have clearer separation between special handling for (linked/certain)
>>   data structures and the usual scenario with the DAG.
> 
> No, I disagree.  We shouldn't be worried about making intrusive changes
> to all invocations or declarations, if that leads to a simpler API.
> 
If VMStateInfo was meant for complete customization, then it was missing
some parts. I think the API is indeed simpler. Just like
definition for VMStateField. Not all of its fields are used all time.

> I agree that overloading .start can be a bit ugly but you can choose to
> overload .num_offset instead, which is already better.
>
I did considered num_offset. But it is associated with an actual field.
On the other hand, start means the start position of the q in the
structure. So it is not that far stretched.

>> I would also suggest unit tests in test/test-vmstate.c. Maybe with
>> that the vmstate migration of QTAILQ could be factored out as a separate
>> patch series.
> 
> Yes, definitely.
> 
This sounds a good idea. Will do it.

> Paolo
> 
Thanks,
Jianjun

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Paolo Bonzini]

> > No, I disagree.  We shouldn't be worried about making intrusive changes
> > to all invocations or declarations, if that leads to a simpler API.
> 
> If VMStateInfo was meant for complete customization, then it was missing
> some parts. I think the API is indeed simpler. Just like
> definition for VMStateField. Not all of its fields are used all time.

Code moves.  We can decide how much customization to allow of VMStateInfo.

You said it: "Not all of its fields are used all time".  Likewise, not
all arguments are used all time for get/put, but it's not an issue if they
are still there!  So this patch is good, but at the same time VMS_LINKED is
pointless.

Paolo

> > I agree that overloading .start can be a bit ugly but you can choose to
> > overload .num_offset instead, which is already better.
> >
> I did considered num_offset. But it is associated with an actual field.
> On the other hand, start means the start position of the q in the
> structure. So it is not that far stretched.
> 
> >> I would also suggest unit tests in test/test-vmstate.c. Maybe with
> >> that the vmstate migration of QTAILQ could be factored out as a separate
> >> patch series.
> > 
> > Yes, definitely.
> > 
> This sounds a good idea. Will do it.
> 
> > Paolo
> > 
> Thanks,
> Jianjun
> 
> 

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Halil Pasic]

[-- Attachment #1: Type: text/plain, Size: 1232 bytes --]



On 10/13/2016 10:22 AM, Paolo Bonzini wrote:
>>> No, I disagree.  We shouldn't be worried about making intrusive changes
>>> > > to all invocations or declarations, if that leads to a simpler API.
>> > 
>> > If VMStateInfo was meant for complete customization, then it was missing
>> > some parts. I think the API is indeed simpler. Just like
>> > definition for VMStateField. Not all of its fields are used all time.
> Code moves.  We can decide how much customization to allow of VMStateInfo.
> 
> You said it: "Not all of its fields are used all time".  Likewise, not
> all arguments are used all time for get/put, but it's not an issue if they
> are still there!  So this patch is good, but at the same time VMS_LINKED is
> pointless.
> 
> Paolo
> 

I'm fine with this. I just think, it would be nice if the contract between
the vmstate-core and the client code implementing VMStateInfo callbacks
could be documented, including when are certain parameters valid, what
they stand for, and how are they supposed to be used for the next version of
the patch. Just to improve readability. Would this be OK with everybody?

By the way the flag VMS_SINGLE is documented as ignored. Should we drop
it?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Paolo Bonzini]

On 13/10/2016 12:48, Halil Pasic wrote:
>> > 
> I'm fine with this. I just think, it would be nice if the contract between
> the vmstate-core and the client code implementing VMStateInfo callbacks
> could be documented, including when are certain parameters valid, what
> they stand for, and how are they supposed to be used for the next version of
> the patch. Just to improve readability. Would this be OK with everybody?
> 
> By the way the flag VMS_SINGLE is documented as ignored. Should we drop
> it?

Yes, I think so.

Paolo

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Jianjun Duan]

On 10/13/2016 03:48 AM, Halil Pasic wrote:
> 
> 
> On 10/13/2016 10:22 AM, Paolo Bonzini wrote:
>>>> No, I disagree.  We shouldn't be worried about making intrusive changes
>>>>>> to all invocations or declarations, if that leads to a simpler API.
>>>>
>>>> If VMStateInfo was meant for complete customization, then it was missing
>>>> some parts. I think the API is indeed simpler. Just like
>>>> definition for VMStateField. Not all of its fields are used all time.
>> Code moves.  We can decide how much customization to allow of VMStateInfo.
>>
>> You said it: "Not all of its fields are used all time".  Likewise, not
>> all arguments are used all time for get/put, but it's not an issue if they
>> are still there!  So this patch is good, but at the same time VMS_LINKED is
>> pointless.
>>
>> Paolo
>>
> 
> I'm fine with this. I just think, it would be nice if the contract between
> the vmstate-core and the client code implementing VMStateInfo callbacks
> could be documented, including when are certain parameters valid, what
> they stand for, and how are they supposed to be used for the next version of
> the patch. Just to improve readability. Would this be OK with everybody?
> 
> By the way the flag VMS_SINGLE is documented as ignored. Should we drop
> it?
> 
I will prepare the VMStateInfo and QTAIL stuff as a separated series. If
indeed VMS_SINGLE is ignored, I can drop it here. It is similar to
VMS_LINKED situation.

Thanks,
Jianjun
> 

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Halil Pasic]

On 10/13/2016 06:23 PM, Jianjun Duan wrote:
> 
> 
> On 10/13/2016 03:48 AM, Halil Pasic wrote:
>>
>>
>> On 10/13/2016 10:22 AM, Paolo Bonzini wrote:
>>>>> No, I disagree.  We shouldn't be worried about making intrusive changes
>>>>>>> to all invocations or declarations, if that leads to a simpler API.
>>>>>
>>>>> If VMStateInfo was meant for complete customization, then it was missing
>>>>> some parts. I think the API is indeed simpler. Just like
>>>>> definition for VMStateField. Not all of its fields are used all time.
>>> Code moves.  We can decide how much customization to allow of VMStateInfo.
>>>
>>> You said it: "Not all of its fields are used all time".  Likewise, not
>>> all arguments are used all time for get/put, but it's not an issue if they
>>> are still there!  So this patch is good, but at the same time VMS_LINKED is
>>> pointless.
>>>
>>> Paolo
>>>
>>
>> I'm fine with this. I just think, it would be nice if the contract between
>> the vmstate-core and the client code implementing VMStateInfo callbacks
>> could be documented, including when are certain parameters valid, what
>> they stand for, and how are they supposed to be used for the next version of
>> the patch. Just to improve readability. Would this be OK with everybody?
>>
>> By the way the flag VMS_SINGLE is documented as ignored. Should we drop
>> it?
>>
> I will prepare the VMStateInfo and QTAIL stuff as a separated series. If
> indeed VMS_SINGLE is ignored, I can drop it here. It is similar to
> VMS_LINKED situation.
> 
> Thanks,
> Jianjun

I think it's completely unrelated, so I would not lump it together with
the QTAILQ stuff.

How do you feel about the apidoc part?

Cheers,
Halil

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 3/6] migration: extend VMStateInfo

[Jianjun Duan]

On 10/13/2016 09:32 AM, Halil Pasic wrote:
> 
> 
> On 10/13/2016 06:23 PM, Jianjun Duan wrote:
>>
>>
>> On 10/13/2016 03:48 AM, Halil Pasic wrote:
>>>
>>>
>>> On 10/13/2016 10:22 AM, Paolo Bonzini wrote:
>>>>>> No, I disagree.  We shouldn't be worried about making intrusive changes
>>>>>>>> to all invocations or declarations, if that leads to a simpler API.
>>>>>>
>>>>>> If VMStateInfo was meant for complete customization, then it was missing
>>>>>> some parts. I think the API is indeed simpler. Just like
>>>>>> definition for VMStateField. Not all of its fields are used all time.
>>>> Code moves.  We can decide how much customization to allow of VMStateInfo.
>>>>
>>>> You said it: "Not all of its fields are used all time".  Likewise, not
>>>> all arguments are used all time for get/put, but it's not an issue if they
>>>> are still there!  So this patch is good, but at the same time VMS_LINKED is
>>>> pointless.
>>>>
>>>> Paolo
>>>>
>>>
>>> I'm fine with this. I just think, it would be nice if the contract between
>>> the vmstate-core and the client code implementing VMStateInfo callbacks
>>> could be documented, including when are certain parameters valid, what
>>> they stand for, and how are they supposed to be used for the next version of
>>> the patch. Just to improve readability. Would this be OK with everybody?
>>>
>>> By the way the flag VMS_SINGLE is documented as ignored. Should we drop
>>> it?
>>>
>> I will prepare the VMStateInfo and QTAIL stuff as a separated series. If
>> indeed VMS_SINGLE is ignored, I can drop it here. It is similar to
>> VMS_LINKED situation.
>>
>> Thanks,
>> Jianjun
> 
> I think it's completely unrelated, so I would not lump it together with
> the QTAILQ stuff.
> 
> How do you feel about the apidoc part?
> 
I will add some comments inside vmstate_save/load_state about it.

Thanks,
Jianjun

> Cheers,
> Halil
> 

[Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

Currently we cannot directly transfer a QTAILQ instance because of the
limitation in the migration code. Here we introduce an approach to
transfer such structures. In our approach such a structure is tagged
with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
so that when VMS_LINKED is encountered, put and get from VMStateInfo are
called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
Similar VMStateInfo can be created for other data structures such as list.
This approach will be used to transfer pending_events and ccs_list in spapr
state.

We also create some macros in qemu/queue.h to access a QTAILQ using pointer
arithmetic. This ensures that we do not depend on the implementation
details about QTAILQ in the migration code.

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
---
 include/migration/vmstate.h | 26 ++++++++++++++++++
 include/qemu/queue.h        | 32 ++++++++++++++++++++++
 migration/trace-events      |  4 +++
 migration/vmstate.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 128 insertions(+)

diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 459dd4a..e60c994 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -186,6 +186,12 @@ enum VMStateFlags {
      * to determine the number of entries in the array. Only valid in
      * combination with one of VMS_VARRAY*. */
     VMS_MULTIPLY_ELEMENTS = 0x4000,
+    /* For fields which need customized handling, such as QTAILQ in queue.h.
+     * When this flag is set in VMStateField, info->get/put will
+     * be used in vmstate_load/save_state instead of recursive call.
+     * User should implement set info to handle the concerned data structure.
+     */
+    VMS_LINKED            = 0x8000,
 };
 
 struct VMStateField {
@@ -246,6 +252,7 @@ extern const VMStateInfo vmstate_info_timer;
 extern const VMStateInfo vmstate_info_buffer;
 extern const VMStateInfo vmstate_info_unused_buffer;
 extern const VMStateInfo vmstate_info_bitmap;
+extern const VMStateInfo vmstate_info_qtailq;
 
 #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
 #define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
@@ -657,6 +664,25 @@ extern const VMStateInfo vmstate_info_bitmap;
     .offset       = offsetof(_state, _field),                        \
 }
 
+/* For QTAILQ that need customized handling
+ * _type: type of QTAILQ element
+ * _next: name of QTAILQ entry field in QTAILQ element
+ * _vmsd: VMSD for QTAILQ element
+ * size: size of QTAILQ element
+ * start: offset of QTAILQ entry in QTAILQ element
+ */
+#define VMSTATE_QTAILQ_V(_field, _state, _version, _vmsd, _type, _next)  \
+{                                                                        \
+    .name         = (stringify(_field)),                                 \
+    .version_id   = (_version),                                          \
+    .vmsd         = &(_vmsd),                                            \
+    .size         = sizeof(_type),                                       \
+    .info         = &vmstate_info_qtailq,                                \
+    .flags        = VMS_LINKED,                                          \
+    .offset       = offsetof(_state, _field),                            \
+    .start        = offsetof(_type, _next),                              \
+}
+
 /* _f : field name
    _f_n : num of elements field_name
    _n : num of elements
diff --git a/include/qemu/queue.h b/include/qemu/queue.h
index 342073f..12c3f80 100644
--- a/include/qemu/queue.h
+++ b/include/qemu/queue.h
@@ -438,4 +438,36 @@ struct {                                                                \
 #define QTAILQ_PREV(elm, headname, field) \
         (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
 
+/*
+ * Offsets of layout of a tail queue head.
+ */
+#define QTAILQ_FIRST_OFFSET 0
+#define QTAILQ_LAST_OFFSET (sizeof(void *))
+
+/*
+ * Offsets of layout of a tail queue element.
+ */
+#define QTAILQ_NEXT_OFFSET 0
+#define QTAILQ_PREV_OFFSET (sizeof(void *))
+
+/*
+ * Tail queue tranversal using pointer arithmetic.
+ */
+#define QTAILQ_RAW_FOREACH(elm, head, entry)                                   \
+        for ((elm) = *((void **) ((char *) (head) + QTAILQ_FIRST_OFFSET));     \
+             (elm);                                                            \
+             (elm) =                                                           \
+                 *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)))
+/*
+ * Tail queue insertion using pointer arithmetic.
+ */
+#define QTAILQ_RAW_INSERT_TAIL(head, elm, entry) do {                          \
+        *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)) = NULL;   \
+        *((void **) ((char *) (elm) + (entry) + QTAILQ_PREV_OFFSET)) =         \
+            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
+        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
+        *((void **) ((char *) (head) + QTAILQ_LAST_OFFSET)) =                  \
+            (void *) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET);          \
+} while (/*CONSTCOND*/0)
+
 #endif /* QEMU_SYS_QUEUE_H */
diff --git a/migration/trace-events b/migration/trace-events
index dfee75a..9a6ec59 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -52,6 +52,10 @@ vmstate_n_elems(const char *name, int n_elems) "%s: %d"
 vmstate_subsection_load(const char *parent) "%s"
 vmstate_subsection_load_bad(const char *parent,  const char *sub, const char *sub2) "%s: %s/%s"
 vmstate_subsection_load_good(const char *parent) "%s"
+get_qtailq(const char *name, int version_id) "%s v%d"
+get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
+put_qtailq(const char *name, int version_id) "%s v%d"
+put_qtailq_end(const char *name, const char *reason) "%s %s"
 
 # migration/qemu-file.c
 qemu_file_fclose(void) ""
diff --git a/migration/vmstate.c b/migration/vmstate.c
index 66802cb..192db8a 100644
--- a/migration/vmstate.c
+++ b/migration/vmstate.c
@@ -5,7 +5,9 @@
 #include "migration/vmstate.h"
 #include "qemu/bitops.h"
 #include "qemu/error-report.h"
+#include "qemu/queue.h"
 #include "trace.h"
+#include "migration/qjson.h"
 
 static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
                                     void *opaque, QJSON *vmdesc);
@@ -121,6 +123,8 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
                 if (field->flags & VMS_STRUCT) {
                     ret = vmstate_load_state(f, field->vmsd, addr,
                                              field->vmsd->version_id);
+                } else if (field->flags & VMS_LINKED) {
+                    ret = field->info->get(f, addr, size, field);
                 } else {
                     ret = field->info->get(f, addr, size, NULL);
 
@@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
 
     if (field->flags & VMS_STRUCT) {
         type = "struct";
+    } else if (field->flags & VMS_LINKED) {
+        type = "linked";
     } else if (field->info->name) {
         type = field->info->name;
     }
@@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
                 }
                 if (field->flags & VMS_STRUCT) {
                     vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
+                } else if  (field->flags & VMS_LINKED) {
+                    field->info->put(f, addr, size, field, vmdesc_loop);
                 } else {
                     field->info->put(f, addr, size, NULL, NULL);
                 }
@@ -939,3 +947,61 @@ const VMStateInfo vmstate_info_bitmap = {
     .get = get_bitmap,
     .put = put_bitmap,
 };
+
+/*get for QTAILQ */
+static int get_qtailq(QEMUFile *f, void *pv, size_t unused_size,
+                      VMStateField *field)
+{
+    int ret = 0;
+    const VMStateDescription *vmsd = field->vmsd;
+    size_t size = field->size;
+    size_t entry = field->start;
+    int version_id = field->version_id;
+    void *elm;
+
+    trace_get_qtailq(vmsd->name, version_id);
+    if (version_id > vmsd->version_id) {
+        trace_get_qtailq_end(vmsd->name, "too new", -EINVAL);
+        return -EINVAL;
+    }
+    if (version_id < vmsd->minimum_version_id) {
+        trace_get_qtailq_end(vmsd->name, "too old", -EINVAL);
+        return -EINVAL;
+    }
+
+    while (qemu_get_byte(f)) {
+        elm =  g_malloc(size);
+        ret = vmstate_load_state(f, vmsd, elm, version_id);
+        if (ret) {
+            return ret;
+        }
+        QTAILQ_RAW_INSERT_TAIL(pv, elm, entry);
+    }
+
+    trace_get_qtailq_end(vmsd->name, "end", ret);
+    return ret;
+}
+
+/* put for QTAILQ */
+static void put_qtailq(QEMUFile *f, void *pv, size_t unused_size,
+                       VMStateField *field, QJSON *vmdesc)
+{
+    const VMStateDescription *vmsd = field->vmsd;
+    size_t entry = field->start;
+    void *elm;
+
+    trace_put_qtailq(vmsd->name, vmsd->version_id);
+
+    QTAILQ_RAW_FOREACH(elm, pv, entry) {
+        qemu_put_byte(f, true);
+        vmstate_save_state(f, vmsd, elm, vmdesc);
+    }
+    qemu_put_byte(f, false);
+
+    trace_put_qtailq_end(vmsd->name, "end");
+}
+const VMStateInfo vmstate_info_qtailq = {
+    .name = "qtailq",
+    .get  = get_qtailq,
+    .put  = put_qtailq,
+};
-- 
1.9.1

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Dr. David Alan Gilbert]

* Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> Currently we cannot directly transfer a QTAILQ instance because of the
> limitation in the migration code. Here we introduce an approach to
> transfer such structures. In our approach such a structure is tagged
> with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
> so that when VMS_LINKED is encountered, put and get from VMStateInfo are
> called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
> Similar VMStateInfo can be created for other data structures such as list.
> This approach will be used to transfer pending_events and ccs_list in spapr
> state.
> 
> We also create some macros in qemu/queue.h to access a QTAILQ using pointer
> arithmetic. This ensures that we do not depend on the implementation
> details about QTAILQ in the migration code.

I think we're going to need a way to have a more flexible
loops; and thus my choice here wouldn't be to use the .get/.put together
with the VMSD; but I think we'll end up needing a new
data structure, maybe a VMStateLoop *loop in VMStateField.

So would it be easier if you added that new member, then you wouldn't have to
modify every get() and put() function that already exists in the previous patch.

Specifically, your format of QTAILQ is perfectly reasonable - a
byte before each entry which is 1 to indicate there's an entry or 0
to indicate termination, but there are lots of other variants, e.g.

   a) put_scsi_requests uses that byte to hold a flag, so it's 0,1,2
      0 still means terminate but 1 or 2 set a flag in the structure.

   b) slirp_state_load also uses a null byte termination but not off a QTAILQ
      (although I think it could be flipped for one) (it uses '42' for the
      non-0 value, but looks like it could become 1)

   c) virtio_blk also rolls it's own linked list but again with the 0/1 byte

  Now how would I modify your QTAILQ load/store to do (a) without copying the whole
thing?

Dave

> 
> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> ---
>  include/migration/vmstate.h | 26 ++++++++++++++++++
>  include/qemu/queue.h        | 32 ++++++++++++++++++++++
>  migration/trace-events      |  4 +++
>  migration/vmstate.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 128 insertions(+)
> 
> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> index 459dd4a..e60c994 100644
> --- a/include/migration/vmstate.h
> +++ b/include/migration/vmstate.h
> @@ -186,6 +186,12 @@ enum VMStateFlags {
>       * to determine the number of entries in the array. Only valid in
>       * combination with one of VMS_VARRAY*. */
>      VMS_MULTIPLY_ELEMENTS = 0x4000,
> +    /* For fields which need customized handling, such as QTAILQ in queue.h.
> +     * When this flag is set in VMStateField, info->get/put will
> +     * be used in vmstate_load/save_state instead of recursive call.
> +     * User should implement set info to handle the concerned data structure.
> +     */
> +    VMS_LINKED            = 0x8000,
>  };
>  
>  struct VMStateField {
> @@ -246,6 +252,7 @@ extern const VMStateInfo vmstate_info_timer;
>  extern const VMStateInfo vmstate_info_buffer;
>  extern const VMStateInfo vmstate_info_unused_buffer;
>  extern const VMStateInfo vmstate_info_bitmap;
> +extern const VMStateInfo vmstate_info_qtailq;
>  
>  #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
>  #define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
> @@ -657,6 +664,25 @@ extern const VMStateInfo vmstate_info_bitmap;
>      .offset       = offsetof(_state, _field),                        \
>  }
>  
> +/* For QTAILQ that need customized handling
> + * _type: type of QTAILQ element
> + * _next: name of QTAILQ entry field in QTAILQ element
> + * _vmsd: VMSD for QTAILQ element
> + * size: size of QTAILQ element
> + * start: offset of QTAILQ entry in QTAILQ element
> + */
> +#define VMSTATE_QTAILQ_V(_field, _state, _version, _vmsd, _type, _next)  \
> +{                                                                        \
> +    .name         = (stringify(_field)),                                 \
> +    .version_id   = (_version),                                          \
> +    .vmsd         = &(_vmsd),                                            \
> +    .size         = sizeof(_type),                                       \
> +    .info         = &vmstate_info_qtailq,                                \
> +    .flags        = VMS_LINKED,                                          \
> +    .offset       = offsetof(_state, _field),                            \
> +    .start        = offsetof(_type, _next),                              \
> +}
> +
>  /* _f : field name
>     _f_n : num of elements field_name
>     _n : num of elements
> diff --git a/include/qemu/queue.h b/include/qemu/queue.h
> index 342073f..12c3f80 100644
> --- a/include/qemu/queue.h
> +++ b/include/qemu/queue.h
> @@ -438,4 +438,36 @@ struct {                                                                \
>  #define QTAILQ_PREV(elm, headname, field) \
>          (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
>  
> +/*
> + * Offsets of layout of a tail queue head.
> + */
> +#define QTAILQ_FIRST_OFFSET 0
> +#define QTAILQ_LAST_OFFSET (sizeof(void *))
> +
> +/*
> + * Offsets of layout of a tail queue element.
> + */
> +#define QTAILQ_NEXT_OFFSET 0
> +#define QTAILQ_PREV_OFFSET (sizeof(void *))
> +
> +/*
> + * Tail queue tranversal using pointer arithmetic.
> + */
> +#define QTAILQ_RAW_FOREACH(elm, head, entry)                                   \
> +        for ((elm) = *((void **) ((char *) (head) + QTAILQ_FIRST_OFFSET));     \
> +             (elm);                                                            \
> +             (elm) =                                                           \
> +                 *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)))
> +/*
> + * Tail queue insertion using pointer arithmetic.
> + */
> +#define QTAILQ_RAW_INSERT_TAIL(head, elm, entry) do {                          \
> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)) = NULL;   \
> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_PREV_OFFSET)) =         \
> +            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
> +        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
> +        *((void **) ((char *) (head) + QTAILQ_LAST_OFFSET)) =                  \
> +            (void *) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET);          \
> +} while (/*CONSTCOND*/0)
> +
>  #endif /* QEMU_SYS_QUEUE_H */
> diff --git a/migration/trace-events b/migration/trace-events
> index dfee75a..9a6ec59 100644
> --- a/migration/trace-events
> +++ b/migration/trace-events
> @@ -52,6 +52,10 @@ vmstate_n_elems(const char *name, int n_elems) "%s: %d"
>  vmstate_subsection_load(const char *parent) "%s"
>  vmstate_subsection_load_bad(const char *parent,  const char *sub, const char *sub2) "%s: %s/%s"
>  vmstate_subsection_load_good(const char *parent) "%s"
> +get_qtailq(const char *name, int version_id) "%s v%d"
> +get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
> +put_qtailq(const char *name, int version_id) "%s v%d"
> +put_qtailq_end(const char *name, const char *reason) "%s %s"
>  
>  # migration/qemu-file.c
>  qemu_file_fclose(void) ""
> diff --git a/migration/vmstate.c b/migration/vmstate.c
> index 66802cb..192db8a 100644
> --- a/migration/vmstate.c
> +++ b/migration/vmstate.c
> @@ -5,7 +5,9 @@
>  #include "migration/vmstate.h"
>  #include "qemu/bitops.h"
>  #include "qemu/error-report.h"
> +#include "qemu/queue.h"
>  #include "trace.h"
> +#include "migration/qjson.h"
>  
>  static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
>                                      void *opaque, QJSON *vmdesc);
> @@ -121,6 +123,8 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
>                  if (field->flags & VMS_STRUCT) {
>                      ret = vmstate_load_state(f, field->vmsd, addr,
>                                               field->vmsd->version_id);
> +                } else if (field->flags & VMS_LINKED) {
> +                    ret = field->info->get(f, addr, size, field);
>                  } else {
>                      ret = field->info->get(f, addr, size, NULL);
>  
> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>  
>      if (field->flags & VMS_STRUCT) {
>          type = "struct";
> +    } else if (field->flags & VMS_LINKED) {
> +        type = "linked";
>      } else if (field->info->name) {
>          type = field->info->name;
>      }
> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>                  }
>                  if (field->flags & VMS_STRUCT) {
>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
> +                } else if  (field->flags & VMS_LINKED) {
> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>                  } else {
>                      field->info->put(f, addr, size, NULL, NULL);
>                  }
> @@ -939,3 +947,61 @@ const VMStateInfo vmstate_info_bitmap = {
>      .get = get_bitmap,
>      .put = put_bitmap,
>  };
> +
> +/*get for QTAILQ */
> +static int get_qtailq(QEMUFile *f, void *pv, size_t unused_size,
> +                      VMStateField *field)
> +{
> +    int ret = 0;
> +    const VMStateDescription *vmsd = field->vmsd;
> +    size_t size = field->size;
> +    size_t entry = field->start;
> +    int version_id = field->version_id;
> +    void *elm;
> +
> +    trace_get_qtailq(vmsd->name, version_id);
> +    if (version_id > vmsd->version_id) {
> +        trace_get_qtailq_end(vmsd->name, "too new", -EINVAL);

Can you make those error_report's please - if it fails we want to
see why in the log.

Dave

> +        return -EINVAL;
> +    }
> +    if (version_id < vmsd->minimum_version_id) {
> +        trace_get_qtailq_end(vmsd->name, "too old", -EINVAL);
> +        return -EINVAL;
> +    }
> +
> +    while (qemu_get_byte(f)) {
> +        elm =  g_malloc(size);
> +        ret = vmstate_load_state(f, vmsd, elm, version_id);
> +        if (ret) {
> +            return ret;
> +        }
> +        QTAILQ_RAW_INSERT_TAIL(pv, elm, entry);
> +    }
> +
> +    trace_get_qtailq_end(vmsd->name, "end", ret);
> +    return ret;
> +}
> +
> +/* put for QTAILQ */
> +static void put_qtailq(QEMUFile *f, void *pv, size_t unused_size,
> +                       VMStateField *field, QJSON *vmdesc)
> +{
> +    const VMStateDescription *vmsd = field->vmsd;
> +    size_t entry = field->start;
> +    void *elm;
> +
> +    trace_put_qtailq(vmsd->name, vmsd->version_id);
> +
> +    QTAILQ_RAW_FOREACH(elm, pv, entry) {
> +        qemu_put_byte(f, true);
> +        vmstate_save_state(f, vmsd, elm, vmdesc);
> +    }
> +    qemu_put_byte(f, false);
> +
> +    trace_put_qtailq_end(vmsd->name, "end");
> +}
> +const VMStateInfo vmstate_info_qtailq = {
> +    .name = "qtailq",
> +    .get  = get_qtailq,
> +    .put  = put_qtailq,
> +};
> -- 
> 1.9.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

On 10/05/2016 09:56 AM, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>> Currently we cannot directly transfer a QTAILQ instance because of the
>> limitation in the migration code. Here we introduce an approach to
>> transfer such structures. In our approach such a structure is tagged
>> with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
>> so that when VMS_LINKED is encountered, put and get from VMStateInfo are
>> called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
>> Similar VMStateInfo can be created for other data structures such as list.
>> This approach will be used to transfer pending_events and ccs_list in spapr
>> state.
>>
>> We also create some macros in qemu/queue.h to access a QTAILQ using pointer
>> arithmetic. This ensures that we do not depend on the implementation
>> details about QTAILQ in the migration code.
> 
> I think we're going to need a way to have a more flexible
> loops; and thus my choice here wouldn't be to use the .get/.put together
> with the VMSD; but I think we'll end up needing a new
> data structure, maybe a VMStateLoop *loop in VMStateField.
> 
> So would it be easier if you added that new member, then you wouldn't have to
> modify every get() and put() function that already exists in the previous patch.
> 
> Specifically, your format of QTAILQ is perfectly reasonable - a
> byte before each entry which is 1 to indicate there's an entry or 0
> to indicate termination, but there are lots of other variants, e.g.
> 
>    a) put_scsi_requests uses that byte to hold a flag, so it's 0,1,2
>       0 still means terminate but 1 or 2 set a flag in the structure.

I quickly take a look of put_scsi_requests. It is transferring a QTAILQ of
SCSIRequest. However it goes into the structure inside to dump the
elements out.
If using my approach, I would have a VMSD for SCSIRequest. The
additional byte used to indicate the end of the queue would lie outside
the SCSCIRequest data block, so there would be no confusion.

>    b) slirp_state_load also uses a null byte termination but not off a QTAILQ
>       (although I think it could be flipped for one) (it uses '42' for the
>       non-0 value, but looks like it could become 1)

>    c) virtio_blk also rolls it's own linked list but again with the 0/1 byte
> 
>   Now how would I modify your QTAILQ load/store to do (a) without copying the whole
> thing?
> 
> Dave
> 
>>
>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>> ---
>>  include/migration/vmstate.h | 26 ++++++++++++++++++
>>  include/qemu/queue.h        | 32 ++++++++++++++++++++++
>>  migration/trace-events      |  4 +++
>>  migration/vmstate.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++
>>  4 files changed, 128 insertions(+)
>>
>> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
>> index 459dd4a..e60c994 100644
>> --- a/include/migration/vmstate.h
>> +++ b/include/migration/vmstate.h
>> @@ -186,6 +186,12 @@ enum VMStateFlags {
>>       * to determine the number of entries in the array. Only valid in
>>       * combination with one of VMS_VARRAY*. */
>>      VMS_MULTIPLY_ELEMENTS = 0x4000,
>> +    /* For fields which need customized handling, such as QTAILQ in queue.h.
>> +     * When this flag is set in VMStateField, info->get/put will
>> +     * be used in vmstate_load/save_state instead of recursive call.
>> +     * User should implement set info to handle the concerned data structure.
>> +     */
>> +    VMS_LINKED            = 0x8000,
>>  };
>>  
>>  struct VMStateField {
>> @@ -246,6 +252,7 @@ extern const VMStateInfo vmstate_info_timer;
>>  extern const VMStateInfo vmstate_info_buffer;
>>  extern const VMStateInfo vmstate_info_unused_buffer;
>>  extern const VMStateInfo vmstate_info_bitmap;
>> +extern const VMStateInfo vmstate_info_qtailq;
>>  
>>  #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
>>  #define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
>> @@ -657,6 +664,25 @@ extern const VMStateInfo vmstate_info_bitmap;
>>      .offset       = offsetof(_state, _field),                        \
>>  }
>>  
>> +/* For QTAILQ that need customized handling
>> + * _type: type of QTAILQ element
>> + * _next: name of QTAILQ entry field in QTAILQ element
>> + * _vmsd: VMSD for QTAILQ element
>> + * size: size of QTAILQ element
>> + * start: offset of QTAILQ entry in QTAILQ element
>> + */
>> +#define VMSTATE_QTAILQ_V(_field, _state, _version, _vmsd, _type, _next)  \
>> +{                                                                        \
>> +    .name         = (stringify(_field)),                                 \
>> +    .version_id   = (_version),                                          \
>> +    .vmsd         = &(_vmsd),                                            \
>> +    .size         = sizeof(_type),                                       \
>> +    .info         = &vmstate_info_qtailq,                                \
>> +    .flags        = VMS_LINKED,                                          \
>> +    .offset       = offsetof(_state, _field),                            \
>> +    .start        = offsetof(_type, _next),                              \
>> +}
>> +
>>  /* _f : field name
>>     _f_n : num of elements field_name
>>     _n : num of elements
>> diff --git a/include/qemu/queue.h b/include/qemu/queue.h
>> index 342073f..12c3f80 100644
>> --- a/include/qemu/queue.h
>> +++ b/include/qemu/queue.h
>> @@ -438,4 +438,36 @@ struct {                                                                \
>>  #define QTAILQ_PREV(elm, headname, field) \
>>          (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
>>  
>> +/*
>> + * Offsets of layout of a tail queue head.
>> + */
>> +#define QTAILQ_FIRST_OFFSET 0
>> +#define QTAILQ_LAST_OFFSET (sizeof(void *))
>> +
>> +/*
>> + * Offsets of layout of a tail queue element.
>> + */
>> +#define QTAILQ_NEXT_OFFSET 0
>> +#define QTAILQ_PREV_OFFSET (sizeof(void *))
>> +
>> +/*
>> + * Tail queue tranversal using pointer arithmetic.
>> + */
>> +#define QTAILQ_RAW_FOREACH(elm, head, entry)                                   \
>> +        for ((elm) = *((void **) ((char *) (head) + QTAILQ_FIRST_OFFSET));     \
>> +             (elm);                                                            \
>> +             (elm) =                                                           \
>> +                 *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)))
>> +/*
>> + * Tail queue insertion using pointer arithmetic.
>> + */
>> +#define QTAILQ_RAW_INSERT_TAIL(head, elm, entry) do {                          \
>> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)) = NULL;   \
>> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_PREV_OFFSET)) =         \
>> +            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
>> +        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
>> +        *((void **) ((char *) (head) + QTAILQ_LAST_OFFSET)) =                  \
>> +            (void *) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET);          \
>> +} while (/*CONSTCOND*/0)
>> +
>>  #endif /* QEMU_SYS_QUEUE_H */
>> diff --git a/migration/trace-events b/migration/trace-events
>> index dfee75a..9a6ec59 100644
>> --- a/migration/trace-events
>> +++ b/migration/trace-events
>> @@ -52,6 +52,10 @@ vmstate_n_elems(const char *name, int n_elems) "%s: %d"
>>  vmstate_subsection_load(const char *parent) "%s"
>>  vmstate_subsection_load_bad(const char *parent,  const char *sub, const char *sub2) "%s: %s/%s"
>>  vmstate_subsection_load_good(const char *parent) "%s"
>> +get_qtailq(const char *name, int version_id) "%s v%d"
>> +get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
>> +put_qtailq(const char *name, int version_id) "%s v%d"
>> +put_qtailq_end(const char *name, const char *reason) "%s %s"
>>  
>>  # migration/qemu-file.c
>>  qemu_file_fclose(void) ""
>> diff --git a/migration/vmstate.c b/migration/vmstate.c
>> index 66802cb..192db8a 100644
>> --- a/migration/vmstate.c
>> +++ b/migration/vmstate.c
>> @@ -5,7 +5,9 @@
>>  #include "migration/vmstate.h"
>>  #include "qemu/bitops.h"
>>  #include "qemu/error-report.h"
>> +#include "qemu/queue.h"
>>  #include "trace.h"
>> +#include "migration/qjson.h"
>>  
>>  static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
>>                                      void *opaque, QJSON *vmdesc);
>> @@ -121,6 +123,8 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
>>                  if (field->flags & VMS_STRUCT) {
>>                      ret = vmstate_load_state(f, field->vmsd, addr,
>>                                               field->vmsd->version_id);
>> +                } else if (field->flags & VMS_LINKED) {
>> +                    ret = field->info->get(f, addr, size, field);
>>                  } else {
>>                      ret = field->info->get(f, addr, size, NULL);
>>  
>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>>  
>>      if (field->flags & VMS_STRUCT) {
>>          type = "struct";
>> +    } else if (field->flags & VMS_LINKED) {
>> +        type = "linked";
>>      } else if (field->info->name) {
>>          type = field->info->name;
>>      }
>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>                  }
>>                  if (field->flags & VMS_STRUCT) {
>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>> +                } else if  (field->flags & VMS_LINKED) {
>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>>                  } else {
>>                      field->info->put(f, addr, size, NULL, NULL);
>>                  }
>> @@ -939,3 +947,61 @@ const VMStateInfo vmstate_info_bitmap = {
>>      .get = get_bitmap,
>>      .put = put_bitmap,
>>  };
>> +
>> +/*get for QTAILQ */
>> +static int get_qtailq(QEMUFile *f, void *pv, size_t unused_size,
>> +                      VMStateField *field)
>> +{
>> +    int ret = 0;
>> +    const VMStateDescription *vmsd = field->vmsd;
>> +    size_t size = field->size;
>> +    size_t entry = field->start;
>> +    int version_id = field->version_id;
>> +    void *elm;
>> +
>> +    trace_get_qtailq(vmsd->name, version_id);
>> +    if (version_id > vmsd->version_id) {
>> +        trace_get_qtailq_end(vmsd->name, "too new", -EINVAL);
> 
> Can you make those error_report's please - if it fails we want to
> see why in the log.
> 
> Dave
> 
>> +        return -EINVAL;
>> +    }
>> +    if (version_id < vmsd->minimum_version_id) {
>> +        trace_get_qtailq_end(vmsd->name, "too old", -EINVAL);
>> +        return -EINVAL;
>> +    }
>> +
>> +    while (qemu_get_byte(f)) {
>> +        elm =  g_malloc(size);
>> +        ret = vmstate_load_state(f, vmsd, elm, version_id);
>> +        if (ret) {
>> +            return ret;
>> +        }
>> +        QTAILQ_RAW_INSERT_TAIL(pv, elm, entry);
>> +    }
>> +
>> +    trace_get_qtailq_end(vmsd->name, "end", ret);
>> +    return ret;
>> +}
>> +
>> +/* put for QTAILQ */
>> +static void put_qtailq(QEMUFile *f, void *pv, size_t unused_size,
>> +                       VMStateField *field, QJSON *vmdesc)
>> +{
>> +    const VMStateDescription *vmsd = field->vmsd;
>> +    size_t entry = field->start;
>> +    void *elm;
>> +
>> +    trace_put_qtailq(vmsd->name, vmsd->version_id);
>> +
>> +    QTAILQ_RAW_FOREACH(elm, pv, entry) {
>> +        qemu_put_byte(f, true);
>> +        vmstate_save_state(f, vmsd, elm, vmdesc);
>> +    }
>> +    qemu_put_byte(f, false);
>> +
>> +    trace_put_qtailq_end(vmsd->name, "end");
>> +}
>> +const VMStateInfo vmstate_info_qtailq = {
>> +    .name = "qtailq",
>> +    .get  = get_qtailq,
>> +    .put  = put_qtailq,
>> +};
>> -- 
>> 1.9.1
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
Thanks,
Jianjun

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Dr. David Alan Gilbert]

* Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> 
> 
> On 10/05/2016 09:56 AM, Dr. David Alan Gilbert wrote:
> > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> >> Currently we cannot directly transfer a QTAILQ instance because of the
> >> limitation in the migration code. Here we introduce an approach to
> >> transfer such structures. In our approach such a structure is tagged
> >> with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
> >> so that when VMS_LINKED is encountered, put and get from VMStateInfo are
> >> called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
> >> Similar VMStateInfo can be created for other data structures such as list.
> >> This approach will be used to transfer pending_events and ccs_list in spapr
> >> state.
> >>
> >> We also create some macros in qemu/queue.h to access a QTAILQ using pointer
> >> arithmetic. This ensures that we do not depend on the implementation
> >> details about QTAILQ in the migration code.
> > 
> > I think we're going to need a way to have a more flexible
> > loops; and thus my choice here wouldn't be to use the .get/.put together
> > with the VMSD; but I think we'll end up needing a new
> > data structure, maybe a VMStateLoop *loop in VMStateField.
> > 
> > So would it be easier if you added that new member, then you wouldn't have to
> > modify every get() and put() function that already exists in the previous patch.
> > 
> > Specifically, your format of QTAILQ is perfectly reasonable - a
> > byte before each entry which is 1 to indicate there's an entry or 0
> > to indicate termination, but there are lots of other variants, e.g.
> > 
> >    a) put_scsi_requests uses that byte to hold a flag, so it's 0,1,2
> >       0 still means terminate but 1 or 2 set a flag in the structure.
> 
> I quickly take a look of put_scsi_requests. It is transferring a QTAILQ of
> SCSIRequest. However it goes into the structure inside to dump the
> elements out.
> If using my approach, I would have a VMSD for SCSIRequest. The
> additional byte used to indicate the end of the queue would lie outside
> the SCSCIRequest data block, so there would be no confusion.

Hmm OK; I don't think it's that easy but we'll see.

However, can I make one much simpler request; please split this patch
so that the VMSTATE_LINKED and vmstate_save_state/vmstate_load_state/vmfield_get_type_name
are in one patch, while the QTAILQ patches are in a separate patch.
(I'd be OK if you moved the VMSTATE_LINKED into the previous patch).

I've just been thinking about a different use for the same mechanism;
I want to do a:
  VMSTATE_WITH_TMP(t1*, type1, type2, vmsd)

which also sets the LINKED, where the .get/.put allocate a temporary
structure (of type/size type2), set up *tmp = t1 and then do the vmstate_load/save
using the vmsd on the temporary; something like (untested):

static int get_tmp(QEMUFile *f, void *pv, size_t unused_size, VMStateField *field)
{
    const VMStateDescription *vmsd = field->vmsd;
    size_t size = field->size;
    int version_id = field->version_id;
    void *tmp = gmalloc(size);
    int ret;
    
    *(void **)tmp = pv;
    ret = vmstate_load_state(f, vmsd, tmp, version_id);
    gfree(tmp);
    return ret;
}

This can be in a generic macro; and we would impose that type2 must be a struct
with the first element is 'type1* parent' (compile checked).
This would work nicely for where we have to do some maths to generate some
temporary results prior to migration; the .pre_save of the vmsd can read the data
from pv->parent and write it to the other fields but not have to use
qemu_get_*/qemu_put_* at all.

Dave

> 
> >    b) slirp_state_load also uses a null byte termination but not off a QTAILQ
> >       (although I think it could be flipped for one) (it uses '42' for the
> >       non-0 value, but looks like it could become 1)
> 
> >    c) virtio_blk also rolls it's own linked list but again with the 0/1 byte
> > 
> >   Now how would I modify your QTAILQ load/store to do (a) without copying the whole
> > thing?
> > 
> > Dave
> > 
> >>
> >> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> >> ---
> >>  include/migration/vmstate.h | 26 ++++++++++++++++++
> >>  include/qemu/queue.h        | 32 ++++++++++++++++++++++
> >>  migration/trace-events      |  4 +++
> >>  migration/vmstate.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++
> >>  4 files changed, 128 insertions(+)
> >>
> >> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> >> index 459dd4a..e60c994 100644
> >> --- a/include/migration/vmstate.h
> >> +++ b/include/migration/vmstate.h
> >> @@ -186,6 +186,12 @@ enum VMStateFlags {
> >>       * to determine the number of entries in the array. Only valid in
> >>       * combination with one of VMS_VARRAY*. */
> >>      VMS_MULTIPLY_ELEMENTS = 0x4000,
> >> +    /* For fields which need customized handling, such as QTAILQ in queue.h.
> >> +     * When this flag is set in VMStateField, info->get/put will
> >> +     * be used in vmstate_load/save_state instead of recursive call.
> >> +     * User should implement set info to handle the concerned data structure.
> >> +     */
> >> +    VMS_LINKED            = 0x8000,
> >>  };
> >>  
> >>  struct VMStateField {
> >> @@ -246,6 +252,7 @@ extern const VMStateInfo vmstate_info_timer;
> >>  extern const VMStateInfo vmstate_info_buffer;
> >>  extern const VMStateInfo vmstate_info_unused_buffer;
> >>  extern const VMStateInfo vmstate_info_bitmap;
> >> +extern const VMStateInfo vmstate_info_qtailq;
> >>  
> >>  #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
> >>  #define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
> >> @@ -657,6 +664,25 @@ extern const VMStateInfo vmstate_info_bitmap;
> >>      .offset       = offsetof(_state, _field),                        \
> >>  }
> >>  
> >> +/* For QTAILQ that need customized handling
> >> + * _type: type of QTAILQ element
> >> + * _next: name of QTAILQ entry field in QTAILQ element
> >> + * _vmsd: VMSD for QTAILQ element
> >> + * size: size of QTAILQ element
> >> + * start: offset of QTAILQ entry in QTAILQ element
> >> + */
> >> +#define VMSTATE_QTAILQ_V(_field, _state, _version, _vmsd, _type, _next)  \
> >> +{                                                                        \
> >> +    .name         = (stringify(_field)),                                 \
> >> +    .version_id   = (_version),                                          \
> >> +    .vmsd         = &(_vmsd),                                            \
> >> +    .size         = sizeof(_type),                                       \
> >> +    .info         = &vmstate_info_qtailq,                                \
> >> +    .flags        = VMS_LINKED,                                          \
> >> +    .offset       = offsetof(_state, _field),                            \
> >> +    .start        = offsetof(_type, _next),                              \
> >> +}
> >> +
> >>  /* _f : field name
> >>     _f_n : num of elements field_name
> >>     _n : num of elements
> >> diff --git a/include/qemu/queue.h b/include/qemu/queue.h
> >> index 342073f..12c3f80 100644
> >> --- a/include/qemu/queue.h
> >> +++ b/include/qemu/queue.h
> >> @@ -438,4 +438,36 @@ struct {                                                                \
> >>  #define QTAILQ_PREV(elm, headname, field) \
> >>          (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
> >>  
> >> +/*
> >> + * Offsets of layout of a tail queue head.
> >> + */
> >> +#define QTAILQ_FIRST_OFFSET 0
> >> +#define QTAILQ_LAST_OFFSET (sizeof(void *))
> >> +
> >> +/*
> >> + * Offsets of layout of a tail queue element.
> >> + */
> >> +#define QTAILQ_NEXT_OFFSET 0
> >> +#define QTAILQ_PREV_OFFSET (sizeof(void *))
> >> +
> >> +/*
> >> + * Tail queue tranversal using pointer arithmetic.
> >> + */
> >> +#define QTAILQ_RAW_FOREACH(elm, head, entry)                                   \
> >> +        for ((elm) = *((void **) ((char *) (head) + QTAILQ_FIRST_OFFSET));     \
> >> +             (elm);                                                            \
> >> +             (elm) =                                                           \
> >> +                 *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)))
> >> +/*
> >> + * Tail queue insertion using pointer arithmetic.
> >> + */
> >> +#define QTAILQ_RAW_INSERT_TAIL(head, elm, entry) do {                          \
> >> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)) = NULL;   \
> >> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_PREV_OFFSET)) =         \
> >> +            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
> >> +        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
> >> +        *((void **) ((char *) (head) + QTAILQ_LAST_OFFSET)) =                  \
> >> +            (void *) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET);          \
> >> +} while (/*CONSTCOND*/0)
> >> +
> >>  #endif /* QEMU_SYS_QUEUE_H */
> >> diff --git a/migration/trace-events b/migration/trace-events
> >> index dfee75a..9a6ec59 100644
> >> --- a/migration/trace-events
> >> +++ b/migration/trace-events
> >> @@ -52,6 +52,10 @@ vmstate_n_elems(const char *name, int n_elems) "%s: %d"
> >>  vmstate_subsection_load(const char *parent) "%s"
> >>  vmstate_subsection_load_bad(const char *parent,  const char *sub, const char *sub2) "%s: %s/%s"
> >>  vmstate_subsection_load_good(const char *parent) "%s"
> >> +get_qtailq(const char *name, int version_id) "%s v%d"
> >> +get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
> >> +put_qtailq(const char *name, int version_id) "%s v%d"
> >> +put_qtailq_end(const char *name, const char *reason) "%s %s"
> >>  
> >>  # migration/qemu-file.c
> >>  qemu_file_fclose(void) ""
> >> diff --git a/migration/vmstate.c b/migration/vmstate.c
> >> index 66802cb..192db8a 100644
> >> --- a/migration/vmstate.c
> >> +++ b/migration/vmstate.c
> >> @@ -5,7 +5,9 @@
> >>  #include "migration/vmstate.h"
> >>  #include "qemu/bitops.h"
> >>  #include "qemu/error-report.h"
> >> +#include "qemu/queue.h"
> >>  #include "trace.h"
> >> +#include "migration/qjson.h"
> >>  
> >>  static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
> >>                                      void *opaque, QJSON *vmdesc);
> >> @@ -121,6 +123,8 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
> >>                  if (field->flags & VMS_STRUCT) {
> >>                      ret = vmstate_load_state(f, field->vmsd, addr,
> >>                                               field->vmsd->version_id);
> >> +                } else if (field->flags & VMS_LINKED) {
> >> +                    ret = field->info->get(f, addr, size, field);
> >>                  } else {
> >>                      ret = field->info->get(f, addr, size, NULL);
> >>  
> >> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
> >>  
> >>      if (field->flags & VMS_STRUCT) {
> >>          type = "struct";
> >> +    } else if (field->flags & VMS_LINKED) {
> >> +        type = "linked";
> >>      } else if (field->info->name) {
> >>          type = field->info->name;
> >>      }
> >> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
> >>                  }
> >>                  if (field->flags & VMS_STRUCT) {
> >>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
> >> +                } else if  (field->flags & VMS_LINKED) {
> >> +                    field->info->put(f, addr, size, field, vmdesc_loop);
> >>                  } else {
> >>                      field->info->put(f, addr, size, NULL, NULL);
> >>                  }
> >> @@ -939,3 +947,61 @@ const VMStateInfo vmstate_info_bitmap = {
> >>      .get = get_bitmap,
> >>      .put = put_bitmap,
> >>  };
> >> +
> >> +/*get for QTAILQ */
> >> +static int get_qtailq(QEMUFile *f, void *pv, size_t unused_size,
> >> +                      VMStateField *field)
> >> +{
> >> +    int ret = 0;
> >> +    const VMStateDescription *vmsd = field->vmsd;
> >> +    size_t size = field->size;
> >> +    size_t entry = field->start;
> >> +    int version_id = field->version_id;
> >> +    void *elm;
> >> +
> >> +    trace_get_qtailq(vmsd->name, version_id);
> >> +    if (version_id > vmsd->version_id) {
> >> +        trace_get_qtailq_end(vmsd->name, "too new", -EINVAL);
> > 
> > Can you make those error_report's please - if it fails we want to
> > see why in the log.
> > 
> > Dave
> > 
> >> +        return -EINVAL;
> >> +    }
> >> +    if (version_id < vmsd->minimum_version_id) {
> >> +        trace_get_qtailq_end(vmsd->name, "too old", -EINVAL);
> >> +        return -EINVAL;
> >> +    }
> >> +
> >> +    while (qemu_get_byte(f)) {
> >> +        elm =  g_malloc(size);
> >> +        ret = vmstate_load_state(f, vmsd, elm, version_id);
> >> +        if (ret) {
> >> +            return ret;
> >> +        }
> >> +        QTAILQ_RAW_INSERT_TAIL(pv, elm, entry);
> >> +    }
> >> +
> >> +    trace_get_qtailq_end(vmsd->name, "end", ret);
> >> +    return ret;
> >> +}
> >> +
> >> +/* put for QTAILQ */
> >> +static void put_qtailq(QEMUFile *f, void *pv, size_t unused_size,
> >> +                       VMStateField *field, QJSON *vmdesc)
> >> +{
> >> +    const VMStateDescription *vmsd = field->vmsd;
> >> +    size_t entry = field->start;
> >> +    void *elm;
> >> +
> >> +    trace_put_qtailq(vmsd->name, vmsd->version_id);
> >> +
> >> +    QTAILQ_RAW_FOREACH(elm, pv, entry) {
> >> +        qemu_put_byte(f, true);
> >> +        vmstate_save_state(f, vmsd, elm, vmdesc);
> >> +    }
> >> +    qemu_put_byte(f, false);
> >> +
> >> +    trace_put_qtailq_end(vmsd->name, "end");
> >> +}
> >> +const VMStateInfo vmstate_info_qtailq = {
> >> +    .name = "qtailq",
> >> +    .get  = get_qtailq,
> >> +    .put  = put_qtailq,
> >> +};
> >> -- 
> >> 1.9.1
> >>
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > 
> Thanks,
> Jianjun
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

On 10/06/2016 12:01 PM, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>>
>>
>> On 10/05/2016 09:56 AM, Dr. David Alan Gilbert wrote:
>>> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>>>> Currently we cannot directly transfer a QTAILQ instance because of the
>>>> limitation in the migration code. Here we introduce an approach to
>>>> transfer such structures. In our approach such a structure is tagged
>>>> with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
>>>> so that when VMS_LINKED is encountered, put and get from VMStateInfo are
>>>> called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
>>>> Similar VMStateInfo can be created for other data structures such as list.
>>>> This approach will be used to transfer pending_events and ccs_list in spapr
>>>> state.
>>>>
>>>> We also create some macros in qemu/queue.h to access a QTAILQ using pointer
>>>> arithmetic. This ensures that we do not depend on the implementation
>>>> details about QTAILQ in the migration code.
>>>
>>> I think we're going to need a way to have a more flexible
>>> loops; and thus my choice here wouldn't be to use the .get/.put together
>>> with the VMSD; but I think we'll end up needing a new
>>> data structure, maybe a VMStateLoop *loop in VMStateField.
>>>
>>> So would it be easier if you added that new member, then you wouldn't have to
>>> modify every get() and put() function that already exists in the previous patch.
>>>
>>> Specifically, your format of QTAILQ is perfectly reasonable - a
>>> byte before each entry which is 1 to indicate there's an entry or 0
>>> to indicate termination, but there are lots of other variants, e.g.
>>>
>>>    a) put_scsi_requests uses that byte to hold a flag, so it's 0,1,2
>>>       0 still means terminate but 1 or 2 set a flag in the structure.
>>
>> I quickly take a look of put_scsi_requests. It is transferring a QTAILQ of
>> SCSIRequest. However it goes into the structure inside to dump the
>> elements out.
>> If using my approach, I would have a VMSD for SCSIRequest. The
>> additional byte used to indicate the end of the queue would lie outside
>> the SCSCIRequest data block, so there would be no confusion.
> 
> Hmm OK; I don't think it's that easy but we'll see.
> 
It is more complicated if we want to use the exact stream as is now.
IMO VMStateInfo provides enough flexibility and is used for migrating
scsi_requests. I would stick with it if the same stream layout is to be
used.
> However, can I make one much simpler request; please split this patch
> so that the VMSTATE_LINKED and vmstate_save_state/vmstate_load_state/vmfield_get_type_name
> are in one patch, while the QTAILQ patches are in a separate patch.
> (I'd be OK if you moved the VMSTATE_LINKED into the previous patch).
>

OK.

> I've just been thinking about a different use for the same mechanism;
> I want to do a:
>   VMSTATE_WITH_TMP(t1*, type1, type2, vmsd)
> 
> which also sets the LINKED, where the .get/.put allocate a temporary
> structure (of type/size type2), set up *tmp = t1 and then do the vmstate_load/save
> using the vmsd on the temporary; something like (untested):
> 
> static int get_tmp(QEMUFile *f, void *pv, size_t unused_size, VMStateField *field)
> {
>     const VMStateDescription *vmsd = field->vmsd;
>     size_t size = field->size;
>     int version_id = field->version_id;
>     void *tmp = gmalloc(size);
>     int ret;
>     
>     *(void **)tmp = pv;
>     ret = vmstate_load_state(f, vmsd, tmp, version_id);
>     gfree(tmp);
>     return ret;
> }
> 
> This can be in a generic macro; and we would impose that type2 must be a struct
> with the first element is 'type1* parent' (compile checked).
> This would work nicely for where we have to do some maths to generate some
> temporary results prior to migration; the .pre_save of the vmsd can read the data
> from pv->parent and write it to the other fields but not have to use
> qemu_get_*/qemu_put_* at all.
>

This could be a special instance of VMStateInfo.

> Dave
> 
>>
>>>    b) slirp_state_load also uses a null byte termination but not off a QTAILQ
>>>       (although I think it could be flipped for one) (it uses '42' for the
>>>       non-0 value, but looks like it could become 1)
>>
>>>    c) virtio_blk also rolls it's own linked list but again with the 0/1 byte
>>>
>>>   Now how would I modify your QTAILQ load/store to do (a) without copying the whole
>>> thing?
>>>
>>> Dave
>>>
>>>>
>>>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>>>> ---
>>>>  include/migration/vmstate.h | 26 ++++++++++++++++++
>>>>  include/qemu/queue.h        | 32 ++++++++++++++++++++++
>>>>  migration/trace-events      |  4 +++
>>>>  migration/vmstate.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++
>>>>  4 files changed, 128 insertions(+)
>>>>
>>>> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
>>>> index 459dd4a..e60c994 100644
>>>> --- a/include/migration/vmstate.h
>>>> +++ b/include/migration/vmstate.h
>>>> @@ -186,6 +186,12 @@ enum VMStateFlags {
>>>>       * to determine the number of entries in the array. Only valid in
>>>>       * combination with one of VMS_VARRAY*. */
>>>>      VMS_MULTIPLY_ELEMENTS = 0x4000,
>>>> +    /* For fields which need customized handling, such as QTAILQ in queue.h.
>>>> +     * When this flag is set in VMStateField, info->get/put will
>>>> +     * be used in vmstate_load/save_state instead of recursive call.
>>>> +     * User should implement set info to handle the concerned data structure.
>>>> +     */
>>>> +    VMS_LINKED            = 0x8000,
>>>>  };
>>>>  
>>>>  struct VMStateField {
>>>> @@ -246,6 +252,7 @@ extern const VMStateInfo vmstate_info_timer;
>>>>  extern const VMStateInfo vmstate_info_buffer;
>>>>  extern const VMStateInfo vmstate_info_unused_buffer;
>>>>  extern const VMStateInfo vmstate_info_bitmap;
>>>> +extern const VMStateInfo vmstate_info_qtailq;
>>>>  
>>>>  #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
>>>>  #define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
>>>> @@ -657,6 +664,25 @@ extern const VMStateInfo vmstate_info_bitmap;
>>>>      .offset       = offsetof(_state, _field),                        \
>>>>  }
>>>>  
>>>> +/* For QTAILQ that need customized handling
>>>> + * _type: type of QTAILQ element
>>>> + * _next: name of QTAILQ entry field in QTAILQ element
>>>> + * _vmsd: VMSD for QTAILQ element
>>>> + * size: size of QTAILQ element
>>>> + * start: offset of QTAILQ entry in QTAILQ element
>>>> + */
>>>> +#define VMSTATE_QTAILQ_V(_field, _state, _version, _vmsd, _type, _next)  \
>>>> +{                                                                        \
>>>> +    .name         = (stringify(_field)),                                 \
>>>> +    .version_id   = (_version),                                          \
>>>> +    .vmsd         = &(_vmsd),                                            \
>>>> +    .size         = sizeof(_type),                                       \
>>>> +    .info         = &vmstate_info_qtailq,                                \
>>>> +    .flags        = VMS_LINKED,                                          \
>>>> +    .offset       = offsetof(_state, _field),                            \
>>>> +    .start        = offsetof(_type, _next),                              \
>>>> +}
>>>> +
>>>>  /* _f : field name
>>>>     _f_n : num of elements field_name
>>>>     _n : num of elements
>>>> diff --git a/include/qemu/queue.h b/include/qemu/queue.h
>>>> index 342073f..12c3f80 100644
>>>> --- a/include/qemu/queue.h
>>>> +++ b/include/qemu/queue.h
>>>> @@ -438,4 +438,36 @@ struct {                                                                \
>>>>  #define QTAILQ_PREV(elm, headname, field) \
>>>>          (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
>>>>  
>>>> +/*
>>>> + * Offsets of layout of a tail queue head.
>>>> + */
>>>> +#define QTAILQ_FIRST_OFFSET 0
>>>> +#define QTAILQ_LAST_OFFSET (sizeof(void *))
>>>> +
>>>> +/*
>>>> + * Offsets of layout of a tail queue element.
>>>> + */
>>>> +#define QTAILQ_NEXT_OFFSET 0
>>>> +#define QTAILQ_PREV_OFFSET (sizeof(void *))
>>>> +
>>>> +/*
>>>> + * Tail queue tranversal using pointer arithmetic.
>>>> + */
>>>> +#define QTAILQ_RAW_FOREACH(elm, head, entry)                                   \
>>>> +        for ((elm) = *((void **) ((char *) (head) + QTAILQ_FIRST_OFFSET));     \
>>>> +             (elm);                                                            \
>>>> +             (elm) =                                                           \
>>>> +                 *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)))
>>>> +/*
>>>> + * Tail queue insertion using pointer arithmetic.
>>>> + */
>>>> +#define QTAILQ_RAW_INSERT_TAIL(head, elm, entry) do {                          \
>>>> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)) = NULL;   \
>>>> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_PREV_OFFSET)) =         \
>>>> +            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
>>>> +        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
>>>> +        *((void **) ((char *) (head) + QTAILQ_LAST_OFFSET)) =                  \
>>>> +            (void *) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET);          \
>>>> +} while (/*CONSTCOND*/0)
>>>> +
>>>>  #endif /* QEMU_SYS_QUEUE_H */
>>>> diff --git a/migration/trace-events b/migration/trace-events
>>>> index dfee75a..9a6ec59 100644
>>>> --- a/migration/trace-events
>>>> +++ b/migration/trace-events
>>>> @@ -52,6 +52,10 @@ vmstate_n_elems(const char *name, int n_elems) "%s: %d"
>>>>  vmstate_subsection_load(const char *parent) "%s"
>>>>  vmstate_subsection_load_bad(const char *parent,  const char *sub, const char *sub2) "%s: %s/%s"
>>>>  vmstate_subsection_load_good(const char *parent) "%s"
>>>> +get_qtailq(const char *name, int version_id) "%s v%d"
>>>> +get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
>>>> +put_qtailq(const char *name, int version_id) "%s v%d"
>>>> +put_qtailq_end(const char *name, const char *reason) "%s %s"
>>>>  
>>>>  # migration/qemu-file.c
>>>>  qemu_file_fclose(void) ""
>>>> diff --git a/migration/vmstate.c b/migration/vmstate.c
>>>> index 66802cb..192db8a 100644
>>>> --- a/migration/vmstate.c
>>>> +++ b/migration/vmstate.c
>>>> @@ -5,7 +5,9 @@
>>>>  #include "migration/vmstate.h"
>>>>  #include "qemu/bitops.h"
>>>>  #include "qemu/error-report.h"
>>>> +#include "qemu/queue.h"
>>>>  #include "trace.h"
>>>> +#include "migration/qjson.h"
>>>>  
>>>>  static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
>>>>                                      void *opaque, QJSON *vmdesc);
>>>> @@ -121,6 +123,8 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
>>>>                  if (field->flags & VMS_STRUCT) {
>>>>                      ret = vmstate_load_state(f, field->vmsd, addr,
>>>>                                               field->vmsd->version_id);
>>>> +                } else if (field->flags & VMS_LINKED) {
>>>> +                    ret = field->info->get(f, addr, size, field);
>>>>                  } else {
>>>>                      ret = field->info->get(f, addr, size, NULL);
>>>>  
>>>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>>>>  
>>>>      if (field->flags & VMS_STRUCT) {
>>>>          type = "struct";
>>>> +    } else if (field->flags & VMS_LINKED) {
>>>> +        type = "linked";
>>>>      } else if (field->info->name) {
>>>>          type = field->info->name;
>>>>      }
>>>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>>>                  }
>>>>                  if (field->flags & VMS_STRUCT) {
>>>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>>>> +                } else if  (field->flags & VMS_LINKED) {
>>>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>>>>                  } else {
>>>>                      field->info->put(f, addr, size, NULL, NULL);
>>>>                  }
>>>> @@ -939,3 +947,61 @@ const VMStateInfo vmstate_info_bitmap = {
>>>>      .get = get_bitmap,
>>>>      .put = put_bitmap,
>>>>  };
>>>> +
>>>> +/*get for QTAILQ */
>>>> +static int get_qtailq(QEMUFile *f, void *pv, size_t unused_size,
>>>> +                      VMStateField *field)
>>>> +{
>>>> +    int ret = 0;
>>>> +    const VMStateDescription *vmsd = field->vmsd;
>>>> +    size_t size = field->size;
>>>> +    size_t entry = field->start;
>>>> +    int version_id = field->version_id;
>>>> +    void *elm;
>>>> +
>>>> +    trace_get_qtailq(vmsd->name, version_id);
>>>> +    if (version_id > vmsd->version_id) {
>>>> +        trace_get_qtailq_end(vmsd->name, "too new", -EINVAL);
>>>
>>> Can you make those error_report's please - if it fails we want to
>>> see why in the log.
>>>
>>> Dave
>>>
>>>> +        return -EINVAL;
>>>> +    }
>>>> +    if (version_id < vmsd->minimum_version_id) {
>>>> +        trace_get_qtailq_end(vmsd->name, "too old", -EINVAL);
>>>> +        return -EINVAL;
>>>> +    }
>>>> +
>>>> +    while (qemu_get_byte(f)) {
>>>> +        elm =  g_malloc(size);
>>>> +        ret = vmstate_load_state(f, vmsd, elm, version_id);
>>>> +        if (ret) {
>>>> +            return ret;
>>>> +        }
>>>> +        QTAILQ_RAW_INSERT_TAIL(pv, elm, entry);
>>>> +    }
>>>> +
>>>> +    trace_get_qtailq_end(vmsd->name, "end", ret);
>>>> +    return ret;
>>>> +}
>>>> +
>>>> +/* put for QTAILQ */
>>>> +static void put_qtailq(QEMUFile *f, void *pv, size_t unused_size,
>>>> +                       VMStateField *field, QJSON *vmdesc)
>>>> +{
>>>> +    const VMStateDescription *vmsd = field->vmsd;
>>>> +    size_t entry = field->start;
>>>> +    void *elm;
>>>> +
>>>> +    trace_put_qtailq(vmsd->name, vmsd->version_id);
>>>> +
>>>> +    QTAILQ_RAW_FOREACH(elm, pv, entry) {
>>>> +        qemu_put_byte(f, true);
>>>> +        vmstate_save_state(f, vmsd, elm, vmdesc);
>>>> +    }
>>>> +    qemu_put_byte(f, false);
>>>> +
>>>> +    trace_put_qtailq_end(vmsd->name, "end");
>>>> +}
>>>> +const VMStateInfo vmstate_info_qtailq = {
>>>> +    .name = "qtailq",
>>>> +    .get  = get_qtailq,
>>>> +    .put  = put_qtailq,
>>>> +};
>>>> -- 
>>>> 1.9.1
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>> Thanks,
>> Jianjun
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
Thanks,
Jianjun

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5051 bytes --]

On Thu, Oct 06, 2016 at 08:01:56PM +0100, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > 
> > 
> > On 10/05/2016 09:56 AM, Dr. David Alan Gilbert wrote:
> > > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> > >> Currently we cannot directly transfer a QTAILQ instance because of the
> > >> limitation in the migration code. Here we introduce an approach to
> > >> transfer such structures. In our approach such a structure is tagged
> > >> with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
> > >> so that when VMS_LINKED is encountered, put and get from VMStateInfo are
> > >> called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
> > >> Similar VMStateInfo can be created for other data structures such as list.
> > >> This approach will be used to transfer pending_events and ccs_list in spapr
> > >> state.
> > >>
> > >> We also create some macros in qemu/queue.h to access a QTAILQ using pointer
> > >> arithmetic. This ensures that we do not depend on the implementation
> > >> details about QTAILQ in the migration code.
> > > 
> > > I think we're going to need a way to have a more flexible
> > > loops; and thus my choice here wouldn't be to use the .get/.put together
> > > with the VMSD; but I think we'll end up needing a new
> > > data structure, maybe a VMStateLoop *loop in VMStateField.
> > > 
> > > So would it be easier if you added that new member, then you wouldn't have to
> > > modify every get() and put() function that already exists in the previous patch.
> > > 
> > > Specifically, your format of QTAILQ is perfectly reasonable - a
> > > byte before each entry which is 1 to indicate there's an entry or 0
> > > to indicate termination, but there are lots of other variants, e.g.
> > > 
> > >    a) put_scsi_requests uses that byte to hold a flag, so it's 0,1,2
> > >       0 still means terminate but 1 or 2 set a flag in the structure.
> > 
> > I quickly take a look of put_scsi_requests. It is transferring a QTAILQ of
> > SCSIRequest. However it goes into the structure inside to dump the
> > elements out.
> > If using my approach, I would have a VMSD for SCSIRequest. The
> > additional byte used to indicate the end of the queue would lie outside
> > the SCSCIRequest data block, so there would be no confusion.
> 
> Hmm OK; I don't think it's that easy but we'll see.
> 
> However, can I make one much simpler request; please split this patch
> so that the VMSTATE_LINKED and vmstate_save_state/vmstate_load_state/vmfield_get_type_name
> are in one patch, while the QTAILQ patches are in a separate patch.
> (I'd be OK if you moved the VMSTATE_LINKED into the previous patch).
> 
> I've just been thinking about a different use for the same mechanism;
> I want to do a:
>   VMSTATE_WITH_TMP(t1*, type1, type2, vmsd)
> 
> which also sets the LINKED, where the .get/.put allocate a temporary
> structure (of type/size type2), set up *tmp = t1 and then do the vmstate_load/save
> using the vmsd on the temporary; something like (untested):
> 
> static int get_tmp(QEMUFile *f, void *pv, size_t unused_size, VMStateField *field)
> {
>     const VMStateDescription *vmsd = field->vmsd;
>     size_t size = field->size;
>     int version_id = field->version_id;
>     void *tmp = gmalloc(size);
>     int ret;
>     
>     *(void **)tmp = pv;
>     ret = vmstate_load_state(f, vmsd, tmp, version_id);
>     gfree(tmp);
>     return ret;
> }
> 
> This can be in a generic macro; and we would impose that type2 must be a struct
> with the first element is 'type1* parent' (compile checked).
> This would work nicely for where we have to do some maths to generate some
> temporary results prior to migration; the .pre_save of the vmsd can read the data
> from pv->parent and write it to the other fields but not have to use
> qemu_get_*/qemu_put_* at all.
> 
> Dave

Oh, I like this idea.  I know there are a number of places where
should-be-obsolete fields are still present in structures purely to
catch incoming migration info which is then converted to the modern
representation in post_load.  This would allow cleaning a bunch of
those up.

It would also mean we don't necessarily need explicit handling of
queues/lists.  I objected to early versions of this series which
dumped the qtailq into an array and used the existing array vmstate
types, because it meant not just an only-for-migration field in the
structure, but a substantial slab of only-for-migration data.

If we added the concept of temporary "catching" structures to the
vmsd, that objection would go away.  I'd be happy enough to
temporarily dump the queue into an array, transfer that over the
stream into another temporary array, then load it into the destination
queue.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Paolo Bonzini]

On 06/10/2016 21:01, Dr. David Alan Gilbert wrote:
>> >> +                } else if (field->flags & VMS_LINKED) {
>> >> +                    ret = field->info->get(f, addr, size, field);
>> >>                  } else {
>> >>                      ret = field->info->get(f, addr, size, NULL);
>> >>  
>> >> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>> >>  
>> >>      if (field->flags & VMS_STRUCT) {
>> >>          type = "struct";
>> >> +    } else if (field->flags & VMS_LINKED) {
>> >> +        type = "linked";
>> >>      } else if (field->info->name) {
>> >>          type = field->info->name;
>> >>      }
>> >> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>> >>                  }
>> >>                  if (field->flags & VMS_STRUCT) {
>> >>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>> >> +                } else if  (field->flags & VMS_LINKED) {
>> >> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>> >>                  } else {
>> >>                      field->info->put(f, addr, size, NULL, NULL);
>> >>                  }

Is VMS_LINKED needed at all, since the fields are unused for every
VMStateInfo except qtailq?

Paolo

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Dr. David Alan Gilbert]

* Paolo Bonzini (pbonzini@redhat.com) wrote:
> 
> 
> On 06/10/2016 21:01, Dr. David Alan Gilbert wrote:
> >> >> +                } else if (field->flags & VMS_LINKED) {
> >> >> +                    ret = field->info->get(f, addr, size, field);
> >> >>                  } else {
> >> >>                      ret = field->info->get(f, addr, size, NULL);
> >> >>  
> >> >> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
> >> >>  
> >> >>      if (field->flags & VMS_STRUCT) {
> >> >>          type = "struct";
> >> >> +    } else if (field->flags & VMS_LINKED) {
> >> >> +        type = "linked";
> >> >>      } else if (field->info->name) {
> >> >>          type = field->info->name;
> >> >>      }
> >> >> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
> >> >>                  }
> >> >>                  if (field->flags & VMS_STRUCT) {
> >> >>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
> >> >> +                } else if  (field->flags & VMS_LINKED) {
> >> >> +                    field->info->put(f, addr, size, field, vmdesc_loop);
> >> >>                  } else {
> >> >>                      field->info->put(f, addr, size, NULL, NULL);
> >> >>                  }
> 
> Is VMS_LINKED needed at all, since the fields are unused for every
> VMStateInfo except qtailq?

No, I think you could easily drop the VMS_LINKED and just always pass them in.

Dave

> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

On 10/07/2016 07:34 AM, Dr. David Alan Gilbert wrote:
> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>
>>
>> On 06/10/2016 21:01, Dr. David Alan Gilbert wrote:
>>>>>> +                } else if (field->flags & VMS_LINKED) {
>>>>>> +                    ret = field->info->get(f, addr, size, field);
>>>>>>                  } else {
>>>>>>                      ret = field->info->get(f, addr, size, NULL);
>>>>>>  
>>>>>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>>>>>>  
>>>>>>      if (field->flags & VMS_STRUCT) {
>>>>>>          type = "struct";
>>>>>> +    } else if (field->flags & VMS_LINKED) {
>>>>>> +        type = "linked";
>>>>>>      } else if (field->info->name) {
>>>>>>          type = field->info->name;
>>>>>>      }
>>>>>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>>>>>                  }
>>>>>>                  if (field->flags & VMS_STRUCT) {
>>>>>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>>>>>> +                } else if  (field->flags & VMS_LINKED) {
>>>>>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>>>>>>                  } else {
>>>>>>                      field->info->put(f, addr, size, NULL, NULL);
>>>>>>                  }
>>
>> Is VMS_LINKED needed at all, since the fields are unused for every
>> VMStateInfo except qtailq?
> 
> No, I think you could easily drop the VMS_LINKED and just always pass them in.

It is needed if we want to use vmdesc_loop.

Thanks,
Jianjun

> Dave
> 
>> Paolo
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Paolo Bonzini]

On 07/10/2016 18:31, Jianjun Duan wrote:
> 
> 
> On 10/07/2016 07:34 AM, Dr. David Alan Gilbert wrote:
>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>>
>>>
>>> On 06/10/2016 21:01, Dr. David Alan Gilbert wrote:
>>>>>>> +                } else if (field->flags & VMS_LINKED) {
>>>>>>> +                    ret = field->info->get(f, addr, size, field);
>>>>>>>                  } else {
>>>>>>>                      ret = field->info->get(f, addr, size, NULL);
>>>>>>>  
>>>>>>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>>>>>>>  
>>>>>>>      if (field->flags & VMS_STRUCT) {
>>>>>>>          type = "struct";
>>>>>>> +    } else if (field->flags & VMS_LINKED) {
>>>>>>> +        type = "linked";
>>>>>>>      } else if (field->info->name) {
>>>>>>>          type = field->info->name;
>>>>>>>      }
>>>>>>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>>>>>>                  }
>>>>>>>                  if (field->flags & VMS_STRUCT) {
>>>>>>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>>>>>>> +                } else if  (field->flags & VMS_LINKED) {
>>>>>>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>>>>>>>                  } else {
>>>>>>>                      field->info->put(f, addr, size, NULL, NULL);
>>>>>>>                  }
>>>
>>> Is VMS_LINKED needed at all, since the fields are unused for every
>>> VMStateInfo except qtailq?
>>
>> No, I think you could easily drop the VMS_LINKED and just always pass them in.
> 
> It is needed if we want to use vmdesc_loop.

Just always pass it in, can't you?

Paolo

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

On 10/07/2016 09:32 AM, Paolo Bonzini wrote:
> 
> 
> On 07/10/2016 18:31, Jianjun Duan wrote:
>>
>>
>> On 10/07/2016 07:34 AM, Dr. David Alan Gilbert wrote:
>>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>>>
>>>>
>>>> On 06/10/2016 21:01, Dr. David Alan Gilbert wrote:
>>>>>>>> +                } else if (field->flags & VMS_LINKED) {
>>>>>>>> +                    ret = field->info->get(f, addr, size, field);
>>>>>>>>                  } else {
>>>>>>>>                      ret = field->info->get(f, addr, size, NULL);
>>>>>>>>  
>>>>>>>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>>>>>>>>  
>>>>>>>>      if (field->flags & VMS_STRUCT) {
>>>>>>>>          type = "struct";
>>>>>>>> +    } else if (field->flags & VMS_LINKED) {
>>>>>>>> +        type = "linked";
>>>>>>>>      } else if (field->info->name) {
>>>>>>>>          type = field->info->name;
>>>>>>>>      }
>>>>>>>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>>>>>>>                  }
>>>>>>>>                  if (field->flags & VMS_STRUCT) {
>>>>>>>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>>>>>>>> +                } else if  (field->flags & VMS_LINKED) {
>>>>>>>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>>>>>>>>                  } else {
>>>>>>>>                      field->info->put(f, addr, size, NULL, NULL);
>>>>>>>>                  }
>>>>
>>>> Is VMS_LINKED needed at all, since the fields are unused for every
>>>> VMStateInfo except qtailq?
>>>
>>> No, I think you could easily drop the VMS_LINKED and just always pass them in.
>>
>> It is needed if we want to use vmdesc_loop.
> 
> Just always pass it in, can't you?
> 
Could that lead to repetition of array elements? I know it depends on
individual put details.

Thanks,
Jianjun
> Paolo
> 

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Dr. David Alan Gilbert]

* Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> 
> 
> On 10/07/2016 09:32 AM, Paolo Bonzini wrote:
> > 
> > 
> > On 07/10/2016 18:31, Jianjun Duan wrote:
> >>
> >>
> >> On 10/07/2016 07:34 AM, Dr. David Alan Gilbert wrote:
> >>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
> >>>>
> >>>>
> >>>> On 06/10/2016 21:01, Dr. David Alan Gilbert wrote:
> >>>>>>>> +                } else if (field->flags & VMS_LINKED) {
> >>>>>>>> +                    ret = field->info->get(f, addr, size, field);
> >>>>>>>>                  } else {
> >>>>>>>>                      ret = field->info->get(f, addr, size, NULL);
> >>>>>>>>  
> >>>>>>>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
> >>>>>>>>  
> >>>>>>>>      if (field->flags & VMS_STRUCT) {
> >>>>>>>>          type = "struct";
> >>>>>>>> +    } else if (field->flags & VMS_LINKED) {
> >>>>>>>> +        type = "linked";
> >>>>>>>>      } else if (field->info->name) {
> >>>>>>>>          type = field->info->name;
> >>>>>>>>      }
> >>>>>>>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
> >>>>>>>>                  }
> >>>>>>>>                  if (field->flags & VMS_STRUCT) {
> >>>>>>>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
> >>>>>>>> +                } else if  (field->flags & VMS_LINKED) {
> >>>>>>>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
> >>>>>>>>                  } else {
> >>>>>>>>                      field->info->put(f, addr, size, NULL, NULL);
> >>>>>>>>                  }
> >>>>
> >>>> Is VMS_LINKED needed at all, since the fields are unused for every
> >>>> VMStateInfo except qtailq?
> >>>
> >>> No, I think you could easily drop the VMS_LINKED and just always pass them in.
> >>
> >> It is needed if we want to use vmdesc_loop.
> > 
> > Just always pass it in, can't you?
> > 
> Could that lead to repetition of array elements? I know it depends on
> individual put details.

I don't think so; Paolo is just suggesting replacing:

 +                } else if  (field->flags & VMS_LINKED) {
 +                    field->info->put(f, addr, size, field, vmdesc_loop);
                  } else {
                      field->info->put(f, addr, size, NULL, NULL);
                  }

by:

                  } else {
 -                    field->info->put(f, addr, size, NULL, NULL);
 +                    field->info->put(f, addr, size, field, vmdesc_loop);
                  }

most get/put won't use the field/vmdesc_loop so it wont matter.

Dave

> 
> Thanks,
> Jianjun
> > Paolo
> > 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

On 10/07/2016 10:34 AM, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>>
>>
>> On 10/07/2016 09:32 AM, Paolo Bonzini wrote:
>>>
>>>
>>> On 07/10/2016 18:31, Jianjun Duan wrote:
>>>>
>>>>
>>>> On 10/07/2016 07:34 AM, Dr. David Alan Gilbert wrote:
>>>>> * Paolo Bonzini (pbonzini@redhat.com) wrote:
>>>>>>
>>>>>>
>>>>>> On 06/10/2016 21:01, Dr. David Alan Gilbert wrote:
>>>>>>>>>> +                } else if (field->flags & VMS_LINKED) {
>>>>>>>>>> +                    ret = field->info->get(f, addr, size, field);
>>>>>>>>>>                  } else {
>>>>>>>>>>                      ret = field->info->get(f, addr, size, NULL);
>>>>>>>>>>  
>>>>>>>>>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>>>>>>>>>>  
>>>>>>>>>>      if (field->flags & VMS_STRUCT) {
>>>>>>>>>>          type = "struct";
>>>>>>>>>> +    } else if (field->flags & VMS_LINKED) {
>>>>>>>>>> +        type = "linked";
>>>>>>>>>>      } else if (field->info->name) {
>>>>>>>>>>          type = field->info->name;
>>>>>>>>>>      }
>>>>>>>>>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>>>>>>>>>                  }
>>>>>>>>>>                  if (field->flags & VMS_STRUCT) {
>>>>>>>>>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>>>>>>>>>> +                } else if  (field->flags & VMS_LINKED) {
>>>>>>>>>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>>>>>>>>>>                  } else {
>>>>>>>>>>                      field->info->put(f, addr, size, NULL, NULL);
>>>>>>>>>>                  }
>>>>>>
>>>>>> Is VMS_LINKED needed at all, since the fields are unused for every
>>>>>> VMStateInfo except qtailq?
>>>>>
>>>>> No, I think you could easily drop the VMS_LINKED and just always pass them in.
>>>>
>>>> It is needed if we want to use vmdesc_loop.
>>>
>>> Just always pass it in, can't you?
>>>
>> Could that lead to repetition of array elements? I know it depends on
>> individual put details.
> 
> I don't think so; Paolo is just suggesting replacing:
> 
>  +                } else if  (field->flags & VMS_LINKED) {
>  +                    field->info->put(f, addr, size, field, vmdesc_loop);
>                   } else {
>                       field->info->put(f, addr, size, NULL, NULL);
>                   }
> 
> by:
> 
>                   } else {
>  -                    field->info->put(f, addr, size, NULL, NULL);
>  +                    field->info->put(f, addr, size, field, vmdesc_loop);
>                   }
> 
> most get/put won't use the field/vmdesc_loop so it wont matter.
> 
> Dave
> 

Even though most put/get have no issues now, when somebody writes a new
put, he or she could run into issues if only checking the type
signature. It makes the code more readable.

I am OK either way.

Thanks,
Jianjun
>>
>> Thanks,
>> Jianjun
>>> Paolo
>>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Paolo Bonzini]

On 07/10/2016 19:43, Jianjun Duan wrote:
> Even though most put/get have no issues now, when somebody writes a new
> put, he or she could run into issues if only checking the type
> signature. It makes the code more readable.

No, it doesn't because one is left wondering what is VMS_LINKED about.
What is the relation between linked datastructures and passing some
arguments as NULL/noon-NULL?

When somebody writes a new get/put, they'll make an informed choice on
what arguments they need. :)

Paolo

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Halil Pasic]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]



On 10/08/2016 01:37 PM, Paolo Bonzini wrote:
>> Even though most put/get have no issues now, when somebody writes a new
>> > put, he or she could run into issues if only checking the type
>> > signature. It makes the code more readable.

> No, it doesn't because one is left wondering what is VMS_LINKED about.

I agree with Paolo. IMHO VMS_LINKED is conceptually difficult in  a sense
that it's quite different that what we have currently. I have the feeling,
conceptually, we are trying to fit in something like data structures with a
type parameter (the element type) here. AFAIU what vmstate currently can is
directed acyclic graphs of certain stuff (and also completely custom
handling based on custom put/get).

> What is the relation between linked datastructures and passing some
> arguments as NULL/noon-NULL?

IFAIU we need those for the datastructure because it's linked and has
a type parameter (element type). The two last arguments are for
the element type. These were added by the previous patch because
the old VMStateInfo did not need these. So we drastically changed
the scope of VMStateInfo with the previous patch and I'm not
sure I like this.

I will have to stare at this a bit longer to bring something more
constructive than these (largely feeling-based) remarks.

Cheers,
Halil


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

On 10/08/2016 12:28 PM, Halil Pasic wrote:
> 
> 
> On 10/08/2016 01:37 PM, Paolo Bonzini wrote:
>>> Even though most put/get have no issues now, when somebody writes a new
>>>> put, he or she could run into issues if only checking the type
>>>> signature. It makes the code more readable.
> 
>> No, it doesn't because one is left wondering what is VMS_LINKED about.
> 
> I agree with Paolo. IMHO VMS_LINKED is conceptually difficult in  a sense
> that it's quite different that what we have currently. I have the feeling,
> conceptually, we are trying to fit in something like data structures with a
> type parameter (the element type) here. AFAIU what vmstate currently can is
> directed acyclic graphs of certain stuff (and also completely custom
> handling based on custom put/get).
> 
You are right. What we have in VMSTATE now cannot handle a recursive (or
cyclic as you call it) structure. The idea was to use VMS_LINKED to
indicate a recursive structure. In this patch is used on a queue. It can
also be used on list or trees.
In this regard, VMS_LINKED does represent something.

Thanks,
Jianjun

>> What is the relation between linked datastructures and passing some
>> arguments as NULL/noon-NULL?
> 
> IFAIU we need those for the datastructure because it's linked and has
> a type parameter (element type). The two last arguments are for
> the element type. These were added by the previous patch because
> the old VMStateInfo did not need these. So we drastically changed
> the scope of VMStateInfo with the previous patch and I'm not
> sure I like this.
> 
> I will have to stare at this a bit longer to bring something more
> constructive than these (largely feeling-based) remarks.
> 
> Cheers,
> Halil
> 

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Paolo Bonzini]

On 10/10/2016 23:29, Jianjun Duan wrote:
> You are right. What we have in VMSTATE now cannot handle a recursive (or
> cyclic as you call it) structure. The idea was to use VMS_LINKED to
> indicate a recursive structure.

Sure, but it's unnecessary.  If you didn't have VMS_LINKED, no one would
notice the difference.

Yes, existing VMStateInfos didn't need the new arguments, the new one
does.  So you add the new arguments.  There's no need to add the flag.

Paolo

> In this patch is used on a queue. It can
> also be used on list or trees.
> In this regard, VMS_LINKED does represent something.

Re: [Qemu-devel] [Qemu-ppc] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Jianjun Duan]

On 10/08/2016 12:28 PM, Halil Pasic wrote:
> 
> 
> On 10/08/2016 01:37 PM, Paolo Bonzini wrote:
>>> Even though most put/get have no issues now, when somebody writes a new
>>>> put, he or she could run into issues if only checking the type
>>>> signature. It makes the code more readable.
> 
>> No, it doesn't because one is left wondering what is VMS_LINKED about.
> 
> I agree with Paolo. IMHO VMS_LINKED is conceptually difficult in  a sense
> that it's quite different that what we have currently. I have the feeling,
> conceptually, we are trying to fit in something like data structures with a
> type parameter (the element type) here. AFAIU what vmstate currently can is
> directed acyclic graphs of certain stuff (and also completely custom
> handling based on custom put/get).
> 
You are right. What we have in vmstate now cannot handle a recursive or
cyclic structure in a generic way. VMS_LINKED is intended to indicate a
recursive structure, with the newly added parameters of put/get
providing information about the element type in the recursive structure.

Thanks,
Jianjun
>> What is the relation between linked datastructures and passing some
>> arguments as NULL/noon-NULL?
> 
> IFAIU we need those for the datastructure because it's linked and has
> a type parameter (element type). The two last arguments are for
> the element type. These were added by the previous patch because
> the old VMStateInfo did not need these. So we drastically changed
> the scope of VMStateInfo with the previous patch and I'm not
> sure I like this.
> 
> I will have to stare at this a bit longer to bring something more
> constructive than these (largely feeling-based) remarks.
> 
> Cheers,
> Halil
> 

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Paolo Bonzini]

On 05/10/2016 18:56, Dr. David Alan Gilbert wrote:
> * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
>> Currently we cannot directly transfer a QTAILQ instance because of the
>> limitation in the migration code. Here we introduce an approach to
>> transfer such structures. In our approach such a structure is tagged
>> with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
>> so that when VMS_LINKED is encountered, put and get from VMStateInfo are
>> called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
>> Similar VMStateInfo can be created for other data structures such as list.
>> This approach will be used to transfer pending_events and ccs_list in spapr
>> state.
>>
>> We also create some macros in qemu/queue.h to access a QTAILQ using pointer
>> arithmetic. This ensures that we do not depend on the implementation
>> details about QTAILQ in the migration code.
> 
> I think we're going to need a way to have a more flexible
> loops; and thus my choice here wouldn't be to use the .get/.put together
> with the VMSD; but I think we'll end up needing a new
> data structure, maybe a VMStateLoop *loop in VMStateField.

Or just realize that we already have a Turing-complete programming
language at our disposal, and it's called C. :)

> Specifically, your format of QTAILQ is perfectly reasonable - a
> byte before each entry which is 1 to indicate there's an entry or 0
> to indicate termination, but there are lots of other variants, e.g.
> 
>    a) put_scsi_requests uses that byte to hold a flag, so it's 0,1,2
>       0 still means terminate but 1 or 2 set a flag in the structure.

Seriously, the way I'd implement (a) is to make the QTAILQ reader peek
for the next byte.  If 0, eat it and exit the loop.  Otherwise, go ahead
with vmstate_load_state which will parse the byte again.

Likewise, on saving let the VMState write a non-zero byte at the
beginning, and then write a zero at the end.

Yes, it's sickening but that's what you do to honor backwards compatibility.

The other possibility is just to bump the version and make the SCSI
request flag a separate byte after the "is there another entry" byte.

Paolo

>    b) slirp_state_load also uses a null byte termination but not off a QTAILQ
>       (although I think it could be flipped for one) (it uses '42' for the
>       non-0 value, but looks like it could become 1)
> 
>    c) virtio_blk also rolls it's own linked list but again with the 0/1 byte
> 
>   Now how would I modify your QTAILQ load/store to do (a) without copying the whole
> thing?
> 
> Dave
> 
>>
>> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
>> ---
>>  include/migration/vmstate.h | 26 ++++++++++++++++++
>>  include/qemu/queue.h        | 32 ++++++++++++++++++++++
>>  migration/trace-events      |  4 +++
>>  migration/vmstate.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++
>>  4 files changed, 128 insertions(+)
>>
>> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
>> index 459dd4a..e60c994 100644
>> --- a/include/migration/vmstate.h
>> +++ b/include/migration/vmstate.h
>> @@ -186,6 +186,12 @@ enum VMStateFlags {
>>       * to determine the number of entries in the array. Only valid in
>>       * combination with one of VMS_VARRAY*. */
>>      VMS_MULTIPLY_ELEMENTS = 0x4000,
>> +    /* For fields which need customized handling, such as QTAILQ in queue.h.
>> +     * When this flag is set in VMStateField, info->get/put will
>> +     * be used in vmstate_load/save_state instead of recursive call.
>> +     * User should implement set info to handle the concerned data structure.
>> +     */
>> +    VMS_LINKED            = 0x8000,
>>  };
>>  
>>  struct VMStateField {
>> @@ -246,6 +252,7 @@ extern const VMStateInfo vmstate_info_timer;
>>  extern const VMStateInfo vmstate_info_buffer;
>>  extern const VMStateInfo vmstate_info_unused_buffer;
>>  extern const VMStateInfo vmstate_info_bitmap;
>> +extern const VMStateInfo vmstate_info_qtailq;
>>  
>>  #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
>>  #define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
>> @@ -657,6 +664,25 @@ extern const VMStateInfo vmstate_info_bitmap;
>>      .offset       = offsetof(_state, _field),                        \
>>  }
>>  
>> +/* For QTAILQ that need customized handling
>> + * _type: type of QTAILQ element
>> + * _next: name of QTAILQ entry field in QTAILQ element
>> + * _vmsd: VMSD for QTAILQ element
>> + * size: size of QTAILQ element
>> + * start: offset of QTAILQ entry in QTAILQ element
>> + */
>> +#define VMSTATE_QTAILQ_V(_field, _state, _version, _vmsd, _type, _next)  \
>> +{                                                                        \
>> +    .name         = (stringify(_field)),                                 \
>> +    .version_id   = (_version),                                          \
>> +    .vmsd         = &(_vmsd),                                            \
>> +    .size         = sizeof(_type),                                       \
>> +    .info         = &vmstate_info_qtailq,                                \
>> +    .flags        = VMS_LINKED,                                          \
>> +    .offset       = offsetof(_state, _field),                            \
>> +    .start        = offsetof(_type, _next),                              \
>> +}
>> +
>>  /* _f : field name
>>     _f_n : num of elements field_name
>>     _n : num of elements
>> diff --git a/include/qemu/queue.h b/include/qemu/queue.h
>> index 342073f..12c3f80 100644
>> --- a/include/qemu/queue.h
>> +++ b/include/qemu/queue.h
>> @@ -438,4 +438,36 @@ struct {                                                                \
>>  #define QTAILQ_PREV(elm, headname, field) \
>>          (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
>>  
>> +/*
>> + * Offsets of layout of a tail queue head.
>> + */
>> +#define QTAILQ_FIRST_OFFSET 0
>> +#define QTAILQ_LAST_OFFSET (sizeof(void *))
>> +
>> +/*
>> + * Offsets of layout of a tail queue element.
>> + */
>> +#define QTAILQ_NEXT_OFFSET 0
>> +#define QTAILQ_PREV_OFFSET (sizeof(void *))
>> +
>> +/*
>> + * Tail queue tranversal using pointer arithmetic.
>> + */
>> +#define QTAILQ_RAW_FOREACH(elm, head, entry)                                   \
>> +        for ((elm) = *((void **) ((char *) (head) + QTAILQ_FIRST_OFFSET));     \
>> +             (elm);                                                            \
>> +             (elm) =                                                           \
>> +                 *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)))
>> +/*
>> + * Tail queue insertion using pointer arithmetic.
>> + */
>> +#define QTAILQ_RAW_INSERT_TAIL(head, elm, entry) do {                          \
>> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)) = NULL;   \
>> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_PREV_OFFSET)) =         \
>> +            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
>> +        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
>> +        *((void **) ((char *) (head) + QTAILQ_LAST_OFFSET)) =                  \
>> +            (void *) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET);          \
>> +} while (/*CONSTCOND*/0)
>> +
>>  #endif /* QEMU_SYS_QUEUE_H */
>> diff --git a/migration/trace-events b/migration/trace-events
>> index dfee75a..9a6ec59 100644
>> --- a/migration/trace-events
>> +++ b/migration/trace-events
>> @@ -52,6 +52,10 @@ vmstate_n_elems(const char *name, int n_elems) "%s: %d"
>>  vmstate_subsection_load(const char *parent) "%s"
>>  vmstate_subsection_load_bad(const char *parent,  const char *sub, const char *sub2) "%s: %s/%s"
>>  vmstate_subsection_load_good(const char *parent) "%s"
>> +get_qtailq(const char *name, int version_id) "%s v%d"
>> +get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
>> +put_qtailq(const char *name, int version_id) "%s v%d"
>> +put_qtailq_end(const char *name, const char *reason) "%s %s"
>>  
>>  # migration/qemu-file.c
>>  qemu_file_fclose(void) ""
>> diff --git a/migration/vmstate.c b/migration/vmstate.c
>> index 66802cb..192db8a 100644
>> --- a/migration/vmstate.c
>> +++ b/migration/vmstate.c
>> @@ -5,7 +5,9 @@
>>  #include "migration/vmstate.h"
>>  #include "qemu/bitops.h"
>>  #include "qemu/error-report.h"
>> +#include "qemu/queue.h"
>>  #include "trace.h"
>> +#include "migration/qjson.h"
>>  
>>  static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
>>                                      void *opaque, QJSON *vmdesc);
>> @@ -121,6 +123,8 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
>>                  if (field->flags & VMS_STRUCT) {
>>                      ret = vmstate_load_state(f, field->vmsd, addr,
>>                                               field->vmsd->version_id);
>> +                } else if (field->flags & VMS_LINKED) {
>> +                    ret = field->info->get(f, addr, size, field);
>>                  } else {
>>                      ret = field->info->get(f, addr, size, NULL);
>>  
>> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
>>  
>>      if (field->flags & VMS_STRUCT) {
>>          type = "struct";
>> +    } else if (field->flags & VMS_LINKED) {
>> +        type = "linked";
>>      } else if (field->info->name) {
>>          type = field->info->name;
>>      }
>> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
>>                  }
>>                  if (field->flags & VMS_STRUCT) {
>>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
>> +                } else if  (field->flags & VMS_LINKED) {
>> +                    field->info->put(f, addr, size, field, vmdesc_loop);
>>                  } else {
>>                      field->info->put(f, addr, size, NULL, NULL);
>>                  }
>> @@ -939,3 +947,61 @@ const VMStateInfo vmstate_info_bitmap = {
>>      .get = get_bitmap,
>>      .put = put_bitmap,
>>  };
>> +
>> +/*get for QTAILQ */
>> +static int get_qtailq(QEMUFile *f, void *pv, size_t unused_size,
>> +                      VMStateField *field)
>> +{
>> +    int ret = 0;
>> +    const VMStateDescription *vmsd = field->vmsd;
>> +    size_t size = field->size;
>> +    size_t entry = field->start;
>> +    int version_id = field->version_id;
>> +    void *elm;
>> +
>> +    trace_get_qtailq(vmsd->name, version_id);
>> +    if (version_id > vmsd->version_id) {
>> +        trace_get_qtailq_end(vmsd->name, "too new", -EINVAL);
> 
> Can you make those error_report's please - if it fails we want to
> see why in the log.
> 
> Dave
> 
>> +        return -EINVAL;
>> +    }
>> +    if (version_id < vmsd->minimum_version_id) {
>> +        trace_get_qtailq_end(vmsd->name, "too old", -EINVAL);
>> +        return -EINVAL;
>> +    }
>> +
>> +    while (qemu_get_byte(f)) {
>> +        elm =  g_malloc(size);
>> +        ret = vmstate_load_state(f, vmsd, elm, version_id);
>> +        if (ret) {
>> +            return ret;
>> +        }
>> +        QTAILQ_RAW_INSERT_TAIL(pv, elm, entry);
>> +    }
>> +
>> +    trace_get_qtailq_end(vmsd->name, "end", ret);
>> +    return ret;
>> +}
>> +
>> +/* put for QTAILQ */
>> +static void put_qtailq(QEMUFile *f, void *pv, size_t unused_size,
>> +                       VMStateField *field, QJSON *vmdesc)
>> +{
>> +    const VMStateDescription *vmsd = field->vmsd;
>> +    size_t entry = field->start;
>> +    void *elm;
>> +
>> +    trace_put_qtailq(vmsd->name, vmsd->version_id);
>> +
>> +    QTAILQ_RAW_FOREACH(elm, pv, entry) {
>> +        qemu_put_byte(f, true);
>> +        vmstate_save_state(f, vmsd, elm, vmdesc);
>> +    }
>> +    qemu_put_byte(f, false);
>> +
>> +    trace_put_qtailq_end(vmsd->name, "end");
>> +}
>> +const VMStateInfo vmstate_info_qtailq = {
>> +    .name = "qtailq",
>> +    .get  = get_qtailq,
>> +    .put  = put_qtailq,
>> +};
>> -- 
>> 1.9.1
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> 

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Dr. David Alan Gilbert]

* Paolo Bonzini (pbonzini@redhat.com) wrote:
> 
> 
> On 05/10/2016 18:56, Dr. David Alan Gilbert wrote:
> > * Jianjun Duan (duanj@linux.vnet.ibm.com) wrote:
> >> Currently we cannot directly transfer a QTAILQ instance because of the
> >> limitation in the migration code. Here we introduce an approach to
> >> transfer such structures. In our approach such a structure is tagged
> >> with VMS_LINKED. We then modified vmstate_save_state and vmstate_load_state
> >> so that when VMS_LINKED is encountered, put and get from VMStateInfo are
> >> called respectively. We created VMStateInfo vmstate_info_qtailq for QTAILQ.
> >> Similar VMStateInfo can be created for other data structures such as list.
> >> This approach will be used to transfer pending_events and ccs_list in spapr
> >> state.
> >>
> >> We also create some macros in qemu/queue.h to access a QTAILQ using pointer
> >> arithmetic. This ensures that we do not depend on the implementation
> >> details about QTAILQ in the migration code.
> > 
> > I think we're going to need a way to have a more flexible
> > loops; and thus my choice here wouldn't be to use the .get/.put together
> > with the VMSD; but I think we'll end up needing a new
> > data structure, maybe a VMStateLoop *loop in VMStateField.
> 
> Or just realize that we already have a Turing-complete programming
> language at our disposal, and it's called C. :)

Yes, and it does seem that the virtio migration code has used it to its
full abilities.

> > Specifically, your format of QTAILQ is perfectly reasonable - a
> > byte before each entry which is 1 to indicate there's an entry or 0
> > to indicate termination, but there are lots of other variants, e.g.
> > 
> >    a) put_scsi_requests uses that byte to hold a flag, so it's 0,1,2
> >       0 still means terminate but 1 or 2 set a flag in the structure.
> 
> Seriously, the way I'd implement (a) is to make the QTAILQ reader peek
> for the next byte.  If 0, eat it and exit the loop.  Otherwise, go ahead
> with vmstate_load_state which will parse the byte again.
> 
> Likewise, on saving let the VMState write a non-zero byte at the
> beginning, and then write a zero at the end.
> 
> Yes, it's sickening but that's what you do to honor backwards compatibility.

Actually, that's not *that* bad an idea.

Lets go with Jianjun's structure for the moment; we can always expand on it.

It seems we have ~3 concepts that feel partially independent:

    a) The format of the loop on the wire (eg one byte per iteration, 0 terminates)
    b) The way the list is represented (QTAILQ, simple array, device specific linked-list)
    c) The data gathered in each iteration
    d) The allocation of (c)

This patch has a,b,d all wrapped up together in the get/put functions -
where I was hoping to find a way to separate them a bit so that we
could say; I want a loop, with this format, into this data structure, using this allocator.

Dave

> The other possibility is just to bump the version and make the SCSI
> request flag a separate byte after the "is there another entry" byte.
> 
> Paolo
> 
> >    b) slirp_state_load also uses a null byte termination but not off a QTAILQ
> >       (although I think it could be flipped for one) (it uses '42' for the
> >       non-0 value, but looks like it could become 1)
> > 
> >    c) virtio_blk also rolls it's own linked list but again with the 0/1 byte
> > 
> >   Now how would I modify your QTAILQ load/store to do (a) without copying the whole
> > thing?
> > 
> > Dave
> > 
> >>
> >> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> >> ---
> >>  include/migration/vmstate.h | 26 ++++++++++++++++++
> >>  include/qemu/queue.h        | 32 ++++++++++++++++++++++
> >>  migration/trace-events      |  4 +++
> >>  migration/vmstate.c         | 66 +++++++++++++++++++++++++++++++++++++++++++++
> >>  4 files changed, 128 insertions(+)
> >>
> >> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> >> index 459dd4a..e60c994 100644
> >> --- a/include/migration/vmstate.h
> >> +++ b/include/migration/vmstate.h
> >> @@ -186,6 +186,12 @@ enum VMStateFlags {
> >>       * to determine the number of entries in the array. Only valid in
> >>       * combination with one of VMS_VARRAY*. */
> >>      VMS_MULTIPLY_ELEMENTS = 0x4000,
> >> +    /* For fields which need customized handling, such as QTAILQ in queue.h.
> >> +     * When this flag is set in VMStateField, info->get/put will
> >> +     * be used in vmstate_load/save_state instead of recursive call.
> >> +     * User should implement set info to handle the concerned data structure.
> >> +     */
> >> +    VMS_LINKED            = 0x8000,
> >>  };
> >>  
> >>  struct VMStateField {
> >> @@ -246,6 +252,7 @@ extern const VMStateInfo vmstate_info_timer;
> >>  extern const VMStateInfo vmstate_info_buffer;
> >>  extern const VMStateInfo vmstate_info_unused_buffer;
> >>  extern const VMStateInfo vmstate_info_bitmap;
> >> +extern const VMStateInfo vmstate_info_qtailq;
> >>  
> >>  #define type_check_2darray(t1,t2,n,m) ((t1(*)[n][m])0 - (t2*)0)
> >>  #define type_check_array(t1,t2,n) ((t1(*)[n])0 - (t2*)0)
> >> @@ -657,6 +664,25 @@ extern const VMStateInfo vmstate_info_bitmap;
> >>      .offset       = offsetof(_state, _field),                        \
> >>  }
> >>  
> >> +/* For QTAILQ that need customized handling
> >> + * _type: type of QTAILQ element
> >> + * _next: name of QTAILQ entry field in QTAILQ element
> >> + * _vmsd: VMSD for QTAILQ element
> >> + * size: size of QTAILQ element
> >> + * start: offset of QTAILQ entry in QTAILQ element
> >> + */
> >> +#define VMSTATE_QTAILQ_V(_field, _state, _version, _vmsd, _type, _next)  \
> >> +{                                                                        \
> >> +    .name         = (stringify(_field)),                                 \
> >> +    .version_id   = (_version),                                          \
> >> +    .vmsd         = &(_vmsd),                                            \
> >> +    .size         = sizeof(_type),                                       \
> >> +    .info         = &vmstate_info_qtailq,                                \
> >> +    .flags        = VMS_LINKED,                                          \
> >> +    .offset       = offsetof(_state, _field),                            \
> >> +    .start        = offsetof(_type, _next),                              \
> >> +}
> >> +
> >>  /* _f : field name
> >>     _f_n : num of elements field_name
> >>     _n : num of elements
> >> diff --git a/include/qemu/queue.h b/include/qemu/queue.h
> >> index 342073f..12c3f80 100644
> >> --- a/include/qemu/queue.h
> >> +++ b/include/qemu/queue.h
> >> @@ -438,4 +438,36 @@ struct {                                                                \
> >>  #define QTAILQ_PREV(elm, headname, field) \
> >>          (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
> >>  
> >> +/*
> >> + * Offsets of layout of a tail queue head.
> >> + */
> >> +#define QTAILQ_FIRST_OFFSET 0
> >> +#define QTAILQ_LAST_OFFSET (sizeof(void *))
> >> +
> >> +/*
> >> + * Offsets of layout of a tail queue element.
> >> + */
> >> +#define QTAILQ_NEXT_OFFSET 0
> >> +#define QTAILQ_PREV_OFFSET (sizeof(void *))
> >> +
> >> +/*
> >> + * Tail queue tranversal using pointer arithmetic.
> >> + */
> >> +#define QTAILQ_RAW_FOREACH(elm, head, entry)                                   \
> >> +        for ((elm) = *((void **) ((char *) (head) + QTAILQ_FIRST_OFFSET));     \
> >> +             (elm);                                                            \
> >> +             (elm) =                                                           \
> >> +                 *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)))
> >> +/*
> >> + * Tail queue insertion using pointer arithmetic.
> >> + */
> >> +#define QTAILQ_RAW_INSERT_TAIL(head, elm, entry) do {                          \
> >> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET)) = NULL;   \
> >> +        *((void **) ((char *) (elm) + (entry) + QTAILQ_PREV_OFFSET)) =         \
> >> +            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
> >> +        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
> >> +        *((void **) ((char *) (head) + QTAILQ_LAST_OFFSET)) =                  \
> >> +            (void *) ((char *) (elm) + (entry) + QTAILQ_NEXT_OFFSET);          \
> >> +} while (/*CONSTCOND*/0)
> >> +
> >>  #endif /* QEMU_SYS_QUEUE_H */
> >> diff --git a/migration/trace-events b/migration/trace-events
> >> index dfee75a..9a6ec59 100644
> >> --- a/migration/trace-events
> >> +++ b/migration/trace-events
> >> @@ -52,6 +52,10 @@ vmstate_n_elems(const char *name, int n_elems) "%s: %d"
> >>  vmstate_subsection_load(const char *parent) "%s"
> >>  vmstate_subsection_load_bad(const char *parent,  const char *sub, const char *sub2) "%s: %s/%s"
> >>  vmstate_subsection_load_good(const char *parent) "%s"
> >> +get_qtailq(const char *name, int version_id) "%s v%d"
> >> +get_qtailq_end(const char *name, const char *reason, int val) "%s %s/%d"
> >> +put_qtailq(const char *name, int version_id) "%s v%d"
> >> +put_qtailq_end(const char *name, const char *reason) "%s %s"
> >>  
> >>  # migration/qemu-file.c
> >>  qemu_file_fclose(void) ""
> >> diff --git a/migration/vmstate.c b/migration/vmstate.c
> >> index 66802cb..192db8a 100644
> >> --- a/migration/vmstate.c
> >> +++ b/migration/vmstate.c
> >> @@ -5,7 +5,9 @@
> >>  #include "migration/vmstate.h"
> >>  #include "qemu/bitops.h"
> >>  #include "qemu/error-report.h"
> >> +#include "qemu/queue.h"
> >>  #include "trace.h"
> >> +#include "migration/qjson.h"
> >>  
> >>  static void vmstate_subsection_save(QEMUFile *f, const VMStateDescription *vmsd,
> >>                                      void *opaque, QJSON *vmdesc);
> >> @@ -121,6 +123,8 @@ int vmstate_load_state(QEMUFile *f, const VMStateDescription *vmsd,
> >>                  if (field->flags & VMS_STRUCT) {
> >>                      ret = vmstate_load_state(f, field->vmsd, addr,
> >>                                               field->vmsd->version_id);
> >> +                } else if (field->flags & VMS_LINKED) {
> >> +                    ret = field->info->get(f, addr, size, field);
> >>                  } else {
> >>                      ret = field->info->get(f, addr, size, NULL);
> >>  
> >> @@ -193,6 +197,8 @@ static const char *vmfield_get_type_name(VMStateField *field)
> >>  
> >>      if (field->flags & VMS_STRUCT) {
> >>          type = "struct";
> >> +    } else if (field->flags & VMS_LINKED) {
> >> +        type = "linked";
> >>      } else if (field->info->name) {
> >>          type = field->info->name;
> >>      }
> >> @@ -327,6 +333,8 @@ void vmstate_save_state(QEMUFile *f, const VMStateDescription *vmsd,
> >>                  }
> >>                  if (field->flags & VMS_STRUCT) {
> >>                      vmstate_save_state(f, field->vmsd, addr, vmdesc_loop);
> >> +                } else if  (field->flags & VMS_LINKED) {
> >> +                    field->info->put(f, addr, size, field, vmdesc_loop);
> >>                  } else {
> >>                      field->info->put(f, addr, size, NULL, NULL);
> >>                  }
> >> @@ -939,3 +947,61 @@ const VMStateInfo vmstate_info_bitmap = {
> >>      .get = get_bitmap,
> >>      .put = put_bitmap,
> >>  };
> >> +
> >> +/*get for QTAILQ */
> >> +static int get_qtailq(QEMUFile *f, void *pv, size_t unused_size,
> >> +                      VMStateField *field)
> >> +{
> >> +    int ret = 0;
> >> +    const VMStateDescription *vmsd = field->vmsd;
> >> +    size_t size = field->size;
> >> +    size_t entry = field->start;
> >> +    int version_id = field->version_id;
> >> +    void *elm;
> >> +
> >> +    trace_get_qtailq(vmsd->name, version_id);
> >> +    if (version_id > vmsd->version_id) {
> >> +        trace_get_qtailq_end(vmsd->name, "too new", -EINVAL);
> > 
> > Can you make those error_report's please - if it fails we want to
> > see why in the log.
> > 
> > Dave
> > 
> >> +        return -EINVAL;
> >> +    }
> >> +    if (version_id < vmsd->minimum_version_id) {
> >> +        trace_get_qtailq_end(vmsd->name, "too old", -EINVAL);
> >> +        return -EINVAL;
> >> +    }
> >> +
> >> +    while (qemu_get_byte(f)) {
> >> +        elm =  g_malloc(size);
> >> +        ret = vmstate_load_state(f, vmsd, elm, version_id);
> >> +        if (ret) {
> >> +            return ret;
> >> +        }
> >> +        QTAILQ_RAW_INSERT_TAIL(pv, elm, entry);
> >> +    }
> >> +
> >> +    trace_get_qtailq_end(vmsd->name, "end", ret);
> >> +    return ret;
> >> +}
> >> +
> >> +/* put for QTAILQ */
> >> +static void put_qtailq(QEMUFile *f, void *pv, size_t unused_size,
> >> +                       VMStateField *field, QJSON *vmdesc)
> >> +{
> >> +    const VMStateDescription *vmsd = field->vmsd;
> >> +    size_t entry = field->start;
> >> +    void *elm;
> >> +
> >> +    trace_put_qtailq(vmsd->name, vmsd->version_id);
> >> +
> >> +    QTAILQ_RAW_FOREACH(elm, pv, entry) {
> >> +        qemu_put_byte(f, true);
> >> +        vmstate_save_state(f, vmsd, elm, vmdesc);
> >> +    }
> >> +    qemu_put_byte(f, false);
> >> +
> >> +    trace_put_qtailq_end(vmsd->name, "end");
> >> +}
> >> +const VMStateInfo vmstate_info_qtailq = {
> >> +    .name = "qtailq",
> >> +    .get  = get_qtailq,
> >> +    .put  = put_qtailq,
> >> +};
> >> -- 
> >> 1.9.1
> >>
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > 
> > 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Paolo Bonzini]

On 06/10/2016 13:56, Dr. David Alan Gilbert wrote:
>> > Yes, it's sickening but that's what you do to honor backwards compatibility.
> Actually, that's not *that* bad an idea.
> 
> Lets go with Jianjun's structure for the moment; we can always expand on it.
> 
> It seems we have ~3 concepts that feel partially independent:
> 
>     a) The format of the loop on the wire (eg one byte per iteration, 0 terminates)
>     b) The way the list is represented (QTAILQ, simple array, device specific linked-list)
>     c) The data gathered in each iteration
>     d) The allocation of (c)
> 
> This patch has a,b,d all wrapped up together in the get/put functions -
> where I was hoping to find a way to separate them a bit so that we
> could say; I want a loop, with this format, into this data structure, using this allocator.

Yes, the sickening part is when the format of the loop intersects with
the format of the datastructure.

I agree with moving the allocator out of VMStateInfo and back into
VMStateField, but only as long as VMStateAllocator could replace other
VMS_* flags.

I'm not sure about the value in separating (a) and (b), but we can do
things one step at a time.

By the way, regarding this:

> The other possibility is just to bump the version and make the SCSI
> request flag a separate byte after the "is there another entry" byte.

There is another way to do it that is much more backwards-compatible.
Choose a "default" value of retry corresponding to what QEMU encodes as
a "1".  If it's different, use a subsection to encode that.  Migration
from old to new will fail if the wrong value of retry is used, because
it will see a 2 where the QTAILQ loop expects a zero or one.  Migration
from new to old will fail if the wrong value of retry is used, because
it will see a subsection header where the QTAILQ loop expects a zero or one.

I think this is acceptable, and it would only affect migration of USB
storage devices.

Paolo

Re: [Qemu-devel] [QEMU PATCH v5 4/6] migration: migrate QTAILQ

[Dr. David Alan Gilbert]

* Paolo Bonzini (pbonzini@redhat.com) wrote:
> 
> 
> On 06/10/2016 13:56, Dr. David Alan Gilbert wrote:
> >> > Yes, it's sickening but that's what you do to honor backwards compatibility.
> > Actually, that's not *that* bad an idea.
> > 
> > Lets go with Jianjun's structure for the moment; we can always expand on it.
> > 
> > It seems we have ~3 concepts that feel partially independent:
> > 
> >     a) The format of the loop on the wire (eg one byte per iteration, 0 terminates)
> >     b) The way the list is represented (QTAILQ, simple array, device specific linked-list)
> >     c) The data gathered in each iteration
> >     d) The allocation of (c)
> > 
> > This patch has a,b,d all wrapped up together in the get/put functions -
> > where I was hoping to find a way to separate them a bit so that we
> > could say; I want a loop, with this format, into this data structure, using this allocator.
> 
> Yes, the sickening part is when the format of the loop intersects with
> the format of the datastructure.

Yes.

> I agree with moving the allocator out of VMStateInfo and back into
> VMStateField, but only as long as VMStateAllocator could replace other
> VMS_* flags.
> 
> I'm not sure about the value in separating (a) and (b), but we can do
> things one step at a time.

The other observation is that in many of the cases the loop body uses
some state present in the outer state or in a value read prior to the
start of the loop.
For example virtio_blk_load_device uses the vdev pointer inside
the loop during the initialisation of each loaded request (I can see
some hacky ways of avoiding it but it's messy).

> By the way, regarding this:
> 
> > The other possibility is just to bump the version and make the SCSI
> > request flag a separate byte after the "is there another entry" byte.
> 
> There is another way to do it that is much more backwards-compatible.
> Choose a "default" value of retry corresponding to what QEMU encodes as
> a "1".  If it's different, use a subsection to encode that.  Migration
> from old to new will fail if the wrong value of retry is used, because
> it will see a 2 where the QTAILQ loop expects a zero or one.  Migration
> from new to old will fail if the wrong value of retry is used, because
> it will see a subsection header where the QTAILQ loop expects a zero or one.
> 
> I think this is acceptable, and it would only affect migration of USB
> storage devices.

Yes, it might be worth it in some of these cases; although I do try
and avoid breaking format at almost all costs.

Dave

> Paolo
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

[Qemu-devel] [QEMU PATCH v5 5/6] migration: spapr: migrate ccs_list in spapr state

[Jianjun Duan]

ccs_list in spapr state maintains the device tree related
information on the rtas side for hotplugged devices. In racing
situations between hotplug events and migration operation, a rtas
hotplug event could be migrated from the source guest to target
guest, or the source guest could have not yet finished fetching
the device tree when migration is started, the target will try
to finish fetching the device tree. By migrating ccs_list, the
target can fetch the device tree properly.

ccs_list is put in a subsection in the spapr state VMSD to make
sure migration across different versions is not broken.

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
---
 hw/ppc/spapr.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 63b6a0d..1847d35 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1255,6 +1255,36 @@ static bool version_before_3(void *opaque, int version_id)
     return version_id < 3;
 }
 
+static bool spapr_ccs_list_needed(void *opaque)
+{
+    sPAPRMachineState *spapr = (sPAPRMachineState *)opaque;
+    return !QTAILQ_EMPTY(&spapr->ccs_list);
+}
+
+static const VMStateDescription vmstate_spapr_ccs = {
+    .name = "spaprconfigureconnectorstate",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_UINT32(drc_index, sPAPRConfigureConnectorState),
+        VMSTATE_INT32(fdt_offset, sPAPRConfigureConnectorState),
+        VMSTATE_INT32(fdt_depth, sPAPRConfigureConnectorState),
+        VMSTATE_END_OF_LIST()
+    },
+};
+
+static const VMStateDescription vmstate_spapr_ccs_list = {
+    .name = "spaprccslist",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = spapr_ccs_list_needed,
+    .fields = (VMStateField[]) {
+        VMSTATE_QTAILQ_V(ccs_list, sPAPRMachineState, 1,
+                         vmstate_spapr_ccs, sPAPRConfigureConnectorState, next),
+        VMSTATE_END_OF_LIST()
+    },
+};
+
 static const VMStateDescription vmstate_spapr = {
     .name = "spapr",
     .version_id = 3,
@@ -1270,6 +1300,10 @@ static const VMStateDescription vmstate_spapr = {
         VMSTATE_PPC_TIMEBASE_V(tb, sPAPRMachineState, 2),
         VMSTATE_END_OF_LIST()
     },
+    .subsections = (const VMStateDescription*[]) {
+        &vmstate_spapr_ccs_list,
+        NULL
+    }
 };
 
 static int htab_save_setup(QEMUFile *f, void *opaque)
-- 
1.9.1

Re: [Qemu-devel] [QEMU PATCH v5 5/6] migration: spapr: migrate ccs_list in spapr state

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3843 bytes --]

On Mon, Oct 03, 2016 at 11:24:56AM -0700, Jianjun Duan wrote:
> ccs_list in spapr state maintains the device tree related
> information on the rtas side for hotplugged devices. In racing
> situations between hotplug events and migration operation, a rtas
> hotplug event could be migrated from the source guest to target
> guest, or the source guest could have not yet finished fetching
> the device tree when migration is started, the target will try
> to finish fetching the device tree. By migrating ccs_list, the
> target can fetch the device tree properly.
> 
> ccs_list is put in a subsection in the spapr state VMSD to make
> sure migration across different versions is not broken.
> 
> Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>

I'm still not entirely convinced we need to migrate the ccs_list.
What would happen if we did this:

   * Keep a flag which indicates whether the guest is in the middle of
     the configure_connector process.
       - I'm not sure if that would need to be a new bit of state, or
         if we could deduce it from the value of the isolation and
	 allocation states
       - If it's new state, we'd need to migrate it, obviously not if
         we can derive it from other state flags

   * On the destination during post_load, if there was an in-progress
     configure_connector on the source, we set another "stale
     configure" flag

   * When a configure_connector call is attempted on the destination
     with the stale configure flag set, return an error

The question is, if we choose the right error, can we get the guest to
either restart the configure from scratch, or fail gracefully, so the
operator can restart the hotplug

> ---
>  hw/ppc/spapr.c | 34 ++++++++++++++++++++++++++++++++++
>  1 file changed, 34 insertions(+)
> 
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 63b6a0d..1847d35 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -1255,6 +1255,36 @@ static bool version_before_3(void *opaque, int version_id)
>      return version_id < 3;
>  }
>  
> +static bool spapr_ccs_list_needed(void *opaque)
> +{
> +    sPAPRMachineState *spapr = (sPAPRMachineState *)opaque;
> +    return !QTAILQ_EMPTY(&spapr->ccs_list);
> +}
> +
> +static const VMStateDescription vmstate_spapr_ccs = {
> +    .name = "spaprconfigureconnectorstate",
> +    .version_id = 1,
> +    .minimum_version_id = 1,
> +    .fields = (VMStateField[]) {
> +        VMSTATE_UINT32(drc_index, sPAPRConfigureConnectorState),
> +        VMSTATE_INT32(fdt_offset, sPAPRConfigureConnectorState),
> +        VMSTATE_INT32(fdt_depth, sPAPRConfigureConnectorState),
> +        VMSTATE_END_OF_LIST()
> +    },
> +};
> +
> +static const VMStateDescription vmstate_spapr_ccs_list = {
> +    .name = "spaprccslist",
> +    .version_id = 1,
> +    .minimum_version_id = 1,
> +    .needed = spapr_ccs_list_needed,
> +    .fields = (VMStateField[]) {
> +        VMSTATE_QTAILQ_V(ccs_list, sPAPRMachineState, 1,
> +                         vmstate_spapr_ccs, sPAPRConfigureConnectorState, next),
> +        VMSTATE_END_OF_LIST()
> +    },
> +};
> +
>  static const VMStateDescription vmstate_spapr = {
>      .name = "spapr",
>      .version_id = 3,
> @@ -1270,6 +1300,10 @@ static const VMStateDescription vmstate_spapr = {
>          VMSTATE_PPC_TIMEBASE_V(tb, sPAPRMachineState, 2),
>          VMSTATE_END_OF_LIST()
>      },
> +    .subsections = (const VMStateDescription*[]) {
> +        &vmstate_spapr_ccs_list,
> +        NULL
> +    }
>  };
>  
>  static int htab_save_setup(QEMUFile *f, void *opaque)

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] [QEMU PATCH v5 5/6] migration: spapr: migrate ccs_list in spapr state

[Michael Roth]

Quoting David Gibson (2016-10-06 22:36:07)
> On Mon, Oct 03, 2016 at 11:24:56AM -0700, Jianjun Duan wrote:
> > ccs_list in spapr state maintains the device tree related
> > information on the rtas side for hotplugged devices. In racing
> > situations between hotplug events and migration operation, a rtas
> > hotplug event could be migrated from the source guest to target
> > guest, or the source guest could have not yet finished fetching
> > the device tree when migration is started, the target will try
> > to finish fetching the device tree. By migrating ccs_list, the
> > target can fetch the device tree properly.
> > 
> > ccs_list is put in a subsection in the spapr state VMSD to make
> > sure migration across different versions is not broken.
> > 
> > Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> 
> I'm still not entirely convinced we need to migrate the ccs_list.
> What would happen if we did this:
> 
>    * Keep a flag which indicates whether the guest is in the middle of
>      the configure_connector process.
>        - I'm not sure if that would need to be a new bit of state, or
>          if we could deduce it from the value of the isolation and
>          allocation states
>        - If it's new state, we'd need to migrate it, obviously not if
>          we can derive it from other state flags
> 
>    * On the destination during post_load, if there was an in-progress
>      configure_connector on the source, we set another "stale
>      configure" flag
> 
>    * When a configure_connector call is attempted on the destination
>      with the stale configure flag set, return an error
> 
> The question is, if we choose the right error, can we get the guest to
> either restart the configure from scratch, or fail gracefully, so the
> operator can restart the hotplug

To get the configure to restart, the guest's configure_connector
implementation would need to changed. Current code in drmgr would just
bail on any error, and I'd imagine the in-kernel version does the same.

So at least for existing guests, the only option is failing the command
at the operator's interface, namely device_add. device_add is
asynchronous to the actual hotplug event handling however. So if we want
to convey failure to the user, it would have to be either in the form of
a new QMP event emitted to convey hotplug success or error, or through
device_add itself by implementing something like the async QMP rework
that Marc-Andre posted some time back (which still seems to be a topic
of debate). Which either approach, something like libvirt, with adequate
state-tracking for pending hotplug events, could handle an error event
on the target side post-migrate and convey that to the user somehow.

That's probably a much larger discussion if it comes to that, but doable
in theory.

But even that wouldn't get us totally out of the woods: DRC state can
still be modified outside of hotplug. For instance, a guest should be
able to do:

  drmgr -c pci -s <drc_index> -r
  drmgr -c pci -s <drc_index> -a

to return a device to firmware and then later take it back and
reconfigure it. I'm not aware of any common case where this would occur,
but it's not disallowed by the specification, and performing a migration
between these 2 operations would currently break this since the default
coldplug state on target assumes a configured state in source.

> 
> > ---
> >  hw/ppc/spapr.c | 34 ++++++++++++++++++++++++++++++++++
> >  1 file changed, 34 insertions(+)
> > 
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index 63b6a0d..1847d35 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -1255,6 +1255,36 @@ static bool version_before_3(void *opaque, int version_id)
> >      return version_id < 3;
> >  }
> >  
> > +static bool spapr_ccs_list_needed(void *opaque)
> > +{
> > +    sPAPRMachineState *spapr = (sPAPRMachineState *)opaque;
> > +    return !QTAILQ_EMPTY(&spapr->ccs_list);
> > +}
> > +
> > +static const VMStateDescription vmstate_spapr_ccs = {
> > +    .name = "spaprconfigureconnectorstate",
> > +    .version_id = 1,
> > +    .minimum_version_id = 1,
> > +    .fields = (VMStateField[]) {
> > +        VMSTATE_UINT32(drc_index, sPAPRConfigureConnectorState),
> > +        VMSTATE_INT32(fdt_offset, sPAPRConfigureConnectorState),
> > +        VMSTATE_INT32(fdt_depth, sPAPRConfigureConnectorState),
> > +        VMSTATE_END_OF_LIST()
> > +    },
> > +};
> > +
> > +static const VMStateDescription vmstate_spapr_ccs_list = {
> > +    .name = "spaprccslist",
> > +    .version_id = 1,
> > +    .minimum_version_id = 1,
> > +    .needed = spapr_ccs_list_needed,
> > +    .fields = (VMStateField[]) {
> > +        VMSTATE_QTAILQ_V(ccs_list, sPAPRMachineState, 1,
> > +                         vmstate_spapr_ccs, sPAPRConfigureConnectorState, next),
> > +        VMSTATE_END_OF_LIST()
> > +    },
> > +};
> > +
> >  static const VMStateDescription vmstate_spapr = {
> >      .name = "spapr",
> >      .version_id = 3,
> > @@ -1270,6 +1300,10 @@ static const VMStateDescription vmstate_spapr = {
> >          VMSTATE_PPC_TIMEBASE_V(tb, sPAPRMachineState, 2),
> >          VMSTATE_END_OF_LIST()
> >      },
> > +    .subsections = (const VMStateDescription*[]) {
> > +        &vmstate_spapr_ccs_list,
> > +        NULL
> > +    }
> >  };
> >  
> >  static int htab_save_setup(QEMUFile *f, void *opaque)
> 
> -- 
> David Gibson                    | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
>                                 | _way_ _around_!
> http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] [QEMU PATCH v5 5/6] migration: spapr: migrate ccs_list in spapr state

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3987 bytes --]

On Fri, Oct 07, 2016 at 09:52:51AM -0500, Michael Roth wrote:
> Quoting David Gibson (2016-10-06 22:36:07)
> > On Mon, Oct 03, 2016 at 11:24:56AM -0700, Jianjun Duan wrote:
> > > ccs_list in spapr state maintains the device tree related
> > > information on the rtas side for hotplugged devices. In racing
> > > situations between hotplug events and migration operation, a rtas
> > > hotplug event could be migrated from the source guest to target
> > > guest, or the source guest could have not yet finished fetching
> > > the device tree when migration is started, the target will try
> > > to finish fetching the device tree. By migrating ccs_list, the
> > > target can fetch the device tree properly.
> > > 
> > > ccs_list is put in a subsection in the spapr state VMSD to make
> > > sure migration across different versions is not broken.
> > > 
> > > Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
> > 
> > I'm still not entirely convinced we need to migrate the ccs_list.
> > What would happen if we did this:
> > 
> >    * Keep a flag which indicates whether the guest is in the middle of
> >      the configure_connector process.
> >        - I'm not sure if that would need to be a new bit of state, or
> >          if we could deduce it from the value of the isolation and
> >          allocation states
> >        - If it's new state, we'd need to migrate it, obviously not if
> >          we can derive it from other state flags
> > 
> >    * On the destination during post_load, if there was an in-progress
> >      configure_connector on the source, we set another "stale
> >      configure" flag
> > 
> >    * When a configure_connector call is attempted on the destination
> >      with the stale configure flag set, return an error
> > 
> > The question is, if we choose the right error, can we get the guest to
> > either restart the configure from scratch, or fail gracefully, so the
> > operator can restart the hotplug
> 
> To get the configure to restart, the guest's configure_connector
> implementation would need to changed. Current code in drmgr would just
> bail on any error, and I'd imagine the in-kernel version does the same.
> 
> So at least for existing guests, the only option is failing the command
> at the operator's interface, namely device_add. device_add is
> asynchronous to the actual hotplug event handling however. So if we want
> to convey failure to the user, it would have to be either in the form of
> a new QMP event emitted to convey hotplug success or error, or through
> device_add itself by implementing something like the async QMP rework
> that Marc-Andre posted some time back (which still seems to be a topic
> of debate). Which either approach, something like libvirt, with adequate
> state-tracking for pending hotplug events, could handle an error event
> on the target side post-migrate and convey that to the user somehow.
> 
> That's probably a much larger discussion if it comes to that, but doable
> in theory.
> 
> But even that wouldn't get us totally out of the woods: DRC state can
> still be modified outside of hotplug. For instance, a guest should be
> able to do:
> 
>   drmgr -c pci -s <drc_index> -r
>   drmgr -c pci -s <drc_index> -a
> 
> to return a device to firmware and then later take it back and
> reconfigure it. I'm not aware of any common case where this would occur,
> but it's not disallowed by the specification, and performing a migration
> between these 2 operations would currently break this since the default
> coldplug state on target assumes a configured state in source.

Thanks, that's a good case for why we need this.  Can you please fold
this description into your commit messages so it's there for posterity.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[Qemu-devel] [QEMU PATCH v5 6/6] migration: spapr: migrate pending_events of spapr state

[Jianjun Duan]

In racing situations between hotplug events and migration operation,
a rtas hotplug event could have not yet be delivered to the source
guest when migration is started. In this case the pending_events of
spapr state need be transmitted to the target so that the hotplug
event can be finished on the target.

All the different fields of the events are encoded as defined by
PAPR. We can migrate them as uint8_t binary stream without any
concerns about data padding or endianess.

pending_events is put in a subsection in the spapr state VMSD to make
sure migration across different versions is not broken.

Signed-off-by: Jianjun Duan <duanj@linux.vnet.ibm.com>
---
 hw/ppc/spapr.c         | 33 +++++++++++++++++++++++++++++++++
 hw/ppc/spapr_events.c  | 22 +++++++++++++---------
 include/hw/ppc/spapr.h |  3 ++-
 3 files changed, 48 insertions(+), 10 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 1847d35..5b57e5a 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1255,12 +1255,32 @@ static bool version_before_3(void *opaque, int version_id)
     return version_id < 3;
 }
 
+static bool spapr_pending_events_needed(void *opaque)
+{
+    sPAPRMachineState *spapr = (sPAPRMachineState *)opaque;
+    return !QTAILQ_EMPTY(&spapr->pending_events);
+}
+
 static bool spapr_ccs_list_needed(void *opaque)
 {
     sPAPRMachineState *spapr = (sPAPRMachineState *)opaque;
     return !QTAILQ_EMPTY(&spapr->ccs_list);
 }
 
+static const VMStateDescription vmstate_spapr_event_entry = {
+    .name = "spapreventlogentry",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (VMStateField[]) {
+        VMSTATE_INT32(log_type, sPAPREventLogEntry),
+        VMSTATE_BOOL(exception, sPAPREventLogEntry),
+        VMSTATE_UINT32(data_size, sPAPREventLogEntry),
+        VMSTATE_VARRAY_UINT32_ALLOC(data, sPAPREventLogEntry, data_size,
+                                    0, vmstate_info_uint8, uint8_t),
+        VMSTATE_END_OF_LIST()
+  },
+};
+
 static const VMStateDescription vmstate_spapr_ccs = {
     .name = "spaprconfigureconnectorstate",
     .version_id = 1,
@@ -1273,6 +1293,18 @@ static const VMStateDescription vmstate_spapr_ccs = {
     },
 };
 
+static const VMStateDescription vmstate_spapr_pending_events = {
+    .name = "spaprpendingevents",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = spapr_pending_events_needed,
+    .fields = (VMStateField[]) {
+        VMSTATE_QTAILQ_V(pending_events, sPAPRMachineState, 1,
+                         vmstate_spapr_event_entry, sPAPREventLogEntry, next),
+        VMSTATE_END_OF_LIST()
+    },
+};
+
 static const VMStateDescription vmstate_spapr_ccs_list = {
     .name = "spaprccslist",
     .version_id = 1,
@@ -1301,6 +1333,7 @@ static const VMStateDescription vmstate_spapr = {
         VMSTATE_END_OF_LIST()
     },
     .subsections = (const VMStateDescription*[]) {
+        &vmstate_spapr_pending_events,
         &vmstate_spapr_ccs_list,
         NULL
     }
diff --git a/hw/ppc/spapr_events.c b/hw/ppc/spapr_events.c
index 4c7b6ae..3f45744 100644
--- a/hw/ppc/spapr_events.c
+++ b/hw/ppc/spapr_events.c
@@ -230,7 +230,8 @@ void spapr_events_fdt_skel(void *fdt, uint32_t check_exception_irq)
     _FDT((fdt_end_node(fdt)));
 }
 
-static void rtas_event_log_queue(int log_type, void *data, bool exception)
+static void rtas_event_log_queue(int log_type, void *data, bool exception,
+				 int data_size)
 {
     sPAPRMachineState *spapr = SPAPR_MACHINE(qdev_get_machine());
     sPAPREventLogEntry *entry = g_new(sPAPREventLogEntry, 1);
@@ -239,6 +240,7 @@ static void rtas_event_log_queue(int log_type, void *data, bool exception)
     entry->log_type = log_type;
     entry->exception = exception;
     entry->data = data;
+    entry->data_size = data_size;
     QTAILQ_INSERT_TAIL(&spapr->pending_events, entry, next);
 }
 
@@ -341,6 +343,7 @@ static void spapr_powerdown_req(Notifier *n, void *opaque)
     struct rtas_event_log_v6_mainb *mainb;
     struct rtas_event_log_v6_epow *epow;
     struct epow_log_full *new_epow;
+    uint32_t data_size;
 
     new_epow = g_malloc0(sizeof(*new_epow));
     hdr = &new_epow->hdr;
@@ -349,13 +352,13 @@ static void spapr_powerdown_req(Notifier *n, void *opaque)
     mainb = &new_epow->mainb;
     epow = &new_epow->epow;
 
+    data_size = sizeof(*new_epow);
     hdr->summary = cpu_to_be32(RTAS_LOG_VERSION_6
                                | RTAS_LOG_SEVERITY_EVENT
                                | RTAS_LOG_DISPOSITION_NOT_RECOVERED
                                | RTAS_LOG_OPTIONAL_PART_PRESENT
                                | RTAS_LOG_TYPE_EPOW);
-    hdr->extended_length = cpu_to_be32(sizeof(*new_epow)
-                                       - sizeof(new_epow->hdr));
+    hdr->extended_length = cpu_to_be32(data_size - sizeof(new_epow->hdr));
 
     spapr_init_v6hdr(v6hdr);
     spapr_init_maina(maina, 3 /* Main-A, Main-B and EPOW */);
@@ -375,7 +378,7 @@ static void spapr_powerdown_req(Notifier *n, void *opaque)
     epow->event_modifier = RTAS_LOG_V6_EPOW_MODIFIER_NORMAL;
     epow->extended_modifier = RTAS_LOG_V6_EPOW_XMODIFIER_PARTITION_SPECIFIC;
 
-    rtas_event_log_queue(RTAS_LOG_TYPE_EPOW, new_epow, true);
+    rtas_event_log_queue(RTAS_LOG_TYPE_EPOW, new_epow, true, data_size);
 
     qemu_irq_pulse(xics_get_qirq(spapr->xics, spapr->check_exception_irq));
 }
@@ -398,6 +401,7 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action,
     struct rtas_event_log_v6_maina *maina;
     struct rtas_event_log_v6_mainb *mainb;
     struct rtas_event_log_v6_hp *hp;
+    uint32_t data_size;
 
     new_hp = g_malloc0(sizeof(struct hp_log_full));
     hdr = &new_hp->hdr;
@@ -406,14 +410,14 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action,
     mainb = &new_hp->mainb;
     hp = &new_hp->hp;
 
+    data_size = sizeof(*new_hp);
     hdr->summary = cpu_to_be32(RTAS_LOG_VERSION_6
                                | RTAS_LOG_SEVERITY_EVENT
                                | RTAS_LOG_DISPOSITION_NOT_RECOVERED
                                | RTAS_LOG_OPTIONAL_PART_PRESENT
                                | RTAS_LOG_INITIATOR_HOTPLUG
                                | RTAS_LOG_TYPE_HOTPLUG);
-    hdr->extended_length = cpu_to_be32(sizeof(*new_hp)
-                                       - sizeof(new_hp->hdr));
+    hdr->extended_length = cpu_to_be32(data_size - sizeof(new_hp->hdr));
 
     spapr_init_v6hdr(v6hdr);
     spapr_init_maina(maina, 3 /* Main-A, Main-B, HP */);
@@ -457,7 +461,7 @@ static void spapr_hotplug_req_event(uint8_t hp_id, uint8_t hp_action,
         hp->drc.index = cpu_to_be32(drc);
     }
 
-    rtas_event_log_queue(RTAS_LOG_TYPE_HOTPLUG, new_hp, true);
+    rtas_event_log_queue(RTAS_LOG_TYPE_HOTPLUG, new_hp, true, data_size);
 
     qemu_irq_pulse(xics_get_qirq(spapr->xics, spapr->check_exception_irq));
 }
@@ -524,7 +528,7 @@ static void check_exception(PowerPCCPU *cpu, sPAPRMachineState *spapr,
         goto out_no_events;
     }
 
-    hdr = event->data;
+    hdr = (struct rtas_error_log *)event->data;
     event_len = be32_to_cpu(hdr->extended_length) + sizeof(*hdr);
 
     if (event_len < len) {
@@ -574,7 +578,7 @@ static void event_scan(PowerPCCPU *cpu, sPAPRMachineState *spapr,
         goto out_no_events;
     }
 
-    hdr = event->data;
+    hdr = (struct rtas_error_log *)event->data;
     event_len = be32_to_cpu(hdr->extended_length) + sizeof(*hdr);
 
     if (event_len < len) {
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 6289d50..8772f9d 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -568,7 +568,8 @@ sPAPRTCETable *spapr_tce_find_by_liobn(target_ulong liobn);
 struct sPAPREventLogEntry {
     int log_type;
     bool exception;
-    void *data;
+    uint32_t data_size;
+    uint8_t *data;
     QTAILQ_ENTRY(sPAPREventLogEntry) next;
 };
 
-- 
1.9.1

Re: [Qemu-devel] [QEMU PATCH v5 0/6] migration: ensure hotplug and migration work together

[no-reply]

Hi,

Your series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 1475519097-27611-1-git-send-email-duanj@linux.vnet.ibm.com
Subject: [Qemu-devel] [QEMU PATCH v5 0/6] migration: ensure hotplug and migration work together

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

# Useful git options
git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git show --no-patch --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]         patchew/1475519097-27611-1-git-send-email-duanj@linux.vnet.ibm.com -> patchew/1475519097-27611-1-git-send-email-duanj@linux.vnet.ibm.com
Switched to a new branch 'test'
d935e1f migration: spapr: migrate pending_events of spapr state
64aa229 migration: spapr: migrate ccs_list in spapr state
ecf2d6c migration: migrate QTAILQ
6f641b8 migration: extend VMStateInfo
803e704 migration: spapr_drc: defined VMStateDescription struct
08b0c64 migration: alternative way to set instance_id in SaveStateEntry

=== OUTPUT BEGIN ===
Checking PATCH 1/6: migration: alternative way to set instance_id in SaveStateEntry...
Checking PATCH 2/6: migration: spapr_drc: defined VMStateDescription struct...
ERROR: space required before the open parenthesis '('
#69: FILE: hw/ppc/spapr_drc.c:634:
+    switch(drc->type) {

total: 1 errors, 0 warnings, 135 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 3/6: migration: extend VMStateInfo...
Checking PATCH 4/6: migration: migrate QTAILQ...
ERROR: spaces required around that '+' (ctx:WxV)
#108: FILE: include/qemu/queue.h:467:
+            *((void **) ((char *) (head) +QTAILQ_LAST_OFFSET));                \
                                          ^

ERROR: spaces required around that '+' (ctx:WxV)
#109: FILE: include/qemu/queue.h:468:
+        **((void ***)((char *) (head) +QTAILQ_LAST_OFFSET)) = (elm);           \
                                       ^

total: 2 errors, 0 warnings, 184 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 5/6: migration: spapr: migrate ccs_list in spapr state...
ERROR: spaces required around that '*' (ctx:VxV)
#66: FILE: hw/ppc/spapr.c:1287:
+    .subsections = (const VMStateDescription*[]) {
                                             ^

total: 1 errors, 0 warnings, 46 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 6/6: migration: spapr: migrate pending_events of spapr state...
ERROR: code indent should never use tabs
#96: FILE: hw/ppc/spapr_events.c:234:
+^I^I^I^I int data_size)$

total: 1 errors, 0 warnings, 159 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [QEMU PATCH v5 0/6] migration: ensure hotplug and migration work together

[no-reply]

Hi,

Your series failed automatic build test. Please find the testing commands and
their output below. If you have docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 1475519097-27611-1-git-send-email-duanj@linux.vnet.ibm.com
Subject: [Qemu-devel] [QEMU PATCH v5 0/6] migration: ensure hotplug and migration work together

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
make J=8 docker-test-quick@centos6
make J=8 docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
d935e1f migration: spapr: migrate pending_events of spapr state
64aa229 migration: spapr: migrate ccs_list in spapr state
ecf2d6c migration: migrate QTAILQ
6f641b8 migration: extend VMStateInfo
803e704 migration: spapr_drc: defined VMStateDescription struct
08b0c64 migration: alternative way to set instance_id in SaveStateEntry

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into 'dtc'...
Submodule path 'dtc': checked out '65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf'
  BUILD centos6
  ARCHIVE qemu.tgz
  ARCHIVE dtc.tgz
  COPY RUNNER
  RUN test-quick in centos6
Packages installed:
SDL-devel-1.2.14-7.el6_7.1.x86_64
ccache-3.1.6-2.el6.x86_64
epel-release-6-8.noarch
gcc-4.4.7-17.el6.x86_64
git-1.7.1-4.el6_7.1.x86_64
glib2-devel-2.28.8-5.el6.x86_64
libfdt-devel-1.4.0-1.el6.x86_64
make-3.81-23.el6.x86_64
package g++ is not installed
pixman-devel-0.32.8-1.el6.x86_64
tar-1.23-15.el6_8.x86_64
zlib-devel-1.2.3-29.el6.x86_64

Environment variables:
PACKAGES=libfdt-devel ccache     tar git make gcc g++     zlib-devel glib2-devel SDL-devel pixman-devel     epel-release
HOSTNAME=cb3993c582fb
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
LANG=en_US.UTF-8
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES= dtc
DEBUG=
G_BROKEN_FILENAMES=1
CCACHE_HASHDIR=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/src/tests/docker/install
No C++ compiler available; disabling C++ specific optional code
Install prefix    /tmp/qemu-test/src/tests/docker/install
BIOS directory    /tmp/qemu-test/src/tests/docker/install/share/qemu
binary directory  /tmp/qemu-test/src/tests/docker/install/bin
library directory /tmp/qemu-test/src/tests/docker/install/lib
module directory  /tmp/qemu-test/src/tests/docker/install/lib/qemu
libexec directory /tmp/qemu-test/src/tests/docker/install/libexec
include directory /tmp/qemu-test/src/tests/docker/install/include
config directory  /tmp/qemu-test/src/tests/docker/install/etc
local state directory   /tmp/qemu-test/src/tests/docker/install/var
Manual directory  /tmp/qemu-test/src/tests/docker/install/share/man
ELF interp prefix /usr/gnemul/qemu-%M
Source path       /tmp/qemu-test/src
C compiler        cc
Host C compiler   cc
C++ compiler      
Objective-C compiler cc
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/include/pixman-1    -pthread -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include   -fPIE -DPIE -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-all
LDFLAGS           -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -pie -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
tcg debug enabled no
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
pixman            system
SDL support       yes (1.2.14)
GTK support       no 
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    no
GNUTLS rnd        no
libgcrypt         no
libgcrypt kdf     no
nettle            no 
nettle kdf        no
libtasn1          no
curses support    no
virgl support     no
curl support      no
mingw32 support   no
Audio drivers     oss
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
VNC support       yes
VNC SASL support  no
VNC JPEG support  no
VNC PNG support   no
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               yes
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support yes
Install blobs     yes
KVM support       yes
RDMA support      no
TCG interpreter   no
fdt support       yes
preadv support    yes
fdatasync         yes
madvise           yes
posix_madvise     yes
libcap-ng support no
vhost-net support yes
vhost-scsi support yes
vhost-vsock support yes
Trace backends    log
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info no
QGA MSI support   no
seccomp support   no
coroutine backend ucontext
coroutine pool    yes
GlusterFS support no
Archipelago support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   no
TPM passthrough   yes
QOM debugging     yes
lzo support       no
snappy support    no
bzip2 support     no
NUMA host support no
tcmalloc support  no
jemalloc support  no
avx2 optimization no
replication support yes
  GEN   x86_64-softmmu/config-devices.mak.tmp
  GEN   aarch64-softmmu/config-devices.mak.tmp
  GEN   config-host.h
  GEN   qemu-options.def
  GEN   qmp-commands.h
  GEN   qapi-types.h
  GEN   qapi-visit.h
  GEN   qapi-event.h
  GEN   x86_64-softmmu/config-devices.mak
  GEN   aarch64-softmmu/config-devices.mak
  GEN   qmp-introspect.h
  GEN   module_block.h
  GEN   tests/test-qapi-types.h
  GEN   tests/test-qapi-visit.h
  GEN   tests/test-qmp-commands.h
  GEN   tests/test-qapi-event.h
  GEN   tests/test-qmp-introspect.h
  GEN   config-all-devices.mak
  GEN   trace/generated-events.h
  GEN   trace/generated-tracers.h
  GEN   trace/generated-tcg-tracers.h
  GEN   trace/generated-helpers-wrappers.h
  GEN   trace/generated-helpers.h
  CC    tests/qemu-iotests/socket_scm_helper.o
  GEN   qga/qapi-generated/qga-qapi-visit.h
  GEN   qga/qapi-generated/qga-qapi-types.h
  GEN   qga/qapi-generated/qga-qmp-commands.h
  GEN   qga/qapi-generated/qga-qapi-types.c
  GEN   qga/qapi-generated/qga-qapi-visit.c
  GEN   qga/qapi-generated/qga-qmp-marshal.c
  GEN   qmp-introspect.c
  GEN   qapi-types.c
  GEN   qapi-visit.c
  GEN   qapi-event.c
  CC    qapi/qapi-visit-core.o
  CC    qapi/qapi-dealloc-visitor.o
  CC    qapi/qmp-input-visitor.o
  CC    qapi/qmp-output-visitor.o
  CC    qapi/qmp-registry.o
  CC    qapi/qmp-dispatch.o
  CC    qapi/string-input-visitor.o
  CC    qapi/string-output-visitor.o
  CC    qapi/opts-visitor.o
  CC    qapi/qapi-clone-visitor.o
  CC    qapi/qmp-event.o
  CC    qapi/qapi-util.o
  CC    qobject/qnull.o
  CC    qobject/qint.o
  CC    qobject/qstring.o
  CC    qobject/qdict.o
  CC    qobject/qlist.o
  CC    qobject/qfloat.o
  CC    qobject/qbool.o
  CC    qobject/qjson.o
  CC    qobject/qobject.o
  CC    qobject/json-lexer.o
  CC    qobject/json-streamer.o
  CC    qobject/json-parser.o
  GEN   trace/generated-events.c
  CC    trace/control.o
  CC    trace/qmp.o
  CC    util/osdep.o
  CC    util/cutils.o
  CC    util/unicode.o
  CC    util/qemu-timer-common.o
  CC    util/bufferiszero.o
  CC    util/compatfd.o
  CC    util/event_notifier-posix.o
  CC    util/mmap-alloc.o
  CC    util/oslib-posix.o
  CC    util/qemu-openpty.o
  CC    util/qemu-thread-posix.o
  CC    util/memfd.o
  CC    util/envlist.o
  CC    util/path.o
  CC    util/module.o
  CC    util/bitmap.o
  CC    util/bitops.o
  CC    util/hbitmap.o
  CC    util/fifo8.o
  CC    util/acl.o
  CC    util/error.o
  CC    util/qemu-error.o
  CC    util/id.o
  CC    util/iov.o
  CC    util/qemu-config.o
  CC    util/qemu-sockets.o
  CC    util/uri.o
  CC    util/notify.o
  CC    util/qemu-option.o
  CC    util/qemu-progress.o
  CC    util/hexdump.o
  CC    util/crc32c.o
  CC    util/uuid.o
  CC    util/throttle.o
  CC    util/getauxval.o
  CC    util/readline.o
  CC    util/rfifolock.o
  CC    util/rcu.o
  CC    util/qemu-coroutine.o
  CC    util/qemu-coroutine-lock.o
  CC    util/qemu-coroutine-io.o
  CC    util/qemu-coroutine-sleep.o
  CC    util/coroutine-ucontext.o
  CC    util/buffer.o
  CC    util/timed-average.o
  CC    util/base64.o
  CC    util/log.o
  CC    util/qdist.o
  CC    util/qht.o
  CC    util/range.o
  CC    crypto/pbkdf-stub.o
  CC    stubs/arch-query-cpu-def.o
  CC    stubs/arch-query-cpu-model-expansion.o
  CC    stubs/arch-query-cpu-model-comparison.o
/tmp/qemu-test/src/util/qht.c: In function ‘qht_reset_size’:
/tmp/qemu-test/src/util/qht.c:413: warning: ‘new’ may be used uninitialized in this function
  CC    stubs/arch-query-cpu-model-baseline.o
  CC    stubs/bdrv-next-monitor-owned.o
  CC    stubs/blk-commit-all.o
  CC    stubs/blockdev-close-all-bdrv-states.o
  CC    stubs/clock-warp.o
  CC    stubs/cpu-get-clock.o
  CC    stubs/cpu-get-icount.o
  CC    stubs/dump.o
  CC    stubs/fdset-add-fd.o
  CC    stubs/fdset-find-fd.o
  CC    stubs/fdset-get-fd.o
  CC    stubs/fdset-remove-fd.o
  CC    stubs/gdbstub.o
  CC    stubs/get-fd.o
  CC    stubs/get-next-serial.o
  CC    stubs/get-vm-name.o
  CC    stubs/iothread-lock.o
  CC    stubs/is-daemonized.o
  CC    stubs/machine-init-done.o
  CC    stubs/migr-blocker.o
  CC    stubs/mon-is-qmp.o
  CC    stubs/mon-printf.o
  CC    stubs/monitor-init.o
  CC    stubs/notify-event.o
  CC    stubs/qtest.o
  CC    stubs/replay.o
  CC    stubs/replay-user.o
  CC    stubs/reset.o
  CC    stubs/runstate-check.o
  CC    stubs/set-fd-handler.o
  CC    stubs/slirp.o
  CC    stubs/sysbus.o
  CC    stubs/trace-control.o
  CC    stubs/uuid.o
  CC    stubs/vm-stop.o
  CC    stubs/vmstate.o
  CC    stubs/cpus.o
  CC    stubs/kvm.o
  CC    stubs/qmp_pc_dimm_device_list.o
  CC    stubs/target-monitor-defs.o
  CC    stubs/target-get-monitor-def.o
  CC    stubs/vhost.o
  CC    stubs/iohandler.o
  CC    stubs/smbios_type_38.o
  CC    stubs/ipmi.o
  CC    stubs/pc_madt_cpu_entry.o
  CC    contrib/ivshmem-client/ivshmem-client.o
  CC    contrib/ivshmem-client/main.o
  CC    contrib/ivshmem-server/ivshmem-server.o
  CC    contrib/ivshmem-server/main.o
  CC    qemu-nbd.o
  CC    async.o
  CC    thread-pool.o
  CC    block.o
  CC    blockjob.o
  CC    main-loop.o
  CC    iohandler.o
  CC    qemu-timer.o
  CC    aio-posix.o
  CC    qemu-io-cmds.o
  CC    replication.o
  CC    block/raw_bsd.o
  CC    block/qcow.o
  CC    block/vdi.o
  CC    block/vmdk.o
  CC    block/cloop.o
  CC    block/bochs.o
  CC    block/vpc.o
  CC    block/vvfat.o
  CC    block/dmg.o
  CC    block/qcow2.o
  CC    block/qcow2-refcount.o
  CC    block/qcow2-cluster.o
  CC    block/qcow2-snapshot.o
  CC    block/qcow2-cache.o
  CC    block/qed.o
  CC    block/qed-gencb.o
  CC    block/qed-l2-cache.o
  CC    block/qed-table.o
  CC    block/qed-cluster.o
  CC    block/qed-check.o
  CC    block/vhdx.o
  CC    block/vhdx-endian.o
  CC    block/vhdx-log.o
  CC    block/quorum.o
  CC    block/parallels.o
  CC    block/blkdebug.o
  CC    block/blkverify.o
  CC    block/blkreplay.o
  CC    block/block-backend.o
  CC    block/snapshot.o
  CC    block/qapi.o
  CC    block/raw-posix.o
  CC    block/null.o
  CC    block/mirror.o
  CC    block/commit.o
  CC    block/io.o
  CC    block/throttle-groups.o
  CC    block/nbd.o
  CC    block/nbd-client.o
  CC    block/sheepdog.o
  CC    block/accounting.o
  CC    block/dirty-bitmap.o
  CC    block/write-threshold.o
  CC    block/backup.o
  CC    block/replication.o
  CC    block/crypto.o
  CC    nbd/server.o
  CC    nbd/client.o
  CC    nbd/common.o
  CC    crypto/init.o
  CC    crypto/hash.o
  CC    crypto/hash-glib.o
  CC    crypto/aes.o
  CC    crypto/desrfb.o
  CC    crypto/cipher.o
  CC    crypto/tlscreds.o
  CC    crypto/tlscredsanon.o
  CC    crypto/tlscredsx509.o
  CC    crypto/tlssession.o
  CC    crypto/secret.o
  CC    crypto/random-platform.o
  CC    crypto/pbkdf.o
  CC    crypto/ivgen.o
  CC    crypto/ivgen-essiv.o
  CC    crypto/ivgen-plain.o
  CC    crypto/ivgen-plain64.o
  CC    crypto/afsplit.o
  CC    crypto/xts.o
  CC    crypto/block.o
  CC    crypto/block-qcow.o
  CC    crypto/block-luks.o
  CC    io/channel.o
  CC    io/channel-buffer.o
  CC    io/channel-command.o
  CC    io/channel-file.o
  CC    io/channel-socket.o
  CC    io/channel-tls.o
  CC    io/channel-watch.o
  CC    io/channel-websock.o
  CC    io/channel-util.o
  CC    io/task.o
  CC    qom/object.o
  CC    qom/container.o
  CC    qom/qom-qobject.o
  CC    qom/object_interfaces.o
  GEN   qemu-img-cmds.h
  CC    qemu-io.o
  CC    qemu-bridge-helper.o
  CC    blockdev.o
  CC    blockdev-nbd.o
  CC    iothread.o
  CC    qdev-monitor.o
  CC    device-hotplug.o
  CC    os-posix.o
  CC    qemu-char.o
  CC    page_cache.o
  CC    accel.o
  CC    bt-host.o
  CC    bt-vhci.o
  CC    dma-helpers.o
  CC    vl.o
  CC    tpm.o
  CC    device_tree.o
  GEN   qmp-marshal.c
  CC    qmp.o
  CC    hmp.o
  CC    tcg-runtime.o
  CC    cpus-common.o
  CC    audio/audio.o
  CC    audio/noaudio.o
  CC    audio/wavaudio.o
  CC    audio/mixeng.o
  CC    audio/sdlaudio.o
  CC    audio/ossaudio.o
  CC    audio/wavcapture.o
  CC    backends/rng.o
  CC    backends/rng-egd.o
  CC    backends/rng-random.o
  CC    backends/msmouse.o
  CC    backends/testdev.o
  CC    backends/tpm.o
  CC    backends/hostmem.o
  CC    backends/hostmem-ram.o
  CC    backends/hostmem-file.o
  CC    block/stream.o
  CC    disas/arm.o
  CC    disas/i386.o
  CC    fsdev/qemu-fsdev-dummy.o
  CC    fsdev/qemu-fsdev-opts.o
  CC    hw/acpi/core.o
  CC    hw/acpi/piix4.o
  CC    hw/acpi/pcihp.o
  CC    hw/acpi/ich9.o
  CC    hw/acpi/tco.o
  CC    hw/acpi/cpu_hotplug.o
  CC    hw/acpi/memory_hotplug.o
  CC    hw/acpi/memory_hotplug_acpi_table.o
  CC    hw/acpi/cpu.o
  CC    hw/acpi/acpi_interface.o
  CC    hw/acpi/bios-linker-loader.o
  CC    hw/acpi/aml-build.o
  CC    hw/acpi/ipmi.o
  CC    hw/audio/sb16.o
  CC    hw/audio/es1370.o
  CC    hw/audio/ac97.o
  CC    hw/audio/fmopl.o
  CC    hw/audio/adlib.o
  CC    hw/audio/gus.o
  CC    hw/audio/gusemu_hal.o
  CC    hw/audio/gusemu_mixer.o
  CC    hw/audio/cs4231a.o
  CC    hw/audio/intel-hda.o
  CC    hw/audio/hda-codec.o
  CC    hw/audio/pcspk.o
  CC    hw/audio/wm8750.o
  CC    hw/audio/pl041.o
  CC    hw/audio/lm4549.o
  CC    hw/audio/marvell_88w8618.o
  CC    hw/block/block.o
  CC    hw/block/cdrom.o
  CC    hw/block/hd-geometry.o
  CC    hw/block/fdc.o
  CC    hw/block/m25p80.o
  CC    hw/block/nand.o
  CC    hw/block/pflash_cfi01.o
  CC    hw/block/pflash_cfi02.o
  CC    hw/block/ecc.o
  CC    hw/block/onenand.o
  CC    hw/block/nvme.o
  CC    hw/bt/core.o
  CC    hw/bt/l2cap.o
  CC    hw/bt/sdp.o
  CC    hw/bt/hci.o
  CC    hw/bt/hid.o
  CC    hw/bt/hci-csr.o
  CC    hw/char/ipoctal232.o
  CC    hw/char/parallel.o
  CC    hw/char/pl011.o
  CC    hw/char/serial.o
  CC    hw/char/serial-isa.o
  CC    hw/char/serial-pci.o
  CC    hw/char/virtio-console.o
  CC    hw/char/cadence_uart.o
  CC    hw/char/debugcon.o
  CC    hw/char/imx_serial.o
  CC    hw/core/qdev.o
  CC    hw/core/qdev-properties.o
  CC    hw/core/bus.o
  CC    hw/core/fw-path-provider.o
  CC    hw/core/irq.o
  CC    hw/core/hotplug.o
  CC    hw/core/ptimer.o
  CC    hw/core/sysbus.o
  CC    hw/core/machine.o
  CC    hw/core/null-machine.o
  CC    hw/core/loader.o
  CC    hw/core/qdev-properties-system.o
  CC    hw/core/register.o
  CC    hw/core/platform-bus.o
  CC    hw/display/ads7846.o
  CC    hw/display/cirrus_vga.o
  CC    hw/display/pl110.o
  CC    hw/display/ssd0303.o
  CC    hw/display/ssd0323.o
  CC    hw/display/vga-pci.o
  CC    hw/display/vga-isa.o
  CC    hw/display/vmware_vga.o
  CC    hw/display/blizzard.o
  CC    hw/display/exynos4210_fimd.o
  CC    hw/display/framebuffer.o
  CC    hw/display/tc6393xb.o
  CC    hw/dma/pl080.o
  CC    hw/dma/pl330.o
  CC    hw/dma/i8257.o
  CC    hw/dma/xlnx-zynq-devcfg.o
  CC    hw/gpio/max7310.o
  CC    hw/gpio/pl061.o
  CC    hw/gpio/zaurus.o
  CC    hw/gpio/gpio_key.o
  CC    hw/i2c/core.o
  CC    hw/i2c/smbus.o
  CC    hw/i2c/smbus_eeprom.o
  CC    hw/i2c/i2c-ddc.o
  CC    hw/i2c/versatile_i2c.o
  CC    hw/i2c/smbus_ich9.o
  CC    hw/i2c/pm_smbus.o
  CC    hw/i2c/bitbang_i2c.o
  CC    hw/i2c/exynos4210_i2c.o
  CC    hw/i2c/imx_i2c.o
  CC    hw/i2c/aspeed_i2c.o
  CC    hw/ide/core.o
  CC    hw/ide/atapi.o
  CC    hw/ide/qdev.o
  CC    hw/ide/pci.o
  CC    hw/ide/isa.o
  CC    hw/ide/piix.o
  CC    hw/ide/microdrive.o
  CC    hw/ide/ahci.o
  CC    hw/ide/ich.o
  CC    hw/input/hid.o
  CC    hw/input/lm832x.o
  CC    hw/input/pckbd.o
  CC    hw/input/pl050.o
  CC    hw/input/ps2.o
  CC    hw/input/stellaris_input.o
  CC    hw/input/tsc2005.o
  CC    hw/input/vmmouse.o
  CC    hw/input/virtio-input.o
  CC    hw/input/virtio-input-hid.o
  CC    hw/input/virtio-input-host.o
  CC    hw/intc/i8259_common.o
  CC    hw/intc/i8259.o
  CC    hw/intc/pl190.o
  CC    hw/intc/realview_gic.o
  CC    hw/intc/imx_avic.o
  CC    hw/intc/ioapic_common.o
  CC    hw/intc/arm_gic_common.o
  CC    hw/intc/arm_gic.o
  CC    hw/intc/arm_gicv2m.o
/tmp/qemu-test/src/hw/input/virtio-input.c:299: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/input/virtio-input.c:299: warning: initialization from incompatible pointer type
  CC    hw/intc/arm_gicv3_common.o
  CC    hw/intc/arm_gicv3.o
  CC    hw/intc/arm_gicv3_dist.o
  CC    hw/intc/arm_gicv3_redist.o
  CC    hw/ipack/ipack.o
  CC    hw/ipack/tpci200.o
  CC    hw/ipmi/ipmi.o
  CC    hw/ipmi/ipmi_bmc_sim.o
  CC    hw/ipmi/ipmi_bmc_extern.o
  CC    hw/ipmi/isa_ipmi_kcs.o
  CC    hw/ipmi/isa_ipmi_bt.o
  CC    hw/isa/isa-bus.o
  CC    hw/isa/apm.o
  CC    hw/mem/pc-dimm.o
  CC    hw/mem/nvdimm.o
  CC    hw/misc/applesmc.o
  CC    hw/misc/max111x.o
  CC    hw/misc/tmp105.o
  CC    hw/misc/debugexit.o
  CC    hw/misc/sga.o
  CC    hw/misc/pc-testdev.o
  CC    hw/misc/pci-testdev.o
  CC    hw/misc/arm_l2x0.o
  CC    hw/misc/arm_integrator_debug.o
  CC    hw/misc/a9scu.o
  CC    hw/misc/arm11scu.o
  CC    hw/net/ne2000.o
  CC    hw/net/eepro100.o
  CC    hw/net/pcnet-pci.o
  CC    hw/net/pcnet.o
  CC    hw/net/e1000.o
  CC    hw/net/e1000x_common.o
  CC    hw/net/net_tx_pkt.o
  CC    hw/net/net_rx_pkt.o
  CC    hw/net/e1000e.o
  CC    hw/net/e1000e_core.o
  CC    hw/net/rtl8139.o
  CC    hw/net/vmxnet3.o
  CC    hw/net/smc91c111.o
  CC    hw/net/lan9118.o
  CC    hw/net/ne2000-isa.o
  CC    hw/net/xgmac.o
  CC    hw/net/allwinner_emac.o
  CC    hw/net/imx_fec.o
  CC    hw/net/cadence_gem.o
  CC    hw/net/stellaris_enet.o
  CC    hw/net/rocker/rocker.o
  CC    hw/net/rocker/rocker_fp.o
  CC    hw/net/rocker/rocker_desc.o
  CC    hw/net/rocker/rocker_world.o
  CC    hw/net/rocker/rocker_of_dpa.o
  CC    hw/nvram/eeprom93xx.o
  CC    hw/nvram/fw_cfg.o
  CC    hw/pci-bridge/pci_bridge_dev.o
  CC    hw/pci-bridge/pci_expander_bridge.o
  CC    hw/pci-bridge/xio3130_upstream.o
  CC    hw/pci-bridge/xio3130_downstream.o
  CC    hw/pci-bridge/ioh3420.o
  CC    hw/pci-bridge/i82801b11.o
  CC    hw/pci-host/pam.o
  CC    hw/pci-host/versatile.o
  CC    hw/pci-host/piix.o
  CC    hw/pci-host/q35.o
  CC    hw/pci-host/gpex.o
  CC    hw/pci/pci.o
  CC    hw/pci/pci_bridge.o
  CC    hw/pci/msix.o
/tmp/qemu-test/src/hw/nvram/fw_cfg.c: In function ‘fw_cfg_dma_transfer’:
/tmp/qemu-test/src/hw/nvram/fw_cfg.c:330: warning: ‘read’ may be used uninitialized in this function
  CC    hw/pci/msi.o
  CC    hw/pci/shpc.o
  CC    hw/pci/slotid_cap.o
  CC    hw/pci/pci_host.o
  CC    hw/pci/pcie_host.o
  CC    hw/pci/pcie.o
  CC    hw/pci/pcie_aer.o
  CC    hw/pci/pcie_port.o
  CC    hw/pci/pci-stub.o
  CC    hw/pcmcia/pcmcia.o
  CC    hw/scsi/scsi-disk.o
  CC    hw/scsi/scsi-generic.o
  CC    hw/scsi/scsi-bus.o
  CC    hw/scsi/lsi53c895a.o
  CC    hw/scsi/mptsas.o
  CC    hw/scsi/mptconfig.o
  CC    hw/scsi/mptendian.o
  CC    hw/scsi/megasas.o
  CC    hw/scsi/vmw_pvscsi.o
  CC    hw/scsi/esp.o
  CC    hw/scsi/esp-pci.o
  CC    hw/sd/pl181.o
  CC    hw/sd/ssi-sd.o
  CC    hw/sd/sd.o
  CC    hw/sd/core.o
  CC    hw/sd/sdhci.o
  CC    hw/smbios/smbios.o
  CC    hw/smbios/smbios_type_38.o
  CC    hw/ssi/pl022.o
  CC    hw/ssi/ssi.o
  CC    hw/ssi/xilinx_spips.o
  CC    hw/ssi/aspeed_smc.o
  CC    hw/timer/arm_timer.o
  CC    hw/timer/arm_mptimer.o
  CC    hw/timer/a9gtimer.o
  CC    hw/timer/cadence_ttc.o
  CC    hw/timer/ds1338.o
  CC    hw/timer/hpet.o
  CC    hw/timer/i8254_common.o
  CC    hw/timer/i8254.o
  CC    hw/timer/pl031.o
  CC    hw/timer/twl92230.o
  CC    hw/timer/imx_epit.o
  CC    hw/timer/imx_gpt.o
  CC    hw/timer/stm32f2xx_timer.o
  CC    hw/timer/aspeed_timer.o
  CC    hw/tpm/tpm_tis.o
  CC    hw/tpm/tpm_passthrough.o
  CC    hw/tpm/tpm_util.o
  CC    hw/usb/core.o
  CC    hw/usb/combined-packet.o
  CC    hw/usb/bus.o
  CC    hw/usb/libhw.o
  CC    hw/usb/desc.o
  CC    hw/usb/desc-msos.o
  CC    hw/usb/hcd-uhci.o
  CC    hw/usb/hcd-ohci.o
  CC    hw/usb/hcd-ehci.o
  CC    hw/usb/hcd-ehci-pci.o
  CC    hw/usb/hcd-ehci-sysbus.o
  CC    hw/usb/hcd-xhci.o
  CC    hw/usb/hcd-musb.o
  CC    hw/usb/dev-hub.o
  CC    hw/usb/dev-hid.o
  CC    hw/usb/dev-wacom.o
  CC    hw/usb/dev-storage.o
  CC    hw/usb/dev-uas.o
  CC    hw/usb/dev-audio.o
  CC    hw/usb/dev-serial.o
  CC    hw/usb/dev-network.o
  CC    hw/usb/dev-bluetooth.o
  CC    hw/usb/dev-smartcard-reader.o
  CC    hw/usb/dev-mtp.o
  CC    hw/usb/host-stub.o
  CC    hw/virtio/virtio-rng.o
  CC    hw/virtio/virtio-pci.o
  CC    hw/virtio/virtio-bus.o
  CC    hw/virtio/virtio-mmio.o
  CC    hw/watchdog/watchdog.o
  CC    hw/watchdog/wdt_i6300esb.o
  CC    hw/watchdog/wdt_ib700.o
  CC    migration/migration.o
  CC    migration/socket.o
  CC    migration/fd.o
  CC    migration/exec.o
  CC    migration/tls.o
  CC    migration/vmstate.o
  CC    migration/qemu-file.o
/tmp/qemu-test/src/hw/virtio/virtio-rng.c:219: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/virtio/virtio-rng.c:219: warning: initialization from incompatible pointer type
  CC    migration/qemu-file-channel.o
  CC    migration/xbzrle.o
  CC    migration/postcopy-ram.o
  CC    migration/qjson.o
  CC    migration/block.o
  CC    net/net.o
  CC    net/queue.o
  CC    net/checksum.o
  CC    net/util.o
  CC    net/hub.o
  CC    net/socket.o
  CC    net/dump.o
  CC    net/eth.o
  CC    net/l2tpv3.o
  CC    net/tap.o
  CC    net/vhost-user.o
  CC    net/tap-linux.o
  CC    net/slirp.o
  CC    net/filter.o
  CC    net/filter-buffer.o
  CC    net/filter-mirror.o
  CC    net/colo-compare.o
  CC    net/colo.o
  CC    net/filter-rewriter.o
  CC    qom/cpu.o
  CC    replay/replay.o
  CC    replay/replay-internal.o
  CC    replay/replay-events.o
  CC    replay/replay-time.o
  CC    replay/replay-input.o
  CC    replay/replay-char.o
/tmp/qemu-test/src/replay/replay-internal.c: In function ‘replay_put_array’:
/tmp/qemu-test/src/replay/replay-internal.c:65: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
  CC    replay/replay-snapshot.o
  CC    slirp/cksum.o
  CC    slirp/if.o
  CC    slirp/ip_icmp.o
  CC    slirp/ip6_icmp.o
  CC    slirp/ip6_input.o
  CC    slirp/ip6_output.o
  CC    slirp/ip_input.o
  CC    slirp/ip_output.o
  CC    slirp/dnssearch.o
  CC    slirp/dhcpv6.o
  CC    slirp/slirp.o
  CC    slirp/mbuf.o
  CC    slirp/misc.o
  CC    slirp/sbuf.o
  CC    slirp/socket.o
  CC    slirp/tcp_input.o
  CC    slirp/tcp_output.o
  CC    slirp/tcp_subr.o
  CC    slirp/tcp_timer.o
  CC    slirp/udp.o
  CC    slirp/udp6.o
  CC    slirp/bootp.o
/tmp/qemu-test/src/slirp/tcp_input.c: In function ‘tcp_input’:
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_p’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_len’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_tos’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_id’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_off’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_ttl’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_sum’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_src.s_addr’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:219: warning: ‘save_ip.ip_dst.s_addr’ may be used uninitialized in this function
/tmp/qemu-test/src/slirp/tcp_input.c:220: warning: ‘save_ip6.ip_nh’ may be used uninitialized in this function
  CC    slirp/tftp.o
  CC    slirp/arp_table.o
  CC    slirp/ndp_table.o
  CC    ui/keymaps.o
  CC    ui/console.o
  CC    ui/cursor.o
  CC    ui/qemu-pixman.o
  CC    ui/input.o
  CC    ui/input-keymap.o
  CC    ui/input-legacy.o
  CC    ui/input-linux.o
  CC    ui/sdl.o
  CC    ui/sdl_zoom.o
  CC    ui/x_keymap.o
  CC    ui/vnc.o
  CC    ui/vnc-enc-zlib.o
  CC    ui/vnc-enc-hextile.o
  CC    ui/vnc-enc-tight.o
  CC    ui/vnc-palette.o
  CC    ui/vnc-enc-zrle.o
  CC    ui/vnc-auth-vencrypt.o
  CC    ui/vnc-ws.o
  CC    ui/vnc-jobs.o
  LINK  tests/qemu-iotests/socket_scm_helper
  CC    qga/commands.o
  CC    qga/guest-agent-command-state.o
  AS    optionrom/multiboot.o
  AS    optionrom/linuxboot.o
  CC    optionrom/linuxboot_dma.o
  AS    optionrom/kvmvapic.o
cc: unrecognized option '-no-integrated-as'
cc: unrecognized option '-no-integrated-as'
  CC    qga/main.o
  Building optionrom/multiboot.img
  Building optionrom/linuxboot.img
  CC    qga/commands-posix.o
  Building optionrom/linuxboot_dma.img
  Building optionrom/kvmvapic.img
  Building optionrom/multiboot.raw
  Building optionrom/linuxboot.raw
  Building optionrom/linuxboot_dma.raw
  Building optionrom/kvmvapic.raw
  Signing optionrom/multiboot.bin
  Signing optionrom/linuxboot.bin
  Signing optionrom/linuxboot_dma.bin
  CC    qga/channel-posix.o
  Signing optionrom/kvmvapic.bin
  CC    qga/qapi-generated/qga-qapi-types.o
  CC    qga/qapi-generated/qga-qapi-visit.o
  CC    qga/qapi-generated/qga-qmp-marshal.o
  CC    qmp-introspect.o
  CC    qapi-types.o
  CC    qapi-visit.o
  CC    qapi-event.o
  AR    libqemustub.a
  CC    qemu-img.o
  CC    qmp-marshal.o
  CC    trace/generated-events.o
  AR    libqemuutil.a
  LINK  qemu-ga
  LINK  ivshmem-client
  LINK  ivshmem-server
  LINK  qemu-nbd
  LINK  qemu-io
  LINK  qemu-bridge-helper
  LINK  qemu-img
  GEN   x86_64-softmmu/hmp-commands.h
  GEN   x86_64-softmmu/hmp-commands-info.h
  GEN   x86_64-softmmu/config-target.h
  CC    x86_64-softmmu/exec.o
  CC    x86_64-softmmu/translate-all.o
  CC    x86_64-softmmu/cpu-exec.o
  CC    x86_64-softmmu/translate-common.o
  CC    x86_64-softmmu/cpu-exec-common.o
  CC    x86_64-softmmu/tcg/tcg.o
  CC    x86_64-softmmu/tcg/tcg-op.o
  GEN   aarch64-softmmu/hmp-commands.h
  GEN   aarch64-softmmu/hmp-commands-info.h
  GEN   aarch64-softmmu/config-target.h
  CC    aarch64-softmmu/exec.o
  CC    x86_64-softmmu/tcg/optimize.o
  CC    x86_64-softmmu/tcg/tcg-common.o
  CC    x86_64-softmmu/fpu/softfloat.o
  CC    x86_64-softmmu/disas.o
  CC    x86_64-softmmu/arch_init.o
  CC    aarch64-softmmu/translate-all.o
  CC    x86_64-softmmu/cpus.o
  CC    x86_64-softmmu/monitor.o
  CC    x86_64-softmmu/gdbstub.o
  CC    x86_64-softmmu/balloon.o
  CC    aarch64-softmmu/cpu-exec.o
  CC    aarch64-softmmu/translate-common.o
  CC    aarch64-softmmu/cpu-exec-common.o
  CC    aarch64-softmmu/tcg/tcg.o
  CC    x86_64-softmmu/ioport.o
  CC    x86_64-softmmu/numa.o
  CC    aarch64-softmmu/tcg/tcg-op.o
  CC    x86_64-softmmu/qtest.o
  CC    aarch64-softmmu/tcg/optimize.o
  CC    aarch64-softmmu/tcg/tcg-common.o
  CC    x86_64-softmmu/bootdevice.o
  CC    x86_64-softmmu/kvm-all.o
  CC    aarch64-softmmu/fpu/softfloat.o
  CC    aarch64-softmmu/disas.o
  GEN   aarch64-softmmu/gdbstub-xml.c
  CC    x86_64-softmmu/memory.o
  CC    aarch64-softmmu/kvm-stub.o
  CC    aarch64-softmmu/arch_init.o
  CC    x86_64-softmmu/cputlb.o
  CC    aarch64-softmmu/cpus.o
  CC    x86_64-softmmu/memory_mapping.o
  CC    aarch64-softmmu/monitor.o
  CC    aarch64-softmmu/gdbstub.o
  CC    aarch64-softmmu/balloon.o
  CC    x86_64-softmmu/dump.o
  CC    aarch64-softmmu/ioport.o
  CC    aarch64-softmmu/numa.o
  CC    aarch64-softmmu/qtest.o
  CC    aarch64-softmmu/bootdevice.o
  CC    x86_64-softmmu/migration/ram.o
  CC    x86_64-softmmu/migration/savevm.o
  CC    aarch64-softmmu/memory.o
  CC    aarch64-softmmu/cputlb.o
  CC    x86_64-softmmu/xen-common-stub.o
  CC    aarch64-softmmu/memory_mapping.o
  CC    x86_64-softmmu/xen-hvm-stub.o
  CC    aarch64-softmmu/dump.o
  CC    x86_64-softmmu/hw/acpi/nvdimm.o
  CC    aarch64-softmmu/migration/ram.o
  CC    x86_64-softmmu/hw/block/virtio-blk.o
  CC    x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC    aarch64-softmmu/migration/savevm.o
  CC    x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC    x86_64-softmmu/hw/core/nmi.o
  CC    aarch64-softmmu/xen-common-stub.o
  CC    x86_64-softmmu/hw/cpu/core.o
  CC    aarch64-softmmu/xen-hvm-stub.o
  CC    aarch64-softmmu/hw/block/virtio-blk.o
/tmp/qemu-test/src/hw/block/virtio-blk.c:959: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/block/virtio-blk.c:959: warning: initialization from incompatible pointer type
  CC    x86_64-softmmu/hw/display/vga.o
  CC    aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC    x86_64-softmmu/hw/display/virtio-gpu.o
  CC    x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC    aarch64-softmmu/hw/char/exynos4210_uart.o
/tmp/qemu-test/src/hw/char/virtio-serial-bus.c:1078: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/char/virtio-serial-bus.c:1078: warning: initialization from incompatible pointer type
  CC    aarch64-softmmu/hw/char/omap_uart.o
  CC    aarch64-softmmu/hw/char/digic-uart.o
  CC    x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC    x86_64-softmmu/hw/display/virtio-vga.o
  CC    x86_64-softmmu/hw/intc/apic.o
  CC    x86_64-softmmu/hw/intc/apic_common.o
  CC    x86_64-softmmu/hw/intc/ioapic.o
/tmp/qemu-test/src/hw/block/virtio-blk.c:959: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/block/virtio-blk.c:959: warning: initialization from incompatible pointer type
  CC    aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC    aarch64-softmmu/hw/char/bcm2835_aux.o
  CC    aarch64-softmmu/hw/char/virtio-serial-bus.o
/tmp/qemu-test/src/hw/display/virtio-gpu.c:1222: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/display/virtio-gpu.c:1222: warning: initialization from incompatible pointer type
  CC    aarch64-softmmu/hw/core/nmi.o
  CC    x86_64-softmmu/hw/isa/lpc_ich9.o
  CC    x86_64-softmmu/hw/misc/vmport.o
  CC    x86_64-softmmu/hw/misc/ivshmem.o
  CC    aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC    aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC    x86_64-softmmu/hw/misc/pvpanic.o
  CC    x86_64-softmmu/hw/misc/edu.o
  CC    x86_64-softmmu/hw/misc/hyperv_testdev.o
  CC    aarch64-softmmu/hw/cpu/a9mpcore.o
  CC    aarch64-softmmu/hw/cpu/a15mpcore.o
  CC    aarch64-softmmu/hw/cpu/core.o
  CC    x86_64-softmmu/hw/net/virtio-net.o
  CC    x86_64-softmmu/hw/net/vhost_net.o
  CC    aarch64-softmmu/hw/display/omap_dss.o
/tmp/qemu-test/src/hw/char/virtio-serial-bus.c:1078: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/char/virtio-serial-bus.c:1078: warning: initialization from incompatible pointer type
  CC    x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC    x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC    x86_64-softmmu/hw/scsi/vhost-scsi.o
  CC    aarch64-softmmu/hw/display/omap_lcdc.o
  CC    x86_64-softmmu/hw/timer/mc146818rtc.o
  CC    aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC    x86_64-softmmu/hw/vfio/common.o
  CC    aarch64-softmmu/hw/display/bcm2835_fb.o
  CC    aarch64-softmmu/hw/display/vga.o
/tmp/qemu-test/src/hw/scsi/virtio-scsi.c:924: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/scsi/virtio-scsi.c:924: warning: initialization from incompatible pointer type
  CC    x86_64-softmmu/hw/vfio/pci.o
  CC    aarch64-softmmu/hw/display/virtio-gpu.o
  CC    x86_64-softmmu/hw/vfio/pci-quirks.o
/tmp/qemu-test/src/hw/net/virtio-net.c:1857: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/net/virtio-net.c:1857: warning: initialization from incompatible pointer type
  CC    aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC    aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC    aarch64-softmmu/hw/display/dpcd.o
  CC    aarch64-softmmu/hw/display/xlnx_dp.o
  CC    x86_64-softmmu/hw/vfio/platform.o
  CC    x86_64-softmmu/hw/vfio/calxeda-xgmac.o
/tmp/qemu-test/src/hw/display/virtio-gpu.c:1222: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/display/virtio-gpu.c:1222: warning: initialization from incompatible pointer type
  CC    x86_64-softmmu/hw/vfio/amd-xgbe.o
  CC    x86_64-softmmu/hw/vfio/spapr.o
  CC    aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC    aarch64-softmmu/hw/dma/omap_dma.o
  CC    aarch64-softmmu/hw/dma/soc_dma.o
  CC    aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC    aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC    x86_64-softmmu/hw/virtio/virtio.o
  CC    aarch64-softmmu/hw/gpio/omap_gpio.o
  CC    x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC    x86_64-softmmu/hw/virtio/vhost.o
  CC    aarch64-softmmu/hw/gpio/imx_gpio.o
  CC    aarch64-softmmu/hw/i2c/omap_i2c.o
  CC    aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC    x86_64-softmmu/hw/virtio/vhost-backend.o
  CC    aarch64-softmmu/hw/input/tsc210x.o
  CC    aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC    x86_64-softmmu/hw/virtio/vhost-user.o
  CC    x86_64-softmmu/hw/virtio/vhost-vsock.o
  CC    aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC    x86_64-softmmu/hw/i386/multiboot.o
/tmp/qemu-test/src/hw/virtio/virtio-balloon.c:497: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/virtio/virtio-balloon.c:497: warning: initialization from incompatible pointer type
  CC    x86_64-softmmu/hw/i386/pc.o
  CC    x86_64-softmmu/hw/i386/pc_piix.o
/tmp/qemu-test/src/hw/virtio/vhost-vsock.c:295: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/virtio/vhost-vsock.c:295: warning: initialization from incompatible pointer type
  CC    x86_64-softmmu/hw/i386/pc_q35.o
  CC    x86_64-softmmu/hw/i386/pc_sysfw.o
  CC    x86_64-softmmu/hw/i386/x86-iommu.o
  CC    aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC    aarch64-softmmu/hw/intc/omap_intc.o
  CC    aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC    x86_64-softmmu/hw/i386/intel_iommu.o
  CC    aarch64-softmmu/hw/intc/bcm2836_control.o
  CC    aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC    x86_64-softmmu/hw/i386/amd_iommu.o
/tmp/qemu-test/src/hw/i386/pc_piix.c: In function ‘igd_passthrough_isa_bridge_create’:
/tmp/qemu-test/src/hw/i386/pc_piix.c:1046: warning: ‘pch_rev_id’ may be used uninitialized in this function
  CC    x86_64-softmmu/hw/i386/kvmvapic.o
  CC    aarch64-softmmu/hw/intc/aspeed_vic.o
  CC    x86_64-softmmu/hw/i386/acpi-build.o
  CC    x86_64-softmmu/hw/i386/pci-assign-load-rom.o
  CC    aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC    aarch64-softmmu/hw/misc/ivshmem.o
  CC    x86_64-softmmu/hw/i386/kvm/clock.o
  CC    aarch64-softmmu/hw/misc/arm_sysctl.o
  CC    x86_64-softmmu/hw/i386/kvm/apic.o
  CC    aarch64-softmmu/hw/misc/cbus.o
  CC    aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC    aarch64-softmmu/hw/misc/imx_ccm.o
  CC    x86_64-softmmu/hw/i386/kvm/i8259.o
  CC    aarch64-softmmu/hw/misc/imx31_ccm.o
  CC    aarch64-softmmu/hw/misc/imx25_ccm.o
/tmp/qemu-test/src/hw/i386/acpi-build.c: In function ‘build_append_pci_bus_devices’:
/tmp/qemu-test/src/hw/i386/acpi-build.c:472: warning: ‘notify_method’ may be used uninitialized in this function
  CC    aarch64-softmmu/hw/misc/imx6_ccm.o
  CC    x86_64-softmmu/hw/i386/kvm/ioapic.o
  CC    aarch64-softmmu/hw/misc/imx6_src.o
  CC    x86_64-softmmu/hw/i386/kvm/i8254.o
  CC    x86_64-softmmu/hw/i386/kvm/pci-assign.o
  CC    x86_64-softmmu/target-i386/translate.o
  CC    aarch64-softmmu/hw/misc/mst_fpga.o
  CC    x86_64-softmmu/target-i386/helper.o
  CC    x86_64-softmmu/target-i386/cpu.o
  CC    x86_64-softmmu/target-i386/bpt_helper.o
  CC    aarch64-softmmu/hw/misc/omap_clk.o
  CC    aarch64-softmmu/hw/misc/omap_gpmc.o
  CC    x86_64-softmmu/target-i386/excp_helper.o
  CC    aarch64-softmmu/hw/misc/omap_l4.o
  CC    aarch64-softmmu/hw/misc/omap_sdrc.o
  CC    aarch64-softmmu/hw/misc/omap_tap.o
  CC    aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC    x86_64-softmmu/target-i386/fpu_helper.o
  CC    aarch64-softmmu/hw/misc/bcm2835_property.o
  CC    aarch64-softmmu/hw/misc/zynq_slcr.o
  CC    x86_64-softmmu/target-i386/cc_helper.o
  CC    x86_64-softmmu/target-i386/int_helper.o
  CC    x86_64-softmmu/target-i386/svm_helper.o
  CC    x86_64-softmmu/target-i386/smm_helper.o
  CC    aarch64-softmmu/hw/misc/zynq-xadc.o
  CC    aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC    aarch64-softmmu/hw/misc/edu.o
  CC    x86_64-softmmu/target-i386/misc_helper.o
  CC    x86_64-softmmu/target-i386/mem_helper.o
  CC    x86_64-softmmu/target-i386/seg_helper.o
  CC    x86_64-softmmu/target-i386/mpx_helper.o
  CC    aarch64-softmmu/hw/misc/auxbus.o
  CC    aarch64-softmmu/hw/misc/aspeed_scu.o
  CC    aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC    aarch64-softmmu/hw/net/virtio-net.o
  CC    x86_64-softmmu/target-i386/gdbstub.o
  CC    aarch64-softmmu/hw/net/vhost_net.o
  CC    aarch64-softmmu/hw/pcmcia/pxa2xx.o
/tmp/qemu-test/src/hw/net/virtio-net.c:1857: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/net/virtio-net.c:1857: warning: initialization from incompatible pointer type
  CC    x86_64-softmmu/target-i386/machine.o
  CC    aarch64-softmmu/hw/scsi/virtio-scsi.o
  CC    x86_64-softmmu/target-i386/arch_memory_mapping.o
  CC    aarch64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC    aarch64-softmmu/hw/scsi/vhost-scsi.o
  CC    x86_64-softmmu/target-i386/arch_dump.o
  CC    x86_64-softmmu/target-i386/monitor.o
  CC    aarch64-softmmu/hw/sd/omap_mmc.o
  CC    x86_64-softmmu/target-i386/kvm.o
/tmp/qemu-test/src/hw/scsi/virtio-scsi.c:924: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/scsi/virtio-scsi.c:924: warning: initialization from incompatible pointer type
  CC    aarch64-softmmu/hw/sd/pxa2xx_mmci.o
  CC    aarch64-softmmu/hw/ssi/omap_spi.o
  CC    aarch64-softmmu/hw/ssi/imx_spi.o
  CC    aarch64-softmmu/hw/timer/exynos4210_mct.o
  CC    x86_64-softmmu/target-i386/hyperv.o
  CC    aarch64-softmmu/hw/timer/exynos4210_pwm.o
  CC    aarch64-softmmu/hw/timer/exynos4210_rtc.o
  GEN   trace/generated-helpers.c
  CC    x86_64-softmmu/trace/control-target.o
  CC    aarch64-softmmu/hw/timer/omap_gptimer.o
  CC    aarch64-softmmu/hw/timer/omap_synctimer.o
  CC    aarch64-softmmu/hw/timer/pxa2xx_timer.o
  CC    aarch64-softmmu/hw/timer/digic-timer.o
  CC    x86_64-softmmu/trace/generated-helpers.o
  CC    aarch64-softmmu/hw/timer/allwinner-a10-pit.o
  CC    aarch64-softmmu/hw/usb/tusb6010.o
  CC    aarch64-softmmu/hw/vfio/common.o
  CC    aarch64-softmmu/hw/vfio/pci.o
  CC    aarch64-softmmu/hw/vfio/pci-quirks.o
  CC    aarch64-softmmu/hw/vfio/platform.o
  CC    aarch64-softmmu/hw/vfio/calxeda-xgmac.o
  CC    aarch64-softmmu/hw/vfio/amd-xgbe.o
  LINK  x86_64-softmmu/qemu-system-x86_64
  CC    aarch64-softmmu/hw/vfio/spapr.o
  CC    aarch64-softmmu/hw/virtio/virtio.o
  CC    aarch64-softmmu/hw/virtio/virtio-balloon.o
  CC    aarch64-softmmu/hw/virtio/vhost.o
  CC    aarch64-softmmu/hw/virtio/vhost-backend.o
  CC    aarch64-softmmu/hw/virtio/vhost-user.o
  CC    aarch64-softmmu/hw/virtio/vhost-vsock.o
  CC    aarch64-softmmu/hw/arm/boot.o
  CC    aarch64-softmmu/hw/arm/collie.o
  CC    aarch64-softmmu/hw/arm/exynos4_boards.o
/tmp/qemu-test/src/hw/virtio/virtio-balloon.c:497: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/virtio/virtio-balloon.c:497: warning: initialization from incompatible pointer type
  CC    aarch64-softmmu/hw/arm/gumstix.o
/tmp/qemu-test/src/hw/virtio/vhost-vsock.c:295: warning: initialization from incompatible pointer type
/tmp/qemu-test/src/hw/virtio/vhost-vsock.c:295: warning: initialization from incompatible pointer type
  CC    aarch64-softmmu/hw/arm/highbank.o
  CC    aarch64-softmmu/hw/arm/digic_boards.o
  CC    aarch64-softmmu/hw/arm/integratorcp.o
  CC    aarch64-softmmu/hw/arm/mainstone.o
  CC    aarch64-softmmu/hw/arm/musicpal.o
  CC    aarch64-softmmu/hw/arm/nseries.o
  CC    aarch64-softmmu/hw/arm/omap_sx1.o
  CC    aarch64-softmmu/hw/arm/palm.o
  CC    aarch64-softmmu/hw/arm/realview.o
  CC    aarch64-softmmu/hw/arm/spitz.o
  CC    aarch64-softmmu/hw/arm/stellaris.o
  CC    aarch64-softmmu/hw/arm/tosa.o
  CC    aarch64-softmmu/hw/arm/versatilepb.o
  CC    aarch64-softmmu/hw/arm/vexpress.o
  CC    aarch64-softmmu/hw/arm/virt.o
  CC    aarch64-softmmu/hw/arm/xilinx_zynq.o
  CC    aarch64-softmmu/hw/arm/z2.o
  CC    aarch64-softmmu/hw/arm/virt-acpi-build.o
  CC    aarch64-softmmu/hw/arm/netduino2.o
  CC    aarch64-softmmu/hw/arm/sysbus-fdt.o
  CC    aarch64-softmmu/hw/arm/armv7m.o
  CC    aarch64-softmmu/hw/arm/exynos4210.o
  CC    aarch64-softmmu/hw/arm/pxa2xx.o
  CC    aarch64-softmmu/hw/arm/pxa2xx_gpio.o
  CC    aarch64-softmmu/hw/arm/pxa2xx_pic.o
  CC    aarch64-softmmu/hw/arm/digic.o
  CC    aarch64-softmmu/hw/arm/omap1.o
  CC    aarch64-softmmu/hw/arm/omap2.o
  CC    aarch64-softmmu/hw/arm/strongarm.o
  CC    aarch64-softmmu/hw/arm/allwinner-a10.o
  CC    aarch64-softmmu/hw/arm/cubieboard.o
  CC    aarch64-softmmu/hw/arm/bcm2835_peripherals.o
  CC    aarch64-softmmu/hw/arm/bcm2836.o
  CC    aarch64-softmmu/hw/arm/raspi.o
  CC    aarch64-softmmu/hw/arm/stm32f205_soc.o
  CC    aarch64-softmmu/hw/arm/xlnx-zynqmp.o
  CC    aarch64-softmmu/hw/arm/xlnx-ep108.o
  CC    aarch64-softmmu/hw/arm/fsl-imx25.o
  CC    aarch64-softmmu/hw/arm/imx25_pdk.o
  CC    aarch64-softmmu/hw/arm/fsl-imx31.o
  CC    aarch64-softmmu/hw/arm/kzm.o
  CC    aarch64-softmmu/hw/arm/fsl-imx6.o
  CC    aarch64-softmmu/hw/arm/sabrelite.o
  CC    aarch64-softmmu/hw/arm/aspeed_soc.o
  CC    aarch64-softmmu/hw/arm/aspeed.o
  CC    aarch64-softmmu/target-arm/arm-semi.o
  CC    aarch64-softmmu/target-arm/machine.o
  CC    aarch64-softmmu/target-arm/psci.o
  CC    aarch64-softmmu/target-arm/arch_dump.o
  CC    aarch64-softmmu/target-arm/monitor.o
  CC    aarch64-softmmu/target-arm/kvm-stub.o
  CC    aarch64-softmmu/target-arm/translate.o
  CC    aarch64-softmmu/target-arm/op_helper.o
  CC    aarch64-softmmu/target-arm/helper.o
  CC    aarch64-softmmu/target-arm/cpu.o
  CC    aarch64-softmmu/target-arm/neon_helper.o
  CC    aarch64-softmmu/target-arm/iwmmxt_helper.o
  CC    aarch64-softmmu/target-arm/gdbstub.o
  CC    aarch64-softmmu/target-arm/cpu64.o
  CC    aarch64-softmmu/target-arm/translate-a64.o
  CC    aarch64-softmmu/target-arm/helper-a64.o
  CC    aarch64-softmmu/target-arm/gdbstub64.o
  CC    aarch64-softmmu/target-arm/crypto_helper.o
  CC    aarch64-softmmu/target-arm/arm-powerctl.o
  GEN   trace/generated-helpers.c
  CC    aarch64-softmmu/trace/control-target.o
  CC    aarch64-softmmu/gdbstub-xml.o
  CC    aarch64-softmmu/trace/generated-helpers.o
/tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘handle_shri_with_rndacc’:
/tmp/qemu-test/src/target-arm/translate-a64.c:6333: warning: ‘tcg_src_hi’ may be used uninitialized in this function
/tmp/qemu-test/src/target-arm/translate-a64.c: In function ‘disas_simd_scalar_two_reg_misc’:
/tmp/qemu-test/src/target-arm/translate-a64.c:8060: warning: ‘rmode’ may be used uninitialized in this function
  LINK  aarch64-softmmu/qemu-system-aarch64
  TEST  tests/qapi-schema/alternate-any.out
  TEST  tests/qapi-schema/alternate-array.out
  TEST  tests/qapi-schema/alternate-base.out
  TEST  tests/qapi-schema/alternate-clash.out
  TEST  tests/qapi-schema/alternate-conflict-dict.out
  TEST  tests/qapi-schema/alternate-conflict-string.out
  TEST  tests/qapi-schema/alternate-empty.out
  TEST  tests/qapi-schema/alternate-nested.out
  TEST  tests/qapi-schema/alternate-unknown.out
  TEST  tests/qapi-schema/args-alternate.out
  TEST  tests/qapi-schema/args-any.out
  TEST  tests/qapi-schema/args-array-empty.out
  TEST  tests/qapi-schema/args-array-unknown.out
  TEST  tests/qapi-schema/args-bad-boxed.out
  TEST  tests/qapi-schema/args-boxed-anon.out
  TEST  tests/qapi-schema/args-boxed-empty.out
  TEST  tests/qapi-schema/args-boxed-string.out
  TEST  tests/qapi-schema/args-int.out
  TEST  tests/qapi-schema/args-invalid.out
  TEST  tests/qapi-schema/args-member-array-bad.out
  TEST  tests/qapi-schema/args-member-case.out
  TEST  tests/qapi-schema/args-member-unknown.out
  TEST  tests/qapi-schema/args-name-clash.out
  TEST  tests/qapi-schema/args-union.out
  TEST  tests/qapi-schema/args-unknown.out
  TEST  tests/qapi-schema/bad-base.out
  TEST  tests/qapi-schema/bad-data.out
  TEST  tests/qapi-schema/bad-ident.out
  TEST  tests/qapi-schema/bad-type-bool.out
  TEST  tests/qapi-schema/bad-type-dict.out
  TEST  tests/qapi-schema/bad-type-int.out
  TEST  tests/qapi-schema/base-cycle-direct.out
  TEST  tests/qapi-schema/base-cycle-indirect.out
  TEST  tests/qapi-schema/command-int.out
  TEST  tests/qapi-schema/comments.out
  TEST  tests/qapi-schema/double-data.out
  TEST  tests/qapi-schema/double-type.out
  TEST  tests/qapi-schema/duplicate-key.out
  TEST  tests/qapi-schema/empty.out
  TEST  tests/qapi-schema/enum-bad-name.out
  TEST  tests/qapi-schema/enum-bad-prefix.out
  TEST  tests/qapi-schema/enum-clash-member.out
  TEST  tests/qapi-schema/enum-dict-member.out
  TEST  tests/qapi-schema/enum-int-member.out
  TEST  tests/qapi-schema/enum-member-case.out
  TEST  tests/qapi-schema/enum-missing-data.out
  TEST  tests/qapi-schema/enum-wrong-data.out
  TEST  tests/qapi-schema/escape-outside-string.out
  TEST  tests/qapi-schema/escape-too-big.out
  TEST  tests/qapi-schema/escape-too-short.out
  TEST  tests/qapi-schema/event-boxed-empty.out
  TEST  tests/qapi-schema/event-case.out
  TEST  tests/qapi-schema/event-nest-struct.out
  TEST  tests/qapi-schema/flat-union-array-branch.out
  TEST  tests/qapi-schema/flat-union-bad-base.out
  TEST  tests/qapi-schema/flat-union-bad-discriminator.out
  TEST  tests/qapi-schema/flat-union-base-any.out
  TEST  tests/qapi-schema/flat-union-base-union.out
  TEST  tests/qapi-schema/flat-union-clash-member.out
  TEST  tests/qapi-schema/flat-union-empty.out
  TEST  tests/qapi-schema/flat-union-incomplete-branch.out
  TEST  tests/qapi-schema/flat-union-inline.out
  TEST  tests/qapi-schema/flat-union-int-branch.out
  TEST  tests/qapi-schema/flat-union-invalid-branch-key.out
  TEST  tests/qapi-schema/flat-union-invalid-discriminator.out
  TEST  tests/qapi-schema/flat-union-no-base.out
  TEST  tests/qapi-schema/flat-union-optional-discriminator.out
  TEST  tests/qapi-schema/flat-union-string-discriminator.out
  TEST  tests/qapi-schema/funny-char.out
  TEST  tests/qapi-schema/ident-with-escape.out
  TEST  tests/qapi-schema/include-before-err.out
  TEST  tests/qapi-schema/include-cycle.out
  TEST  tests/qapi-schema/include-format-err.out
  TEST  tests/qapi-schema/include-nested-err.out
  TEST  tests/qapi-schema/include-no-file.out
  TEST  tests/qapi-schema/include-non-file.out
  TEST  tests/qapi-schema/include-relpath.out
  TEST  tests/qapi-schema/include-repetition.out
  TEST  tests/qapi-schema/include-self-cycle.out
  TEST  tests/qapi-schema/include-simple.out
  TEST  tests/qapi-schema/indented-expr.out
  TEST  tests/qapi-schema/leading-comma-list.out
  TEST  tests/qapi-schema/leading-comma-object.out
  TEST  tests/qapi-schema/missing-colon.out
  TEST  tests/qapi-schema/missing-comma-list.out
  TEST  tests/qapi-schema/missing-comma-object.out
  TEST  tests/qapi-schema/missing-type.out
  TEST  tests/qapi-schema/nested-struct-data.out
  TEST  tests/qapi-schema/non-objects.out
  TEST  tests/qapi-schema/qapi-schema-test.out
  TEST  tests/qapi-schema/quoted-structural-chars.out
  TEST  tests/qapi-schema/redefined-builtin.out
  TEST  tests/qapi-schema/redefined-command.out
  TEST  tests/qapi-schema/redefined-event.out
  TEST  tests/qapi-schema/redefined-type.out
  TEST  tests/qapi-schema/reserved-command-q.out
  TEST  tests/qapi-schema/reserved-enum-q.out
  TEST  tests/qapi-schema/reserved-member-has.out
  TEST  tests/qapi-schema/reserved-member-q.out
  TEST  tests/qapi-schema/reserved-member-u.out
  TEST  tests/qapi-schema/reserved-member-underscore.out
  TEST  tests/qapi-schema/reserved-type-kind.out
  TEST  tests/qapi-schema/reserved-type-list.out
  TEST  tests/qapi-schema/returns-alternate.out
  TEST  tests/qapi-schema/returns-array-bad.out
  TEST  tests/qapi-schema/returns-dict.out
  TEST  tests/qapi-schema/returns-unknown.out
  TEST  tests/qapi-schema/returns-whitelist.out
  TEST  tests/qapi-schema/struct-base-clash-deep.out
  TEST  tests/qapi-schema/struct-base-clash.out
  TEST  tests/qapi-schema/struct-data-invalid.out
  TEST  tests/qapi-schema/struct-member-invalid.out
  TEST  tests/qapi-schema/trailing-comma-list.out
  TEST  tests/qapi-schema/trailing-comma-object.out
  TEST  tests/qapi-schema/type-bypass-bad-gen.out
  TEST  tests/qapi-schema/unclosed-list.out
  TEST  tests/qapi-schema/unclosed-object.out
  TEST  tests/qapi-schema/unclosed-string.out
  TEST  tests/qapi-schema/unicode-str.out
  TEST  tests/qapi-schema/union-base-no-discriminator.out
  TEST  tests/qapi-schema/union-branch-case.out
  TEST  tests/qapi-schema/union-clash-branches.out
  TEST  tests/qapi-schema/union-empty.out
  TEST  tests/qapi-schema/union-invalid-base.out
  TEST  tests/qapi-schema/union-optional-branch.out
  TEST  tests/qapi-schema/union-unknown.out
  TEST  tests/qapi-schema/unknown-escape.out
  TEST  tests/qapi-schema/unknown-expr-key.out
  CC    tests/check-qdict.o
  CC    tests/check-qfloat.o
  CC    tests/check-qint.o
  CC    tests/check-qstring.o
  CC    tests/check-qlist.o
  CC    tests/check-qnull.o
  CC    tests/check-qjson.o
  CC    tests/test-qmp-output-visitor.o
  GEN   tests/test-qapi-visit.c
  GEN   tests/test-qapi-types.c
  GEN   tests/test-qapi-event.c
  GEN   tests/test-qmp-introspect.c
  CC    tests/test-clone-visitor.o
  CC    tests/test-qmp-input-visitor.o
  CC    tests/test-qmp-input-strict.o
  CC    tests/test-qmp-commands.o
  GEN   tests/test-qmp-marshal.c
  CC    tests/test-string-input-visitor.o
  CC    tests/test-string-output-visitor.o
  CC    tests/test-qmp-event.o
  CC    tests/test-opts-visitor.o
  CC    tests/test-coroutine.o
  CC    tests/test-visitor-serialization.o
  CC    tests/test-iov.o
  CC    tests/test-aio.o
  CC    tests/test-rfifolock.o
  CC    tests/test-throttle.o
  CC    tests/test-thread-pool.o
  CC    tests/test-hbitmap.o
  CC    tests/test-blockjob.o
  CC    tests/test-blockjob-txn.o
  CC    tests/test-x86-cpuid.o
  CC    tests/test-xbzrle.o
  CC    tests/test-vmstate.o
  CC    tests/test-cutils.o
  CC    tests/test-mul64.o
  CC    tests/test-int128.o
  CC    tests/rcutorture.o
  CC    tests/test-rcu-list.o
  CC    tests/test-qdist.o
  CC    tests/test-qht.o
  CC    tests/test-qht-par.o
/tmp/qemu-test/src/tests/test-int128.c:180: warning: ‘__noclone__’ attribute directive ignored
  CC    tests/qht-bench.o
  CC    tests/test-bitops.o
  CC    tests/check-qom-interface.o
  CC    tests/check-qom-proplist.o
  CC    tests/test-qemu-opts.o
  CC    tests/test-write-threshold.o
  CC    tests/test-crypto-hash.o
  CC    tests/test-crypto-cipher.o
  CC    tests/test-crypto-secret.o
  CC    tests/test-qga.o
  CC    tests/test-timed-average.o
  CC    tests/libqtest.o
  CC    tests/test-io-task.o
  CC    tests/test-io-channel-socket.o
  CC    tests/io-channel-helpers.o
  CC    tests/test-io-channel-file.o
  CC    tests/test-io-channel-command.o
  CC    tests/test-io-channel-buffer.o
  CC    tests/test-base64.o
  CC    tests/test-crypto-ivgen.o
  CC    tests/test-crypto-afsplit.o
  CC    tests/test-crypto-xts.o
  CC    tests/test-crypto-block.o
  CC    tests/test-logging.o
  CC    tests/test-replication.o
  CC    tests/test-bufferiszero.o
  CC    tests/test-uuid.o
  CC    tests/vhost-user-test.o
  CC    tests/libqos/pci.o
  CC    tests/libqos/fw_cfg.o
  CC    tests/libqos/malloc.o
  CC    tests/libqos/i2c.o
  CC    tests/libqos/libqos.o
  CC    tests/libqos/pci-pc.o
  CC    tests/libqos/malloc-pc.o
  CC    tests/libqos/libqos-pc.o
  CC    tests/libqos/ahci.o
  CC    tests/libqos/virtio.o
  CC    tests/libqos/virtio-pci.o
  CC    tests/libqos/virtio-mmio.o
  CC    tests/libqos/malloc-generic.o
  CC    tests/endianness-test.o
  CC    tests/fdc-test.o
  CC    tests/ide-test.o
  CC    tests/ahci-test.o
  CC    tests/hd-geo-test.o
  CC    tests/boot-order-test.o
  CC    tests/bios-tables-test.o
  CC    tests/boot-sector.o
/tmp/qemu-test/src/tests/ide-test.c: In function ‘cdrom_pio_impl’:
/tmp/qemu-test/src/tests/ide-test.c:739: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
/tmp/qemu-test/src/tests/ide-test.c: In function ‘test_cdrom_dma’:
/tmp/qemu-test/src/tests/ide-test.c:832: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
  CC    tests/boot-serial-test.o
  CC    tests/pxe-test.o
  CC    tests/rtc-test.o
  CC    tests/ipmi-kcs-test.o
  CC    tests/ipmi-bt-test.o
  CC    tests/i440fx-test.o
/tmp/qemu-test/src/tests/boot-sector.c: In function ‘boot_sector_init’:
/tmp/qemu-test/src/tests/boot-sector.c:80: warning: ignoring return value of ‘fwrite’, declared with attribute warn_unused_result
  CC    tests/fw_cfg-test.o
  CC    tests/drive_del-test.o
  CC    tests/wdt_ib700-test.o
  CC    tests/tco-test.o
  CC    tests/e1000-test.o
  CC    tests/e1000e-test.o
  CC    tests/rtl8139-test.o
  CC    tests/pcnet-test.o
  CC    tests/eepro100-test.o
  CC    tests/ne2000-test.o
  CC    tests/nvme-test.o
  CC    tests/ac97-test.o
  CC    tests/es1370-test.o
  CC    tests/virtio-net-test.o
  CC    tests/virtio-balloon-test.o
  CC    tests/virtio-blk-test.o
  CC    tests/virtio-rng-test.o
  CC    tests/virtio-scsi-test.o
  CC    tests/virtio-serial-test.o
  CC    tests/virtio-console-test.o
  CC    tests/tpci200-test.o
  CC    tests/ipoctal232-test.o
  CC    tests/display-vga-test.o
  CC    tests/intel-hda-test.o
  CC    tests/ivshmem-test.o
  CC    tests/vmxnet3-test.o
  CC    tests/pvpanic-test.o
  CC    tests/i82801b11-test.o
  CC    tests/ioh3420-test.o
  CC    tests/usb-hcd-ohci-test.o
  CC    tests/libqos/usb.o
  CC    tests/usb-hcd-uhci-test.o
  CC    tests/usb-hcd-ehci-test.o
  CC    tests/usb-hcd-xhci-test.o
  CC    tests/pc-cpu-test.o
  CC    tests/q35-test.o
  CC    tests/test-netfilter.o
  CC    tests/test-filter-mirror.o
  CC    tests/test-filter-redirector.o
  CC    tests/postcopy-test.o
  CC    tests/test-x86-cpuid-compat.o
  CC    tests/device-introspect-test.o
  CC    tests/qom-test.o
  CC    tests/ptimer-test.o
  CC    tests/ptimer-test-stubs.o
  LINK  tests/check-qdict
  LINK  tests/check-qfloat
  LINK  tests/check-qint
  LINK  tests/check-qstring
  LINK  tests/check-qlist
  LINK  tests/check-qnull
  LINK  tests/check-qjson
  CC    tests/test-qapi-visit.o
  CC    tests/test-qapi-types.o
  CC    tests/test-qapi-event.o
  CC    tests/test-qmp-introspect.o
  CC    tests/test-qmp-marshal.o
  LINK  tests/test-coroutine
  LINK  tests/test-iov
  LINK  tests/test-aio
  LINK  tests/test-rfifolock
  LINK  tests/test-throttle
  LINK  tests/test-thread-pool
  LINK  tests/test-hbitmap
  LINK  tests/test-blockjob
  LINK  tests/test-blockjob-txn
  LINK  tests/test-x86-cpuid
  LINK  tests/test-xbzrle
  LINK  tests/test-vmstate
  LINK  tests/test-cutils
  LINK  tests/test-mul64
  LINK  tests/test-int128
  LINK  tests/rcutorture
  LINK  tests/test-rcu-list
  LINK  tests/test-qdist
  LINK  tests/test-qht
  LINK  tests/qht-bench
  LINK  tests/test-bitops
  LINK  tests/check-qom-interface
  LINK  tests/check-qom-proplist
  LINK  tests/test-qemu-opts
  LINK  tests/test-write-threshold
  LINK  tests/test-crypto-hash
  LINK  tests/test-crypto-cipher
  LINK  tests/test-crypto-secret
  LINK  tests/test-qga
  LINK  tests/test-timed-average
  LINK  tests/test-io-task
  LINK  tests/test-io-channel-socket
  LINK  tests/test-io-channel-file
  LINK  tests/test-io-channel-command
  LINK  tests/test-io-channel-buffer
  LINK  tests/test-base64
  LINK  tests/test-crypto-ivgen
  LINK  tests/test-crypto-afsplit
  LINK  tests/test-crypto-xts
  LINK  tests/test-crypto-block
  LINK  tests/test-logging
  LINK  tests/test-replication
  LINK  tests/test-bufferiszero
  LINK  tests/test-uuid
  LINK  tests/vhost-user-test
  LINK  tests/endianness-test
  LINK  tests/fdc-test
  LINK  tests/ide-test
  LINK  tests/ahci-test
  LINK  tests/hd-geo-test
  LINK  tests/boot-order-test
  LINK  tests/bios-tables-test
  LINK  tests/boot-serial-test
  LINK  tests/pxe-test
  LINK  tests/rtc-test
  LINK  tests/ipmi-kcs-test
  LINK  tests/ipmi-bt-test
  LINK  tests/i440fx-test
  LINK  tests/fw_cfg-test
  LINK  tests/drive_del-test
  LINK  tests/wdt_ib700-test
  LINK  tests/tco-test
  LINK  tests/e1000-test
  LINK  tests/e1000e-test
  LINK  tests/rtl8139-test
  LINK  tests/pcnet-test
  LINK  tests/eepro100-test
  LINK  tests/ne2000-test
  LINK  tests/nvme-test
  LINK  tests/ac97-test
  LINK  tests/es1370-test
  LINK  tests/virtio-net-test
  LINK  tests/virtio-balloon-test
  LINK  tests/virtio-blk-test
  LINK  tests/virtio-rng-test
  LINK  tests/virtio-scsi-test
  LINK  tests/virtio-serial-test
  LINK  tests/virtio-console-test
  LINK  tests/tpci200-test
  LINK  tests/ipoctal232-test
  LINK  tests/display-vga-test
  LINK  tests/intel-hda-test
  LINK  tests/ivshmem-test
  LINK  tests/vmxnet3-test
  LINK  tests/pvpanic-test
  LINK  tests/i82801b11-test
  LINK  tests/ioh3420-test
  LINK  tests/usb-hcd-ohci-test
  LINK  tests/usb-hcd-uhci-test
  LINK  tests/usb-hcd-ehci-test
  LINK  tests/usb-hcd-xhci-test
  LINK  tests/pc-cpu-test
  LINK  tests/q35-test
  LINK  tests/test-netfilter
  LINK  tests/test-filter-mirror
  LINK  tests/test-filter-redirector
  LINK  tests/postcopy-test
  LINK  tests/test-x86-cpuid-compat
  LINK  tests/device-introspect-test
  LINK  tests/qom-test
  LINK  tests/ptimer-test
GTESTER tests/check-qdict
GTESTER tests/check-qfloat
GTESTER tests/check-qint
GTESTER tests/check-qstring
GTESTER tests/check-qlist
GTESTER tests/check-qnull
GTESTER tests/check-qjson
  LINK  tests/test-qmp-output-visitor
  LINK  tests/test-clone-visitor
  LINK  tests/test-qmp-input-visitor
  LINK  tests/test-qmp-input-strict
  LINK  tests/test-qmp-commands
  LINK  tests/test-string-input-visitor
  LINK  tests/test-string-output-visitor
  LINK  tests/test-qmp-event
  LINK  tests/test-opts-visitor
GTESTER tests/test-coroutine
  LINK  tests/test-visitor-serialization
GTESTER tests/test-iov
GTESTER tests/test-aio
GTESTER tests/test-rfifolock
GTESTER tests/test-throttle
GTESTER tests/test-thread-pool
GTESTER tests/test-hbitmap
GTESTER tests/test-blockjob
GTESTER tests/test-blockjob-txn
GTESTER tests/test-x86-cpuid
GTESTER tests/test-xbzrle
GTESTER tests/test-vmstate
GTESTER tests/test-cutils
GTESTER tests/test-mul64
GTESTER tests/test-int128
GTESTER tests/rcutorture
GTESTER tests/test-rcu-list
GTESTER tests/test-qdist
GTESTER tests/test-qht
  LINK  tests/test-qht-par
GTESTER tests/test-bitops
GTESTER tests/check-qom-interface
GTESTER tests/check-qom-proplist
GTESTER tests/test-qemu-opts
GTESTER tests/test-write-threshold
GTESTER tests/test-crypto-hash
GTESTER tests/test-crypto-cipher
GTESTER tests/test-crypto-secret
GTESTER tests/test-qga
GTESTER tests/test-timed-average
GTESTER tests/test-io-task
GTESTER tests/test-io-channel-socket
GTESTER tests/test-io-channel-file
GTESTER tests/test-io-channel-command
GTESTER tests/test-io-channel-buffer
GTESTER tests/test-base64
GTESTER tests/test-crypto-ivgen
GTESTER tests/test-crypto-afsplit
GTESTER tests/test-crypto-xts
GTESTER tests/test-crypto-block
GTESTER tests/test-logging
GTESTER tests/test-replication
GTESTER tests/test-bufferiszero
GTESTER tests/test-uuid
GTESTER check-qtest-x86_64
GTESTER check-qtest-aarch64
GTESTER tests/test-qmp-output-visitor
GTESTER tests/test-clone-visitor
GTESTER tests/test-qmp-input-visitor
GTESTER tests/test-qmp-input-strict
GTESTER tests/test-qmp-commands
GTESTER tests/test-string-input-visitor
GTESTER tests/test-string-output-visitor
GTESTER tests/test-qmp-event
GTESTER tests/test-opts-visitor
GTESTER tests/test-visitor-serialization
GTESTER tests/test-qht-par
qemu-system-x86_64: Failed to unlink socket .: Is a directory
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
  BUILD fedora
  ARCHIVE qemu.tgz
  ARCHIVE dtc.tgz
  COPY RUNNER
  RUN test-mingw in fedora
Packages installed:
PyYAML-3.11-12.fc24.x86_64
SDL-devel-1.2.15-21.fc24.x86_64
bc-1.06.95-16.fc24.x86_64
bison-3.0.4-4.fc24.x86_64
ccache-3.3.1-1.fc24.x86_64
clang-3.8.0-2.fc24.x86_64
findutils-4.6.0-7.fc24.x86_64
flex-2.6.0-2.fc24.x86_64
gcc-6.1.1-3.fc24.x86_64
gcc-c++-6.1.1-3.fc24.x86_64
git-2.7.4-2.fc24.x86_64
glib2-devel-2.48.2-1.fc24.x86_64
libfdt-devel-1.4.2-1.fc24.x86_64
make-4.1-5.fc24.x86_64
mingw32-SDL-1.2.15-7.fc24.noarch
mingw32-bzip2-1.0.6-7.fc24.noarch
mingw32-curl-7.47.0-1.fc24.noarch
mingw32-glib2-2.48.2-1.fc24.noarch
mingw32-gmp-6.1.0-1.fc24.noarch
mingw32-gnutls-3.4.14-1.fc24.noarch
mingw32-gtk2-2.24.30-1.fc24.noarch
mingw32-gtk3-3.20.9-1.fc24.noarch
mingw32-libjpeg-turbo-1.4.2-1.fc24.noarch
mingw32-libpng-1.6.23-1.fc24.noarch
mingw32-libssh2-1.4.3-5.fc24.noarch
mingw32-libtasn1-4.5-2.fc24.noarch
mingw32-nettle-3.2-1.fc24.noarch
mingw32-pixman-0.34.0-1.fc24.noarch
mingw32-pkg-config-0.28-6.fc24.x86_64
mingw64-SDL-1.2.15-7.fc24.noarch
mingw64-bzip2-1.0.6-7.fc24.noarch
mingw64-curl-7.47.0-1.fc24.noarch
mingw64-glib2-2.48.2-1.fc24.noarch
mingw64-gmp-6.1.0-1.fc24.noarch
mingw64-gnutls-3.4.14-1.fc24.noarch
mingw64-gtk2-2.24.30-1.fc24.noarch
mingw64-gtk3-3.20.9-1.fc24.noarch
mingw64-libjpeg-turbo-1.4.2-1.fc24.noarch
mingw64-libpng-1.6.23-1.fc24.noarch
mingw64-libssh2-1.4.3-5.fc24.noarch
mingw64-libtasn1-4.5-2.fc24.noarch
mingw64-nettle-3.2-1.fc24.noarch
mingw64-pixman-0.34.0-1.fc24.noarch
mingw64-pkg-config-0.28-6.fc24.x86_64
perl-5.22.2-362.fc24.x86_64
pixman-devel-0.34.0-2.fc24.x86_64
sparse-0.5.0-7.fc24.x86_64
tar-1.28-7.fc24.x86_64
which-2.20-13.fc24.x86_64
zlib-devel-1.2.8-10.fc24.x86_64

Environment variables:
PACKAGES=ccache git tar PyYAML sparse flex bison     glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel     gcc gcc-c++ clang make perl which bc findutils     mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config     mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1     mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2     mingw32-bzip2     mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config     mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1     mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2     mingw64-bzip2
HOSTNAME=
TERM=xterm
MAKEFLAGS= -j8
HISTSIZE=1000
J=8
USER=root
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
MAIL=/var/spool/mail/root
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
FEATURES=mingw clang pyyaml dtc
DEBUG=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/src/tests/docker/install --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-debug --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0
Install prefix    /tmp/qemu-test/src/tests/docker/install
BIOS directory    /tmp/qemu-test/src/tests/docker/install
binary directory  /tmp/qemu-test/src/tests/docker/install
library directory /tmp/qemu-test/src/tests/docker/install/lib
module directory  /tmp/qemu-test/src/tests/docker/install/lib
libexec directory /tmp/qemu-test/src/tests/docker/install/libexec
include directory /tmp/qemu-test/src/tests/docker/install/include
config directory  /tmp/qemu-test/src/tests/docker/install
local state directory   queried at runtime
Windows SDK       no
Source path       /tmp/qemu-test/src
C compiler        x86_64-w64-mingw32-gcc
Host C compiler   cc
C++ compiler      x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS           rv
CFLAGS            -g 
QEMU_CFLAGS       -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1  -I$(SRC_PATH)/dtc/libfdt -Werror -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -m64 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-shift-negative-value -Wmissing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -I/usr/x86_64-w64-mingw32/sys-root/mingw/include   -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 
LDFLAGS           -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
tcg debug enabled yes
gprof enabled     no
sparse enabled    no
strip binaries    no
profiler          no
static build      no
pixman            system
SDL support       yes (1.2.15)
GTK support       yes (2.24.30)
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.2)
nettle kdf        yes
libtasn1          yes
curses support    no
virgl support     no
curl support      yes
mingw32 support   yes
Audio drivers     dsound
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
VNC support       yes
VNC SASL support  no
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               no
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support no
Install blobs     yes
KVM support       no
RDMA support      no
TCG interpreter   no
fdt support       yes
preadv support    no
fdatasync         no
madvise           no
posix_madvise     no
libcap-ng support no
vhost-net support no
vhost-scsi support no
vhost-vsock support no
Trace backends    simple
Trace output file trace-<pid>
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info yes
QGA MSI support   no
seccomp support   no
coroutine backend win32
coroutine pool    yes
GlusterFS support no
Archipelago support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
QOM debugging     yes
lzo support       no
snappy support    no
bzip2 support     yes
NUMA host support no
tcmalloc support  no
jemalloc support  no
avx2 optimization yes
replication support yes
mkdir -p dtc/libfdt
  GEN   x86_64-softmmu/config-devices.mak.tmp
mkdir -p dtc/tests
  GEN   aarch64-softmmu/config-devices.mak.tmp
  GEN   config-host.h
  GEN   qemu-options.def
  GEN   qmp-commands.h
  GEN   qapi-types.h
  GEN   qapi-visit.h
  GEN   qapi-event.h
  GEN   x86_64-softmmu/config-devices.mak
  GEN   aarch64-softmmu/config-devices.mak
  GEN   qmp-introspect.h
  GEN   module_block.h
  GEN   tests/test-qapi-types.h
  GEN   tests/test-qapi-visit.h
  GEN   tests/test-qmp-commands.h
  GEN   tests/test-qapi-event.h
  GEN   tests/test-qmp-introspect.h
  GEN   config-all-devices.mak
  GEN   trace/generated-events.h
	 DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
	 DEP /tmp/qemu-test/src/dtc/tests/trees.S
	 DEP /tmp/qemu-test/src/dtc/tests/testutils.c
	 DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
  GEN   trace/generated-tracers.h
	 DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
	 DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
  GEN   trace/generated-tcg-tracers.h
	 DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
	 DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
  GEN   trace/generated-helpers-wrappers.h
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
  GEN   trace/generated-helpers.h
	 DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
	 DEP /tmp/qemu-test/src/dtc/tests/incbin.c
	 DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
	 DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
	 DEP /tmp/qemu-test/src/dtc/tests/path-references.c
	 DEP /tmp/qemu-test/src/dtc/tests/references.c
	 DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/set_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
	 DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
	 DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
	 DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
	 DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
	 DEP /tmp/qemu-test/src/dtc/tests/notfound.c
	 DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
	 DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
	 DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_path.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/getprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/find_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/root_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
	 DEP /tmp/qemu-test/src/dtc/util.c
	 DEP /tmp/qemu-test/src/dtc/fdtput.c
	 DEP /tmp/qemu-test/src/dtc/fdtget.c
	 DEP /tmp/qemu-test/src/dtc/fdtdump.c
	 LEX convert-dtsv0-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/srcpos.c
	 BISON dtc-parser.tab.c
	 LEX dtc-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/treesource.c
	 DEP /tmp/qemu-test/src/dtc/livetree.c
	 DEP /tmp/qemu-test/src/dtc/fstree.c
	 DEP /tmp/qemu-test/src/dtc/flattree.c
	 DEP /tmp/qemu-test/src/dtc/dtc.c
	 DEP /tmp/qemu-test/src/dtc/data.c
	 DEP /tmp/qemu-test/src/dtc/checks.c
	 DEP convert-dtsv0-lexer.lex.c
	 DEP dtc-parser.tab.c
	 DEP dtc-lexer.lex.c
	CHK version_gen.h
	UPD version_gen.h
	 DEP /tmp/qemu-test/src/dtc/util.c
	 CC libfdt/fdt.o
	 CC libfdt/fdt_sw.o
	 CC libfdt/fdt_wip.o
	 CC libfdt/fdt_ro.o
	 CC libfdt/fdt_rw.o
	 CC libfdt/fdt_strerror.o
	 CC libfdt/fdt_empty_tree.o
	 AR libfdt/libfdt.a
x86_64-w64-mingw32-ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
  RC    version.lo
  RC    version.o
  GEN   qga/qapi-generated/qga-qapi-types.h
  GEN   qga/qapi-generated/qga-qapi-visit.h
  GEN   qga/qapi-generated/qga-qmp-commands.h
  GEN   qga/qapi-generated/qga-qapi-types.c
  GEN   qga/qapi-generated/qga-qapi-visit.c
  GEN   qga/qapi-generated/qga-qmp-marshal.c
  GEN   qmp-introspect.c
  GEN   qapi-types.c
  GEN   qapi-visit.c
  GEN   qapi-event.c
  CC    qapi/qapi-visit-core.o
  CC    qapi/qapi-dealloc-visitor.o
  CC    qapi/qmp-input-visitor.o
  CC    qapi/qmp-output-visitor.o
  CC    qapi/qmp-registry.o
  CC    qapi/qmp-dispatch.o
  CC    qapi/string-input-visitor.o
  CC    qapi/string-output-visitor.o
  CC    qapi/opts-visitor.o
  CC    qapi/qapi-clone-visitor.o
  CC    qapi/qmp-event.o
  CC    qapi/qapi-util.o
  CC    qobject/qnull.o
  CC    qobject/qint.o
  CC    qobject/qstring.o
  CC    qobject/qdict.o
  CC    qobject/qlist.o
  CC    qobject/qfloat.o
  CC    qobject/qbool.o
  CC    qobject/qobject.o
  CC    qobject/qjson.o
  CC    qobject/json-lexer.o
  CC    qobject/json-streamer.o
  CC    qobject/json-parser.o
  GEN   trace/generated-events.c
  GEN   trace/generated-tracers.c
  CC    trace/simple.o
  CC    trace/control.o
  CC    trace/qmp.o
  CC    util/osdep.o
  CC    util/cutils.o
  CC    util/unicode.o
  CC    util/qemu-timer-common.o
  CC    util/bufferiszero.o
  CC    util/event_notifier-win32.o
  CC    util/oslib-win32.o
  CC    util/qemu-thread-win32.o
  CC    util/envlist.o
  CC    util/path.o
  CC    util/module.o
  CC    util/bitmap.o
  CC    util/bitops.o
  CC    util/hbitmap.o
  CC    util/fifo8.o
  CC    util/acl.o
  CC    util/error.o
  CC    util/qemu-error.o
  CC    util/id.o
  CC    util/iov.o
  CC    util/qemu-config.o
  CC    util/qemu-sockets.o
  CC    util/uri.o
  CC    util/notify.o
  CC    util/qemu-option.o
  CC    util/qemu-progress.o
  CC    util/hexdump.o
  CC    util/crc32c.o
  CC    util/uuid.o
  CC    util/throttle.o
  CC    util/getauxval.o
  CC    util/readline.o
  CC    util/rfifolock.o
  CC    util/rcu.o
  CC    util/qemu-coroutine.o
  CC    util/qemu-coroutine-lock.o
  CC    util/qemu-coroutine-io.o
  CC    util/qemu-coroutine-sleep.o
  CC    util/coroutine-win32.o
  CC    util/buffer.o
  CC    util/timed-average.o
  CC    util/base64.o
  CC    util/log.o
  CC    util/qdist.o
  CC    util/qht.o
  CC    util/range.o
  CC    crypto/pbkdf-stub.o
  CC    stubs/arch-query-cpu-def.o
  CC    stubs/arch-query-cpu-model-expansion.o
  CC    stubs/arch-query-cpu-model-comparison.o
  CC    stubs/arch-query-cpu-model-baseline.o
  CC    stubs/bdrv-next-monitor-owned.o
  CC    stubs/blk-commit-all.o
  CC    stubs/blockdev-close-all-bdrv-states.o
  CC    stubs/clock-warp.o
  CC    stubs/cpu-get-clock.o
  CC    stubs/cpu-get-icount.o
  CC    stubs/dump.o
  CC    stubs/fdset-add-fd.o
  CC    stubs/fdset-find-fd.o
  CC    stubs/fdset-get-fd.o
  CC    stubs/fdset-remove-fd.o
  CC    stubs/gdbstub.o
  CC    stubs/get-fd.o
  CC    stubs/get-next-serial.o
  CC    stubs/get-vm-name.o
  CC    stubs/iothread-lock.o
  CC    stubs/is-daemonized.o
  CC    stubs/machine-init-done.o
  CC    stubs/migr-blocker.o
  CC    stubs/mon-is-qmp.o
  CC    stubs/mon-printf.o
  CC    stubs/monitor-init.o
  CC    stubs/notify-event.o
  CC    stubs/qtest.o
  CC    stubs/replay.o
  CC    stubs/replay-user.o
  CC    stubs/reset.o
  CC    stubs/runstate-check.o
  CC    stubs/set-fd-handler.o
  CC    stubs/slirp.o
  CC    stubs/sysbus.o
  CC    stubs/trace-control.o
  CC    stubs/uuid.o
  CC    stubs/vm-stop.o
  CC    stubs/vmstate.o
  CC    stubs/fd-register.o
  CC    stubs/cpus.o
  CC    stubs/kvm.o
  CC    stubs/qmp_pc_dimm_device_list.o
  CC    stubs/target-monitor-defs.o
  CC    stubs/target-get-monitor-def.o
  CC    stubs/vhost.o
  CC    stubs/iohandler.o
  CC    stubs/smbios_type_38.o
  CC    stubs/ipmi.o
  CC    stubs/pc_madt_cpu_entry.o
  GEN   qemu-img-cmds.h
  CC    async.o
  CC    thread-pool.o
  CC    blockjob.o
  CC    block.o
  CC    main-loop.o
  CC    iohandler.o
  CC    qemu-timer.o
  CC    aio-win32.o
  CC    qemu-io-cmds.o
  CC    replication.o
  CC    block/raw_bsd.o
  CC    block/qcow.o
  CC    block/vdi.o
  CC    block/vmdk.o
  CC    block/cloop.o
  CC    block/bochs.o
  CC    block/vpc.o
  CC    block/vvfat.o
  CC    block/dmg.o
  CC    block/qcow2.o
  CC    block/qcow2-refcount.o
  CC    block/qcow2-cluster.o
  CC    block/qcow2-snapshot.o
  CC    block/qcow2-cache.o
  CC    block/qed.o
  CC    block/qed-gencb.o
  CC    block/qed-l2-cache.o
  CC    block/qed-table.o
  CC    block/qed-cluster.o
  CC    block/qed-check.o
  CC    block/vhdx.o
  CC    block/vhdx-endian.o
  CC    block/vhdx-log.o
  CC    block/quorum.o
  CC    block/parallels.o
  CC    block/blkdebug.o
  CC    block/blkverify.o
  CC    block/blkreplay.o
  CC    block/block-backend.o
  CC    block/snapshot.o
  CC    block/qapi.o
  CC    block/raw-win32.o
  CC    block/win32-aio.o
  CC    block/null.o
  CC    block/mirror.o
  CC    block/commit.o
  CC    block/io.o
  CC    block/throttle-groups.o
  CC    block/nbd.o
  CC    block/nbd-client.o
  CC    block/sheepdog.o
  CC    block/accounting.o
  CC    block/dirty-bitmap.o
  CC    block/write-threshold.o
  CC    block/backup.o
  CC    block/replication.o
  CC    block/crypto.o
  CC    nbd/server.o
  CC    nbd/client.o
  CC    nbd/common.o
  CC    block/curl.o
  CC    block/ssh.o
  CC    crypto/init.o
  CC    crypto/hash.o
  CC    crypto/hash-nettle.o
  CC    crypto/aes.o
  CC    crypto/desrfb.o
  CC    crypto/cipher.o
  CC    crypto/tlscreds.o
  CC    crypto/tlscredsanon.o
  CC    crypto/tlscredsx509.o
  CC    crypto/tlssession.o
  CC    crypto/secret.o
  CC    crypto/random-gnutls.o
  CC    crypto/pbkdf.o
  CC    crypto/pbkdf-nettle.o
  CC    crypto/ivgen.o
  CC    crypto/ivgen-essiv.o
  CC    crypto/ivgen-plain.o
  CC    crypto/ivgen-plain64.o
  CC    crypto/afsplit.o
  CC    crypto/xts.o
  CC    crypto/block.o
  CC    crypto/block-qcow.o
  CC    crypto/block-luks.o
  CC    io/channel.o
  CC    io/channel-buffer.o
  CC    io/channel-command.o
  CC    io/channel-file.o
  CC    io/channel-socket.o
  CC    io/channel-tls.o
  CC    io/channel-watch.o
  CC    io/channel-websock.o
  CC    io/channel-util.o
  CC    io/task.o
  CC    qom/object.o
  CC    qom/container.o
  CC    qom/qom-qobject.o
  CC    qom/object_interfaces.o
  CC    qemu-io.o
  CC    blockdev.o
  CC    blockdev-nbd.o
  CC    iothread.o
  CC    qdev-monitor.o
  CC    device-hotplug.o
  CC    os-win32.o
  CC    qemu-char.o
  CC    page_cache.o
  CC    accel.o
  CC    bt-host.o
  CC    bt-vhci.o
  CC    dma-helpers.o
  CC    vl.o
  CC    tpm.o
  CC    device_tree.o
  GEN   qmp-marshal.c
  CC    qmp.o
  CC    hmp.o
  CC    tcg-runtime.o
  CC    cpus-common.o
  CC    audio/audio.o
  CC    audio/noaudio.o
  CC    audio/wavaudio.o
  CC    audio/mixeng.o
  CC    audio/sdlaudio.o
  CC    audio/dsoundaudio.o
  CC    audio/audio_win_int.o
  CC    audio/wavcapture.o
  CC    backends/rng.o
  CC    backends/rng-egd.o
  CC    backends/msmouse.o
  CC    backends/testdev.o
  CC    backends/tpm.o
  CC    backends/hostmem.o
  CC    backends/hostmem-ram.o
  CC    block/stream.o
  CC    disas/arm.o
  CXX   disas/arm-a64.o
  CC    disas/i386.o
  CXX   disas/libvixl/vixl/utils.o
  CXX   disas/libvixl/vixl/compiler-intrinsics.o
  CXX   disas/libvixl/vixl/a64/instructions-a64.o
  CXX   disas/libvixl/vixl/a64/decoder-a64.o
  CXX   disas/libvixl/vixl/a64/disasm-a64.o
  CC    hw/acpi/core.o
  CC    hw/acpi/piix4.o
  CC    hw/acpi/pcihp.o
  CC    hw/acpi/ich9.o
  CC    hw/acpi/tco.o
  CC    hw/acpi/cpu_hotplug.o
  CC    hw/acpi/memory_hotplug.o
  CC    hw/acpi/memory_hotplug_acpi_table.o
  CC    hw/acpi/cpu.o
  CC    hw/acpi/acpi_interface.o
  CC    hw/acpi/bios-linker-loader.o
  CC    hw/acpi/aml-build.o
  CC    hw/acpi/ipmi.o
  CC    hw/audio/sb16.o
  CC    hw/audio/es1370.o
  CC    hw/audio/ac97.o
  CC    hw/audio/fmopl.o
  CC    hw/audio/adlib.o
  CC    hw/audio/gus.o
  CC    hw/audio/gusemu_hal.o
  CC    hw/audio/gusemu_mixer.o
  CC    hw/audio/cs4231a.o
  CC    hw/audio/intel-hda.o
  CC    hw/audio/hda-codec.o
  CC    hw/audio/pcspk.o
  CC    hw/audio/wm8750.o
  CC    hw/audio/pl041.o
  CC    hw/audio/lm4549.o
  CC    hw/audio/marvell_88w8618.o
  CC    hw/block/block.o
  CC    hw/block/cdrom.o
  CC    hw/block/hd-geometry.o
  CC    hw/block/fdc.o
  CC    hw/block/m25p80.o
  CC    hw/block/nand.o
  CC    hw/block/pflash_cfi01.o
  CC    hw/block/ecc.o
  CC    hw/block/pflash_cfi02.o
  CC    hw/block/onenand.o
  CC    hw/block/nvme.o
  CC    hw/bt/core.o
  CC    hw/bt/l2cap.o
  CC    hw/bt/sdp.o
  CC    hw/bt/hci.o
  CC    hw/bt/hid.o
  CC    hw/bt/hci-csr.o
  CC    hw/char/ipoctal232.o
  CC    hw/char/parallel.o
  CC    hw/char/pl011.o
  CC    hw/char/serial.o
  CC    hw/char/serial-isa.o
  CC    hw/char/serial-pci.o
  CC    hw/char/virtio-console.o
  CC    hw/char/cadence_uart.o
  CC    hw/char/debugcon.o
  CC    hw/char/imx_serial.o
  CC    hw/core/qdev.o
  CC    hw/core/qdev-properties.o
  CC    hw/core/bus.o
  CC    hw/core/fw-path-provider.o
  CC    hw/core/irq.o
  CC    hw/core/hotplug.o
  CC    hw/core/ptimer.o
  CC    hw/core/sysbus.o
  CC    hw/core/machine.o
  CC    hw/core/loader.o
  CC    hw/core/null-machine.o
  CC    hw/core/qdev-properties-system.o
  CC    hw/core/register.o
  CC    hw/core/platform-bus.o
  CC    hw/display/ads7846.o
  CC    hw/display/cirrus_vga.o
  CC    hw/display/pl110.o
  CC    hw/display/ssd0303.o
  CC    hw/display/ssd0323.o
  CC    hw/display/vga-pci.o
  CC    hw/display/vga-isa.o
  CC    hw/display/vmware_vga.o
  CC    hw/display/blizzard.o
  CC    hw/display/exynos4210_fimd.o
  CC    hw/display/framebuffer.o
  CC    hw/display/tc6393xb.o
  CC    hw/dma/pl080.o
  CC    hw/dma/pl330.o
  CC    hw/dma/i8257.o
  CC    hw/dma/xlnx-zynq-devcfg.o
  CC    hw/gpio/max7310.o
  CC    hw/gpio/pl061.o
  CC    hw/gpio/zaurus.o
  CC    hw/gpio/gpio_key.o
  CC    hw/i2c/core.o
  CC    hw/i2c/smbus.o
  CC    hw/i2c/smbus_eeprom.o
  CC    hw/i2c/i2c-ddc.o
  CC    hw/i2c/versatile_i2c.o
  CC    hw/i2c/smbus_ich9.o
  CC    hw/i2c/pm_smbus.o
  CC    hw/i2c/bitbang_i2c.o
  CC    hw/i2c/exynos4210_i2c.o
  CC    hw/i2c/imx_i2c.o
  CC    hw/i2c/aspeed_i2c.o
  CC    hw/ide/core.o
  CC    hw/ide/atapi.o
  CC    hw/ide/qdev.o
  CC    hw/ide/pci.o
  CC    hw/ide/isa.o
  CC    hw/ide/piix.o
  CC    hw/ide/microdrive.o
  CC    hw/ide/ahci.o
  CC    hw/ide/ich.o
  CC    hw/input/hid.o
  CC    hw/input/lm832x.o
  CC    hw/input/pckbd.o
  CC    hw/input/pl050.o
  CC    hw/input/ps2.o
  CC    hw/input/stellaris_input.o
  CC    hw/input/tsc2005.o
  CC    hw/input/vmmouse.o
  CC    hw/input/virtio-input.o
  CC    hw/input/virtio-input-hid.o
  CC    hw/intc/i8259_common.o
  CC    hw/intc/i8259.o
  CC    hw/intc/pl190.o
  CC    hw/intc/imx_avic.o
  CC    hw/intc/realview_gic.o
  CC    hw/intc/ioapic_common.o
In file included from /tmp/qemu-test/src/hw/input/virtio-input.c:13:0:
/tmp/qemu-test/src/hw/input/virtio-input.c:299:55: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
 VMSTATE_VIRTIO_DEVICE(input, VIRTIO_INPUT_VM_VERSION, virtio_input_load,
                                                       ^
/tmp/qemu-test/src/include/hw/virtio/virtio.h:190:32: note: in definition of macro 'VMSTATE_VIRTIO_DEVICE'
                         .get = getf,          \
                                ^~~~
/tmp/qemu-test/src/hw/input/virtio-input.c:299:55: note: (near initialization for '(anonymous).get')
 VMSTATE_VIRTIO_DEVICE(input, VIRTIO_INPUT_VM_VERSION, virtio_input_load,
                                                       ^
/tmp/qemu-test/src/include/hw/virtio/virtio.h:190:32: note: in definition of macro 'VMSTATE_VIRTIO_DEVICE'
                         .get = getf,          \
                                ^~~~
/tmp/qemu-test/src/hw/input/virtio-input.c:300:23: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
                       virtio_vmstate_save);
                       ^
/tmp/qemu-test/src/include/hw/virtio/virtio.h:191:32: note: in definition of macro 'VMSTATE_VIRTIO_DEVICE'
                         .put = putf,          \
                                ^~~~
/tmp/qemu-test/src/hw/input/virtio-input.c:300:23: note: (near initialization for '(anonymous).put')
                       virtio_vmstate_save);
                       ^
/tmp/qemu-test/src/include/hw/virtio/virtio.h:191:32: note: in definition of macro 'VMSTATE_VIRTIO_DEVICE'
                         .put = putf,          \
                                ^~~~
cc1: all warnings being treated as errors
/tmp/qemu-test/src/rules.mak:60: recipe for target 'hw/input/virtio-input.o' failed
make: *** [hw/input/virtio-input.o] Error 1
make: *** Waiting for unfinished jobs....
tests/docker/Makefile.include:107: recipe for target 'docker-run-test-mingw@fedora' failed
make: *** [docker-run-test-mingw@fedora] Error 2
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [QEMU PATCH v5 0/6] migration: ensure hotplug and migration work together

[Jianjun Duan]

I will address the style issues together with any other possible
comments. The build test failed on code not in my patches.

Thanks,
Jianjun

On 10/03/2016 11:24 AM, Jianjun Duan wrote:
> Hi all,
>    The previous patches seem to get buried deep somewhere. I am sending the lated rebased version. Comments are welcome.
> 
> v5: - Rebased to David's ppc-for-2.8. Previous versions are:
> 
> v4: - Introduce a way to set customized instance_id in SaveStateEntry. Use it
>       to set instance_id for DRC using its unique index to address David 
>       Gibson's concern.
>     - Rename VMS_CSTM to VMS_LINKED based on Paolo Bonzini's suggestions.
>     - Clean up qjson stuff in put_qtailq. 
>     - Add trace for put_qtailq and get_qtailq based on David Gilbert's 
>       suggestion.
> 
>     - Based on David's ppc-for-2.7. 
> 
> v3: - Simplify overall design followng discussion with Paolo. No longer need
>       metadata to migrate QTAILQ.
>     - Extend VMStateInfo instead of adding similar fields to VMStateField.
>     - Clean up macros in qemu/queue.h.
> (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg05695.html)
> 
> v2: - Introduce a general approach to migrate QTAILQ in qemu/queue.h.
>     - Migrate signalled field in the DRC state.
>     - Put the newly added migrating fields in subsections so that backward 
>       migration is not broken.  
>     - Set detach_cb field right after migration so that a migrated hot-unplug
>       event could finish its course.
> (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-05/msg04188.html)
> 
> v1: - Inital version.
> (link: https://lists.nongnu.org/archive/html/qemu-devel/2016-04/msg02601.html)
> 
> To make guest device (PCI, CPU and memory) hotplug work together 
> with guest migration, spapr drc state needs be transmitted in
> migration. This patch defines the VMStateDescription struct for
> spapr drc state to enable it.
> 
> To fix the potential racing between hotplug events on guest and 
> guest migration, ccs_list and pending_events of spapr state need be 
> transmitted in migration. This patch also takes care of it.
> 
> Jianjun Duan (6):
>   migration: alternative way to set instance_id in SaveStateEntry
>   migration: spapr_drc: defined VMStateDescription struct
>   migration: extend VMStateInfo
>   migration: migrate QTAILQ
>   migration: spapr: migrate ccs_list in spapr state
>   migration: spapr: migrate pending_events of spapr state
> 
>  hw/net/vmxnet3.c            |  18 +++--
>  hw/nvram/eeprom93xx.c       |   6 +-
>  hw/nvram/fw_cfg.c           |   6 +-
>  hw/pci/msix.c               |   6 +-
>  hw/pci/pci.c                |  12 ++--
>  hw/pci/shpc.c               |   5 +-
>  hw/ppc/spapr.c              |  67 ++++++++++++++++++
>  hw/ppc/spapr_drc.c          |  69 +++++++++++++++++++
>  hw/ppc/spapr_events.c       |  22 +++---
>  hw/ppc/spapr_pci.c          |  22 ++++++
>  hw/scsi/scsi-bus.c          |   6 +-
>  hw/timer/twl92230.c         |   6 +-
>  hw/usb/redirect.c           |  18 +++--
>  hw/virtio/virtio-pci.c      |   6 +-
>  hw/virtio/virtio.c          |   6 +-
>  include/hw/ppc/spapr.h      |   3 +-
>  include/hw/ppc/spapr_drc.h  |   9 +++
>  include/hw/qdev-core.h      |   6 ++
>  include/migration/vmstate.h |  36 ++++++++--
>  include/qemu/queue.h        |  32 +++++++++
>  migration/savevm.c          |  25 +++++--
>  migration/trace-events      |   4 ++
>  migration/vmstate.c         | 161 ++++++++++++++++++++++++++++++++++----------
>  target-alpha/machine.c      |   5 +-
>  target-arm/machine.c        |  12 ++--
>  target-i386/machine.c       |  21 ++++--
>  target-mips/machine.c       |  10 +--
>  target-ppc/machine.c        |  10 +--
>  target-sparc/machine.c      |   5 +-
>  29 files changed, 505 insertions(+), 109 deletions(-)
>