[Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Jianfeng Tan]

Existing VMs with virtio devices and vhost-kernel as the backend
are always started with mem config:

  "-m xG"
  (with a ram block named "pc.ram")

while new VMs with virtio devices and vhost-user as the backend
are always started with mem config:

  "-m xG -numa node,memdev=pc.ram -object memory-backend-file,id=pc.ram,..."
  (with a ram block named "/object/pc.ram")

As we migrate from vhost-kernel to vhost-user, it failes as:

  Unknown ramblock "pc.ram", cannot accept migration
  error while loading state for instance 0x0 of device 'ram'
  load of migration failed: Invalid argument

Here are some options to fix this:

1. When we do ram name comparison, we truncate the prefix as this patch shows.
It cannot cover the corner case: the source VM could have two ram blocks
with name of "pc.ram" and "/object/pc.ram".

2. We add an alias name to RAMBlock; when we do name comparison, not only
idstr is compared, but also compared to the alias. But this will add more
complexity to upper layer stack OpenStack/libvirt.

Any thoughts?

Cc: Jason Wang <jasowang@redhat.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Suggested-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
---
 exec.c | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/exec.c b/exec.c
index 4722e52..d294e5c 100644
--- a/exec.c
+++ b/exec.c
@@ -2334,13 +2334,24 @@ found:
 RAMBlock *qemu_ram_block_by_name(const char *name)
 {
     RAMBlock *block;
+    char *name1, *id1;
+    char *name2, *id2;
+
+    name1 = strdup(name);
+    id1 = basename(name1);
 
     RAMBLOCK_FOREACH(block) {
-        if (!strcmp(name, block->idstr)) {
+        name2 = strdup(block->idstr);
+	id2 = basename(name2);
+        if (!strcmp(id1, id2)) {
+            free(name1);
+	    free(name2);
             return block;
         }
+	free(name2);
     }
 
+    free(name1);
     return NULL;
 }
 
-- 
2.7.4

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 1517842735-9011-1-git-send-email-jianfeng.tan@intel.com
Subject: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/1517842735-9011-1-git-send-email-jianfeng.tan@intel.com -> patchew/1517842735-9011-1-git-send-email-jianfeng.tan@intel.com
Switched to a new branch 'test'
2a645f0bd4 exec: eliminate ram naming issue as migration

=== OUTPUT BEGIN ===
Checking PATCH 1/1: exec: eliminate ram naming issue as migration...
ERROR: code indent should never use tabs
#61: FILE: exec.c:2377:
+^Iid2 = basename(name2);$

ERROR: code indent should never use tabs
#64: FILE: exec.c:2380:
+^I    free(name2);$

ERROR: code indent should never use tabs
#67: FILE: exec.c:2383:
+^Ifree(name2);$

total: 3 errors, 0 warnings, 25 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Paolo Bonzini]

On 05/02/2018 15:58, Jianfeng Tan wrote:
> Here are some options to fix this:
> 
> 1. When we do ram name comparison, we truncate the prefix as this patch shows.
> It cannot cover the corner case: the source VM could have two ram blocks
> with name of "pc.ram" and "/object/pc.ram".

That shouldn't happen ("pc.ram" exists even in the "-numa
node,memdev=..." case, but it has no RAM block).

>      RAMBLOCK_FOREACH(block) {
> -        if (!strcmp(name, block->idstr)) {
> +        name2 = strdup(block->idstr);
> +	id2 = basename(name2);
> +        if (!strcmp(id1, id2)) {
> +            free(name1);
> +	    free(name2);
>              return block;
>          }
> +	free(name2);

Instead of this, perhaps just skip "/object/" in both id1 and
block->idstr?  This also removes the need for strdup/free.

However, note that

  -m xG -numa node,memdev=pc.ram \
  -object memory-backend-file,id=pc.ram,...

works for both vhost-kernel and vhost-user, so I'd rather consider this
a configuration problem and not do anything.

Thanks,

Paolo

>      }
>  
> +    free(name1);
>      return NULL;
>  }
>  
> 

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

Hi Paolo,

On 2/5/2018 11:53 PM, Paolo Bonzini wrote:
> On 05/02/2018 15:58, Jianfeng Tan wrote:
>> Here are some options to fix this:
>>
>> 1. When we do ram name comparison, we truncate the prefix as this patch shows.
>> It cannot cover the corner case: the source VM could have two ram blocks
>> with name of "pc.ram" and "/object/pc.ram".
> That shouldn't happen ("pc.ram" exists even in the "-numa
> node,memdev=..." case, but it has no RAM block).

Suppose we have a VM started with "-m xG", and then hot plugged with a 
ram block:
   (qemu) object_add 
memory-backend-file,id=pc.ram,size=1G,mem-path=/dev/hugepages
   (qemu) device_add pc-dimm,id=pc.ram,memdev=pc.ram

Then we would have both ram block named pc.ram:
               Block Name    PSize
                       pc.ram     4 KiB
       /objects/pc.ram    2 MiB

But I assume it's a corner case which not really happen.

>
>>       RAMBLOCK_FOREACH(block) {
>> -        if (!strcmp(name, block->idstr)) {
>> +        name2 = strdup(block->idstr);
>> +	id2 = basename(name2);
>> +        if (!strcmp(id1, id2)) {
>> +            free(name1);
>> +	    free(name2);
>>               return block;
>>           }
>> +	free(name2);
> Instead of this, perhaps just skip "/object/" in both id1 and
> block->idstr?  This also removes the need for strdup/free.

Not familiar with this before, so there would not be something like 
/object/xxx/id? In that case, we can just skip "/object/". Thanks!

>
> However, note that
>
>    -m xG -numa node,memdev=pc.ram \
>    -object memory-backend-file,id=pc.ram,...
>
> works for both vhost-kernel and vhost-user, so I'd rather consider this
> a configuration problem and not do anything.

That configuration indeed works for both. But in the production env, 
lots of VMs are already started with previous mem config. If we do 
nothing, it will take a long time (shutdown/start for each VM) to 
migrate to the new setup. This patch is to make this process more smooth 
without any bad effect if possible.

Thanks,
Jianfeng

>
> Thanks,
>
> Paolo
>
>>       }
>>   
>> +    free(name1);
>>       return NULL;
>>   }
>>   
>>

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Paolo Bonzini]

On 05/02/2018 17:12, Tan, Jianfeng wrote:
> Hi Paolo,
> 
> On 2/5/2018 11:53 PM, Paolo Bonzini wrote:
>> On 05/02/2018 15:58, Jianfeng Tan wrote:
>>> Here are some options to fix this:
>>>
>>> 1. When we do ram name comparison, we truncate the prefix as this
>>> patch shows.
>>> It cannot cover the corner case: the source VM could have two ram blocks
>>> with name of "pc.ram" and "/object/pc.ram".
>> That shouldn't happen ("pc.ram" exists even in the "-numa
>> node,memdev=..." case, but it has no RAM block).
> 
> Suppose we have a VM started with "-m xG", and then hot plugged with a
> ram block:
>   (qemu) object_add
> memory-backend-file,id=pc.ram,size=1G,mem-path=/dev/hugepages
>   (qemu) device_add pc-dimm,id=pc.ram,memdev=pc.ram
> 
> Then we would have both ram block named pc.ram:
>               Block Name    PSize
>                       pc.ram     4 KiB
>       /objects/pc.ram    2 MiB
> 
> But I assume it's a corner case which not really happen.

Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.

>> However, note that
>>
>>    -m xG -numa node,memdev=pc.ram \
>>    -object memory-backend-file,id=pc.ram,...
>>
>> works for both vhost-kernel and vhost-user, so I'd rather consider this
>> a configuration problem and not do anything.
> 
> That configuration indeed works for both. But in the production env,
> lots of VMs are already started with previous mem config. If we do
> nothing, it will take a long time (shutdown/start for each VM) to
> migrate to the new setup. This patch is to make this process more smooth
> without any bad effect if possible.

I understand.  However it's not as bad as "there's no possibility at all
to migrate from vhost-kernel to vhost-user".  There are cases that are
more problematic: for example, there's no possibility at all to add
memory NUMA policy during a live migration, unless -object
memory-backend-* was used on the source.

Paolo

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

On 2/6/2018 12:19 AM, Paolo Bonzini wrote:
> On 05/02/2018 17:12, Tan, Jianfeng wrote:
>> Hi Paolo,
>>
>> On 2/5/2018 11:53 PM, Paolo Bonzini wrote:
>>> On 05/02/2018 15:58, Jianfeng Tan wrote:
>>>> Here are some options to fix this:
>>>>
>>>> 1. When we do ram name comparison, we truncate the prefix as this
>>>> patch shows.
>>>> It cannot cover the corner case: the source VM could have two ram blocks
>>>> with name of "pc.ram" and "/object/pc.ram".
>>> That shouldn't happen ("pc.ram" exists even in the "-numa
>>> node,memdev=..." case, but it has no RAM block).
>> Suppose we have a VM started with "-m xG", and then hot plugged with a
>> ram block:
>>    (qemu) object_add
>> memory-backend-file,id=pc.ram,size=1G,mem-path=/dev/hugepages
>>    (qemu) device_add pc-dimm,id=pc.ram,memdev=pc.ram
>>
>> Then we would have both ram block named pc.ram:
>>                Block Name    PSize
>>                        pc.ram     4 KiB
>>        /objects/pc.ram    2 MiB
>>
>> But I assume it's a corner case which not really happen.
> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.
>
>>> However, note that
>>>
>>>     -m xG -numa node,memdev=pc.ram \
>>>     -object memory-backend-file,id=pc.ram,...
>>>
>>> works for both vhost-kernel and vhost-user, so I'd rather consider this
>>> a configuration problem and not do anything.
>> That configuration indeed works for both. But in the production env,
>> lots of VMs are already started with previous mem config. If we do
>> nothing, it will take a long time (shutdown/start for each VM) to
>> migrate to the new setup. This patch is to make this process more smooth
>> without any bad effect if possible.
> I understand.  However it's not as bad as "there's no possibility at all
> to migrate from vhost-kernel to vhost-user".  There are cases that are
> more problematic: for example, there's no possibility at all to add
> memory NUMA policy during a live migration, unless -object
> memory-backend-* was used on the source.

Please help me to understand: Are you saying it's always recommended to 
use -object memory-backend-* configuration even with vhost-kernel 
backend for the reason you mentioned? Or just another more serious 
problem that we shall work on in priority?

Thanks,
Jianfeng

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Paolo Bonzini]

On 05/02/2018 17:44, Tan, Jianfeng wrote:
>>>
>> I understand.  However it's not as bad as "there's no possibility at all
>> to migrate from vhost-kernel to vhost-user".  There are cases that are
>> more problematic: for example, there's no possibility at all to add
>> memory NUMA policy during a live migration, unless -object
>> memory-backend-* was used on the source.
> 
> Please help me to understand: Are you saying it's always recommended to
> use -object memory-backend-* configuration even with vhost-kernel
> backend for the reason you mentioned?

Indeed---in general, there's no downside to always using -object to
start a virtual machine.  Of course when migrating a machine that was
started without -object, you need to use it on the destination too.

Paolo

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Mon, 5 Feb 2018 17:19:09 +0100
Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 05/02/2018 17:12, Tan, Jianfeng wrote:
> > Hi Paolo,
> > 
> > On 2/5/2018 11:53 PM, Paolo Bonzini wrote:  
> >> On 05/02/2018 15:58, Jianfeng Tan wrote:  
> >>> Here are some options to fix this:
> >>>
> >>> 1. When we do ram name comparison, we truncate the prefix as this
> >>> patch shows.
> >>> It cannot cover the corner case: the source VM could have two ram blocks
> >>> with name of "pc.ram" and "/object/pc.ram".  
> >> That shouldn't happen ("pc.ram" exists even in the "-numa
> >> node,memdev=..." case, but it has no RAM block).  
> > 
> > Suppose we have a VM started with "-m xG", and then hot plugged with a
> > ram block:
> >   (qemu) object_add
> > memory-backend-file,id=pc.ram,size=1G,mem-path=/dev/hugepages
> >   (qemu) device_add pc-dimm,id=pc.ram,memdev=pc.ram
> > 
> > Then we would have both ram block named pc.ram:
> >               Block Name    PSize
> >                       pc.ram     4 KiB
> >       /objects/pc.ram    2 MiB
> > 
> > But I assume it's a corner case which not really happen.  
> 
> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.
perhaps we should fail object_add memory-backend-foo if it resulted
in creating ramblock with duplicate id
 
> 
> >> However, note that
> >>
> >>    -m xG -numa node,memdev=pc.ram \
> >>    -object memory-backend-file,id=pc.ram,...
> >>
> >> works for both vhost-kernel and vhost-user, so I'd rather consider this
> >> a configuration problem and not do anything.  
> > 
> > That configuration indeed works for both. But in the production env,
> > lots of VMs are already started with previous mem config. If we do
> > nothing, it will take a long time (shutdown/start for each VM) to
> > migrate to the new setup. This patch is to make this process more smooth
> > without any bad effect if possible.  
> 
> I understand.  However it's not as bad as "there's no possibility at all
> to migrate from vhost-kernel to vhost-user".  There are cases that are
> more problematic: for example, there's no possibility at all to add
> memory NUMA policy during a live migration, unless -object
> memory-backend-* was used on the source.
> 
> Paolo
> 

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Paolo Bonzini]

On 05/02/2018 18:15, Igor Mammedov wrote:
>>>
>>> Then we would have both ram block named pc.ram:
>>>               Block Name    PSize
>>>                       pc.ram     4 KiB
>>>       /objects/pc.ram    2 MiB
>>>
>>> But I assume it's a corner case which not really happen.  
>> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.
>  
> perhaps we should fail object_add memory-backend-foo if it resulted
> in creating ramblock with duplicate id

Note that it would only be duplicated with Jianfeng's patch.  So I'm
worried that his patch is worse than what we have now, because it may
create conflicts with system RAMBlock names are not necessarily
predictable.  Right now, -object creates RAMBlock names that are nicely
constrained within /object/.

Paolo

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

> -----Original Message-----
> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> Sent: Tuesday, February 6, 2018 1:32 AM
> To: Igor Mammedov
> Cc: Tan, Jianfeng; qemu-devel@nongnu.org; Jason Wang; Maxime Coquelin;
> Michael S . Tsirkin
> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> migration
> 
> On 05/02/2018 18:15, Igor Mammedov wrote:
> >>>
> >>> Then we would have both ram block named pc.ram:
> >>>               Block Name    PSize
> >>>                       pc.ram     4 KiB
> >>>       /objects/pc.ram    2 MiB
> >>>
> >>> But I assume it's a corner case which not really happen.
> >> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.
> >
> > perhaps we should fail object_add memory-backend-foo if it resulted
> > in creating ramblock with duplicate id
> 
> Note that it would only be duplicated with Jianfeng's patch.  So I'm
> worried that his patch is worse than what we have now, because it may
> create conflicts with system RAMBlock names are not necessarily
> predictable.  Right now, -object creates RAMBlock names that are nicely
> constrained within /object/.

So we are trading off between the benefit it takes and the bad effect it brings.

I'm wondering if the above example is the only failed case this patch leads to, i.e, only there is a ram named "pc.ram" and "/object/pc.ram" in the src VM?

Please also consider the second option, that adding an alias name for RAMBlock; I'm not a big fan for that one, as it just pushes the problem to OpenStack/Libvirt.

Or any other suggestions?

Thanks,
Jianfeng

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Wed, 7 Feb 2018 07:49:58 +0000
"Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:

> > -----Original Message-----
> > From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> > Sent: Tuesday, February 6, 2018 1:32 AM
> > To: Igor Mammedov
> > Cc: Tan, Jianfeng; qemu-devel@nongnu.org; Jason Wang; Maxime Coquelin;
> > Michael S . Tsirkin
> > Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > migration
> > 
> > On 05/02/2018 18:15, Igor Mammedov wrote:  
> > >>>
> > >>> Then we would have both ram block named pc.ram:
> > >>>               Block Name    PSize
> > >>>                       pc.ram     4 KiB
> > >>>       /objects/pc.ram    2 MiB
> > >>>
> > >>> But I assume it's a corner case which not really happen.  
> > >> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.  
> > >
> > > perhaps we should fail object_add memory-backend-foo if it resulted
> > > in creating ramblock with duplicate id  
> > 
> > Note that it would only be duplicated with Jianfeng's patch.  So I'm
> > worried that his patch is worse than what we have now, because it may
> > create conflicts with system RAMBlock names are not necessarily
> > predictable.  Right now, -object creates RAMBlock names that are nicely
> > constrained within /object/.  
> 
> So we are trading off between the benefit it takes and the bad effect it brings.
> 
> I'm wondering if the above example is the only failed case this patch leads to, i.e, only there is a ram named "pc.ram" and "/object/pc.ram" in the src VM?
> 
> Please also consider the second option, that adding an alias name for RAMBlock; I'm not a big fan for that one, as it just pushes the problem to OpenStack/Libvirt.
looking at provided CLI examples it's configuration issue on src and dst,
one shall not mix numa and non numa variants.

> Or any other suggestions?
Fix configuration, namely dst side of it (i.e. use the same -m only variant
without -numa as it's on src).

BTW, what are you trying to achieve adding -numa on dst?

> Thanks,
> Jianfeng

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

On 2/7/2018 8:06 PM, Igor Mammedov wrote:
> On Wed, 7 Feb 2018 07:49:58 +0000
> "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
>
>>> -----Original Message-----
>>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>>> Sent: Tuesday, February 6, 2018 1:32 AM
>>> To: Igor Mammedov
>>> Cc: Tan, Jianfeng; qemu-devel@nongnu.org; Jason Wang; Maxime Coquelin;
>>> Michael S . Tsirkin
>>> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
>>> migration
>>>
>>> On 05/02/2018 18:15, Igor Mammedov wrote:
>>>>>> Then we would have both ram block named pc.ram:
>>>>>>                Block Name    PSize
>>>>>>                        pc.ram     4 KiB
>>>>>>        /objects/pc.ram    2 MiB
>>>>>>
>>>>>> But I assume it's a corner case which not really happen.
>>>>> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.
>>>> perhaps we should fail object_add memory-backend-foo if it resulted
>>>> in creating ramblock with duplicate id
>>> Note that it would only be duplicated with Jianfeng's patch.  So I'm
>>> worried that his patch is worse than what we have now, because it may
>>> create conflicts with system RAMBlock names are not necessarily
>>> predictable.  Right now, -object creates RAMBlock names that are nicely
>>> constrained within /object/.
>> So we are trading off between the benefit it takes and the bad effect it brings.
>>
>> I'm wondering if the above example is the only failed case this patch leads to, i.e, only there is a ram named "pc.ram" and "/object/pc.ram" in the src VM?
>>
>> Please also consider the second option, that adding an alias name for RAMBlock; I'm not a big fan for that one, as it just pushes the problem to OpenStack/Libvirt.
> looking at provided CLI examples it's configuration issue on src and dst,
> one shall not mix numa and non numa variants.

Aha, that's another thing we also want to change. We now add numa at dst 
node, only because without -numa, we cannot set up the file-baked memory 
with share=on.

For example, "-m xG -mem-path xxx" can set up a file-baked memory, but 
the file is not share-able.

>
>> Or any other suggestions?
> Fix configuration, namely dst side of it (i.e. use the same -m only variant
> without -numa as it's on src).
>
> BTW, what are you trying to achieve adding -numa on dst?

See above reply.

Thanks,
Jianfeng

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Thu, 8 Feb 2018 09:20:45 +0800
"Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:

> On 2/7/2018 8:06 PM, Igor Mammedov wrote:
> > On Wed, 7 Feb 2018 07:49:58 +0000
> > "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
> >  
> >>> -----Original Message-----
> >>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> >>> Sent: Tuesday, February 6, 2018 1:32 AM
> >>> To: Igor Mammedov
> >>> Cc: Tan, Jianfeng; qemu-devel@nongnu.org; Jason Wang; Maxime Coquelin;
> >>> Michael S . Tsirkin
> >>> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> >>> migration
> >>>
> >>> On 05/02/2018 18:15, Igor Mammedov wrote:  
> >>>>>> Then we would have both ram block named pc.ram:
> >>>>>>                Block Name    PSize
> >>>>>>                        pc.ram     4 KiB
> >>>>>>        /objects/pc.ram    2 MiB
> >>>>>>
> >>>>>> But I assume it's a corner case which not really happen.  
> >>>>> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.  
> >>>> perhaps we should fail object_add memory-backend-foo if it resulted
> >>>> in creating ramblock with duplicate id  
> >>> Note that it would only be duplicated with Jianfeng's patch.  So I'm
> >>> worried that his patch is worse than what we have now, because it may
> >>> create conflicts with system RAMBlock names are not necessarily
> >>> predictable.  Right now, -object creates RAMBlock names that are nicely
> >>> constrained within /object/.  
> >> So we are trading off between the benefit it takes and the bad effect it brings.
> >>
> >> I'm wondering if the above example is the only failed case this patch leads to, i.e, only there is a ram named "pc.ram" and "/object/pc.ram" in the src VM?
> >>
> >> Please also consider the second option, that adding an alias name for RAMBlock; I'm not a big fan for that one, as it just pushes the problem to OpenStack/Libvirt.  
> > looking at provided CLI examples it's configuration issue on src and dst,
> > one shall not mix numa and non numa variants.  
> 
> Aha, that's another thing we also want to change. We now add numa at dst 
> node, only because without -numa, we cannot set up the file-baked memory 
> with share=on.
then shouldn't you start src with the same -numa to begin with,
changing such things on the fly is not supported.
General rule is that machine on dst has to be the same as on src.
(with backend not visible to guest it possible might be changed
but it's hard to tell if something would break due to that
or would continue working in future since doesn't go along with above rule)

> For example, "-m xG -mem-path xxx" can set up a file-baked memory, but 
> the file is not share-able.
It could be solved by adding memdev option to machine,
which would allow to specify backend object. And then on
top make -mem-path alias new option to clean thing up.

But then again, You'd need to start both src and dst
with the same option.
 
> >  
> >> Or any other suggestions?  
> > Fix configuration, namely dst side of it (i.e. use the same -m only variant
> > without -numa as it's on src).
> >
> > BTW, what are you trying to achieve adding -numa on dst?  
> 
> See above reply.
> 
> Thanks,
> Jianfeng

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

On 2/8/2018 5:51 PM, Igor Mammedov wrote:
> On Thu, 8 Feb 2018 09:20:45 +0800
> "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
>
>> On 2/7/2018 8:06 PM, Igor Mammedov wrote:
>>> On Wed, 7 Feb 2018 07:49:58 +0000
>>> "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
>>>   
>>>>> -----Original Message-----
>>>>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
>>>>> Sent: Tuesday, February 6, 2018 1:32 AM
>>>>> To: Igor Mammedov
>>>>> Cc: Tan, Jianfeng; qemu-devel@nongnu.org; Jason Wang; Maxime Coquelin;
>>>>> Michael S . Tsirkin
>>>>> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
>>>>> migration
>>>>>
>>>>> On 05/02/2018 18:15, Igor Mammedov wrote:
>>>>>>>> Then we would have both ram block named pc.ram:
>>>>>>>>                 Block Name    PSize
>>>>>>>>                         pc.ram     4 KiB
>>>>>>>>         /objects/pc.ram    2 MiB
>>>>>>>>
>>>>>>>> But I assume it's a corner case which not really happen.
>>>>>>> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.
>>>>>> perhaps we should fail object_add memory-backend-foo if it resulted
>>>>>> in creating ramblock with duplicate id
>>>>> Note that it would only be duplicated with Jianfeng's patch.  So I'm
>>>>> worried that his patch is worse than what we have now, because it may
>>>>> create conflicts with system RAMBlock names are not necessarily
>>>>> predictable.  Right now, -object creates RAMBlock names that are nicely
>>>>> constrained within /object/.
>>>> So we are trading off between the benefit it takes and the bad effect it brings.
>>>>
>>>> I'm wondering if the above example is the only failed case this patch leads to, i.e, only there is a ram named "pc.ram" and "/object/pc.ram" in the src VM?
>>>>
>>>> Please also consider the second option, that adding an alias name for RAMBlock; I'm not a big fan for that one, as it just pushes the problem to OpenStack/Libvirt.
>>> looking at provided CLI examples it's configuration issue on src and dst,
>>> one shall not mix numa and non numa variants.
>> Aha, that's another thing we also want to change. We now add numa at dst
>> node, only because without -numa, we cannot set up the file-baked memory
>> with share=on.
> then shouldn't you start src with the same -numa to begin with,
> changing such things on the fly is not supported.

Yes, you are describing the best practice. But we are originally trying 
to migrate without any changes to QEMU.

> General rule is that machine on dst has to be the same as on src.

OK.

> (with backend not visible to guest it possible might be changed
> but it's hard to tell if something would break due to that
> or would continue working in future since doesn't go along with above rule)
>
>> For example, "-m xG -mem-path xxx" can set up a file-baked memory, but
>> the file is not share-able.
> It could be solved by adding memdev option to machine,
> which would allow to specify backend object. And then on
> top make -mem-path alias new option to clean thing up.

Do you mean?

src vm: -m xG
dst vm: -m xG,memdev=pc.ram -object 
memory-backend-file,id=pc.ram,size=xG,mem-path=xxx,share=on ...


>
> But then again, You'd need to start both src and dst
> with the same option.

Yeah, got it :-)

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Thu, 8 Feb 2018 18:18:20 +0800
"Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:

> On 2/8/2018 5:51 PM, Igor Mammedov wrote:
> > On Thu, 8 Feb 2018 09:20:45 +0800
> > "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
> >  
> >> On 2/7/2018 8:06 PM, Igor Mammedov wrote:  
> >>> On Wed, 7 Feb 2018 07:49:58 +0000
> >>> "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
> >>>     
> >>>>> -----Original Message-----
> >>>>> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> >>>>> Sent: Tuesday, February 6, 2018 1:32 AM
> >>>>> To: Igor Mammedov
> >>>>> Cc: Tan, Jianfeng; qemu-devel@nongnu.org; Jason Wang; Maxime Coquelin;
> >>>>> Michael S . Tsirkin
> >>>>> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> >>>>> migration
> >>>>>
> >>>>> On 05/02/2018 18:15, Igor Mammedov wrote:  
> >>>>>>>> Then we would have both ram block named pc.ram:
> >>>>>>>>                 Block Name    PSize
> >>>>>>>>                         pc.ram     4 KiB
> >>>>>>>>         /objects/pc.ram    2 MiB
> >>>>>>>>
> >>>>>>>> But I assume it's a corner case which not really happen.  
> >>>>>>> Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.  
> >>>>>> perhaps we should fail object_add memory-backend-foo if it resulted
> >>>>>> in creating ramblock with duplicate id  
> >>>>> Note that it would only be duplicated with Jianfeng's patch.  So I'm
> >>>>> worried that his patch is worse than what we have now, because it may
> >>>>> create conflicts with system RAMBlock names are not necessarily
> >>>>> predictable.  Right now, -object creates RAMBlock names that are nicely
> >>>>> constrained within /object/.  
> >>>> So we are trading off between the benefit it takes and the bad effect it brings.
> >>>>
> >>>> I'm wondering if the above example is the only failed case this patch leads to, i.e, only there is a ram named "pc.ram" and "/object/pc.ram" in the src VM?
> >>>>
> >>>> Please also consider the second option, that adding an alias name for RAMBlock; I'm not a big fan for that one, as it just pushes the problem to OpenStack/Libvirt.  
> >>> looking at provided CLI examples it's configuration issue on src and dst,
> >>> one shall not mix numa and non numa variants.  
> >> Aha, that's another thing we also want to change. We now add numa at dst
> >> node, only because without -numa, we cannot set up the file-baked memory
> >> with share=on.  
> > then shouldn't you start src with the same -numa to begin with,
> > changing such things on the fly is not supported.  
> 
> Yes, you are describing the best practice. But we are originally trying 
> to migrate without any changes to QEMU.
> 
> > General rule is that machine on dst has to be the same as on src.  
> 
> OK.
> 
> > (with backend not visible to guest it possible might be changed
> > but it's hard to tell if something would break due to that
> > or would continue working in future since doesn't go along with above rule)
> >  
> >> For example, "-m xG -mem-path xxx" can set up a file-baked memory, but
> >> the file is not share-able.  
> > It could be solved by adding memdev option to machine,
> > which would allow to specify backend object. And then on
> > top make -mem-path alias new option to clean thing up.  
> 
> Do you mean?
> 
> src vm: -m xG
> dst vm: -m xG,memdev=pc.ram -object 
> memory-backend-file,id=pc.ram,size=xG,mem-path=xxx,share=on ...
Yep, I've meant something like it

src vm: -m xG,memdev=SHARED_RAM -object memory-backend-file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
dst vm: -m xG,memdev=SHARED_RAM -object memory-backend-file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on

or it could be -machine FOO,inital_ram_memdev=...
maybe making -M optional in this case as size is specified by backend

PS:
it's not a good idea to use QEMU's internal id 'pc.ram'
for user specified objects as it might cause problems.

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

Hi Igor and all,

> -----Original Message-----
> From: Igor Mammedov [mailto:imammedo@redhat.com]
> Sent: Thursday, February 8, 2018 7:30 PM
> To: Tan, Jianfeng
> Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-devel@nongnu.org;
> Michael S . Tsirkin
> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> migration
> 
[...]
> > > It could be solved by adding memdev option to machine,
> > > which would allow to specify backend object. And then on
> > > top make -mem-path alias new option to clean thing up.
> >
> > Do you mean?
> >
> > src vm: -m xG
> > dst vm: -m xG,memdev=pc.ram -object memory-backend-file,id=pc.ram,size=xG,mem-path=xxx,share=on ...
> Yep, I've meant something like it
> 
> src vm: -m xG,memdev=SHARED_RAM -object memory-backend-file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
> dst vm: -m xG,memdev=SHARED_RAM -object memory-backend-file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on

After a second thought, I find adding a backend for nonnuma pc RAM is roundabout way.

And we actually have an existing way to add a file-backed RAM: commit c902760fb25f ("Add option to use file backed guest memory"). Basically, this commit adds two options, --mem-path and --mem-prealloc, without specify a backend explicitly.

So how about just adding a new option --mem-share to decide if that's a private memory or shared memory? That seems much straightforward way to me; after this change we can migrate like:

src vm: -m xG
dst vm: -m xG --mem-path xxx --mem-share

Thanks,
Jianfeng

> 
> or it could be -machine FOO,inital_ram_memdev=...
> maybe making -M optional in this case as size is specified by backend
> 
> PS:
> it's not a good idea to use QEMU's internal id 'pc.ram'
> for user specified objects as it might cause problems.

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

> -----Original Message-----
> From: Tan, Jianfeng
> Sent: Saturday, February 24, 2018 11:08 AM
> To: 'Igor Mammedov'
> Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-devel@nongnu.org;
> Michael S . Tsirkin
> Subject: RE: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> migration
> 
> Hi Igor and all,
> 
> > -----Original Message-----
> > From: Igor Mammedov [mailto:imammedo@redhat.com]
> > Sent: Thursday, February 8, 2018 7:30 PM
> > To: Tan, Jianfeng
> > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-
> devel@nongnu.org;
> > Michael S . Tsirkin
> > Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > migration
> >
> [...]
> > > > It could be solved by adding memdev option to machine,
> > > > which would allow to specify backend object. And then on
> > > > top make -mem-path alias new option to clean thing up.
> > >
> > > Do you mean?
> > >
> > > src vm: -m xG
> > > dst vm: -m xG,memdev=pc.ram -object memory-backend-
> file,id=pc.ram,size=xG,mem-path=xxx,share=on ...
> > Yep, I've meant something like it
> >
> > src vm: -m xG,memdev=SHARED_RAM -object memory-backend-
> file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
> > dst vm: -m xG,memdev=SHARED_RAM -object memory-backend-
> file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
> 
> After a second thought, I find adding a backend for nonnuma pc RAM is
> roundabout way.
> 
> And we actually have an existing way to add a file-backed RAM: commit
> c902760fb25f ("Add option to use file backed guest memory"). Basically, this
> commit adds two options, --mem-path and --mem-prealloc, without specify
> a backend explicitly.
> 
> So how about just adding a new option --mem-share to decide if that's a
> private memory or shared memory? That seems much straightforward way
> to me; after this change we can migrate like:
> 
> src vm: -m xG
> dst vm: -m xG --mem-path xxx --mem-share
> 

Attach the patch FYI. Look forward to your thoughts.

diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 31612ca..5eaf367 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -127,6 +127,7 @@ extern bool enable_mlock;
 extern uint8_t qemu_extra_params_fw[2];
 extern QEMUClockType rtc_clock;
 extern const char *mem_path;
+extern int mem_share;
 extern int mem_prealloc;
 
 #define MAX_NODES 128
diff --git a/numa.c b/numa.c
index 7b9c33a..322289f 100644
--- a/numa.c
+++ b/numa.c
@@ -456,7 +456,7 @@ static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
     if (mem_path) {
 #ifdef __linux__
         Error *err = NULL;
-        memory_region_init_ram_from_file(mr, owner, name, ram_size, false,
+        memory_region_init_ram_from_file(mr, owner, name, ram_size, mem_share,
                                          mem_path, &err);
         if (err) {
             error_report_err(err);
diff --git a/qemu-options.hx b/qemu-options.hx
index 678181c..c968c53 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -389,6 +389,15 @@ STEXI
 Allocate guest RAM from a temporarily created file in @var{path}.
 ETEXI
 
+DEF("mem-share", 0, QEMU_OPTION_memshare,
+    "-mem-share   make guest memory shareable (use with -mem-path)\n",
+    QEMU_ARCH_ALL)
+STEXI
+@item -mem-share
+@findex -mem-share
+Make file-backed guest RAM shareable when using -mem-path.
+ETEXI
+
 DEF("mem-prealloc", 0, QEMU_OPTION_mem_prealloc,
     "-mem-prealloc   preallocate guest memory (use with -mem-path)\n",
     QEMU_ARCH_ALL)
diff --git a/vl.c b/vl.c
index 444b750..0ff06c2 100644
--- a/vl.c
+++ b/vl.c
@@ -140,6 +140,7 @@ int display_opengl;
 const char* keyboard_layout = NULL;
 ram_addr_t ram_size;
 const char *mem_path = NULL;
+int mem_share = 0;
 int mem_prealloc = 0; /* force preallocation of physical target memory */
 bool enable_mlock = false;
 int nb_nics;
@@ -3395,6 +3396,9 @@ int main(int argc, char **argv, char **envp)
             case QEMU_OPTION_mempath:
                 mem_path = optarg;
                 break;
+            case QEMU_OPTION_memshare:
+                mem_share = 1;
+                break;
             case QEMU_OPTION_mem_prealloc:
                 mem_prealloc = 1;
                 break;

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Sat, 24 Feb 2018 03:11:30 +0000
"Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:

> > -----Original Message-----
> > From: Tan, Jianfeng
> > Sent: Saturday, February 24, 2018 11:08 AM
> > To: 'Igor Mammedov'
> > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-devel@nongnu.org;
> > Michael S . Tsirkin
> > Subject: RE: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > migration
> > 
> > Hi Igor and all,
> >   
> > > -----Original Message-----
> > > From: Igor Mammedov [mailto:imammedo@redhat.com]
> > > Sent: Thursday, February 8, 2018 7:30 PM
> > > To: Tan, Jianfeng
> > > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-  
> > devel@nongnu.org;  
> > > Michael S . Tsirkin
> > > Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > > migration
> > >  
> > [...]  
> > > > > It could be solved by adding memdev option to machine,
> > > > > which would allow to specify backend object. And then on
> > > > > top make -mem-path alias new option to clean thing up.  
> > > >
> > > > Do you mean?
> > > >
> > > > src vm: -m xG
> > > > dst vm: -m xG,memdev=pc.ram -object memory-backend-  
> > file,id=pc.ram,size=xG,mem-path=xxx,share=on ...  
> > > Yep, I've meant something like it
> > >
> > > src vm: -m xG,memdev=SHARED_RAM -object memory-backend-  
> > file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on  
> > > dst vm: -m xG,memdev=SHARED_RAM -object memory-backend-  
> > file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
> > 
> > After a second thought, I find adding a backend for nonnuma pc RAM is
> > roundabout way.
> > 
> > And we actually have an existing way to add a file-backed RAM: commit
> > c902760fb25f ("Add option to use file backed guest memory"). Basically, this
> > commit adds two options, --mem-path and --mem-prealloc, without specify
> > a backend explicitly.
> > 
> > So how about just adding a new option --mem-share to decide if that's a
> > private memory or shared memory? That seems much straightforward way
Above options are legacy (which we can't remove for compat reasons),
their replacement is 'memory-backend-file' backend which has all of
the above including 'share' property.

So just add 'memdev' property to machine and reuse memory-backend-file
with it instead of duplicating functionality in the legacy code.

> > to me; after this change we can migrate like:
> > 
> > src vm: -m xG
> > dst vm: -m xG --mem-path xxx --mem-share
Even though it might work for now, that's still invalid configuration
for migration, src side must include the same
  "--mem-path xxx --mem-share"
options as dst.

It'd be better to fix management application to start QEMU
properly on SRC side.

 
> Attach the patch FYI. Look forward to your thoughts.
> 
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index 31612ca..5eaf367 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -127,6 +127,7 @@ extern bool enable_mlock;
>  extern uint8_t qemu_extra_params_fw[2];
>  extern QEMUClockType rtc_clock;
>  extern const char *mem_path;
> +extern int mem_share;
>  extern int mem_prealloc;
>  
>  #define MAX_NODES 128
> diff --git a/numa.c b/numa.c
> index 7b9c33a..322289f 100644
> --- a/numa.c
> +++ b/numa.c
> @@ -456,7 +456,7 @@ static void allocate_system_memory_nonnuma(MemoryRegion *mr, Object *owner,
>      if (mem_path) {
>  #ifdef __linux__
>          Error *err = NULL;
> -        memory_region_init_ram_from_file(mr, owner, name, ram_size, false,
> +        memory_region_init_ram_from_file(mr, owner, name, ram_size, mem_share,
>                                           mem_path, &err);
>          if (err) {
>              error_report_err(err);
> diff --git a/qemu-options.hx b/qemu-options.hx
> index 678181c..c968c53 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -389,6 +389,15 @@ STEXI
>  Allocate guest RAM from a temporarily created file in @var{path}.
>  ETEXI
>  
> +DEF("mem-share", 0, QEMU_OPTION_memshare,
> +    "-mem-share   make guest memory shareable (use with -mem-path)\n",
> +    QEMU_ARCH_ALL)
> +STEXI
> +@item -mem-share
> +@findex -mem-share
> +Make file-backed guest RAM shareable when using -mem-path.
> +ETEXI
> +
>  DEF("mem-prealloc", 0, QEMU_OPTION_mem_prealloc,
>      "-mem-prealloc   preallocate guest memory (use with -mem-path)\n",
>      QEMU_ARCH_ALL)
> diff --git a/vl.c b/vl.c
> index 444b750..0ff06c2 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -140,6 +140,7 @@ int display_opengl;
>  const char* keyboard_layout = NULL;
>  ram_addr_t ram_size;
>  const char *mem_path = NULL;
> +int mem_share = 0;
>  int mem_prealloc = 0; /* force preallocation of physical target memory */
>  bool enable_mlock = false;
>  int nb_nics;
> @@ -3395,6 +3396,9 @@ int main(int argc, char **argv, char **envp)
>              case QEMU_OPTION_mempath:
>                  mem_path = optarg;
>                  break;
> +            case QEMU_OPTION_memshare:
> +                mem_share = 1;
> +                break;
>              case QEMU_OPTION_mem_prealloc:
>                  mem_prealloc = 1;
>                  break;

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Paolo Bonzini]

On 26/02/2018 13:55, Igor Mammedov wrote:
>>> So how about just adding a new option --mem-share to decide if that's a
>>> private memory or shared memory? That seems much straightforward way
> Above options are legacy (which we can't remove for compat reasons),
> their replacement is 'memory-backend-file' backend which has all of
> the above including 'share' property.

More precisely, we have added "-object memory-backend-file" to avoid
proliferation of options related to memory.  Besides unifying the cases
of 1 and >1 NUMA node, using -object also has the advantage of
supporting memory hotplug.

You wrote "I find adding a backend for nonnuma pc RAM is roundabout way"
but basically the command line says "this VM has only one NUMA node,
backed by this memory object" which is a precise description of what the
VM memory looks like.

> So just add 'memdev' property to machine and reuse memory-backend-file
> with it instead of duplicating functionality in the legacy code.

That would however also have a different RAMBlock id, effectively
producing the same output as "-numa node,memdev=...".

I think this should be solved at the libvirt level.  Libvirt should
write in the migration XML cookie whether the VM is using -object or
-mem-path to declare its memory, and newly-started VMs should always use
-object.  This won't fix the problem for VMs that are already running,
but it will fix it the next time they are started.

Paolo

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

> -----Original Message-----
> From: Paolo Bonzini [mailto:pbonzini@redhat.com]
> Sent: Monday, February 26, 2018 10:43 PM
> To: Igor Mammedov; Tan, Jianfeng
> Cc: Jason Wang; Maxime Coquelin; qemu-devel@nongnu.org; Michael S .
> Tsirkin
> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> migration
> 
> On 26/02/2018 13:55, Igor Mammedov wrote:
> >>> So how about just adding a new option --mem-share to decide if that's a
> >>> private memory or shared memory? That seems much straightforward
> way
> > Above options are legacy (which we can't remove for compat reasons),
> > their replacement is 'memory-backend-file' backend which has all of
> > the above including 'share' property.
> 
> More precisely, we have added "-object memory-backend-file" to avoid
> proliferation of options related to memory.  Besides unifying the cases
> of 1 and >1 NUMA node, using -object also has the advantage of
> supporting memory hotplug.
> 
> You wrote "I find adding a backend for nonnuma pc RAM is roundabout way"
> but basically the command line says "this VM has only one NUMA node,
> backed by this memory object" which is a precise description of what the
> VM memory looks like.
> 
> > So just add 'memdev' property to machine and reuse memory-backend-file
> > with it instead of duplicating functionality in the legacy code.
> 
> That would however also have a different RAMBlock id, effectively
> producing the same output as "-numa node,memdev=...".

Is it possible that we force that RAMBlock id to be "pc.ram"?

> 
> I think this should be solved at the libvirt level.  Libvirt should
> write in the migration XML cookie whether the VM is using -object or
> -mem-path to declare its memory, and newly-started VMs should always use
> -object.  This won't fix the problem for VMs that are already running,
> but it will fix it the next time they are started.

In this case, we return to the start point :-)

Thanks,
Jianfeng

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

> -----Original Message-----
> From: Igor Mammedov [mailto:imammedo@redhat.com]
> Sent: Monday, February 26, 2018 8:56 PM
> To: Tan, Jianfeng
> Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-devel@nongnu.org;
> Michael S . Tsirkin
> Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> migration
> 
> On Sat, 24 Feb 2018 03:11:30 +0000
> "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
> 
> > > -----Original Message-----
> > > From: Tan, Jianfeng
> > > Sent: Saturday, February 24, 2018 11:08 AM
> > > To: 'Igor Mammedov'
> > > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-
> devel@nongnu.org;
> > > Michael S . Tsirkin
> > > Subject: RE: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > > migration
> > >
> > > Hi Igor and all,
> > >
> > > > -----Original Message-----
> > > > From: Igor Mammedov [mailto:imammedo@redhat.com]
> > > > Sent: Thursday, February 8, 2018 7:30 PM
> > > > To: Tan, Jianfeng
> > > > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-
> > > devel@nongnu.org;
> > > > Michael S . Tsirkin
> > > > Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > > > migration
> > > >
> > > [...]
> > > > > > It could be solved by adding memdev option to machine,
> > > > > > which would allow to specify backend object. And then on
> > > > > > top make -mem-path alias new option to clean thing up.
> > > > >
> > > > > Do you mean?
> > > > >
> > > > > src vm: -m xG
> > > > > dst vm: -m xG,memdev=pc.ram -object memory-backend-
> > > file,id=pc.ram,size=xG,mem-path=xxx,share=on ...
> > > > Yep, I've meant something like it
> > > >
> > > > src vm: -m xG,memdev=SHARED_RAM -object memory-backend-
> > > file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
> > > > dst vm: -m xG,memdev=SHARED_RAM -object memory-backend-
> > > file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
> > >
> > > After a second thought, I find adding a backend for nonnuma pc RAM is
> > > roundabout way.
> > >
> > > And we actually have an existing way to add a file-backed RAM: commit
> > > c902760fb25f ("Add option to use file backed guest memory"). Basically,
> this
> > > commit adds two options, --mem-path and --mem-prealloc, without
> specify
> > > a backend explicitly.
> > >
> > > So how about just adding a new option --mem-share to decide if that's a
> > > private memory or shared memory? That seems much straightforward
> way
> Above options are legacy (which we can't remove for compat reasons),
> their replacement is 'memory-backend-file' backend which has all of
> the above including 'share' property.

OK, such options are legacy. I've no idea of that. Thanks! That makes sense.

> 
> So just add 'memdev' property to machine and reuse memory-backend-file
> with it instead of duplicating functionality in the legacy code.

To "-m" or "-machine"?

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Tue, 27 Feb 2018 04:36:45 +0000
"Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:

> > -----Original Message-----
> > From: Igor Mammedov [mailto:imammedo@redhat.com]
> > Sent: Monday, February 26, 2018 8:56 PM
> > To: Tan, Jianfeng
> > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-devel@nongnu.org;
> > Michael S . Tsirkin
> > Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > migration
> > 
> > On Sat, 24 Feb 2018 03:11:30 +0000
> > "Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:
> >   
> > > > -----Original Message-----
> > > > From: Tan, Jianfeng
> > > > Sent: Saturday, February 24, 2018 11:08 AM
> > > > To: 'Igor Mammedov'
> > > > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-  
> > devel@nongnu.org;  
> > > > Michael S . Tsirkin
> > > > Subject: RE: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > > > migration
> > > >
> > > > Hi Igor and all,
> > > >  
> > > > > -----Original Message-----
> > > > > From: Igor Mammedov [mailto:imammedo@redhat.com]
> > > > > Sent: Thursday, February 8, 2018 7:30 PM
> > > > > To: Tan, Jianfeng
> > > > > Cc: Paolo Bonzini; Jason Wang; Maxime Coquelin; qemu-  
> > > > devel@nongnu.org;  
> > > > > Michael S . Tsirkin
> > > > > Subject: Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as
> > > > > migration
> > > > >  
> > > > [...]  
> > > > > > > It could be solved by adding memdev option to machine,
> > > > > > > which would allow to specify backend object. And then on
> > > > > > > top make -mem-path alias new option to clean thing up.  
> > > > > >
> > > > > > Do you mean?
> > > > > >
> > > > > > src vm: -m xG
> > > > > > dst vm: -m xG,memdev=pc.ram -object memory-backend-  
> > > > file,id=pc.ram,size=xG,mem-path=xxx,share=on ...  
> > > > > Yep, I've meant something like it
> > > > >
> > > > > src vm: -m xG,memdev=SHARED_RAM -object memory-backend-  
> > > > file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on  
> > > > > dst vm: -m xG,memdev=SHARED_RAM -object memory-backend-  
> > > > file,id=SHARED_RAM,size=xG,mem-path=xxx,share=on
> > > >
> > > > After a second thought, I find adding a backend for nonnuma pc RAM is
> > > > roundabout way.
> > > >
> > > > And we actually have an existing way to add a file-backed RAM: commit
> > > > c902760fb25f ("Add option to use file backed guest memory"). Basically,  
> > this  
> > > > commit adds two options, --mem-path and --mem-prealloc, without  
> > specify  
> > > > a backend explicitly.
> > > >
> > > > So how about just adding a new option --mem-share to decide if that's a
> > > > private memory or shared memory? That seems much straightforward  
> > way
> > Above options are legacy (which we can't remove for compat reasons),
> > their replacement is 'memory-backend-file' backend which has all of
> > the above including 'share' property.  
> 
> OK, such options are legacy. I've no idea of that. Thanks! That makes sense.
> 
> > 
> > So just add 'memdev' property to machine and reuse memory-backend-file
> > with it instead of duplicating functionality in the legacy code.  
> 
> To "-m" or "-machine"?
"-machine", I plan to convert -m to machine options as well (it's somewhere on my TODO list)

but as Paolo pointed out that will help only to avoid using -numa
and won't help with your case, which should be solved at upper layer
(i.e. starting QEMU on src with shared memory from the begging).

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Dr. David Alan Gilbert]

* Igor Mammedov (imammedo@redhat.com) wrote:
> On Mon, 5 Feb 2018 17:19:09 +0100
> Paolo Bonzini <pbonzini@redhat.com> wrote:
> 
> > On 05/02/2018 17:12, Tan, Jianfeng wrote:
> > > Hi Paolo,
> > > 
> > > On 2/5/2018 11:53 PM, Paolo Bonzini wrote:  
> > >> On 05/02/2018 15:58, Jianfeng Tan wrote:  
> > >>> Here are some options to fix this:
> > >>>
> > >>> 1. When we do ram name comparison, we truncate the prefix as this
> > >>> patch shows.
> > >>> It cannot cover the corner case: the source VM could have two ram blocks
> > >>> with name of "pc.ram" and "/object/pc.ram".  
> > >> That shouldn't happen ("pc.ram" exists even in the "-numa
> > >> node,memdev=..." case, but it has no RAM block).  
> > > 
> > > Suppose we have a VM started with "-m xG", and then hot plugged with a
> > > ram block:
> > >   (qemu) object_add
> > > memory-backend-file,id=pc.ram,size=1G,mem-path=/dev/hugepages
> > >   (qemu) device_add pc-dimm,id=pc.ram,memdev=pc.ram
> > > 
> > > Then we would have both ram block named pc.ram:
> > >               Block Name    PSize
> > >                       pc.ram     4 KiB
> > >       /objects/pc.ram    2 MiB
> > > 
> > > But I assume it's a corner case which not really happen.  
> > 
> > Yeah, you're right. :/  I hadn't thought of hotplug.  It can happen indeed.
> perhaps we should fail object_add memory-backend-foo if it resulted
> in creating ramblock with duplicate id
>  
That's probably not a bad idea;  I thought I'd hit a simpliar problem a
while ago; I'd ended up (through a different problem) of having
RAMBlocks with empty names and ended up with two of them.

Dave

> > 
> > >> However, note that
> > >>
> > >>    -m xG -numa node,memdev=pc.ram \
> > >>    -object memory-backend-file,id=pc.ram,...
> > >>
> > >> works for both vhost-kernel and vhost-user, so I'd rather consider this
> > >> a configuration problem and not do anything.  
> > > 
> > > That configuration indeed works for both. But in the production env,
> > > lots of VMs are already started with previous mem config. If we do
> > > nothing, it will take a long time (shutdown/start for each VM) to
> > > migrate to the new setup. This patch is to make this process more smooth
> > > without any bad effect if possible.  
> > 
> > I understand.  However it's not as bad as "there's no possibility at all
> > to migrate from vhost-kernel to vhost-user".  There are cases that are
> > more problematic: for example, there's no possibility at all to add
> > memory NUMA policy during a live migration, unless -object
> > memory-backend-* was used on the source.
> > 
> > Paolo
> > 
> 
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[no-reply]

Hi,

This series failed docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

Type: series
Message-id: 1517842735-9011-1-git-send-email-jianfeng.tan@intel.com
Subject: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

=== TEST SCRIPT BEGIN ===
#!/bin/bash
set -e
git submodule update --init dtc
# Let docker tests dump environment info
export SHOW_ENV=1
export J=8
time make docker-test-mingw@fedora
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
2a645f0bd4 exec: eliminate ram naming issue as migration

=== OUTPUT BEGIN ===
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-83d1w41a/src/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
  BUILD   fedora
  GEN     /var/tmp/patchew-tester-tmp-83d1w41a/src/docker-src.2018-02-05-11.09.50.15886/qemu.tar
Cloning into '/var/tmp/patchew-tester-tmp-83d1w41a/src/docker-src.2018-02-05-11.09.50.15886/qemu.tar.vroot'...
done.
Your branch is up-to-date with 'origin/test'.
Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
Cloning into '/var/tmp/patchew-tester-tmp-83d1w41a/src/docker-src.2018-02-05-11.09.50.15886/qemu.tar.vroot/dtc'...
Submodule path 'dtc': checked out 'e54388015af1fb4bf04d0bca99caba1074d9cc42'
Submodule 'ui/keycodemapdb' (git://git.qemu.org/keycodemapdb.git) registered for path 'ui/keycodemapdb'
Cloning into '/var/tmp/patchew-tester-tmp-83d1w41a/src/docker-src.2018-02-05-11.09.50.15886/qemu.tar.vroot/ui/keycodemapdb'...
Submodule path 'ui/keycodemapdb': checked out '10739aa26051a5d49d88132604539d3ed085e72e'
  COPY    RUNNER
    RUN test-mingw in qemu:fedora 
Packages installed:
PyYAML-3.11-13.fc25.x86_64
SDL-devel-1.2.15-21.fc24.x86_64
bc-1.06.95-16.fc24.x86_64
bison-3.0.4-4.fc24.x86_64
bzip2-1.0.6-21.fc25.x86_64
ccache-3.3.4-1.fc25.x86_64
clang-3.9.1-2.fc25.x86_64
findutils-4.6.0-8.fc25.x86_64
flex-2.6.0-3.fc25.x86_64
gcc-6.4.1-1.fc25.x86_64
gcc-c++-6.4.1-1.fc25.x86_64
gettext-0.19.8.1-3.fc25.x86_64
git-2.9.5-3.fc25.x86_64
glib2-devel-2.50.3-1.fc25.x86_64
hostname-3.15-8.fc25.x86_64
libaio-devel-0.3.110-6.fc24.x86_64
libasan-6.4.1-1.fc25.x86_64
libfdt-devel-1.4.2-1.fc25.x86_64
libubsan-6.4.1-1.fc25.x86_64
make-4.1-6.fc25.x86_64
mingw32-SDL-1.2.15-7.fc24.noarch
mingw32-bzip2-1.0.6-7.fc24.noarch
mingw32-curl-7.47.0-1.fc24.noarch
mingw32-glib2-2.50.3-1.fc25.noarch
mingw32-gmp-6.1.1-1.fc25.noarch
mingw32-gnutls-3.5.5-2.fc25.noarch
mingw32-gtk2-2.24.31-2.fc25.noarch
mingw32-gtk3-3.22.17-1.fc25.noarch
mingw32-libjpeg-turbo-1.5.1-1.fc25.noarch
mingw32-libpng-1.6.27-1.fc25.noarch
mingw32-libssh2-1.4.3-5.fc24.noarch
mingw32-libtasn1-4.9-1.fc25.noarch
mingw32-nettle-3.3-1.fc25.noarch
mingw32-pixman-0.34.0-1.fc25.noarch
mingw32-pkg-config-0.28-6.fc24.x86_64
mingw64-SDL-1.2.15-7.fc24.noarch
mingw64-bzip2-1.0.6-7.fc24.noarch
mingw64-curl-7.47.0-1.fc24.noarch
mingw64-glib2-2.50.3-1.fc25.noarch
mingw64-gmp-6.1.1-1.fc25.noarch
mingw64-gnutls-3.5.5-2.fc25.noarch
mingw64-gtk2-2.24.31-2.fc25.noarch
mingw64-gtk3-3.22.17-1.fc25.noarch
mingw64-libjpeg-turbo-1.5.1-1.fc25.noarch
mingw64-libpng-1.6.27-1.fc25.noarch
mingw64-libssh2-1.4.3-5.fc24.noarch
mingw64-libtasn1-4.9-1.fc25.noarch
mingw64-nettle-3.3-1.fc25.noarch
mingw64-pixman-0.34.0-1.fc25.noarch
mingw64-pkg-config-0.28-6.fc24.x86_64
nettle-devel-3.3-1.fc25.x86_64
package python2 is not installed
perl-5.24.3-389.fc25.x86_64
pixman-devel-0.34.0-2.fc24.x86_64
sparse-0.5.0-10.fc25.x86_64
tar-1.29-3.fc25.x86_64
which-2.21-1.fc25.x86_64
zlib-devel-1.2.8-10.fc24.x86_64

Environment variables:
PACKAGES=ccache gettext git tar PyYAML sparse flex bison python2 bzip2 hostname     glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel     gcc gcc-c++ clang make perl which bc findutils libaio-devel     nettle-devel libasan libubsan     mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config     mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1     mingw32-libjpeg-turbo mingw32-libpng mingw32-curl mingw32-libssh2     mingw32-bzip2     mingw64-pixman mingw64-glib2 mingw64-gmp mingw64-SDL mingw64-pkg-config     mingw64-gtk2 mingw64-gtk3 mingw64-gnutls mingw64-nettle mingw64-libtasn1     mingw64-libjpeg-turbo mingw64-libpng mingw64-curl mingw64-libssh2     mingw64-bzip2
HOSTNAME=f36211f496cf
MAKEFLAGS= -j8
J=8
CCACHE_DIR=/var/tmp/ccache
EXTRA_CONFIGURE_OPTS=
V=
SHOW_ENV=1
PATH=/usr/lib/ccache:/usr/lib64/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
TARGET_LIST=
FGC=f25
SHLVL=1
HOME=/root
TEST_DIR=/tmp/qemu-test
DISTTAG=f25container
FEATURES=mingw clang pyyaml asan dtc
DEBUG=
_=/usr/bin/env

Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent --with-sdlabi=1.2 --with-gtkabi=2.0
Install prefix    /tmp/qemu-test/install
BIOS directory    /tmp/qemu-test/install
firmware path     /tmp/qemu-test/install/share/qemu-firmware
binary directory  /tmp/qemu-test/install
library directory /tmp/qemu-test/install/lib
module directory  /tmp/qemu-test/install/lib
libexec directory /tmp/qemu-test/install/libexec
include directory /tmp/qemu-test/install/include
config directory  /tmp/qemu-test/install
local state directory   queried at runtime
Windows SDK       no
Source path       /tmp/qemu-test/src
GIT binary        git
GIT submodules    
C compiler        x86_64-w64-mingw32-gcc
Host C compiler   cc
C++ compiler      x86_64-w64-mingw32-g++
Objective-C compiler clang
ARFLAGS           rv
CFLAGS            -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -g 
QEMU_CFLAGS       -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/pixman-1  -I$(SRC_PATH)/dtc/libfdt -Werror -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -m64 -mcx16 -mthreads -D__USE_MINGW_ANSI_STDIO=1 -DWIN32_LEAN_AND_MEAN -DWINVER=0x501 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv  -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include  -I/usr/x86_64-w64-mingw32/sys-root/mingw/include   -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 
LDFLAGS           -Wl,--nxcompat -Wl,--no-seh -Wl,--dynamicbase -Wl,--warn-common -m64 -g 
make              make
install           install
python            python -B
smbd              /usr/sbin/smbd
module support    no
host CPU          x86_64
host big endian   no
target list       x86_64-softmmu aarch64-softmmu
gprof enabled     no
sparse enabled    no
strip binaries    yes
profiler          no
static build      no
SDL support       yes (1.2.15)
GTK support       yes (2.24.31)
GTK GL support    no
VTE support       no 
TLS priority      NORMAL
GNUTLS support    yes
GNUTLS rnd        yes
libgcrypt         no
libgcrypt kdf     no
nettle            yes (3.3)
nettle kdf        yes
libtasn1          yes
curses support    no
virgl support     no
curl support      yes
mingw32 support   yes
Audio drivers     dsound
Block whitelist (rw) 
Block whitelist (ro) 
VirtFS support    no
Multipath support no
VNC support       yes
VNC SASL support  no
VNC JPEG support  yes
VNC PNG support   yes
xen support       no
brlapi support    no
bluez  support    no
Documentation     no
PIE               no
vde support       no
netmap support    no
Linux AIO support no
ATTR/XATTR support no
Install blobs     yes
KVM support       no
HAX support       yes
HVF support       no
TCG support       yes
TCG debug enabled no
TCG interpreter   no
malloc trim support no
RDMA support      no
fdt support       yes
preadv support    no
fdatasync         no
madvise           no
posix_madvise     no
libcap-ng support no
vhost-net support no
vhost-scsi support no
vhost-vsock support no
vhost-user support no
Trace backends    simple
Trace output file trace-<pid>
spice support     no 
rbd support       no
xfsctl support    no
smartcard support no
libusb            no
usb net redir     no
OpenGL support    no
OpenGL dmabufs    no
libiscsi support  no
libnfs support    no
build guest agent yes
QGA VSS support   no
QGA w32 disk info yes
QGA MSI support   no
seccomp support   no
coroutine backend win32
coroutine pool    yes
debug stack usage no
crypto afalg      no
GlusterFS support no
gcov              gcov
gcov enabled      no
TPM support       yes
libssh2 support   yes
TPM passthrough   no
TPM emulator      no
QOM debugging     yes
Live block migration yes
lzo support       no
snappy support    no
bzip2 support     yes
NUMA host support no
libxml2           no
tcmalloc support  no
jemalloc support  no
avx2 optimization yes
replication support yes
VxHS block device no
capstone          no

WARNING: Use of GTK 2.0 is deprecated and will be removed in
WARNING: future releases. Please switch to using GTK 3.0

WARNING: Use of SDL 1.2 is deprecated and will be removed in
WARNING: future releases. Please switch to using SDL 2.0
  GEN     x86_64-softmmu/config-devices.mak.tmp
  GEN     aarch64-softmmu/config-devices.mak.tmp
  GEN     config-host.h
  GEN     qemu-options.def
  GEN     qmp-commands.h
  GEN     qapi-visit.h
  GEN     qapi-types.h
  GEN     qapi-event.h
  GEN     x86_64-softmmu/config-devices.mak
  GEN     qmp-marshal.c
  GEN     aarch64-softmmu/config-devices.mak
  GEN     qapi-types.c
  GEN     qapi-visit.c
  GEN     qapi-event.c
  GEN     qmp-introspect.h
  GEN     qmp-introspect.c
  GEN     trace/generated-tcg-tracers.h
  GEN     trace/generated-helpers-wrappers.h
  GEN     trace/generated-helpers.h
  GEN     trace/generated-helpers.c
  GEN     module_block.h
  GEN     ui/input-keymap-atset1-to-qcode.c
  GEN     ui/input-keymap-linux-to-qcode.c
  GEN     ui/input-keymap-qcode-to-atset1.c
  GEN     ui/input-keymap-qcode-to-atset2.c
  GEN     ui/input-keymap-qcode-to-atset3.c
  GEN     ui/input-keymap-qcode-to-linux.c
  GEN     ui/input-keymap-qcode-to-qnum.c
  GEN     ui/input-keymap-qcode-to-sun.c
  GEN     ui/input-keymap-qnum-to-qcode.c
  GEN     ui/input-keymap-usb-to-qcode.c
  GEN     ui/input-keymap-win32-to-qcode.c
  GEN     ui/input-keymap-x11-to-qcode.c
  GEN     ui/input-keymap-xorgevdev-to-qcode.c
  GEN     ui/input-keymap-xorgkbd-to-qcode.c
  GEN     ui/input-keymap-xorgxquartz-to-qcode.c
  GEN     ui/input-keymap-xorgxwin-to-qcode.c
  GEN     tests/test-qapi-types.h
  GEN     tests/test-qapi-visit.h
  GEN     tests/test-qmp-commands.h
  GEN     tests/test-qapi-event.h
  GEN     tests/test-qmp-introspect.h
  GEN     trace-root.h
  GEN     util/trace.h
  GEN     crypto/trace.h
  GEN     io/trace.h
  GEN     migration/trace.h
  GEN     block/trace.h
  GEN     chardev/trace.h
  GEN     hw/block/trace.h
  GEN     hw/block/dataplane/trace.h
  GEN     hw/char/trace.h
  GEN     hw/intc/trace.h
  GEN     hw/net/trace.h
  GEN     hw/virtio/trace.h
  GEN     hw/audio/trace.h
  GEN     hw/misc/trace.h
  GEN     hw/usb/trace.h
  GEN     hw/scsi/trace.h
  GEN     hw/nvram/trace.h
  GEN     hw/display/trace.h
  GEN     hw/input/trace.h
  GEN     hw/timer/trace.h
  GEN     hw/dma/trace.h
  GEN     hw/sparc/trace.h
  GEN     hw/sparc64/trace.h
  GEN     hw/sd/trace.h
  GEN     hw/isa/trace.h
  GEN     hw/mem/trace.h
  GEN     hw/i386/trace.h
  GEN     hw/i386/xen/trace.h
  GEN     hw/9pfs/trace.h
  GEN     hw/ppc/trace.h
  GEN     hw/pci/trace.h
  GEN     hw/pci-host/trace.h
  GEN     hw/s390x/trace.h
  GEN     hw/vfio/trace.h
  GEN     hw/acpi/trace.h
  GEN     hw/arm/trace.h
  GEN     hw/alpha/trace.h
  GEN     hw/hppa/trace.h
  GEN     hw/xen/trace.h
  GEN     hw/ide/trace.h
  GEN     ui/trace.h
  GEN     audio/trace.h
  GEN     net/trace.h
  GEN     target/arm/trace.h
  GEN     target/i386/trace.h
  GEN     target/mips/trace.h
  GEN     target/sparc/trace.h
  GEN     target/s390x/trace.h
  GEN     target/ppc/trace.h
  GEN     qom/trace.h
  GEN     linux-user/trace.h
  GEN     qapi/trace.h
  GEN     accel/tcg/trace.h
  GEN     accel/kvm/trace.h
  GEN     nbd/trace.h
  GEN     trace-root.c
  GEN     scsi/trace.h
  GEN     util/trace.c
  GEN     crypto/trace.c
  GEN     io/trace.c
  GEN     migration/trace.c
  GEN     block/trace.c
  GEN     chardev/trace.c
  GEN     hw/block/trace.c
  GEN     hw/block/dataplane/trace.c
  GEN     hw/char/trace.c
  GEN     hw/intc/trace.c
  GEN     hw/net/trace.c
  GEN     hw/virtio/trace.c
  GEN     hw/audio/trace.c
  GEN     hw/misc/trace.c
  GEN     hw/usb/trace.c
  GEN     hw/scsi/trace.c
  GEN     hw/nvram/trace.c
  GEN     hw/display/trace.c
  GEN     hw/input/trace.c
  GEN     hw/timer/trace.c
  GEN     hw/dma/trace.c
  GEN     hw/sparc/trace.c
  GEN     hw/sparc64/trace.c
  GEN     hw/sd/trace.c
  GEN     hw/isa/trace.c
  GEN     hw/mem/trace.c
  GEN     hw/i386/trace.c
  GEN     hw/i386/xen/trace.c
  GEN     hw/9pfs/trace.c
  GEN     hw/ppc/trace.c
  GEN     hw/pci/trace.c
  GEN     hw/pci-host/trace.c
  GEN     hw/s390x/trace.c
  GEN     hw/vfio/trace.c
  GEN     hw/acpi/trace.c
  GEN     hw/arm/trace.c
  GEN     hw/alpha/trace.c
  GEN     hw/hppa/trace.c
  GEN     hw/xen/trace.c
  GEN     hw/ide/trace.c
  GEN     ui/trace.c
  GEN     audio/trace.c
  GEN     net/trace.c
  GEN     target/arm/trace.c
  GEN     target/i386/trace.c
  GEN     target/mips/trace.c
  GEN     target/sparc/trace.c
  GEN     target/s390x/trace.c
  GEN     target/ppc/trace.c
  GEN     qom/trace.c
  GEN     linux-user/trace.c
  GEN     qapi/trace.c
  GEN     accel/tcg/trace.c
  GEN     accel/kvm/trace.c
  GEN     nbd/trace.c
  GEN     config-all-devices.mak
  GEN     scsi/trace.c
	 DEP /tmp/qemu-test/src/dtc/tests/dumptrees.c
	 DEP /tmp/qemu-test/src/dtc/tests/trees.S
	 DEP /tmp/qemu-test/src/dtc/tests/testutils.c
	 DEP /tmp/qemu-test/src/dtc/tests/value-labels.c
	 DEP /tmp/qemu-test/src/dtc/tests/asm_tree_dump.c
	 DEP /tmp/qemu-test/src/dtc/tests/truncated_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/check_path.c
	 DEP /tmp/qemu-test/src/dtc/tests/overlay_bad_fixup.c
	 DEP /tmp/qemu-test/src/dtc/tests/overlay.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_iterate.c
	 DEP /tmp/qemu-test/src/dtc/tests/property_iterate.c
	 DEP /tmp/qemu-test/src/dtc/tests/integer-expressions.c
	 DEP /tmp/qemu-test/src/dtc/tests/utilfdt_test.c
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset_aliases.c
	 DEP /tmp/qemu-test/src/dtc/tests/add_subnode_with_nops.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_unordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtb_reverse.c
	 DEP /tmp/qemu-test/src/dtc/tests/dtbs_equal_ordered.c
	 DEP /tmp/qemu-test/src/dtc/tests/extra-terminating-null.c
	 DEP /tmp/qemu-test/src/dtc/tests/incbin.c
	 DEP /tmp/qemu-test/src/dtc/tests/boot-cpuid.c
	 DEP /tmp/qemu-test/src/dtc/tests/path-references.c
	 DEP /tmp/qemu-test/src/dtc/tests/phandle_format.c
	 DEP /tmp/qemu-test/src/dtc/tests/references.c
	 DEP /tmp/qemu-test/src/dtc/tests/string_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/propname_escapes.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop2.c
	 DEP /tmp/qemu-test/src/dtc/tests/appendprop1.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/del_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/set_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/rw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/open_pack.c
	 DEP /tmp/qemu-test/src/dtc/tests/nopulate.c
	 DEP /tmp/qemu-test/src/dtc/tests/mangle-layout.c
	 DEP /tmp/qemu-test/src/dtc/tests/move_and_save.c
	 DEP /tmp/qemu-test/src/dtc/tests/sw_tree1.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/nop_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/setprop_inplace.c
	 DEP /tmp/qemu-test/src/dtc/tests/stringlist.c
	 DEP /tmp/qemu-test/src/dtc/tests/addr_size_cells.c
	 DEP /tmp/qemu-test/src/dtc/tests/notfound.c
	 DEP /tmp/qemu-test/src/dtc/tests/sized_cells.c
	 DEP /tmp/qemu-test/src/dtc/tests/char_literal.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_alias.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_check_compatible.c
	 DEP /tmp/qemu-test/src/dtc/tests/node_offset_by_prop_value.c
	 DEP /tmp/qemu-test/src/dtc/tests/parent_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/supernode_atdepth_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_path.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_phandle.c
	 DEP /tmp/qemu-test/src/dtc/tests/getprop.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_name.c
	 DEP /tmp/qemu-test/src/dtc/tests/path_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/find_property.c
	 DEP /tmp/qemu-test/src/dtc/tests/subnode_offset.c
	 DEP /tmp/qemu-test/src/dtc/tests/root_node.c
	 DEP /tmp/qemu-test/src/dtc/tests/get_mem_rsv.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_overlay.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_addresses.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_empty_tree.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_strerror.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_rw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_sw.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_wip.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt_ro.c
	 DEP /tmp/qemu-test/src/dtc/libfdt/fdt.c
	 DEP /tmp/qemu-test/src/dtc/util.c
	 DEP /tmp/qemu-test/src/dtc/fdtoverlay.c
	 DEP /tmp/qemu-test/src/dtc/fdtput.c
	 DEP /tmp/qemu-test/src/dtc/fdtget.c
	 DEP /tmp/qemu-test/src/dtc/fdtdump.c
	 LEX convert-dtsv0-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/srcpos.c
	 BISON dtc-parser.tab.c
	 LEX dtc-lexer.lex.c
	 DEP /tmp/qemu-test/src/dtc/treesource.c
	 DEP /tmp/qemu-test/src/dtc/livetree.c
	 DEP /tmp/qemu-test/src/dtc/fstree.c
	 DEP /tmp/qemu-test/src/dtc/flattree.c
	 DEP /tmp/qemu-test/src/dtc/dtc.c
	 DEP /tmp/qemu-test/src/dtc/checks.c
	 DEP /tmp/qemu-test/src/dtc/data.c
	 DEP convert-dtsv0-lexer.lex.c
	 DEP dtc-parser.tab.c
	 DEP dtc-lexer.lex.c
	CHK version_gen.h
	UPD version_gen.h
	 DEP /tmp/qemu-test/src/dtc/util.c
	 CC libfdt/fdt.o
	 CC libfdt/fdt_ro.o
	 CC libfdt/fdt_wip.o
	 CC libfdt/fdt_sw.o
	 CC libfdt/fdt_rw.o
	 CC libfdt/fdt_strerror.o
	 CC libfdt/fdt_addresses.o
	 CC libfdt/fdt_empty_tree.o
	 CC libfdt/fdt_overlay.o
	 AR libfdt/libfdt.a
x86_64-w64-mingw32-ar: creating libfdt/libfdt.a
a - libfdt/fdt.o
a - libfdt/fdt_ro.o
a - libfdt/fdt_wip.o
a - libfdt/fdt_sw.o
a - libfdt/fdt_rw.o
a - libfdt/fdt_strerror.o
a - libfdt/fdt_empty_tree.o
a - libfdt/fdt_addresses.o
a - libfdt/fdt_overlay.o
  RC      version.o
  GEN     qga/qapi-generated/qga-qapi-types.h
  GEN     qga/qapi-generated/qga-qapi-visit.h
  GEN     qga/qapi-generated/qga-qapi-types.c
  GEN     qga/qapi-generated/qga-qapi-visit.c
  GEN     qga/qapi-generated/qga-qmp-commands.h
  GEN     qga/qapi-generated/qga-qmp-marshal.c
  CC      qmp-introspect.o
  CC      qapi-types.o
  CC      qapi-visit.o
  CC      qapi-event.o
  CC      qapi/qapi-visit-core.o
  CC      qapi/qapi-dealloc-visitor.o
  CC      qapi/qobject-input-visitor.o
  CC      qapi/qobject-output-visitor.o
  CC      qapi/qmp-registry.o
  CC      qapi/string-output-visitor.o
  CC      qapi/qmp-dispatch.o
  CC      qapi/string-input-visitor.o
  CC      qapi/opts-visitor.o
  CC      qapi/qapi-clone-visitor.o
  CC      qapi/qmp-event.o
  CC      qapi/qapi-util.o
  CC      qobject/qnull.o
  CC      qobject/qnum.o
  CC      qobject/qstring.o
  CC      qobject/qlist.o
  CC      qobject/qdict.o
  CC      qobject/qbool.o
  CC      qobject/qlit.o
  CC      qobject/qjson.o
  CC      qobject/json-lexer.o
  CC      qobject/json-streamer.o
  CC      qobject/qobject.o
  CC      qobject/json-parser.o
  CC      trace/simple.o
  CC      trace/control.o
  CC      trace/qmp.o
  CC      util/osdep.o
  CC      util/cutils.o
  CC      util/unicode.o
  CC      util/qemu-timer-common.o
  CC      util/bufferiszero.o
  CC      util/lockcnt.o
  CC      util/aiocb.o
  CC      util/async.o
  CC      util/thread-pool.o
  CC      util/qemu-timer.o
  CC      util/main-loop.o
  CC      util/iohandler.o
  CC      util/aio-win32.o
  CC      util/event_notifier-win32.o
  CC      util/oslib-win32.o
  CC      util/qemu-thread-win32.o
  CC      util/envlist.o
  CC      util/path.o
  CC      util/module.o
  CC      util/host-utils.o
  CC      util/bitmap.o
  CC      util/bitops.o
  CC      util/hbitmap.o
  CC      util/fifo8.o
  CC      util/acl.o
  CC      util/cacheinfo.o
  CC      util/error.o
  CC      util/qemu-error.o
  CC      util/id.o
  CC      util/iov.o
  CC      util/qemu-config.o
  CC      util/qemu-sockets.o
  CC      util/uri.o
  CC      util/notify.o
  CC      util/qemu-option.o
  CC      util/qemu-progress.o
  CC      util/keyval.o
  CC      util/hexdump.o
  CC      util/crc32c.o
  CC      util/uuid.o
  CC      util/throttle.o
  CC      util/getauxval.o
  CC      util/readline.o
  CC      util/rcu.o
  CC      util/qemu-coroutine.o
  CC      util/qemu-coroutine-lock.o
  CC      util/qemu-coroutine-io.o
  CC      util/qemu-coroutine-sleep.o
  CC      util/coroutine-win32.o
  CC      util/buffer.o
  CC      util/timed-average.o
  CC      util/base64.o
  CC      util/log.o
  CC      util/pagesize.o
  CC      util/qdist.o
  CC      util/qht.o
  CC      util/range.o
  CC      util/stats64.o
  CC      util/systemd.o
  CC      trace-root.o
  CC      util/trace.o
  CC      crypto/trace.o
  CC      io/trace.o
  CC      migration/trace.o
  CC      block/trace.o
  CC      chardev/trace.o
  CC      hw/block/trace.o
  CC      hw/block/dataplane/trace.o
  CC      hw/char/trace.o
  CC      hw/intc/trace.o
  CC      hw/net/trace.o
  CC      hw/virtio/trace.o
  CC      hw/audio/trace.o
  CC      hw/misc/trace.o
  CC      hw/usb/trace.o
  CC      hw/scsi/trace.o
  CC      hw/nvram/trace.o
  CC      hw/display/trace.o
  CC      hw/input/trace.o
  CC      hw/timer/trace.o
  CC      hw/dma/trace.o
  CC      hw/sparc/trace.o
  CC      hw/sparc64/trace.o
  CC      hw/sd/trace.o
  CC      hw/isa/trace.o
  CC      hw/mem/trace.o
  CC      hw/i386/trace.o
  CC      hw/i386/xen/trace.o
  CC      hw/9pfs/trace.o
  CC      hw/ppc/trace.o
  CC      hw/pci/trace.o
  CC      hw/pci-host/trace.o
  CC      hw/s390x/trace.o
  CC      hw/acpi/trace.o
  CC      hw/vfio/trace.o
  CC      hw/arm/trace.o
  CC      hw/alpha/trace.o
  CC      hw/hppa/trace.o
  CC      hw/xen/trace.o
  CC      hw/ide/trace.o
  CC      ui/trace.o
  CC      audio/trace.o
  CC      net/trace.o
  CC      target/arm/trace.o
  CC      target/i386/trace.o
  CC      target/mips/trace.o
  CC      target/sparc/trace.o
  CC      target/s390x/trace.o
  CC      target/ppc/trace.o
  CC      qom/trace.o
  CC      linux-user/trace.o
  CC      qapi/trace.o
  CC      accel/tcg/trace.o
  CC      accel/kvm/trace.o
  CC      nbd/trace.o
  CC      scsi/trace.o
  CC      crypto/pbkdf-stub.o
  CC      stubs/arch-query-cpu-def.o
  CC      stubs/arch-query-cpu-model-expansion.o
  CC      stubs/arch-query-cpu-model-comparison.o
  CC      stubs/arch-query-cpu-model-baseline.o
  CC      stubs/bdrv-next-monitor-owned.o
  CC      stubs/blk-commit-all.o
  CC      stubs/blockdev-close-all-bdrv-states.o
  CC      stubs/clock-warp.o
  CC      stubs/cpu-get-clock.o
  CC      stubs/cpu-get-icount.o
  CC      stubs/dump.o
  CC      stubs/error-printf.o
  CC      stubs/fdset.o
  CC      stubs/gdbstub.o
  CC      stubs/get-vm-name.o
  CC      stubs/iothread.o
  CC      stubs/iothread-lock.o
  CC      stubs/is-daemonized.o
  CC      stubs/machine-init-done.o
  CC      stubs/migr-blocker.o
  CC      stubs/change-state-handler.o
  CC      stubs/monitor.o
  CC      stubs/notify-event.o
  CC      stubs/qtest.o
  CC      stubs/replay.o
  CC      stubs/runstate-check.o
  CC      stubs/set-fd-handler.o
  CC      stubs/slirp.o
  CC      stubs/sysbus.o
  CC      stubs/tpm.o
  CC      stubs/trace-control.o
  CC      stubs/uuid.o
  CC      stubs/vm-stop.o
  CC      stubs/vmstate.o
  CC      stubs/fd-register.o
  CC      stubs/qmp_pc_dimm.o
  CC      stubs/target-monitor-defs.o
  CC      stubs/target-get-monitor-def.o
  CC      stubs/pc_madt_cpu_entry.o
  CC      stubs/vmgenid.o
  CC      stubs/xen-common.o
  CC      stubs/xen-hvm.o
  CC      stubs/pci-host-piix.o
  GEN     qemu-img-cmds.h
  CC      block.o
  CC      blockjob.o
  CC      qemu-io-cmds.o
  CC      replication.o
  CC      block/raw-format.o
  CC      block/qcow.o
  CC      block/vdi.o
  CC      block/vmdk.o
  CC      block/cloop.o
  CC      block/bochs.o
  CC      block/vpc.o
  CC      block/vvfat.o
  CC      block/dmg.o
  CC      block/qcow2.o
  CC      block/qcow2-refcount.o
  CC      block/qcow2-cluster.o
  CC      block/qcow2-snapshot.o
  CC      block/qcow2-cache.o
  CC      block/qcow2-bitmap.o
  CC      block/qed.o
  CC      block/qed-l2-cache.o
  CC      block/qed-table.o
  CC      block/qed-cluster.o
  CC      block/qed-check.o
  CC      block/vhdx.o
  CC      block/vhdx-endian.o
  CC      block/vhdx-log.o
  CC      block/quorum.o
  CC      block/parallels.o
  CC      block/blkdebug.o
  CC      block/blkverify.o
  CC      block/blkreplay.o
  CC      block/block-backend.o
  CC      block/snapshot.o
  CC      block/qapi.o
  CC      block/file-win32.o
  CC      block/win32-aio.o
  CC      block/null.o
  CC      block/mirror.o
  CC      block/commit.o
  CC      block/io.o
  CC      block/throttle-groups.o
  CC      block/nbd.o
  CC      block/nbd-client.o
  CC      block/sheepdog.o
  CC      block/accounting.o
  CC      block/dirty-bitmap.o
  CC      block/write-threshold.o
  CC      block/backup.o
  CC      block/replication.o
  CC      block/throttle.o
  CC      block/crypto.o
  CC      nbd/server.o
  CC      nbd/client.o
  CC      nbd/common.o
  CC      scsi/utils.o
  CC      block/curl.o
  CC      block/ssh.o
  CC      block/dmg-bz2.o
  CC      crypto/init.o
  CC      crypto/hash.o
  CC      crypto/hash-nettle.o
  CC      crypto/hmac.o
  CC      crypto/hmac-nettle.o
  CC      crypto/aes.o
  CC      crypto/desrfb.o
  CC      crypto/cipher.o
  CC      crypto/tlscreds.o
  CC      crypto/tlscredsanon.o
  CC      crypto/tlscredsx509.o
  CC      crypto/secret.o
  CC      crypto/tlssession.o
  CC      crypto/random-gnutls.o
  CC      crypto/pbkdf.o
  CC      crypto/pbkdf-nettle.o
  CC      crypto/ivgen.o
  CC      crypto/ivgen-essiv.o
  CC      crypto/ivgen-plain.o
  CC      crypto/ivgen-plain64.o
  CC      crypto/afsplit.o
  CC      crypto/xts.o
  CC      crypto/block.o
  CC      crypto/block-luks.o
  CC      crypto/block-qcow.o
  CC      io/channel.o
  CC      io/channel-buffer.o
  CC      io/channel-file.o
  CC      io/channel-command.o
  CC      io/channel-socket.o
  CC      io/channel-tls.o
  CC      io/channel-watch.o
  CC      io/channel-websock.o
  CC      io/channel-util.o
  CC      io/dns-resolver.o
  CC      io/net-listener.o
  CC      io/task.o
  CC      qom/object.o
  CC      qom/container.o
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  CC      qemu-io.o
  CC      blockdev.o
  CC      blockdev-nbd.o
  CC      bootdevice.o
  CC      iothread.o
  CC      qdev-monitor.o
  CC      device-hotplug.o
  CC      bt-host.o
  CC      os-win32.o
  CC      bt-vhci.o
  CC      dma-helpers.o
  CC      vl.o
  CC      tpm.o
  CC      device_tree.o
  CC      qmp-marshal.o
  CC      qmp.o
  CC      hmp.o
  CC      cpus-common.o
  CC      audio/audio.o
  CC      audio/noaudio.o
  CC      audio/wavaudio.o
  CC      audio/mixeng.o
  CC      audio/sdlaudio.o
  CC      audio/dsoundaudio.o
  CC      audio/wavcapture.o
  CC      audio/audio_win_int.o
  CC      backends/rng.o
  CC      backends/rng-egd.o
  CC      backends/tpm.o
  CC      backends/hostmem.o
  CC      backends/hostmem-ram.o
  CC      backends/cryptodev.o
  CC      backends/cryptodev-builtin.o
  CC      block/stream.o
  CC      chardev/msmouse.o
  CC      chardev/wctablet.o
  CC      chardev/testdev.o
  CC      disas/arm.o
  CXX     disas/arm-a64.o
  CC      disas/i386.o
  CXX     disas/libvixl/vixl/utils.o
  CXX     disas/libvixl/vixl/compiler-intrinsics.o
  CXX     disas/libvixl/vixl/a64/instructions-a64.o
  CXX     disas/libvixl/vixl/a64/decoder-a64.o
  CXX     disas/libvixl/vixl/a64/disasm-a64.o
  CC      hw/acpi/core.o
  CC      hw/acpi/piix4.o
  CC      hw/acpi/pcihp.o
  CC      hw/acpi/ich9.o
  CC      hw/acpi/tco.o
  CC      hw/acpi/cpu_hotplug.o
  CC      hw/acpi/memory_hotplug.o
  CC      hw/acpi/cpu.o
  CC      hw/acpi/nvdimm.o
  CC      hw/acpi/vmgenid.o
  CC      hw/acpi/acpi_interface.o
  CC      hw/acpi/bios-linker-loader.o
  CC      hw/acpi/aml-build.o
  CC      hw/acpi/ipmi.o
  CC      hw/acpi/acpi-stub.o
  CC      hw/acpi/ipmi-stub.o
  CC      hw/audio/sb16.o
  CC      hw/audio/es1370.o
  CC      hw/audio/ac97.o
  CC      hw/audio/fmopl.o
  CC      hw/audio/adlib.o
  CC      hw/audio/gus.o
  CC      hw/audio/gusemu_hal.o
  CC      hw/audio/gusemu_mixer.o
  CC      hw/audio/cs4231a.o
  CC      hw/audio/intel-hda.o
  CC      hw/audio/hda-codec.o
  CC      hw/audio/pcspk.o
  CC      hw/audio/wm8750.o
  CC      hw/audio/pl041.o
  CC      hw/audio/lm4549.o
  CC      hw/audio/marvell_88w8618.o
  CC      hw/audio/soundhw.o
  CC      hw/block/block.o
  CC      hw/block/cdrom.o
  CC      hw/block/hd-geometry.o
  CC      hw/block/fdc.o
  CC      hw/block/m25p80.o
  CC      hw/block/nand.o
  CC      hw/block/pflash_cfi01.o
  CC      hw/block/pflash_cfi02.o
  CC      hw/block/ecc.o
  CC      hw/block/onenand.o
  CC      hw/block/nvme.o
  CC      hw/bt/core.o
  CC      hw/bt/l2cap.o
  CC      hw/bt/sdp.o
  CC      hw/bt/hci.o
  CC      hw/bt/hid.o
  CC      hw/bt/hci-csr.o
  CC      hw/char/ipoctal232.o
  CC      hw/char/parallel.o
  CC      hw/char/pl011.o
  CC      hw/char/serial.o
  CC      hw/char/serial-isa.o
  CC      hw/char/serial-pci.o
  CC      hw/char/virtio-console.o
  CC      hw/char/cadence_uart.o
  CC      hw/char/cmsdk-apb-uart.o
  CC      hw/char/debugcon.o
  CC      hw/char/imx_serial.o
  CC      hw/core/qdev.o
  CC      hw/core/qdev-properties.o
  CC      hw/core/bus.o
  CC      hw/core/reset.o
  CC      hw/core/qdev-fw.o
  CC      hw/core/fw-path-provider.o
  CC      hw/core/irq.o
  CC      hw/core/hotplug.o
  CC      hw/core/nmi.o
  CC      hw/core/stream.o
  CC      hw/core/ptimer.o
  CC      hw/core/sysbus.o
  CC      hw/core/machine.o
  CC      hw/core/loader.o
  CC      hw/core/qdev-properties-system.o
  CC      hw/core/register.o
  CC      hw/core/or-irq.o
  CC      hw/core/platform-bus.o
  CC      hw/display/ads7846.o
  CC      hw/cpu/core.o
  CC      hw/display/cirrus_vga.o
  CC      hw/display/pl110.o
  CC      hw/display/ssd0303.o
  CC      hw/display/ssd0323.o
  CC      hw/display/vga-pci.o
  CC      hw/display/vga-isa.o
  CC      hw/display/vmware_vga.o
  CC      hw/display/blizzard.o
  CC      hw/display/exynos4210_fimd.o
  CC      hw/display/framebuffer.o
  CC      hw/display/tc6393xb.o
  CC      hw/dma/pl080.o
  CC      hw/dma/pl330.o
  CC      hw/dma/i8257.o
  CC      hw/dma/xilinx_axidma.o
  CC      hw/dma/xlnx-zynq-devcfg.o
  CC      hw/gpio/max7310.o
  CC      hw/gpio/pl061.o
  CC      hw/gpio/zaurus.o
  CC      hw/gpio/gpio_key.o
  CC      hw/i2c/core.o
  CC      hw/i2c/smbus.o
  CC      hw/i2c/smbus_eeprom.o
  CC      hw/i2c/i2c-ddc.o
  CC      hw/i2c/versatile_i2c.o
  CC      hw/i2c/smbus_ich9.o
  CC      hw/i2c/pm_smbus.o
  CC      hw/i2c/bitbang_i2c.o
  CC      hw/i2c/exynos4210_i2c.o
  CC      hw/i2c/imx_i2c.o
  CC      hw/i2c/aspeed_i2c.o
  CC      hw/ide/core.o
  CC      hw/ide/atapi.o
  CC      hw/ide/qdev.o
  CC      hw/ide/pci.o
  CC      hw/ide/isa.o
  CC      hw/ide/piix.o
  CC      hw/ide/microdrive.o
  CC      hw/ide/ahci.o
  CC      hw/ide/ich.o
  CC      hw/ide/ahci-allwinner.o
  CC      hw/input/hid.o
  CC      hw/input/lm832x.o
  CC      hw/input/pckbd.o
  CC      hw/input/pl050.o
  CC      hw/input/ps2.o
  CC      hw/input/stellaris_input.o
  CC      hw/input/tsc2005.o
  CC      hw/input/virtio-input.o
  CC      hw/input/virtio-input-hid.o
  CC      hw/intc/i8259_common.o
  CC      hw/intc/i8259.o
  CC      hw/intc/pl190.o
  CC      hw/intc/xlnx-pmu-iomod-intc.o
  CC      hw/intc/xlnx-zynqmp-ipi.o
  CC      hw/intc/imx_avic.o
  CC      hw/intc/realview_gic.o
  CC      hw/intc/ioapic_common.o
  CC      hw/intc/arm_gic_common.o
  CC      hw/intc/arm_gic.o
  CC      hw/intc/arm_gicv2m.o
  CC      hw/intc/arm_gicv3_common.o
  CC      hw/intc/arm_gicv3.o
  CC      hw/intc/arm_gicv3_dist.o
  CC      hw/intc/arm_gicv3_redist.o
  CC      hw/intc/arm_gicv3_its_common.o
  CC      hw/intc/intc.o
  CC      hw/ipack/ipack.o
  CC      hw/ipack/tpci200.o
  CC      hw/ipmi/ipmi.o
  CC      hw/ipmi/ipmi_bmc_sim.o
  CC      hw/ipmi/ipmi_bmc_extern.o
  CC      hw/ipmi/isa_ipmi_kcs.o
  CC      hw/ipmi/isa_ipmi_bt.o
  CC      hw/isa/apm.o
  CC      hw/isa/isa-bus.o
  CC      hw/mem/pc-dimm.o
  CC      hw/mem/nvdimm.o
  CC      hw/misc/applesmc.o
  CC      hw/misc/max111x.o
  CC      hw/misc/tmp105.o
  CC      hw/misc/tmp421.o
  CC      hw/misc/debugexit.o
  CC      hw/misc/sga.o
  CC      hw/misc/pc-testdev.o
  CC      hw/misc/pci-testdev.o
  CC      hw/misc/edu.o
  CC      hw/misc/unimp.o
  CC      hw/misc/vmcoreinfo.o
  CC      hw/misc/arm_l2x0.o
  CC      hw/misc/arm_integrator_debug.o
  CC      hw/misc/a9scu.o
  CC      hw/misc/arm11scu.o
  CC      hw/net/ne2000.o
  CC      hw/net/eepro100.o
  CC      hw/net/pcnet-pci.o
  CC      hw/net/pcnet.o
  CC      hw/net/e1000.o
  CC      hw/net/e1000x_common.o
  CC      hw/net/net_tx_pkt.o
  CC      hw/net/net_rx_pkt.o
  CC      hw/net/e1000e.o
  CC      hw/net/e1000e_core.o
  CC      hw/net/rtl8139.o
  CC      hw/net/vmxnet3.o
  CC      hw/net/smc91c111.o
  CC      hw/net/lan9118.o
  CC      hw/net/ne2000-isa.o
  CC      hw/net/xgmac.o
  CC      hw/net/xilinx_axienet.o
  CC      hw/net/allwinner_emac.o
  CC      hw/net/imx_fec.o
  CC      hw/net/cadence_gem.o
  CC      hw/net/stellaris_enet.o
  CC      hw/net/ftgmac100.o
  CC      hw/net/rocker/rocker.o
  CC      hw/net/rocker/rocker_fp.o
  CC      hw/net/rocker/rocker_desc.o
  CC      hw/net/rocker/rocker_world.o
  CC      hw/net/rocker/rocker_of_dpa.o
  CC      hw/nvram/eeprom93xx.o
  CC      hw/nvram/eeprom_at24c.o
  CC      hw/nvram/fw_cfg.o
  CC      hw/nvram/chrp_nvram.o
  CC      hw/pci-bridge/pci_bridge_dev.o
  CC      hw/pci-bridge/pcie_root_port.o
  CC      hw/pci-bridge/gen_pcie_root_port.o
  CC      hw/pci-bridge/pcie_pci_bridge.o
  CC      hw/pci-bridge/pci_expander_bridge.o
  CC      hw/pci-bridge/xio3130_upstream.o
  CC      hw/pci-bridge/xio3130_downstream.o
  CC      hw/pci-bridge/ioh3420.o
  CC      hw/pci-bridge/i82801b11.o
  CC      hw/pci-host/pam.o
  CC      hw/pci-host/versatile.o
  CC      hw/pci-host/piix.o
  CC      hw/pci-host/q35.o
  CC      hw/pci-host/gpex.o
  CC      hw/pci/pci.o
  CC      hw/pci/msix.o
  CC      hw/pci/pci_bridge.o
  CC      hw/pci/msi.o
  CC      hw/pci/shpc.o
  CC      hw/pci/slotid_cap.o
  CC      hw/pci/pci_host.o
  CC      hw/pci/pcie_host.o
  CC      hw/pci/pcie.o
  CC      hw/pci/pcie_port.o
  CC      hw/pci/pcie_aer.o
  CC      hw/pci/pci-stub.o
  CC      hw/pcmcia/pcmcia.o
  CC      hw/scsi/scsi-disk.o
  CC      hw/scsi/scsi-generic.o
  CC      hw/scsi/scsi-bus.o
  CC      hw/scsi/lsi53c895a.o
  CC      hw/scsi/mptsas.o
  CC      hw/scsi/mptconfig.o
  CC      hw/scsi/mptendian.o
  CC      hw/scsi/megasas.o
  CC      hw/scsi/vmw_pvscsi.o
  CC      hw/scsi/esp.o
  CC      hw/scsi/esp-pci.o
  CC      hw/sd/pl181.o
  CC      hw/sd/ssi-sd.o
  CC      hw/sd/sd.o
  CC      hw/sd/core.o
  CC      hw/sd/sdhci.o
  CC      hw/smbios/smbios.o
  CC      hw/smbios/smbios_type_38.o
  CC      hw/smbios/smbios-stub.o
  CC      hw/smbios/smbios_type_38-stub.o
  CC      hw/ssi/pl022.o
  CC      hw/ssi/ssi.o
  CC      hw/ssi/xilinx_spips.o
  CC      hw/ssi/aspeed_smc.o
  CC      hw/ssi/stm32f2xx_spi.o
  CC      hw/ssi/mss-spi.o
  CC      hw/timer/arm_timer.o
  CC      hw/timer/arm_mptimer.o
  CC      hw/timer/armv7m_systick.o
  CC      hw/timer/a9gtimer.o
  CC      hw/timer/cadence_ttc.o
  CC      hw/timer/ds1338.o
  CC      hw/timer/hpet.o
  CC      hw/timer/i8254_common.o
  CC      hw/timer/i8254.o
  CC      hw/timer/pl031.o
  CC      hw/timer/twl92230.o
  CC      hw/timer/imx_epit.o
  CC      hw/timer/imx_gpt.o
  CC      hw/timer/stm32f2xx_timer.o
  CC      hw/timer/aspeed_timer.o
  CC      hw/timer/cmsdk-apb-timer.o
  CC      hw/timer/mss-timer.o
  CC      hw/tpm/tpm_util.o
  CC      hw/tpm/tpm_tis.o
  CC      hw/tpm/tpm_crb.o
  CC      hw/usb/core.o
  CC      hw/usb/combined-packet.o
  CC      hw/usb/bus.o
  CC      hw/usb/libhw.o
  CC      hw/usb/desc.o
  CC      hw/usb/desc-msos.o
  CC      hw/usb/hcd-uhci.o
  CC      hw/usb/hcd-ohci.o
  CC      hw/usb/hcd-ehci-pci.o
  CC      hw/usb/hcd-ehci-sysbus.o
  CC      hw/usb/hcd-ehci.o
  CC      hw/usb/hcd-xhci.o
  CC      hw/usb/hcd-xhci-nec.o
  CC      hw/usb/hcd-musb.o
  CC      hw/usb/dev-hub.o
  CC      hw/usb/dev-hid.o
  CC      hw/usb/dev-wacom.o
  CC      hw/usb/dev-storage.o
  CC      hw/usb/dev-uas.o
  CC      hw/usb/dev-audio.o
  CC      hw/usb/dev-serial.o
  CC      hw/usb/dev-network.o
  CC      hw/usb/dev-bluetooth.o
  CC      hw/usb/dev-smartcard-reader.o
  CC      hw/usb/host-stub.o
  CC      hw/virtio/virtio-rng.o
  CC      hw/virtio/virtio-pci.o
  CC      hw/virtio/virtio-bus.o
  CC      hw/virtio/virtio-mmio.o
  CC      hw/virtio/vhost-stub.o
  CC      hw/watchdog/watchdog.o
  CC      hw/watchdog/wdt_i6300esb.o
  CC      hw/watchdog/wdt_ib700.o
  CC      hw/watchdog/wdt_aspeed.o
  CC      migration/migration.o
  CC      migration/socket.o
  CC      migration/fd.o
  CC      migration/exec.o
  CC      migration/tls.o
  CC      migration/channel.o
  CC      migration/savevm.o
  CC      migration/colo-comm.o
  CC      migration/colo.o
  CC      migration/vmstate.o
  CC      migration/colo-failover.o
  CC      migration/vmstate-types.o
  CC      migration/page_cache.o
  CC      migration/qemu-file.o
  CC      migration/global_state.o
  CC      migration/qemu-file-channel.o
  CC      migration/xbzrle.o
  CC      migration/postcopy-ram.o
  CC      migration/qjson.o
  CC      migration/block.o
  CC      net/net.o
  CC      net/queue.o
  CC      net/checksum.o
  CC      net/util.o
  CC      net/hub.o
  CC      net/socket.o
  CC      net/dump.o
  CC      net/eth.o
  CC      net/slirp.o
  CC      net/filter.o
  CC      net/filter-buffer.o
  CC      net/filter-mirror.o
  CC      net/colo-compare.o
  CC      net/colo.o
  CC      net/filter-rewriter.o
  CC      net/filter-replay.o
  CC      net/tap-win32.o
  CC      qom/cpu.o
  CC      replay/replay.o
  CC      replay/replay-internal.o
  CC      replay/replay-events.o
  CC      replay/replay-time.o
  CC      replay/replay-input.o
  CC      replay/replay-char.o
  CC      replay/replay-snapshot.o
  CC      replay/replay-net.o
  CC      replay/replay-audio.o
  CC      slirp/cksum.o
  CC      slirp/if.o
  CC      slirp/ip_icmp.o
  CC      slirp/ip6_icmp.o
  CC      slirp/ip6_input.o
  CC      slirp/ip6_output.o
  CC      slirp/ip_input.o
  CC      slirp/ip_output.o
  CC      slirp/dnssearch.o
  CC      slirp/dhcpv6.o
  CC      slirp/slirp.o
  CC      slirp/mbuf.o
  CC      slirp/misc.o
  CC      slirp/socket.o
  CC      slirp/tcp_input.o
  CC      slirp/sbuf.o
  CC      slirp/tcp_output.o
  CC      slirp/tcp_subr.o
  CC      slirp/tcp_timer.o
  CC      slirp/udp.o
  CC      slirp/udp6.o
  CC      slirp/bootp.o
  CC      slirp/tftp.o
  CC      slirp/arp_table.o
  CC      slirp/ndp_table.o
  CC      slirp/ncsi.o
  CC      ui/keymaps.o
  CC      ui/console.o
  CC      ui/cursor.o
  CC      ui/qemu-pixman.o
  CC      ui/input.o
  CC      ui/input-keymap.o
  CC      ui/input-legacy.o
  CC      ui/sdl.o
  CC      ui/sdl_zoom.o
  CC      ui/vnc.o
  CC      ui/vnc-enc-zlib.o
  CC      ui/vnc-enc-hextile.o
  CC      ui/vnc-enc-tight.o
  CC      ui/vnc-palette.o
  CC      ui/vnc-enc-zrle.o
  CC      ui/vnc-auth-vencrypt.o
  CC      ui/vnc-ws.o
  CC      ui/vnc-jobs.o
  CC      ui/gtk.o
  CC      chardev/char.o
  CC      chardev/char-console.o
  CC      chardev/char-fe.o
  CC      chardev/char-file.o
  CC      chardev/char-io.o
  CC      chardev/char-mux.o
  CC      chardev/char-null.o
  CC      chardev/char-pipe.o
  CC      chardev/char-ringbuf.o
  CC      chardev/char-serial.o
  CC      chardev/char-socket.o
  CC      chardev/char-stdio.o
  CC      chardev/char-udp.o
  CC      chardev/char-win.o
  CC      chardev/char-win-stdio.o
  CC      qga/commands.o
  CC      qga/guest-agent-command-state.o
  CC      qga/main.o
  CC      qga/commands-win32.o
  CC      qga/channel-win32.o
  CC      qga/service-win32.o
  AS      optionrom/multiboot.o
  AS      optionrom/linuxboot.o
  CC      qga/vss-win32.o
  CC      optionrom/linuxboot_dma.o
  AS      optionrom/kvmvapic.o
  BUILD   optionrom/multiboot.img
  BUILD   optionrom/linuxboot.img
  BUILD   optionrom/kvmvapic.img
  BUILD   optionrom/multiboot.raw
  CC      qga/qapi-generated/qga-qapi-types.o
  BUILD   optionrom/linuxboot.raw
  BUILD   optionrom/linuxboot_dma.img
  BUILD   optionrom/kvmvapic.raw
  CC      qga/qapi-generated/qga-qapi-visit.o
  BUILD   optionrom/linuxboot_dma.raw
  CC      qga/qapi-generated/qga-qmp-marshal.o
  SIGN    optionrom/multiboot.bin
  SIGN    optionrom/linuxboot_dma.bin
  AR      libqemuutil.a
  CC      qemu-img.o
  SIGN    optionrom/kvmvapic.bin
  SIGN    optionrom/linuxboot.bin
  LINK    qemu-ga.exe
  LINK    qemu-img.exe
  LINK    qemu-io.exe
  GEN     x86_64-softmmu/hmp-commands.h
  GEN     x86_64-softmmu/config-target.h
  GEN     x86_64-softmmu/hmp-commands-info.h
  GEN     aarch64-softmmu/hmp-commands-info.h
  GEN     aarch64-softmmu/hmp-commands.h
  GEN     aarch64-softmmu/config-target.h
  CC      aarch64-softmmu/exec.o
  CC      aarch64-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/disas.o
  CC      aarch64-softmmu/fpu/softfloat.o
  CC      aarch64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/exec.o
  GEN     aarch64-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/arch_init.o
  CC      x86_64-softmmu/tcg/tcg.o
  CC      aarch64-softmmu/cpus.o
  CC      x86_64-softmmu/tcg/tcg-op.o
  CC      aarch64-softmmu/monitor.o
  CC      aarch64-softmmu/gdbstub.o
  CC      x86_64-softmmu/tcg/optimize.o
  CC      x86_64-softmmu/tcg/tcg-common.o
  CC      aarch64-softmmu/balloon.o
  CC      aarch64-softmmu/ioport.o
  CC      x86_64-softmmu/fpu/softfloat.o
  CC      x86_64-softmmu/disas.o
  GEN     x86_64-softmmu/gdbstub-xml.c
  CC      aarch64-softmmu/numa.o
  CC      aarch64-softmmu/qtest.o
  CC      x86_64-softmmu/arch_init.o
  CC      aarch64-softmmu/memory.o
  CC      x86_64-softmmu/cpus.o
  CC      x86_64-softmmu/monitor.o
  CC      x86_64-softmmu/gdbstub.o
  CC      aarch64-softmmu/memory_mapping.o
  CC      x86_64-softmmu/balloon.o
  CC      x86_64-softmmu/ioport.o
  CC      x86_64-softmmu/numa.o
  CC      x86_64-softmmu/qtest.o
  CC      x86_64-softmmu/memory.o
  CC      aarch64-softmmu/dump.o
  CC      aarch64-softmmu/migration/ram.o
  CC      aarch64-softmmu/accel/accel.o
  CC      aarch64-softmmu/accel/stubs/hax-stub.o
  CC      x86_64-softmmu/memory_mapping.o
  CC      x86_64-softmmu/dump.o
  CC      aarch64-softmmu/accel/stubs/hvf-stub.o
  CC      aarch64-softmmu/accel/stubs/kvm-stub.o
  CC      x86_64-softmmu/migration/ram.o
  CC      aarch64-softmmu/accel/tcg/tcg-all.o
  CC      aarch64-softmmu/accel/tcg/cputlb.o
  CC      x86_64-softmmu/accel/accel.o
  CC      aarch64-softmmu/accel/tcg/tcg-runtime.o
  CC      x86_64-softmmu/accel/stubs/hvf-stub.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/accel/stubs/kvm-stub.o
  CC      aarch64-softmmu/accel/tcg/translate-all.o
  CC      x86_64-softmmu/accel/tcg/tcg-all.o
  CC      x86_64-softmmu/accel/tcg/cputlb.o
  CC      x86_64-softmmu/accel/tcg/tcg-runtime.o
  CC      aarch64-softmmu/accel/tcg/translator.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec.o
  CC      aarch64-softmmu/hw/adc/stm32f2xx_adc.o
  CC      x86_64-softmmu/accel/tcg/cpu-exec-common.o
  CC      x86_64-softmmu/accel/tcg/translate-all.o
  CC      x86_64-softmmu/accel/tcg/translator.o
  CC      x86_64-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/block/virtio-blk.o
  CC      aarch64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      x86_64-softmmu/hw/block/dataplane/virtio-blk.o
  CC      aarch64-softmmu/hw/char/exynos4210_uart.o
  CC      x86_64-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/hw/char/omap_uart.o
  CC      x86_64-softmmu/hw/core/generic-loader.o
  CC      aarch64-softmmu/hw/char/digic-uart.o
  CC      x86_64-softmmu/hw/core/null-machine.o
  CC      x86_64-softmmu/hw/display/vga.o
  CC      x86_64-softmmu/hw/display/virtio-gpu.o
  CC      aarch64-softmmu/hw/char/stm32f2xx_usart.o
  CC      aarch64-softmmu/hw/char/bcm2835_aux.o
  CC      aarch64-softmmu/hw/char/virtio-serial-bus.o
  CC      aarch64-softmmu/hw/core/generic-loader.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/core/null-machine.o
  CC      x86_64-softmmu/hw/display/virtio-gpu-pci.o
  CC      aarch64-softmmu/hw/cpu/arm11mpcore.o
  CC      aarch64-softmmu/hw/cpu/realview_mpcore.o
  CC      aarch64-softmmu/hw/cpu/a15mpcore.o
  CC      aarch64-softmmu/hw/cpu/a9mpcore.o
  CC      x86_64-softmmu/hw/display/virtio-vga.o
  CC      x86_64-softmmu/hw/intc/apic.o
  CC      x86_64-softmmu/hw/intc/apic_common.o
  CC      aarch64-softmmu/hw/display/omap_dss.o
  CC      aarch64-softmmu/hw/display/omap_lcdc.o
  CC      aarch64-softmmu/hw/display/pxa2xx_lcd.o
  CC      x86_64-softmmu/hw/intc/ioapic.o
  CC      aarch64-softmmu/hw/display/bcm2835_fb.o
  CC      aarch64-softmmu/hw/display/vga.o
  CC      aarch64-softmmu/hw/display/virtio-gpu.o
  CC      x86_64-softmmu/hw/isa/lpc_ich9.o
  CC      x86_64-softmmu/hw/misc/pvpanic.o
  CC      x86_64-softmmu/hw/misc/mmio_interface.o
  CC      x86_64-softmmu/hw/net/virtio-net.o
  CC      x86_64-softmmu/hw/net/vhost_net.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-3d.o
  CC      aarch64-softmmu/hw/display/virtio-gpu-pci.o
  CC      x86_64-softmmu/hw/scsi/virtio-scsi-dataplane.o
  CC      x86_64-softmmu/hw/timer/mc146818rtc.o
  CC      aarch64-softmmu/hw/display/dpcd.o
  CC      x86_64-softmmu/hw/virtio/virtio.o
  CC      aarch64-softmmu/hw/display/xlnx_dp.o
  CC      aarch64-softmmu/hw/dma/xlnx_dpdma.o
  CC      aarch64-softmmu/hw/dma/omap_dma.o
  CC      aarch64-softmmu/hw/dma/soc_dma.o
  CC      x86_64-softmmu/hw/virtio/virtio-balloon.o
  CC      aarch64-softmmu/hw/dma/pxa2xx_dma.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto.o
  CC      aarch64-softmmu/hw/dma/bcm2835_dma.o
  CC      x86_64-softmmu/hw/virtio/virtio-crypto-pci.o
  CC      x86_64-softmmu/hw/i386/multiboot.o
  CC      aarch64-softmmu/hw/gpio/omap_gpio.o
  CC      aarch64-softmmu/hw/gpio/imx_gpio.o
  CC      aarch64-softmmu/hw/gpio/bcm2835_gpio.o
  CC      x86_64-softmmu/hw/i386/pc.o
  CC      x86_64-softmmu/hw/i386/pc_piix.o
  CC      x86_64-softmmu/hw/i386/pc_q35.o
  CC      aarch64-softmmu/hw/i2c/omap_i2c.o
  CC      x86_64-softmmu/hw/i386/pc_sysfw.o
  CC      aarch64-softmmu/hw/input/pxa2xx_keypad.o
  CC      x86_64-softmmu/hw/i386/x86-iommu.o
  CC      x86_64-softmmu/hw/i386/intel_iommu.o
  CC      x86_64-softmmu/hw/i386/amd_iommu.o
  CC      x86_64-softmmu/hw/i386/vmport.o
  CC      aarch64-softmmu/hw/input/tsc210x.o
  CC      x86_64-softmmu/hw/i386/vmmouse.o
  CC      x86_64-softmmu/hw/i386/kvmvapic.o
  CC      aarch64-softmmu/hw/intc/armv7m_nvic.o
  CC      aarch64-softmmu/hw/intc/exynos4210_gic.o
  CC      x86_64-softmmu/hw/i386/acpi-build.o
  CC      x86_64-softmmu/target/i386/helper.o
  CC      x86_64-softmmu/target/i386/cpu.o
  CC      aarch64-softmmu/hw/intc/exynos4210_combiner.o
  CC      x86_64-softmmu/target/i386/gdbstub.o
  CC      aarch64-softmmu/hw/intc/omap_intc.o
  CC      x86_64-softmmu/target/i386/xsave_helper.o
  CC      x86_64-softmmu/target/i386/translate.o
  CC      x86_64-softmmu/target/i386/bpt_helper.o
  CC      x86_64-softmmu/target/i386/cc_helper.o
  CC      x86_64-softmmu/target/i386/excp_helper.o
  CC      aarch64-softmmu/hw/intc/bcm2835_ic.o
  CC      aarch64-softmmu/hw/intc/bcm2836_control.o
  CC      x86_64-softmmu/target/i386/fpu_helper.o
  CC      x86_64-softmmu/target/i386/int_helper.o
  CC      aarch64-softmmu/hw/intc/allwinner-a10-pic.o
  CC      x86_64-softmmu/target/i386/mem_helper.o
  CC      x86_64-softmmu/target/i386/misc_helper.o
  CC      aarch64-softmmu/hw/intc/aspeed_vic.o
  CC      aarch64-softmmu/hw/intc/arm_gicv3_cpuif.o
  CC      x86_64-softmmu/target/i386/mpx_helper.o
  CC      aarch64-softmmu/hw/misc/arm_sysctl.o
  CC      x86_64-softmmu/target/i386/seg_helper.o
  CC      aarch64-softmmu/hw/misc/cbus.o
  CC      aarch64-softmmu/hw/misc/exynos4210_pmu.o
  CC      x86_64-softmmu/target/i386/smm_helper.o
  CC      aarch64-softmmu/hw/misc/exynos4210_clk.o
  CC      aarch64-softmmu/hw/misc/exynos4210_rng.o
  CC      x86_64-softmmu/target/i386/svm_helper.o
  CC      aarch64-softmmu/hw/misc/imx_ccm.o
  CC      aarch64-softmmu/hw/misc/imx31_ccm.o
  CC      aarch64-softmmu/hw/misc/imx25_ccm.o
  CC      x86_64-softmmu/target/i386/machine.o
  CC      x86_64-softmmu/target/i386/arch_memory_mapping.o
  CC      x86_64-softmmu/target/i386/arch_dump.o
  CC      aarch64-softmmu/hw/misc/imx6_ccm.o
  CC      x86_64-softmmu/target/i386/monitor.o
  CC      aarch64-softmmu/hw/misc/imx6_src.o
  CC      x86_64-softmmu/target/i386/kvm-stub.o
  CC      aarch64-softmmu/hw/misc/mst_fpga.o
  CC      x86_64-softmmu/target/i386/hax-all.o
  CC      aarch64-softmmu/hw/misc/omap_clk.o
  CC      x86_64-softmmu/target/i386/hax-mem.o
  CC      aarch64-softmmu/hw/misc/omap_gpmc.o
  CC      aarch64-softmmu/hw/misc/omap_l4.o
  CC      aarch64-softmmu/hw/misc/omap_sdrc.o
  CC      x86_64-softmmu/target/i386/hax-windows.o
  CC      aarch64-softmmu/hw/misc/omap_tap.o
  GEN     trace/generated-helpers.c
  CC      aarch64-softmmu/hw/misc/bcm2835_mbox.o
  CC      aarch64-softmmu/hw/misc/bcm2835_property.o
  CC      aarch64-softmmu/hw/misc/bcm2835_rng.o
/tmp/qemu-test/src/exec.c: In function 'qemu_ram_block_by_name':
/tmp/qemu-test/src/exec.c:2373:11: error: implicit declaration of function 'basename' [-Werror=implicit-function-declaration]
     id1 = basename(name1);
           ^~~~~~~~
/tmp/qemu-test/src/exec.c:2373:5: error: nested extern declaration of 'basename' [-Werror=nested-externs]
     id1 = basename(name1);
     ^~~
/tmp/qemu-test/src/exec.c:2373:9: error: assignment makes pointer from integer without a cast [-Werror=int-conversion]
     id1 = basename(name1);
         ^
/tmp/qemu-test/src/exec.c:2377:6: error: assignment makes pointer from integer without a cast [-Werror=int-conversion]
  id2 = basename(name2);
      ^
cc1: all warnings being treated as errors
  CC      aarch64-softmmu/hw/misc/zynq_slcr.o
/tmp/qemu-test/src/rules.mak:66: recipe for target 'exec.o' failed
make[1]: *** [exec.o] Error 1
make[1]: *** Waiting for unfinished jobs....
Makefile:403: recipe for target 'subdir-x86_64-softmmu' failed
make: *** [subdir-x86_64-softmmu] Error 2
make: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/hw/misc/zynq-xadc.o
  CC      aarch64-softmmu/hw/misc/stm32f2xx_syscfg.o
  CC      aarch64-softmmu/hw/misc/mps2-scc.o
  CC      aarch64-softmmu/hw/misc/auxbus.o
  CC      aarch64-softmmu/hw/misc/aspeed_scu.o
  CC      aarch64-softmmu/hw/misc/aspeed_sdmc.o
  CC      aarch64-softmmu/hw/misc/mmio_interface.o
  CC      aarch64-softmmu/hw/misc/msf2-sysreg.o
  CC      aarch64-softmmu/hw/net/virtio-net.o
  CC      aarch64-softmmu/hw/net/vhost_net.o
/tmp/qemu-test/src/exec.c: In function 'qemu_ram_block_by_name':
/tmp/qemu-test/src/exec.c:2373:11: error: implicit declaration of function 'basename' [-Werror=implicit-function-declaration]
     id1 = basename(name1);
           ^~~~~~~~
/tmp/qemu-test/src/exec.c:2373:5: error: nested extern declaration of 'basename' [-Werror=nested-externs]
     id1 = basename(name1);
     ^~~
/tmp/qemu-test/src/exec.c:2373:9: error: assignment makes pointer from integer without a cast [-Werror=int-conversion]
     id1 = basename(name1);
         ^
/tmp/qemu-test/src/exec.c:2377:6: error: assignment makes pointer from integer without a cast [-Werror=int-conversion]
  id2 = basename(name2);
      ^
cc1: all warnings being treated as errors
  CC      aarch64-softmmu/hw/pcmcia/pxa2xx.o
/tmp/qemu-test/src/rules.mak:66: recipe for target 'exec.o' failed
make[1]: *** [exec.o] Error 1
make[1]: *** Waiting for unfinished jobs....
  CC      aarch64-softmmu/hw/scsi/virtio-scsi.o
Makefile:403: recipe for target 'subdir-aarch64-softmmu' failed
make: *** [subdir-aarch64-softmmu] Error 2
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 407, in <module>
    sys.exit(main())
  File "./tests/docker/docker.py", line 404, in main
    return args.cmdobj.run(args, argv)
  File "./tests/docker/docker.py", line 261, in run
    return Docker().run(argv, args.keep, quiet=args.quiet)
  File "./tests/docker/docker.py", line 229, in run
    quiet=quiet)
  File "./tests/docker/docker.py", line 147, in _do_check
    return subprocess.check_call(self._command + cmd, **kwargs)
  File "/usr/lib64/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'run', '--label', 'com.qemu.instance.uuid=042d3c8c0a8f11e892f952540069c830', '-u', '0', '--security-opt', 'seccomp=unconfined', '--rm', '--net=none', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=8', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/root/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-83d1w41a/src/docker-src.2018-02-05-11.09.50.15886:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2
make[1]: *** [tests/docker/Makefile.include:129: docker-run] Error 1
make: *** [tests/docker/Makefile.include:163: docker-run-test-mingw@fedora] Error 2

real	2m28.722s
user	0m4.904s
sys	0m3.597s
=== OUTPUT END ===

Test command exited with code: 2


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Mon,  5 Feb 2018 14:58:55 +0000
Jianfeng Tan <jianfeng.tan@intel.com> wrote:

> Existing VMs with virtio devices and vhost-kernel as the backend
> are always started with mem config:
> 
>   "-m xG"
>   (with a ram block named "pc.ram")
> 
> while new VMs with virtio devices and vhost-user as the backend
> are always started with mem config:
> 
>   "-m xG -numa node,memdev=pc.ram -object memory-backend-file,id=pc.ram,..."
>   (with a ram block named "/object/pc.ram")
could you elaborate more on what src command line migrating to what dst command line?


> As we migrate from vhost-kernel to vhost-user, it failes as:
> 
>   Unknown ramblock "pc.ram", cannot accept migration
>   error while loading state for instance 0x0 of device 'ram'
>   load of migration failed: Invalid argument
> 
> Here are some options to fix this:
> 
> 1. When we do ram name comparison, we truncate the prefix as this patch shows.
> It cannot cover the corner case: the source VM could have two ram blocks
> with name of "pc.ram" and "/object/pc.ram".
> 
> 2. We add an alias name to RAMBlock; when we do name comparison, not only
> idstr is compared, but also compared to the alias. But this will add more
> complexity to upper layer stack OpenStack/libvirt.
> 
> Any thoughts?
> 
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Michael S. Tsirkin <mst@redhat.com>
> Cc: Maxime Coquelin <maxime.coquelin@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Suggested-by: Michael S. Tsirkin <mst@redhat.com>
> Signed-off-by: Jianfeng Tan <jianfeng.tan@intel.com>
> ---
>  exec.c | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/exec.c b/exec.c
> index 4722e52..d294e5c 100644
> --- a/exec.c
> +++ b/exec.c
> @@ -2334,13 +2334,24 @@ found:
>  RAMBlock *qemu_ram_block_by_name(const char *name)
>  {
>      RAMBlock *block;
> +    char *name1, *id1;
> +    char *name2, *id2;
> +
> +    name1 = strdup(name);
> +    id1 = basename(name1);
>  
>      RAMBLOCK_FOREACH(block) {
> -        if (!strcmp(name, block->idstr)) {
> +        name2 = strdup(block->idstr);
> +	id2 = basename(name2);
> +        if (!strcmp(id1, id2)) {
> +            free(name1);
> +	    free(name2);
>              return block;
>          }
> +	free(name2);
>      }
>  
> +    free(name1);
>      return NULL;
>  }
>  

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Tan, Jianfeng]

On 2/6/2018 12:29 AM, Igor Mammedov wrote:
> On Mon,  5 Feb 2018 14:58:55 +0000
> Jianfeng Tan <jianfeng.tan@intel.com> wrote:
>
>> Existing VMs with virtio devices and vhost-kernel as the backend
>> are always started with mem config:
>>
>>    "-m xG"
>>    (with a ram block named "pc.ram")
>>
>> while new VMs with virtio devices and vhost-user as the backend
>> are always started with mem config:
>>
>>    "-m xG -numa node,memdev=pc.ram -object memory-backend-file,id=pc.ram,..."
>>    (with a ram block named "/object/pc.ram")
> could you elaborate more on what src command line migrating to what dst command line?

The src cmdline:
     $QEMU -enable-kvm -cpu host -smp 4 /path/to/img \
       -m 2G \
       -netdev tap,id=mynet1,vhost=on \
       -device virtio-net-pci,netdev=mynet1,mac=52:54:00:12:34:58 ...

The dst cmdline:
     $QEMU -enable-kvm -cpu host -smp 4 /path/to/img \
       -m 2G -numa node,memdev=pc.ram -mem-prealloc \
       -object 
memory-backend-file,id=pc.ram,size=2G,mem-path=/dev/hugepages,share=on \
       -chardev socket,id=char0,path=/tmp/sock0 \
       -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
       -device virtio-net-pci,netdev=mynet1,mac=52:54:00:12:34:58 \
       -incoming tcp:0:4444 ...

Thanks,
Jianfeng

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Dr. David Alan Gilbert]

* Tan, Jianfeng (jianfeng.tan@intel.com) wrote:
> 
> 
> On 2/6/2018 12:29 AM, Igor Mammedov wrote:
> > On Mon,  5 Feb 2018 14:58:55 +0000
> > Jianfeng Tan <jianfeng.tan@intel.com> wrote:
> > 
> > > Existing VMs with virtio devices and vhost-kernel as the backend
> > > are always started with mem config:
> > > 
> > >    "-m xG"
> > >    (with a ram block named "pc.ram")
> > > 
> > > while new VMs with virtio devices and vhost-user as the backend
> > > are always started with mem config:
> > > 
> > >    "-m xG -numa node,memdev=pc.ram -object memory-backend-file,id=pc.ram,..."
> > >    (with a ram block named "/object/pc.ram")
> > could you elaborate more on what src command line migrating to what dst command line?
> 
> The src cmdline:
>     $QEMU -enable-kvm -cpu host -smp 4 /path/to/img \
>       -m 2G \
>       -netdev tap,id=mynet1,vhost=on \
>       -device virtio-net-pci,netdev=mynet1,mac=52:54:00:12:34:58 ...
> 
> The dst cmdline:
>     $QEMU -enable-kvm -cpu host -smp 4 /path/to/img \
>       -m 2G -numa node,memdev=pc.ram -mem-prealloc \
>       -object
> memory-backend-file,id=pc.ram,size=2G,mem-path=/dev/hugepages,share=on \
>       -chardev socket,id=char0,path=/tmp/sock0 \
>       -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
>       -device virtio-net-pci,netdev=mynet1,mac=52:54:00:12:34:58 \
>       -incoming tcp:0:4444 ...

I'm surprised that it's safe to -numa node  the destination, even with
the hack to the RAMBlock naming.  I'd expect it to have other effects
as well.

Dave

> Thanks,
> Jianfeng
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC] exec: eliminate ram naming issue as migration

[Igor Mammedov]

On Mon, 5 Feb 2018 18:36:31 +0000
"Dr. David Alan Gilbert" <dgilbert@redhat.com> wrote:

> * Tan, Jianfeng (jianfeng.tan@intel.com) wrote:
> > 
> > 
> > On 2/6/2018 12:29 AM, Igor Mammedov wrote:  
> > > On Mon,  5 Feb 2018 14:58:55 +0000
> > > Jianfeng Tan <jianfeng.tan@intel.com> wrote:
> > >   
> > > > Existing VMs with virtio devices and vhost-kernel as the backend
> > > > are always started with mem config:
> > > > 
> > > >    "-m xG"
> > > >    (with a ram block named "pc.ram")
> > > > 
> > > > while new VMs with virtio devices and vhost-user as the backend
> > > > are always started with mem config:
> > > > 
> > > >    "-m xG -numa node,memdev=pc.ram -object memory-backend-file,id=pc.ram,..."
> > > >    (with a ram block named "/object/pc.ram")  
> > > could you elaborate more on what src command line migrating to what dst command line?  
> > 
> > The src cmdline:
> >     $QEMU -enable-kvm -cpu host -smp 4 /path/to/img \
> >       -m 2G \
> >       -netdev tap,id=mynet1,vhost=on \
> >       -device virtio-net-pci,netdev=mynet1,mac=52:54:00:12:34:58 ...
> > 
> > The dst cmdline:
> >     $QEMU -enable-kvm -cpu host -smp 4 /path/to/img \
> >       -m 2G -numa node,memdev=pc.ram -mem-prealloc \
> >       -object
> > memory-backend-file,id=pc.ram,size=2G,mem-path=/dev/hugepages,share=on \
> >       -chardev socket,id=char0,path=/tmp/sock0 \
> >       -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
> >       -device virtio-net-pci,netdev=mynet1,mac=52:54:00:12:34:58 \
> >       -incoming tcp:0:4444 ...  
> 
> I'm surprised that it's safe to -numa node  the destination, even with
> the hack to the RAMBlock naming.  I'd expect it to have other effects
> as well.
it supposed to be 2 mutually exclusive configurations,
but migration stream doesn't have that information explicitly
(different id naming/mapping could fail migration).


> Dave
> 
> > Thanks,
> > Jianfeng
> >   
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK