"xl vcpu-set" not persistent across reboot?

"xl vcpu-set" not persistent across reboot?

[Jan Beulich]

Ian, Wei,

is it intentional that rebooting a (HVM) guest after having altered its
vCPU count will reset it back to the vCPU count it was originally
started with? That doesn't seem very natural - if one hotplugs a CPU
into a physical system and then reboots, that CPU will remain there.

Thanks, Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote:
> Ian, Wei,
> 
> is it intentional that rebooting a (HVM) guest after having altered its
> vCPU count will reset it back to the vCPU count it was originally
> started with? That doesn't seem very natural - if one hotplugs a CPU
> into a physical system and then reboots, that CPU will remain there.
> 

This is probably an oversight.

I've added this to my list of things to look at after the release.

Wei.

> Thanks, Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Jan Beulich]

>>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote:
> On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote:
>> Ian, Wei,
>> 
>> is it intentional that rebooting a (HVM) guest after having altered its
>> vCPU count will reset it back to the vCPU count it was originally
>> started with? That doesn't seem very natural - if one hotplugs a CPU
>> into a physical system and then reboots, that CPU will remain there.
>> 
> 
> This is probably an oversight.
> 
> I've added this to my list of things to look at after the release.

Thanks!

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote:
> >>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote:
> > On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote:
> >> Ian, Wei,
> >> 
> >> is it intentional that rebooting a (HVM) guest after having altered its
> >> vCPU count will reset it back to the vCPU count it was originally
> >> started with? That doesn't seem very natural - if one hotplugs a CPU
> >> into a physical system and then reboots, that CPU will remain there.
> >> 
> > 
> > This is probably an oversight.
> > 
> > I've added this to my list of things to look at after the release.
> 
> Thanks!
> 

I got a patch ready.  But QEMU upstream refuses to start on the receiving end
with following error message:

qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
qemu-system-i386: load of migration failed: Invalid argument

With QEMU traditional HVM guest and PV guest, the guest works fine -- up
and running with all hot plugged cpus available.

So I think the relevant libxl information is transmitted but we also
need to fix QEMU upstream. But that's a separate issue.

Wei.

---8<---
From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
From: Wei Liu <wei.liu2@citrix.com>
Date: Fri, 3 Jun 2016 16:38:32 +0100
Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config

... because the available vcpu bitmap can change during domain life time
due to cpu hotplug and unplug.

Reported-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxl/libxl.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 006b83f..99617f3 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -7270,6 +7270,37 @@ int libxl_retrieve_domain_configuration(libxl_ctx *ctx, uint32_t domid,
         libxl_dominfo_dispose(&info);
     }
 
+    /* VCPUs */
+    {
+        libxl_vcpuinfo *vcpus;
+        libxl_bitmap *map;
+        int nr_vcpus, nr_cpus;
+        unsigned int i;
+
+        vcpus = libxl_list_vcpu(ctx, domid, &nr_vcpus, &nr_cpus);
+        if (!vcpus) {
+            LOG(ERROR, "fail to get vcpu list for domain %d", domid);
+            rc = ERROR_FAIL;
+            goto out;
+        }
+
+        /* Update the avail_vcpus bitmap accordingly */
+        map = &d_config->b_info.avail_vcpus;
+
+        libxl_bitmap_dispose(map);
+
+        libxl_bitmap_alloc(ctx, map, nr_vcpus);
+
+        libxl_bitmap_init(map);
+
+        for (i = 0; i < nr_vcpus; i++) {
+            if (vcpus[i].online)
+                libxl_bitmap_set(map, i);
+        }
+
+        libxl_vcpuinfo_list_free(vcpus, nr_vcpus);
+    }
+
     /* Memory limits:
      *
      * Currently there are three memory limits:
-- 
2.1.4



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Jan Beulich]

>>> On 03.06.16 at 18:35, <wei.liu2@citrix.com> wrote:
> I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> with following error message:
> 
> qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> qemu-system-i386: load of migration failed: Invalid argument
> 
> With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> and running with all hot plugged cpus available.
> 
> So I think the relevant libxl information is transmitted but we also
> need to fix QEMU upstream. But that's a separate issue.

Stefano, Anthony,

any thoughts here?

Thanks, Jan

> ---8<---
> From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
> From: Wei Liu <wei.liu2@citrix.com>
> Date: Fri, 3 Jun 2016 16:38:32 +0100
> Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config
> 
> ... because the available vcpu bitmap can change during domain life time
> due to cpu hotplug and unplug.
> 
> Reported-by: Jan Beulich <jbeulich@suse.com>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  tools/libxl/libxl.c | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 006b83f..99617f3 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -7270,6 +7270,37 @@ int libxl_retrieve_domain_configuration(libxl_ctx 
> *ctx, uint32_t domid,
>          libxl_dominfo_dispose(&info);
>      }
>  
> +    /* VCPUs */
> +    {
> +        libxl_vcpuinfo *vcpus;
> +        libxl_bitmap *map;
> +        int nr_vcpus, nr_cpus;
> +        unsigned int i;
> +
> +        vcpus = libxl_list_vcpu(ctx, domid, &nr_vcpus, &nr_cpus);
> +        if (!vcpus) {
> +            LOG(ERROR, "fail to get vcpu list for domain %d", domid);
> +            rc = ERROR_FAIL;
> +            goto out;
> +        }
> +
> +        /* Update the avail_vcpus bitmap accordingly */
> +        map = &d_config->b_info.avail_vcpus;
> +
> +        libxl_bitmap_dispose(map);
> +
> +        libxl_bitmap_alloc(ctx, map, nr_vcpus);
> +
> +        libxl_bitmap_init(map);
> +
> +        for (i = 0; i < nr_vcpus; i++) {
> +            if (vcpus[i].online)
> +                libxl_bitmap_set(map, i);
> +        }
> +
> +        libxl_vcpuinfo_list_free(vcpus, nr_vcpus);
> +    }
> +
>      /* Memory limits:
>       *
>       * Currently there are three memory limits:
> -- 
> 2.1.4




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote:
> On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote:
> > >>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote:
> > > On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote:
> > >> Ian, Wei,
> > >> 
> > >> is it intentional that rebooting a (HVM) guest after having altered its
> > >> vCPU count will reset it back to the vCPU count it was originally
> > >> started with? That doesn't seem very natural - if one hotplugs a CPU
> > >> into a physical system and then reboots, that CPU will remain there.
> > >> 
> > > 
> > > This is probably an oversight.
> > > 
> > > I've added this to my list of things to look at after the release.
> > 
> > Thanks!
> > 
> 
> I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> with following error message:
> 
> qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> qemu-system-i386: load of migration failed: Invalid argument
> 
> With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> and running with all hot plugged cpus available.
> 
> So I think the relevant libxl information is transmitted but we also
> need to fix QEMU upstream. But that's a separate issue.
> 
> Wei.
> 
> ---8<---
> From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
> From: Wei Liu <wei.liu2@citrix.com>
> Date: Fri, 3 Jun 2016 16:38:32 +0100
> Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config
> 
> ... because the available vcpu bitmap can change during domain life time
> due to cpu hotplug and unplug.
> 
> Reported-by: Jan Beulich <jbeulich@suse.com>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

This patch has two issues:
1. The code to allocate bitmap is wrong
2. The check to see if vcpu is available is wrong

It happens to work on PV and qemu-trad because they don't really rely
on the bitmap provided.

#1 is fixed.

For #2, I haven't really gotten to the point today on how to correctly
get the number of online vcpus.

The current issue I discover is that:

  xl vcpu-set jessie-hvm 4
  xl list -l jessie-hvm | less # search for avail_vcpus

A vcpu is not really considered online from xen's point of view, unless
the guest explicitly activates it, like in guest `echo 1 >
.../cpu1/online`.

This is still not desirable because it would still cause qemu upstream
migration to fail. I will see if there is other way to figure out how
many vcpus are there.

Off the top of my head I might need to interrogate QEMU for that. I will
continue investigation later.

Any hint on how to effectively identify online vcpus is very welcomed.

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

Use Stefano's new email address

On Mon, Jun 06, 2016 at 06:18:06PM +0100, Wei Liu wrote:
> On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote:
> > On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote:
> > > >>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote:
> > > > On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote:
> > > >> Ian, Wei,
> > > >> 
> > > >> is it intentional that rebooting a (HVM) guest after having altered its
> > > >> vCPU count will reset it back to the vCPU count it was originally
> > > >> started with? That doesn't seem very natural - if one hotplugs a CPU
> > > >> into a physical system and then reboots, that CPU will remain there.
> > > >> 
> > > > 
> > > > This is probably an oversight.
> > > > 
> > > > I've added this to my list of things to look at after the release.
> > > 
> > > Thanks!
> > > 
> > 
> > I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> > with following error message:
> > 
> > qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> > qemu-system-i386: load of migration failed: Invalid argument
> > 
> > With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> > and running with all hot plugged cpus available.
> > 
> > So I think the relevant libxl information is transmitted but we also
> > need to fix QEMU upstream. But that's a separate issue.
> > 
> > Wei.
> > 
> > ---8<---
> > From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
> > From: Wei Liu <wei.liu2@citrix.com>
> > Date: Fri, 3 Jun 2016 16:38:32 +0100
> > Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config
> > 
> > ... because the available vcpu bitmap can change during domain life time
> > due to cpu hotplug and unplug.
> > 
> > Reported-by: Jan Beulich <jbeulich@suse.com>
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> 
> This patch has two issues:
> 1. The code to allocate bitmap is wrong
> 2. The check to see if vcpu is available is wrong
> 
> It happens to work on PV and qemu-trad because they don't really rely
> on the bitmap provided.
> 
> #1 is fixed.
> 
> For #2, I haven't really gotten to the point today on how to correctly
> get the number of online vcpus.
> 
> The current issue I discover is that:
> 
>   xl vcpu-set jessie-hvm 4
>   xl list -l jessie-hvm | less # search for avail_vcpus
> 
> A vcpu is not really considered online from xen's point of view, unless
> the guest explicitly activates it, like in guest `echo 1 >
> .../cpu1/online`.
> 
> This is still not desirable because it would still cause qemu upstream
> migration to fail. I will see if there is other way to figure out how
> many vcpus are there.
> 
> Off the top of my head I might need to interrogate QEMU for that. I will
> continue investigation later.
> 
> Any hint on how to effectively identify online vcpus is very welcomed.
> 
> Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Andrew Cooper]

On 06/06/16 18:20, Wei Liu wrote:
> Use Stefano's new email address
>
> On Mon, Jun 06, 2016 at 06:18:06PM +0100, Wei Liu wrote:
>> On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote:
>>> On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote:
>>>>>>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote:
>>>>> On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote:
>>>>>> Ian, Wei,
>>>>>>
>>>>>> is it intentional that rebooting a (HVM) guest after having altered its
>>>>>> vCPU count will reset it back to the vCPU count it was originally
>>>>>> started with? That doesn't seem very natural - if one hotplugs a CPU
>>>>>> into a physical system and then reboots, that CPU will remain there.
>>>>>>
>>>>> This is probably an oversight.
>>>>>
>>>>> I've added this to my list of things to look at after the release.
>>>> Thanks!
>>>>
>>> I got a patch ready.  But QEMU upstream refuses to start on the receiving end
>>> with following error message:
>>>
>>> qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
>>> qemu-system-i386: load of migration failed: Invalid argument
>>>
>>> With QEMU traditional HVM guest and PV guest, the guest works fine -- up
>>> and running with all hot plugged cpus available.
>>>
>>> So I think the relevant libxl information is transmitted but we also
>>> need to fix QEMU upstream. But that's a separate issue.
>>>
>>> Wei.
>>>
>>> ---8<---
>>> From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
>>> From: Wei Liu <wei.liu2@citrix.com>
>>> Date: Fri, 3 Jun 2016 16:38:32 +0100
>>> Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config
>>>
>>> ... because the available vcpu bitmap can change during domain life time
>>> due to cpu hotplug and unplug.
>>>
>>> Reported-by: Jan Beulich <jbeulich@suse.com>
>>> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
>> This patch has two issues:
>> 1. The code to allocate bitmap is wrong
>> 2. The check to see if vcpu is available is wrong
>>
>> It happens to work on PV and qemu-trad because they don't really rely
>> on the bitmap provided.
>>
>> #1 is fixed.
>>
>> For #2, I haven't really gotten to the point today on how to correctly
>> get the number of online vcpus.
>>
>> The current issue I discover is that:
>>
>>   xl vcpu-set jessie-hvm 4
>>   xl list -l jessie-hvm | less # search for avail_vcpus
>>
>> A vcpu is not really considered online from xen's point of view, unless
>> the guest explicitly activates it, like in guest `echo 1 >
>> .../cpu1/online`.
>>
>> This is still not desirable because it would still cause qemu upstream
>> migration to fail. I will see if there is other way to figure out how
>> many vcpus are there.
>>
>> Off the top of my head I might need to interrogate QEMU for that. I will
>> continue investigation later.
>>
>> Any hint on how to effectively identify online vcpus is very welcomed.

Why does qemu even care?  It has nothing to do with vcpu handling. 
There should not be any qemu vcpu records in the first place.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Mon, Jun 06, 2016 at 06:34:44PM +0100, Andrew Cooper wrote:
> On 06/06/16 18:20, Wei Liu wrote:
> > Use Stefano's new email address
> >
> > On Mon, Jun 06, 2016 at 06:18:06PM +0100, Wei Liu wrote:
> >> On Fri, Jun 03, 2016 at 05:35:20PM +0100, Wei Liu wrote:
> >>> On Fri, Jun 03, 2016 at 08:42:11AM -0600, Jan Beulich wrote:
> >>>>>>> On 03.06.16 at 15:41, <wei.liu2@citrix.com> wrote:
> >>>>> On Fri, Jun 03, 2016 at 02:29:12AM -0600, Jan Beulich wrote:
> >>>>>> Ian, Wei,
> >>>>>>
> >>>>>> is it intentional that rebooting a (HVM) guest after having altered its
> >>>>>> vCPU count will reset it back to the vCPU count it was originally
> >>>>>> started with? That doesn't seem very natural - if one hotplugs a CPU
> >>>>>> into a physical system and then reboots, that CPU will remain there.
> >>>>>>
> >>>>> This is probably an oversight.
> >>>>>
> >>>>> I've added this to my list of things to look at after the release.
> >>>> Thanks!
> >>>>
> >>> I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> >>> with following error message:
> >>>
> >>> qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> >>> qemu-system-i386: load of migration failed: Invalid argument
> >>>
> >>> With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> >>> and running with all hot plugged cpus available.
> >>>
> >>> So I think the relevant libxl information is transmitted but we also
> >>> need to fix QEMU upstream. But that's a separate issue.
> >>>
> >>> Wei.
> >>>
> >>> ---8<---
> >>> From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
> >>> From: Wei Liu <wei.liu2@citrix.com>
> >>> Date: Fri, 3 Jun 2016 16:38:32 +0100
> >>> Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config
> >>>
> >>> ... because the available vcpu bitmap can change during domain life time
> >>> due to cpu hotplug and unplug.
> >>>
> >>> Reported-by: Jan Beulich <jbeulich@suse.com>
> >>> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> >> This patch has two issues:
> >> 1. The code to allocate bitmap is wrong
> >> 2. The check to see if vcpu is available is wrong
> >>
> >> It happens to work on PV and qemu-trad because they don't really rely
> >> on the bitmap provided.
> >>
> >> #1 is fixed.
> >>
> >> For #2, I haven't really gotten to the point today on how to correctly
> >> get the number of online vcpus.
> >>
> >> The current issue I discover is that:
> >>
> >>   xl vcpu-set jessie-hvm 4
> >>   xl list -l jessie-hvm | less # search for avail_vcpus
> >>
> >> A vcpu is not really considered online from xen's point of view, unless
> >> the guest explicitly activates it, like in guest `echo 1 >
> >> .../cpu1/online`.
> >>
> >> This is still not desirable because it would still cause qemu upstream
> >> migration to fail. I will see if there is other way to figure out how
> >> many vcpus are there.
> >>
> >> Off the top of my head I might need to interrogate QEMU for that. I will
> >> continue investigation later.
> >>
> >> Any hint on how to effectively identify online vcpus is very welcomed.
> 
> Why does qemu even care?  It has nothing to do with vcpu handling. 
> There should not be any qemu vcpu records in the first place.
> 

IIRC upstream rejected the idea of having no cpu attached to a platform.

Wei.

> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Ian Jackson]

Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> On Mon, Jun 06, 2016 at 06:34:44PM +0100, Andrew Cooper wrote:
> > Why does qemu even care?  It has nothing to do with vcpu handling. 
> > There should not be any qemu vcpu records in the first place.
> 
> IIRC upstream rejected the idea of having no cpu attached to a platform.

That doesn't explain why the number of vcpus that qemu believes in has
to have anything to do with the number of vcpus that the guest has.

The qemu vcpu state is a dummy, regardless of how many of it there
are, surely ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Tue, Jun 14, 2016 at 05:34:22PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> > On Mon, Jun 06, 2016 at 06:34:44PM +0100, Andrew Cooper wrote:
> > > Why does qemu even care?  It has nothing to do with vcpu handling. 
> > > There should not be any qemu vcpu records in the first place.
> > 
> > IIRC upstream rejected the idea of having no cpu attached to a platform.
> 
> That doesn't explain why the number of vcpus that qemu believes in has
> to have anything to do with the number of vcpus that the guest has.
> 
> The qemu vcpu state is a dummy, regardless of how many of it there
> are, surely ?
> 

What Andrew means is that QEMU shouldn't have kept the CPU state
structures in the first place. My response explains why that is not
possible from a QEMU upstream point of view.

Hence the unfortunate fact is that we need to live with it for now. To
start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
All those dummy states need to be kept.

The guest is irrelevant here. We don't care about what guest thinks, but
we need to keep toolstack side view consistent so that migration and
save / restore won't fail.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Ian Jackson]

Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> What Andrew means is that QEMU shouldn't have kept the CPU state
> structures in the first place. My response explains why that is not
> possible from a QEMU upstream point of view.

I don't think it addresses my point.

> Hence the unfortunate fact is that we need to live with it for now. To
> start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
> All those dummy states need to be kept.

Why do we need one dummy state per actual vcpu rather than just one
dummy state no matter how many vcpus ?

Or is qemu involved in hvm cpu hotplug ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Andrew Cooper]

On 14/06/16 17:57, Ian Jackson wrote:
> Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
>> What Andrew means is that QEMU shouldn't have kept the CPU state
>> structures in the first place. My response explains why that is not
>> possible from a QEMU upstream point of view.
> I don't think it addresses my point.
>
>> Hence the unfortunate fact is that we need to live with it for now. To
>> start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
>> All those dummy states need to be kept.
> Why do we need one dummy state per actual vcpu rather than just one
> dummy state no matter how many vcpus ?
>
> Or is qemu involved in hvm cpu hotplug ?

Qemu has nothing to do with vcpus at all, and should not have vcpu state
in its migration stream when acting as a device model.

Someone needs to fix this upstream in Qemu, and that is the *only*
viable option here.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Tue, Jun 14, 2016 at 05:59:30PM +0100, Andrew Cooper wrote:
> On 14/06/16 17:57, Ian Jackson wrote:
> > Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> >> What Andrew means is that QEMU shouldn't have kept the CPU state
> >> structures in the first place. My response explains why that is not
> >> possible from a QEMU upstream point of view.
> > I don't think it addresses my point.
> >
> >> Hence the unfortunate fact is that we need to live with it for now. To
> >> start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
> >> All those dummy states need to be kept.
> > Why do we need one dummy state per actual vcpu rather than just one
> > dummy state no matter how many vcpus ?
> >
> > Or is qemu involved in hvm cpu hotplug ?
> 
> Qemu has nothing to do with vcpus at all, and should not have vcpu state
> in its migration stream when acting as a device model.
> 
> Someone needs to fix this upstream in Qemu, and that is the *only*
> viable option here.
> 

Correct me if I'm wrong -- are you suggesting to add a board / platform
without any CPU to QEMU? If that's the suggestion I think QEMU
maintainers have made clear they won't accept such thing.

Wei.

> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Tue, Jun 14, 2016 at 05:57:00PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> > What Andrew means is that QEMU shouldn't have kept the CPU state
> > structures in the first place. My response explains why that is not
> > possible from a QEMU upstream point of view.
> 
> I don't think it addresses my point.
> 
> > Hence the unfortunate fact is that we need to live with it for now. To
> > start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
> > All those dummy states need to be kept.
> 
> Why do we need one dummy state per actual vcpu rather than just one
> dummy state no matter how many vcpus ?
> 

We can't because ...

> Or is qemu involved in hvm cpu hotplug ?
> 

when doing hotplug, libxl uses QMP command to tell QEMU to create
CPUs.

Whether this can be changed I will let Anthony and Stefano to answer.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Tue, Jun 14, 2016 at 06:03:01PM +0100, Wei Liu wrote:
> On Tue, Jun 14, 2016 at 05:57:00PM +0100, Ian Jackson wrote:
> > Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> > > What Andrew means is that QEMU shouldn't have kept the CPU state
> > > structures in the first place. My response explains why that is not
> > > possible from a QEMU upstream point of view.
> > 
> > I don't think it addresses my point.
> > 
> > > Hence the unfortunate fact is that we need to live with it for now. To
> > > start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
> > > All those dummy states need to be kept.
> > 
> > Why do we need one dummy state per actual vcpu rather than just one
> > dummy state no matter how many vcpus ?
> > 
> 
> We can't because ...
> 
> > Or is qemu involved in hvm cpu hotplug ?
> > 
> 
> when doing hotplug, libxl uses QMP command to tell QEMU to create
> CPUs.
> 
> Whether this can be changed I will let Anthony and Stefano to answer.
> 

OK, a quick check shows that the current state of affairs does require
QEMU to get involved.

If we skip the QMP call, guest gets no new cpus -- CPU hotplug is
completely broken.

Wei.

---8<---
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 0e2c15a..1cd8e00 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -5756,6 +5756,8 @@ static int libxl__set_vcpuonline_qmp(libxl__gc
*gc, uint32_t domid,
     int i, rc;
     libxl_bitmap current_map, final_map;
 
+    return 0;
+
     libxl_bitmap_init(&current_map);
     libxl_bitmap_init(&final_map);

> Wei.
> 
> > Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Anthony PERARD]

On Tue, Jun 14, 2016 at 05:57:00PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [Xen-devel] "xl vcpu-set" not persistent across reboot?"):
> > What Andrew means is that QEMU shouldn't have kept the CPU state
> > structures in the first place. My response explains why that is not
> > possible from a QEMU upstream point of view.
> 
> I don't think it addresses my point.
> 
> > Hence the unfortunate fact is that we need to live with it for now. To
> > start QEMU we need to create a bunch of dummy CPUs to keep QEMU happy.
> > All those dummy states need to be kept.
> 
> Why do we need one dummy state per actual vcpu rather than just one
> dummy state no matter how many vcpus ?
> 
> Or is qemu involved in hvm cpu hotplug ?

It is, QEMU does the notification to the guest of a newly available CPU
via an ACPI GPE or something like that. I think QEMU manage the bitfield
of online CPUs then notify the guest about changes.

-- 
Anthony PERARD

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Jan Beulich]

>>> On 06.06.16 at 19:18, <wei.liu2@citrix.com> wrote:
> The current issue I discover is that:
> 
>   xl vcpu-set jessie-hvm 4
>   xl list -l jessie-hvm | less # search for avail_vcpus
> 
> A vcpu is not really considered online from xen's point of view, unless
> the guest explicitly activates it, like in guest `echo 1 >
> .../cpu1/online`.
> 
> This is still not desirable because it would still cause qemu upstream
> migration to fail. I will see if there is other way to figure out how
> many vcpus are there.
> 
> Off the top of my head I might need to interrogate QEMU for that. I will
> continue investigation later.
> 
> Any hint on how to effectively identify online vcpus is very welcomed.

How come xl itself doesn't know right from the "xl vcpu-set"?

And considering the guest reboot case I originally had the issue
with, the guest not activating the vCPU while still running would
not mean it won't use it post reboot. E.g. if the guest OS is not
CPU-hotplug capable, the reboot may be for the very reason of
activating the extra vCPU(s)...

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Tue, Jun 07, 2016 at 12:38:26AM -0600, Jan Beulich wrote:
> >>> On 06.06.16 at 19:18, <wei.liu2@citrix.com> wrote:
> > The current issue I discover is that:
> > 
> >   xl vcpu-set jessie-hvm 4
> >   xl list -l jessie-hvm | less # search for avail_vcpus
> > 
> > A vcpu is not really considered online from xen's point of view, unless
> > the guest explicitly activates it, like in guest `echo 1 >
> > .../cpu1/online`.
> > 
> > This is still not desirable because it would still cause qemu upstream
> > migration to fail. I will see if there is other way to figure out how
> > many vcpus are there.
> > 
> > Off the top of my head I might need to interrogate QEMU for that. I will
> > continue investigation later.
> > 
> > Any hint on how to effectively identify online vcpus is very welcomed.
> 
> How come xl itself doesn't know right from the "xl vcpu-set"?
> 

That is because:

1. libxl only pulls data from various sources when it needs to right
   at the moment you ask for domain configuration.
2. For QEMU upstream, there is no way in telling whether a hotplug is
   successful or not -- the return value is ignored because response
   from QEMU is not sensible.

Not saying these things are completely unfixable though.

Wei.

> And considering the guest reboot case I originally had the issue
> with, the guest not activating the vCPU while still running would
> not mean it won't use it post reboot. E.g. if the guest OS is not
> CPU-hotplug capable, the reboot may be for the very reason of
> activating the extra vCPU(s)...
> 
> Jan
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Jan Beulich]

(I'm sorry for the re-send - somehow Stefano's old Citrix address got
entered back into my address book.)

>>> On 03.06.16 at 18:35, <wei.liu2@citrix.com> wrote:
> I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> with following error message:
> 
> qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> qemu-system-i386: load of migration failed: Invalid argument
> 
> With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> and running with all hot plugged cpus available.
> 
> So I think the relevant libxl information is transmitted but we also
> need to fix QEMU upstream. But that's a separate issue.

Stefano, Anthony,

any thoughts here?

Thanks, Jan

> ---8<---
> From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
> From: Wei Liu <wei.liu2@citrix.com>
> Date: Fri, 3 Jun 2016 16:38:32 +0100
> Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config
> 
> ... because the available vcpu bitmap can change during domain life time
> due to cpu hotplug and unplug.
> 
> Reported-by: Jan Beulich <jbeulich@suse.com>
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ---
>  tools/libxl/libxl.c | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
> 
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 006b83f..99617f3 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -7270,6 +7270,37 @@ int libxl_retrieve_domain_configuration(libxl_ctx 
> *ctx, uint32_t domid,
>          libxl_dominfo_dispose(&info);
>      }
>  
> +    /* VCPUs */
> +    {
> +        libxl_vcpuinfo *vcpus;
> +        libxl_bitmap *map;
> +        int nr_vcpus, nr_cpus;
> +        unsigned int i;
> +
> +        vcpus = libxl_list_vcpu(ctx, domid, &nr_vcpus, &nr_cpus);
> +        if (!vcpus) {
> +            LOG(ERROR, "fail to get vcpu list for domain %d", domid);
> +            rc = ERROR_FAIL;
> +            goto out;
> +        }
> +
> +        /* Update the avail_vcpus bitmap accordingly */
> +        map = &d_config->b_info.avail_vcpus;
> +
> +        libxl_bitmap_dispose(map);
> +
> +        libxl_bitmap_alloc(ctx, map, nr_vcpus);
> +
> +        libxl_bitmap_init(map);
> +
> +        for (i = 0; i < nr_vcpus; i++) {
> +            if (vcpus[i].online)
> +                libxl_bitmap_set(map, i);
> +        }
> +
> +        libxl_vcpuinfo_list_free(vcpus, nr_vcpus);
> +    }
> +
>      /* Memory limits:
>       *
>       * Currently there are three memory limits:
> -- 
> 2.1.4




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Stefano Stabellini]

On Mon, 6 Jun 2016, Jan Beulich wrote:
> >>> On 03.06.16 at 18:35, <wei.liu2@citrix.com> wrote:
> > I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> > with following error message:
> > 
> > qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> > qemu-system-i386: load of migration failed: Invalid argument
> > 
> > With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> > and running with all hot plugged cpus available.
> > 
> > So I think the relevant libxl information is transmitted but we also
> > need to fix QEMU upstream. But that's a separate issue.

For clarity, you have applied the patch below, started a VM, hotplugged
a vcpu, rebooted the guest, then migrated the VM, but at this point
there is an error?

What are the QEMU command line arguments at the receiving side? Are you
sure that the increased vcpu count is passed to the receiving end by
libxl? It looks like QEMU has been started passing the old vcpu count as
command line argument (-smp etc) at the receiving end.


> Stefano, Anthony,
> 
> any thoughts here?
> 
> Thanks, Jan
> 
> > ---8<---
> > From 790ff77c6307b341dec0b4cc5e2d394e42f82e7c Mon Sep 17 00:00:00 2001
> > From: Wei Liu <wei.liu2@citrix.com>
> > Date: Fri, 3 Jun 2016 16:38:32 +0100
> > Subject: [PATCH] libxl: update vcpus bitmap in retrieved geust config
> > 
> > ... because the available vcpu bitmap can change during domain life time
> > due to cpu hotplug and unplug.
> > 
> > Reported-by: Jan Beulich <jbeulich@suse.com>
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> > ---
> >  tools/libxl/libxl.c | 31 +++++++++++++++++++++++++++++++
> >  1 file changed, 31 insertions(+)
> > 
> > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> > index 006b83f..99617f3 100644
> > --- a/tools/libxl/libxl.c
> > +++ b/tools/libxl/libxl.c
> > @@ -7270,6 +7270,37 @@ int libxl_retrieve_domain_configuration(libxl_ctx 
> > *ctx, uint32_t domid,
> >          libxl_dominfo_dispose(&info);
> >      }
> >  
> > +    /* VCPUs */
> > +    {
> > +        libxl_vcpuinfo *vcpus;
> > +        libxl_bitmap *map;
> > +        int nr_vcpus, nr_cpus;
> > +        unsigned int i;
> > +
> > +        vcpus = libxl_list_vcpu(ctx, domid, &nr_vcpus, &nr_cpus);
> > +        if (!vcpus) {
> > +            LOG(ERROR, "fail to get vcpu list for domain %d", domid);
> > +            rc = ERROR_FAIL;
> > +            goto out;
> > +        }
> > +
> > +        /* Update the avail_vcpus bitmap accordingly */
> > +        map = &d_config->b_info.avail_vcpus;
> > +
> > +        libxl_bitmap_dispose(map);
> > +
> > +        libxl_bitmap_alloc(ctx, map, nr_vcpus);
> > +
> > +        libxl_bitmap_init(map);
> > +
> > +        for (i = 0; i < nr_vcpus; i++) {
> > +            if (vcpus[i].online)
> > +                libxl_bitmap_set(map, i);
> > +        }
> > +
> > +        libxl_vcpuinfo_list_free(vcpus, nr_vcpus);
> > +    }
> > +
> >      /* Memory limits:
> >       *
> >       * Currently there are three memory limits:
> > -- 
> > 2.1.4
> 
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Wei Liu]

On Mon, Jun 06, 2016 at 02:07:46PM +0100, Stefano Stabellini wrote:
> On Mon, 6 Jun 2016, Jan Beulich wrote:
> > >>> On 03.06.16 at 18:35, <wei.liu2@citrix.com> wrote:
> > > I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> > > with following error message:
> > > 
> > > qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> > > qemu-system-i386: load of migration failed: Invalid argument
> > > 
> > > With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> > > and running with all hot plugged cpus available.
> > > 
> > > So I think the relevant libxl information is transmitted but we also
> > > need to fix QEMU upstream. But that's a separate issue.
> 
> For clarity, you have applied the patch below, started a VM, hotplugged
> a vcpu, rebooted the guest, then migrated the VM, but at this point
> there is an error?
> 

Apply this patch, start a guest, hotplug some cpus, make them online
inside guest, and xl migrate guest localhost.

You will see this in qemu log.

> What are the QEMU command line arguments at the receiving side? Are you
> sure that the increased vcpu count is passed to the receiving end by
> libxl? It looks like QEMU has been started passing the old vcpu count as
> command line argument (-smp etc) at the receiving end.
> 

The QEMU command line should be the same on the sending side.

The -smp 1,maxvcpus=4 (something like that).

Does that mean we need to somehow alter QEMU's command line?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: "xl vcpu-set" not persistent across reboot?

[Stefano Stabellini]

On Mon, 6 Jun 2016, Wei Liu wrote:
> On Mon, Jun 06, 2016 at 02:07:46PM +0100, Stefano Stabellini wrote:
> > On Mon, 6 Jun 2016, Jan Beulich wrote:
> > > >>> On 03.06.16 at 18:35, <wei.liu2@citrix.com> wrote:
> > > > I got a patch ready.  But QEMU upstream refuses to start on the receiving end
> > > > with following error message:
> > > > 
> > > > qemu-system-i386: Unknown savevm section or instance 'cpu_common' 1
> > > > qemu-system-i386: load of migration failed: Invalid argument
> > > > 
> > > > With QEMU traditional HVM guest and PV guest, the guest works fine -- up
> > > > and running with all hot plugged cpus available.
> > > > 
> > > > So I think the relevant libxl information is transmitted but we also
> > > > need to fix QEMU upstream. But that's a separate issue.
> > 
> > For clarity, you have applied the patch below, started a VM, hotplugged
> > a vcpu, rebooted the guest, then migrated the VM, but at this point
> > there is an error?
> > 
> 
> Apply this patch, start a guest, hotplug some cpus, make them online
> inside guest, and xl migrate guest localhost.
> 
> You will see this in qemu log.
> 
> > What are the QEMU command line arguments at the receiving side? Are you
> > sure that the increased vcpu count is passed to the receiving end by
> > libxl? It looks like QEMU has been started passing the old vcpu count as
> > command line argument (-smp etc) at the receiving end.
> > 
> 
> The QEMU command line should be the same on the sending side.
> 
> The -smp 1,maxvcpus=4 (something like that).
> 
> Does that mean we need to somehow alter QEMU's command line?

Yes, that's right. The device state that we pass to QEMU is just the
state of the devices specified by the command line args.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel