Error during domain creation on xen 4.8

Error during domain creation on xen 4.8

[Sarah Newman]

Hi,

With xen 4.8.1, I got the error message:

libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2098, wanted dom2097
: No such file or directory

This was while creating an HVM domain with a stub domain, probably concurrent to creating a PV domain. The domains were created as 2098 + 2099 I
believe. I was also using oxenstored.

Please keep me CC'ed as I am not subscribed.

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Error during domain creation on xen 4.8

[Wei Liu]

On Tue, Jun 13, 2017 at 09:29:22AM -0700, Sarah Newman wrote:
> Hi,
> 
> With xen 4.8.1, I got the error message:
> 
> libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2098, wanted dom2097
> : No such file or directory
> 
> This was while creating an HVM domain with a stub domain, probably concurrent to creating a PV domain. The domains were created as 2098 + 2099 I
> believe. I was also using oxenstored.

The xc_domain_getinfolist function is a bit awful to work with.

Can you reliable reproduce this? If so, can you provide xl -vvv create
output?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Error during domain creation on xen 4.8

[Wei Liu]

On Tue, Jun 13, 2017 at 05:56:26PM +0100, Wei Liu wrote:
> On Tue, Jun 13, 2017 at 09:29:22AM -0700, Sarah Newman wrote:
> > Hi,
> > 
> > With xen 4.8.1, I got the error message:
> > 
> > libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2098, wanted dom2097
> > : No such file or directory
> > 
> > This was while creating an HVM domain with a stub domain, probably concurrent to creating a PV domain. The domains were created as 2098 + 2099 I
> > believe. I was also using oxenstored.
> 
> The xc_domain_getinfolist function is a bit awful to work with.

To give you a bit more information: that function searches for N domains
starting from the one you specify.

That the parameter was 2097 but there wasn't dom2097 in Xen, so Xen
returned 2098. The toolstack (correctly) bailed out.

I'm still a bit baffled at this point. Maybe 2097 crashed?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Error during domain creation on xen 4.8

[Sarah Newman]

On 06/13/2017 10:08 AM, Wei Liu wrote:
> On Tue, Jun 13, 2017 at 05:56:26PM +0100, Wei Liu wrote:
>> On Tue, Jun 13, 2017 at 09:29:22AM -0700, Sarah Newman wrote:
>>> Hi,
>>>
>>> With xen 4.8.1, I got the error message:
>>>
>>> libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2098, wanted dom2097
>>> : No such file or directory
>>>
>>> This was while creating an HVM domain with a stub domain, probably concurrent to creating a PV domain. The domains were created as 2098 + 2099 I
>>> believe. I was also using oxenstored.
>>
>> The xc_domain_getinfolist function is a bit awful to work with.
> 
> To give you a bit more information: that function searches for N domains
> starting from the one you specify.
> 
> That the parameter was 2097 but there wasn't dom2097 in Xen, so Xen
> returned 2098. The toolstack (correctly) bailed out.
> 
> I'm still a bit baffled at this point. Maybe 2097 crashed?
> 

It was probably a domain shut down on purpose. Rerunning with -vvv.

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Error during domain creation on xen 4.8

[Sarah Newman]

On 06/13/2017 10:28 AM, Sarah Newman wrote:
> On 06/13/2017 10:08 AM, Wei Liu wrote:
>> On Tue, Jun 13, 2017 at 05:56:26PM +0100, Wei Liu wrote:
>>> On Tue, Jun 13, 2017 at 09:29:22AM -0700, Sarah Newman wrote:
>>>> Hi,
>>>>
>>>> With xen 4.8.1, I got the error message:
>>>>
>>>> libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2098, wanted dom2097
>>>> : No such file or directory
>>>>
>>>> This was while creating an HVM domain with a stub domain, probably concurrent to creating a PV domain. The domains were created as 2098 + 2099 I
>>>> believe. I was also using oxenstored.
>>>
>>> The xc_domain_getinfolist function is a bit awful to work with.
>>
>> To give you a bit more information: that function searches for N domains
>> starting from the one you specify.
>>
>> That the parameter was 2097 but there wasn't dom2097 in Xen, so Xen
>> returned 2098. The toolstack (correctly) bailed out.
>>
>> I'm still a bit baffled at this point. Maybe 2097 crashed?
>>
> 
> It was probably a domain shut down on purpose. Rerunning with -vvv.
> 
> --Sarah
> 

Could this be happening from the block script? There's a function that uses xenstore (I'm using oxenstored) and checks whether a given block device is
in use by the dom0 or another guest.

libxl: debug: libxl_device.c:1157:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
libxl: debug: libxl_device.c:1158:device_hotplug: extra args:
libxl: debug: libxl_device.c:1166:device_hotplug: env:
libxl: debug: libxl_device.c:1173:device_hotplug:       script: /etc/xen/scripts/block
libxl: debug: libxl_device.c:1173:device_hotplug:       XENBUS_TYPE: vbd
libxl: debug: libxl_device.c:1173:device_hotplug:       XENBUS_PATH: backend/vbd/2853/268441856
libxl: debug: libxl_device.c:1173:device_hotplug:       XENBUS_BASE_PATH: backend
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block add
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x1ad98f0: deregister unregistered
libxl: debug: libxl_linux.c:200:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts
libxl: debug: libxl_device.c:1143:device_hotplug: No hotplug script to execute
libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x1ad98f0: deregister unregistered
libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2852, wanted dom2851
: No such file or directory
libxl: debug: libxl_numa.c:502:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=20, nr_vcpus=184, free_memkb=151897
libxl: debug: libxl_numa.c:502:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=20, nr_vcpus=183, free_memkb=157905


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Error during domain creation on xen 4.8

[Wei Liu]

On Tue, Jun 13, 2017 at 11:27:27AM -0700, Sarah Newman wrote:
> On 06/13/2017 10:28 AM, Sarah Newman wrote:
> > On 06/13/2017 10:08 AM, Wei Liu wrote:
> >> On Tue, Jun 13, 2017 at 05:56:26PM +0100, Wei Liu wrote:
> >>> On Tue, Jun 13, 2017 at 09:29:22AM -0700, Sarah Newman wrote:
> >>>> Hi,
> >>>>
> >>>> With xen 4.8.1, I got the error message:
> >>>>
> >>>> libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2098, wanted dom2097
> >>>> : No such file or directory
> >>>>
> >>>> This was while creating an HVM domain with a stub domain, probably concurrent to creating a PV domain. The domains were created as 2098 + 2099 I
> >>>> believe. I was also using oxenstored.
> >>>
> >>> The xc_domain_getinfolist function is a bit awful to work with.
> >>
> >> To give you a bit more information: that function searches for N domains
> >> starting from the one you specify.
> >>
> >> That the parameter was 2097 but there wasn't dom2097 in Xen, so Xen
> >> returned 2098. The toolstack (correctly) bailed out.
> >>
> >> I'm still a bit baffled at this point. Maybe 2097 crashed?
> >>
> > 
> > It was probably a domain shut down on purpose. Rerunning with -vvv.
> > 
> > --Sarah
> > 
> 
> Could this be happening from the block script? There's a function that uses xenstore (I'm using oxenstored) and checks whether a given block device is
> in use by the dom0 or another guest.
> 
> libxl: debug: libxl_device.c:1157:device_hotplug: calling hotplug script: /etc/xen/scripts/block add
> libxl: debug: libxl_device.c:1158:device_hotplug: extra args:
> libxl: debug: libxl_device.c:1166:device_hotplug: env:
> libxl: debug: libxl_device.c:1173:device_hotplug:       script: /etc/xen/scripts/block
> libxl: debug: libxl_device.c:1173:device_hotplug:       XENBUS_TYPE: vbd
> libxl: debug: libxl_device.c:1173:device_hotplug:       XENBUS_PATH: backend/vbd/2853/268441856
> libxl: debug: libxl_device.c:1173:device_hotplug:       XENBUS_BASE_PATH: backend
> libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to execute: /etc/xen/scripts/block add
> libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x1ad98f0: deregister unregistered
> libxl: debug: libxl_linux.c:200:libxl__get_hotplug_script_info: num_exec 1, not running hotplug scripts
> libxl: debug: libxl_device.c:1143:device_hotplug: No hotplug script to execute
> libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch w=0x1ad98f0: deregister unregistered
> libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for dom2852, wanted dom2851
> : No such file or directory
> libxl: debug: libxl_numa.c:502:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=20, nr_vcpus=184, free_memkb=151897
> libxl: debug: libxl_numa.c:502:libxl__get_numa_candidate: New best NUMA placement candidate found: nr_nodes=1, nr_cpus=20, nr_vcpus=183, free_memkb=157905
> 

No likely from the block script. The code in question is trying to get
cpupool / NUMA information. Block does neither of those.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Error during domain creation on xen 4.8

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3383 bytes --]

On Wed, 2017-06-14 at 10:38 +0100, Wei Liu wrote:
> On Tue, Jun 13, 2017 at 11:27:27AM -0700, Sarah Newman wrote:
> > On 06/13/2017 10:28 AM, Sarah Newman wrote:
> > libxl: debug: libxl_device.c:1157:device_hotplug: calling hotplug
> > script: /etc/xen/scripts/block add
> > libxl: debug: libxl_device.c:1158:device_hotplug: extra args:
> > libxl: debug: libxl_device.c:1166:device_hotplug: env:
> > libxl: debug: libxl_device.c:1173:device_hotplug:       script:
> > /etc/xen/scripts/block
> > libxl: debug:
> > libxl_device.c:1173:device_hotplug:       XENBUS_TYPE: vbd
> > libxl: debug:
> > libxl_device.c:1173:device_hotplug:       XENBUS_PATH:
> > backend/vbd/2853/268441856
> > libxl: debug:
> > libxl_device.c:1173:device_hotplug:       XENBUS_BASE_PATH: backend
> > libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking
> > to execute: /etc/xen/scripts/block add
> > libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
> > w=0x1ad98f0: deregister unregistered
> > libxl: debug: libxl_linux.c:200:libxl__get_hotplug_script_info:
> > num_exec 1, not running hotplug scripts
> > libxl: debug: libxl_device.c:1143:device_hotplug: No hotplug script
> > to execute
> > libxl: debug: libxl_event.c:686:libxl__ev_xswatch_deregister: watch
> > w=0x1ad98f0: deregister unregistered
> > libxl: error: libxl_dom.c:60:libxl__domain_cpupool: got info for
> > dom2852, wanted dom2851
> > : No such file or directory
> > libxl: debug: libxl_numa.c:502:libxl__get_numa_candidate: New best
> > NUMA placement candidate found: nr_nodes=1, nr_cpus=20,
> > nr_vcpus=184, free_memkb=151897
> > libxl: debug: libxl_numa.c:502:libxl__get_numa_candidate: New best
> > NUMA placement candidate found: nr_nodes=1, nr_cpus=20,
> > nr_vcpus=183, free_memkb=157905
> > 
> 
> No likely from the block script. The code in question is trying to
> get
> cpupool / NUMA information. Block does neither of those.
>
But then (as, in fact, is also said in the first message of the thread)
  it goes on, and domains are created, aren't they?

I believe this is this message is produced by this call:

static int nr_vcpus_on_nodes()
{
    dinfo = libxl_list_domain(CTX, &nr_doms);
    if (dinfo == NULL)
        return ERROR_FAIL;
    ...
    for (i = 0; i < nr_doms; i++) {
    ...
        cpupool = libxl__domain_cpupool(gc, dinfo[i].domid);
        if (cpupool < 0)
            goto next;
    ...
}

in libxl_numa.c.

What appears to have happened is that a domain disappeared (was
shutdown, was destroyed, crashed, whatever) between when we got the
list of existing domains (with libxl_list_domain()), and when we go
through it, with the for().

Looking at other places, in libxl and xl, where libxl_list_domain() is
used, that happens in a similar way, and I don't think we have anything
in place for preventing this race to happen... and in fact, is not
considered fatal.

Perhaps we should reorg logging so that it will produce a WARN instead
of an ERROR?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel