dom0 kenrel crashes for openstack + libvirt + libxl

dom0 kenrel crashes for openstack + libvirt + libxl

[Xing Lin]

[-- Attachment #1.1: Type: text/plain, Size: 4538 bytes --]

Hi,

I am aware that Xen via libvirt is in the group C support for openstack but
since I am not able to install xenserver iso at compute machines I have, I
have to consider to use xen with libvirt (xcp-xapi is not available for
ubuntu14.04). I have three nodes, each one running ubuntu 14.04. I follow
the instruction to install juno in ubuntu 14.04 and it works (I can create
instances from openstack GUI - horizon) when I use kvm as the hypervisor at
the compute node . However, if I switch to use xen as the hypervisor by
installing xen-hypervisor-amd64 or nova-compute-xen, it will fail to create
instances. It complained "backend for qemu disk is not ready". I checked
and did not find qemu process running in dom0. I could not find
/etc/init.d/xencommons either. Then, I compiled and installed xen-4.4 from
source code and I got xencommons in /etc/init.d. qemu process is running in
dom0. However, the dom0 kernel crashes this time when I try to launch an
instance from openstack GUI. I am able to create a xen guest with
virt-install from command line, with the following command.

"$ virt-install --connect=xen:/// --name u14.04.3 --ram 1024 --disk
u14.04.3.img,size=4,driver_name=qemu --location
http://ftp.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/ --network
bridge=virbr0"


dmesg:

[ 9443.130600] blkfront: xvda: flush diskcache: enabled; persistent grants:
enabled; indirect descriptors: disabled;
[ 9443.132818]  xvda: xvda1
[ 9444.604489] xen:grant_table: WARNING: g.e. 0x30 still in use!
[ 9444.604496] deferring g.e. 0x30 (pfn 0xffffffffffffffff)
[ 9444.604499] xen:grant_table: WARNING: g.e. 0x31 still in use!
[ 9444.604502] deferring g.e. 0x31 (pfn 0xffffffffffffffff)
[ 9444.604505] xen:grant_table: WARNING: g.e. 0x32 still in use!
[ 9444.604508] deferring g.e. 0x32 (pfn 0xffffffffffffffff)
  ==== lots of them====
[ 9444.604719] xen:grant_table: WARNING: g.e. 0xe still in use!
[ 9444.604721] deferring g.e. 0xe (pfn 0xffffffffffffffff)
[ 9444.604723] xen:grant_table: WARNING: g.e. 0xd still in use!
[ 9444.604725] deferring g.e. 0xd (pfn 0xffffffffffffffff)
[ 9448.325408] ------------[ cut here ]------------
[ 9448.325421] WARNING: CPU: 5 PID: 19902 at
/build/buildd/linux-3.13.0/arch/x86/xen/multicalls.c:129 xen_mc_flush+0x
1a9/0x1b0()
[ 9448.325492] CPU: 5 PID: 19902 Comm: sudo Tainted: GF          O
3.13.0-33-generic #58-Ubuntu
[ 9448.325494] Hardware name: Dell Inc. PowerEdge R710/00W9X3, BIOS 2.1.15
09/02/2010
[ 9448.325497]  0000000000000009 ffff8802d13d9aa8 ffffffff8171bd04
0000000000000000
[ 9448.325501]  ffff8802d13d9ae0 ffffffff810676cd 0000000000000000
0000000000000001
[ 9448.325505]  ffff88030418ffe0 ffff88031d6ab180 0000000000000010
ffff8802d13d9af0
[ 9448.325509] Call Trace:
[ 9448.325518]  [<ffffffff8171bd04>] dump_stack+0x45/0x56
[ 9448.325523]  [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
[ 9448.325526]  [<ffffffff810677aa>] warn_slowpath_null+0x1a/0x20
[ 9448.325530]  [<ffffffff81004e99>] xen_mc_flush+0x1a9/0x1b0
[ 9448.325534]  [<ffffffff81006b99>] xen_set_pud_hyper+0x109/0x110
[ 9448.325538]  [<ffffffff81006c3b>] xen_set_pud+0x9b/0xb0
[ 9448.325543]  [<ffffffff81177f06>] __pmd_alloc+0xd6/0x110
[ 9448.325548]  [<ffffffff81182218>] move_page_tables+0x678/0x720
[ 9448.325552]  [<ffffffff8117ddb7>] ? vma_adjust+0x337/0x7d0
[ 9448.325557]  [<ffffffff811c23fd>] shift_arg_pages+0xbd/0x1b0
[ 9448.325560]  [<ffffffff81181671>] ? mprotect_fixup+0x151/0x290
[ 9448.325564]  [<ffffffff811c26cb>] setup_arg_pages+0x1db/0x200
[ 9448.325570]  [<ffffffff81213edc>] load_elf_binary+0x41c/0xd80
[ 9448.325576]  [<ffffffff8131d503>] ? ima_get_action+0x23/0x30
[ 9448.325579]  [<ffffffff8131c7e2>] ? process_measurement+0x82/0x2c0
[ 9448.325584]  [<ffffffff811c2f7f>] search_binary_handler+0x8f/0x1b0
[ 9448.325587]  [<ffffffff811c44f7>] do_execve_common.isra.22+0x5a7/0x7e0
[ 9448.325591]  [<ffffffff811c49c6>] SyS_execve+0x36/0x50
[ 9448.325596]  [<ffffffff8172cc99>] stub_execve+0x69/0xa0
[ 9448.325599] ---[ end trace 53ac16782e751c0a ]---
[ 9448.347994] ------------[ cut here ]------------
[ 9448.348004] WARNING: CPU: 1 PID: 19902 at
/build/buildd/linux-3.13.0/mm/mmap.c:2736 exit_mmap+0x16b/0x170()
==== more message ignored ====


I sent emails to the openstack mailing list and they gave me a few
suggestions. At last, Bob Ball suggested I should probably ask this
question on xen-devel mailing list. Any help will be highly appreciated!
Thanks,

Original discussion on openstack mailing list:
http://www.gossamer-threads.com/lists/openstack/dev/42337


-Xing

[-- Attachment #1.2: Type: text/html, Size: 7177 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Wed, 2014-11-12 at 11:41 -0700, Xing Lin wrote:

> 
> [ 9448.347994] ------------[ cut here ]------------
> [ 9448.348004] WARNING: CPU: 1 PID: 19902 at /build/buildd/linux-3.13.0/mm/mmap.c:2736 exit_mmap+0x16b/0x170()
> ==== more message ignored ====

The most interesting bits of the message are likely to be the lines
which follow here, i.e. the kernel stack trace and associated info.
Please can you paste the complete log from "cut here" until the end.

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[David Vrabel]

On 12/11/14 18:41, Xing Lin wrote:
> Hi, 
> 
> I am aware that Xen via libvirt is in the group C support for openstack
> but since I am not able to install xenserver iso at compute machines I
> have, I have to consider to use xen with libvirt (xcp-xapi is not
> available for ubuntu14.04). I have three nodes, each one running ubuntu
> 14.04. I follow the instruction to install juno in ubuntu 14.04 and it
> works (I can create instances from openstack GUI - horizon) when I use
> kvm as the hypervisor at the compute node . However, if I switch to use
> xen as the hypervisor by installing xen-hypervisor-amd64 or
> nova-compute-xen, it will fail to create instances. It complained
> "backend for qemu disk is not ready". I checked and did not find qemu
> process running in dom0. I could not find /etc/init.d/xencommons
> either. Then, I compiled and installed xen-4.4 from source code and I
> got xencommons in /etc/init.d. qemu process is running in dom0. However,
> the dom0 kernel crashes this time when I try to launch an instance from
> openstack GUI. I am able to create a xen guest with virt-install from
> command line, with the following command. 
> 
> "$ virt-install --connect=xen:/// --name u14.04.3 --ram 1024 --disk
> u14.04.3.img,size=4,driver_name=qemu
> --location http://ftp.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/ --network
> bridge=virbr0"

Does this ubuntu bug look relevant?

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350373

David

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Roger Pau Monné]

El 12/11/14 a les 19.41, Xing Lin ha escrit:
> Hi,
> 
> I am aware that Xen via libvirt is in the group C support for openstack but
> since I am not able to install xenserver iso at compute machines I have, I
> have to consider to use xen with libvirt (xcp-xapi is not available for
> ubuntu14.04). I have three nodes, each one running ubuntu 14.04. I follow
> the instruction to install juno in ubuntu 14.04 and it works (I can create
> instances from openstack GUI - horizon) when I use kvm as the hypervisor at
> the compute node . However, if I switch to use xen as the hypervisor by
> installing xen-hypervisor-amd64 or nova-compute-xen, it will fail to create
> instances. It complained "backend for qemu disk is not ready". I checked
> and did not find qemu process running in dom0. I could not find
> /etc/init.d/xencommons either. Then, I compiled and installed xen-4.4 from
> source code and I got xencommons in /etc/init.d. qemu process is running in
> dom0. However, the dom0 kernel crashes this time when I try to launch an
> instance from openstack GUI. I am able to create a xen guest with
> virt-install from command line, with the following command.
> 
> "$ virt-install --connect=xen:/// --name u14.04.3 --ram 1024 --disk
> u14.04.3.img,size=4,driver_name=qemu --location
> http://ftp.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/ --network
> bridge=virbr0"
> 
> 
> dmesg:
> 
> [ 9443.130600] blkfront: xvda: flush diskcache: enabled; persistent grants:
> enabled; indirect descriptors: disabled;
> [ 9443.132818]  xvda: xvda1
> [ 9444.604489] xen:grant_table: WARNING: g.e. 0x30 still in use!
> [ 9444.604496] deferring g.e. 0x30 (pfn 0xffffffffffffffff)
> [ 9444.604499] xen:grant_table: WARNING: g.e. 0x31 still in use!
> [ 9444.604502] deferring g.e. 0x31 (pfn 0xffffffffffffffff)
> [ 9444.604505] xen:grant_table: WARNING: g.e. 0x32 still in use!
> [ 9444.604508] deferring g.e. 0x32 (pfn 0xffffffffffffffff)
>   ==== lots of them====
> [ 9444.604719] xen:grant_table: WARNING: g.e. 0xe still in use!
> [ 9444.604721] deferring g.e. 0xe (pfn 0xffffffffffffffff)
> [ 9444.604723] xen:grant_table: WARNING: g.e. 0xd still in use!
> [ 9444.604725] deferring g.e. 0xd (pfn 0xffffffffffffffff)
> [ 9448.325408] ------------[ cut here ]------------
> [ 9448.325421] WARNING: CPU: 5 PID: 19902 at
> /build/buildd/linux-3.13.0/arch/x86/xen/multicalls.c:129 xen_mc_flush+0x
> 1a9/0x1b0()
> [ 9448.325492] CPU: 5 PID: 19902 Comm: sudo Tainted: GF          O
> 3.13.0-33-generic #58-Ubuntu
> [ 9448.325494] Hardware name: Dell Inc. PowerEdge R710/00W9X3, BIOS 2.1.15
> 09/02/2010
> [ 9448.325497]  0000000000000009 ffff8802d13d9aa8 ffffffff8171bd04
> 0000000000000000
> [ 9448.325501]  ffff8802d13d9ae0 ffffffff810676cd 0000000000000000
> 0000000000000001
> [ 9448.325505]  ffff88030418ffe0 ffff88031d6ab180 0000000000000010
> ffff8802d13d9af0
> [ 9448.325509] Call Trace:
> [ 9448.325518]  [<ffffffff8171bd04>] dump_stack+0x45/0x56
> [ 9448.325523]  [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
> [ 9448.325526]  [<ffffffff810677aa>] warn_slowpath_null+0x1a/0x20
> [ 9448.325530]  [<ffffffff81004e99>] xen_mc_flush+0x1a9/0x1b0
> [ 9448.325534]  [<ffffffff81006b99>] xen_set_pud_hyper+0x109/0x110
> [ 9448.325538]  [<ffffffff81006c3b>] xen_set_pud+0x9b/0xb0
> [ 9448.325543]  [<ffffffff81177f06>] __pmd_alloc+0xd6/0x110
> [ 9448.325548]  [<ffffffff81182218>] move_page_tables+0x678/0x720
> [ 9448.325552]  [<ffffffff8117ddb7>] ? vma_adjust+0x337/0x7d0
> [ 9448.325557]  [<ffffffff811c23fd>] shift_arg_pages+0xbd/0x1b0
> [ 9448.325560]  [<ffffffff81181671>] ? mprotect_fixup+0x151/0x290
> [ 9448.325564]  [<ffffffff811c26cb>] setup_arg_pages+0x1db/0x200
> [ 9448.325570]  [<ffffffff81213edc>] load_elf_binary+0x41c/0xd80
> [ 9448.325576]  [<ffffffff8131d503>] ? ima_get_action+0x23/0x30
> [ 9448.325579]  [<ffffffff8131c7e2>] ? process_measurement+0x82/0x2c0
> [ 9448.325584]  [<ffffffff811c2f7f>] search_binary_handler+0x8f/0x1b0
> [ 9448.325587]  [<ffffffff811c44f7>] do_execve_common.isra.22+0x5a7/0x7e0
> [ 9448.325591]  [<ffffffff811c49c6>] SyS_execve+0x36/0x50
> [ 9448.325596]  [<ffffffff8172cc99>] stub_execve+0x69/0xa0
> [ 9448.325599] ---[ end trace 53ac16782e751c0a ]---
> [ 9448.347994] ------------[ cut here ]------------
> [ 9448.348004] WARNING: CPU: 1 PID: 19902 at
> /build/buildd/linux-3.13.0/mm/mmap.c:2736 exit_mmap+0x16b/0x170()
> ==== more message ignored ====
> 
> 
> I sent emails to the openstack mailing list and they gave me a few
> suggestions. At last, Bob Ball suggested I should probably ask this
> question on xen-devel mailing list. Any help will be highly appreciated!

This looks like the bug already reported by George Dunlap, could you try
to apply the following patch to qemu-xen and recompile?

http://lists.xenproject.org/archives/html/xen-devel/2014-11/msg01112.html

Roger.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Thu, 2014-11-13 at 08:12 +0000, Ian Campbell wrote:
> On Wed, 2014-11-12 at 11:41 -0700, Xing Lin wrote:
> 
> > 
> > [ 9448.347994] ------------[ cut here ]------------
> > [ 9448.348004] WARNING: CPU: 1 PID: 19902 at /build/buildd/linux-3.13.0/mm/mmap.c:2736 exit_mmap+0x16b/0x170()
> > ==== more message ignored ====
> 
> The most interesting bits of the message are likely to be the lines
> which follow here, i.e. the kernel stack trace and associated info.
> Please can you paste the complete log from "cut here" until the end.

Sorry, it seems my eye somehow jumped from the "g.e. 0x30 still in use!"
bits to the end, missing the stack trace right there in front of me.
Looks like others on the thread are more on the ball and pointing in the
right directions.

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 5386 bytes --]

On 13.11.2014 11:57, Roger Pau Monné wrote:
> El 12/11/14 a les 19.41, Xing Lin ha escrit:
>> Hi,
>>
>> I am aware that Xen via libvirt is in the group C support for openstack but
>> since I am not able to install xenserver iso at compute machines I have, I
>> have to consider to use xen with libvirt (xcp-xapi is not available for
>> ubuntu14.04). I have three nodes, each one running ubuntu 14.04. I follow
>> the instruction to install juno in ubuntu 14.04 and it works (I can create
>> instances from openstack GUI - horizon) when I use kvm as the hypervisor at
>> the compute node . However, if I switch to use xen as the hypervisor by
>> installing xen-hypervisor-amd64 or nova-compute-xen, it will fail to create
>> instances. It complained "backend for qemu disk is not ready". I checked
>> and did not find qemu process running in dom0. I could not find
>> /etc/init.d/xencommons either. Then, I compiled and installed xen-4.4 from
>> source code and I got xencommons in /etc/init.d. qemu process is running in
>> dom0. However, the dom0 kernel crashes this time when I try to launch an
>> instance from openstack GUI. I am able to create a xen guest with
>> virt-install from command line, with the following command.
>>
>> "$ virt-install --connect=xen:/// --name u14.04.3 --ram 1024 --disk
>> u14.04.3.img,size=4,driver_name=qemu --location
>> http://ftp.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/ --network
>> bridge=virbr0"
>>
>>
>> dmesg:
>>
>> [ 9443.130600] blkfront: xvda: flush diskcache: enabled; persistent grants:
>> enabled; indirect descriptors: disabled;
>> [ 9443.132818]  xvda: xvda1
>> [ 9444.604489] xen:grant_table: WARNING: g.e. 0x30 still in use!
>> [ 9444.604496] deferring g.e. 0x30 (pfn 0xffffffffffffffff)
>> [ 9444.604499] xen:grant_table: WARNING: g.e. 0x31 still in use!
>> [ 9444.604502] deferring g.e. 0x31 (pfn 0xffffffffffffffff)
>> [ 9444.604505] xen:grant_table: WARNING: g.e. 0x32 still in use!
>> [ 9444.604508] deferring g.e. 0x32 (pfn 0xffffffffffffffff)
>>   ==== lots of them====
>> [ 9444.604719] xen:grant_table: WARNING: g.e. 0xe still in use!
>> [ 9444.604721] deferring g.e. 0xe (pfn 0xffffffffffffffff)
>> [ 9444.604723] xen:grant_table: WARNING: g.e. 0xd still in use!
>> [ 9444.604725] deferring g.e. 0xd (pfn 0xffffffffffffffff)
>> [ 9448.325408] ------------[ cut here ]------------
>> [ 9448.325421] WARNING: CPU: 5 PID: 19902 at
>> /build/buildd/linux-3.13.0/arch/x86/xen/multicalls.c:129 xen_mc_flush+0x
>> 1a9/0x1b0()
>> [ 9448.325492] CPU: 5 PID: 19902 Comm: sudo Tainted: GF          O
>> 3.13.0-33-generic #58-Ubuntu
>> [ 9448.325494] Hardware name: Dell Inc. PowerEdge R710/00W9X3, BIOS 2.1.15
>> 09/02/2010
>> [ 9448.325497]  0000000000000009 ffff8802d13d9aa8 ffffffff8171bd04
>> 0000000000000000
>> [ 9448.325501]  ffff8802d13d9ae0 ffffffff810676cd 0000000000000000
>> 0000000000000001
>> [ 9448.325505]  ffff88030418ffe0 ffff88031d6ab180 0000000000000010
>> ffff8802d13d9af0
>> [ 9448.325509] Call Trace:
>> [ 9448.325518]  [<ffffffff8171bd04>] dump_stack+0x45/0x56
>> [ 9448.325523]  [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
>> [ 9448.325526]  [<ffffffff810677aa>] warn_slowpath_null+0x1a/0x20
>> [ 9448.325530]  [<ffffffff81004e99>] xen_mc_flush+0x1a9/0x1b0
>> [ 9448.325534]  [<ffffffff81006b99>] xen_set_pud_hyper+0x109/0x110
>> [ 9448.325538]  [<ffffffff81006c3b>] xen_set_pud+0x9b/0xb0
>> [ 9448.325543]  [<ffffffff81177f06>] __pmd_alloc+0xd6/0x110
>> [ 9448.325548]  [<ffffffff81182218>] move_page_tables+0x678/0x720
>> [ 9448.325552]  [<ffffffff8117ddb7>] ? vma_adjust+0x337/0x7d0
>> [ 9448.325557]  [<ffffffff811c23fd>] shift_arg_pages+0xbd/0x1b0
>> [ 9448.325560]  [<ffffffff81181671>] ? mprotect_fixup+0x151/0x290
>> [ 9448.325564]  [<ffffffff811c26cb>] setup_arg_pages+0x1db/0x200
>> [ 9448.325570]  [<ffffffff81213edc>] load_elf_binary+0x41c/0xd80
>> [ 9448.325576]  [<ffffffff8131d503>] ? ima_get_action+0x23/0x30
>> [ 9448.325579]  [<ffffffff8131c7e2>] ? process_measurement+0x82/0x2c0
>> [ 9448.325584]  [<ffffffff811c2f7f>] search_binary_handler+0x8f/0x1b0
>> [ 9448.325587]  [<ffffffff811c44f7>] do_execve_common.isra.22+0x5a7/0x7e0
>> [ 9448.325591]  [<ffffffff811c49c6>] SyS_execve+0x36/0x50
>> [ 9448.325596]  [<ffffffff8172cc99>] stub_execve+0x69/0xa0
>> [ 9448.325599] ---[ end trace 53ac16782e751c0a ]---
>> [ 9448.347994] ------------[ cut here ]------------
>> [ 9448.348004] WARNING: CPU: 1 PID: 19902 at
>> /build/buildd/linux-3.13.0/mm/mmap.c:2736 exit_mmap+0x16b/0x170()
>> ==== more message ignored ====
>>
>>
>> I sent emails to the openstack mailing list and they gave me a few
>> suggestions. At last, Bob Ball suggested I should probably ask this
>> question on xen-devel mailing list. Any help will be highly appreciated!
> 
> This looks like the bug already reported by George Dunlap, could you try
> to apply the following patch to qemu-xen and recompile?
> 
> http://lists.xenproject.org/archives/html/xen-devel/2014-11/msg01112.html

If that turns out to be the problem we need to get a bug reported against the
qemu package as the distro packaged Xen uses stock qemu with xl.

-Stefan

> 
> Roger.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Xing Lin]

[-- Attachment #1.1: Type: text/plain, Size: 6151 bytes --]

Hi,

Roger is correct: after I applied that patch and reinstalled qemu, modified
/etc/init.d/xencommons to use the modified version of qemu to provide disk
backend, I finally succeeded to create a xen guest! I noticed that this
patch was proposed yesterday: just in-time to solve my problem. :-)


root@compute1:/mnt/qemu-2.0.2# virsh --connect=xen:///
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list
 Id    Name                           State
----------------------------------------------------
 1     instance-00000028              running

virsh #

I have been blocked by this problem for almost three weeks and as far as I
know, I could not find any online tutorial on setting up xen based compute
node for openstack juno. I will document steps I take and share them with
the community.

Cheers,
-Xing

On Thu, Nov 13, 2014 at 3:57 AM, Roger Pau Monné <roger.pau@citrix.com>
wrote:

> El 12/11/14 a les 19.41, Xing Lin ha escrit:
> > Hi,
> >
> > I am aware that Xen via libvirt is in the group C support for openstack
> but
> > since I am not able to install xenserver iso at compute machines I have,
> I
> > have to consider to use xen with libvirt (xcp-xapi is not available for
> > ubuntu14.04). I have three nodes, each one running ubuntu 14.04. I follow
> > the instruction to install juno in ubuntu 14.04 and it works (I can
> create
> > instances from openstack GUI - horizon) when I use kvm as the hypervisor
> at
> > the compute node . However, if I switch to use xen as the hypervisor by
> > installing xen-hypervisor-amd64 or nova-compute-xen, it will fail to
> create
> > instances. It complained "backend for qemu disk is not ready". I checked
> > and did not find qemu process running in dom0. I could not find
> > /etc/init.d/xencommons either. Then, I compiled and installed xen-4.4
> from
> > source code and I got xencommons in /etc/init.d. qemu process is running
> in
> > dom0. However, the dom0 kernel crashes this time when I try to launch an
> > instance from openstack GUI. I am able to create a xen guest with
> > virt-install from command line, with the following command.
> >
> > "$ virt-install --connect=xen:/// --name u14.04.3 --ram 1024 --disk
> > u14.04.3.img,size=4,driver_name=qemu --location
> > http://ftp.ubuntu.com/ubuntu/dists/trusty/main/installer-amd64/
> --network
> > bridge=virbr0"
> >
> >
> > dmesg:
> >
> > [ 9443.130600] blkfront: xvda: flush diskcache: enabled; persistent
> grants:
> > enabled; indirect descriptors: disabled;
> > [ 9443.132818]  xvda: xvda1
> > [ 9444.604489] xen:grant_table: WARNING: g.e. 0x30 still in use!
> > [ 9444.604496] deferring g.e. 0x30 (pfn 0xffffffffffffffff)
> > [ 9444.604499] xen:grant_table: WARNING: g.e. 0x31 still in use!
> > [ 9444.604502] deferring g.e. 0x31 (pfn 0xffffffffffffffff)
> > [ 9444.604505] xen:grant_table: WARNING: g.e. 0x32 still in use!
> > [ 9444.604508] deferring g.e. 0x32 (pfn 0xffffffffffffffff)
> >   ==== lots of them====
> > [ 9444.604719] xen:grant_table: WARNING: g.e. 0xe still in use!
> > [ 9444.604721] deferring g.e. 0xe (pfn 0xffffffffffffffff)
> > [ 9444.604723] xen:grant_table: WARNING: g.e. 0xd still in use!
> > [ 9444.604725] deferring g.e. 0xd (pfn 0xffffffffffffffff)
> > [ 9448.325408] ------------[ cut here ]------------
> > [ 9448.325421] WARNING: CPU: 5 PID: 19902 at
> > /build/buildd/linux-3.13.0/arch/x86/xen/multicalls.c:129 xen_mc_flush+0x
> > 1a9/0x1b0()
> > [ 9448.325492] CPU: 5 PID: 19902 Comm: sudo Tainted: GF          O
> > 3.13.0-33-generic #58-Ubuntu
> > [ 9448.325494] Hardware name: Dell Inc. PowerEdge R710/00W9X3, BIOS
> 2.1.15
> > 09/02/2010
> > [ 9448.325497]  0000000000000009 ffff8802d13d9aa8 ffffffff8171bd04
> > 0000000000000000
> > [ 9448.325501]  ffff8802d13d9ae0 ffffffff810676cd 0000000000000000
> > 0000000000000001
> > [ 9448.325505]  ffff88030418ffe0 ffff88031d6ab180 0000000000000010
> > ffff8802d13d9af0
> > [ 9448.325509] Call Trace:
> > [ 9448.325518]  [<ffffffff8171bd04>] dump_stack+0x45/0x56
> > [ 9448.325523]  [<ffffffff810676cd>] warn_slowpath_common+0x7d/0xa0
> > [ 9448.325526]  [<ffffffff810677aa>] warn_slowpath_null+0x1a/0x20
> > [ 9448.325530]  [<ffffffff81004e99>] xen_mc_flush+0x1a9/0x1b0
> > [ 9448.325534]  [<ffffffff81006b99>] xen_set_pud_hyper+0x109/0x110
> > [ 9448.325538]  [<ffffffff81006c3b>] xen_set_pud+0x9b/0xb0
> > [ 9448.325543]  [<ffffffff81177f06>] __pmd_alloc+0xd6/0x110
> > [ 9448.325548]  [<ffffffff81182218>] move_page_tables+0x678/0x720
> > [ 9448.325552]  [<ffffffff8117ddb7>] ? vma_adjust+0x337/0x7d0
> > [ 9448.325557]  [<ffffffff811c23fd>] shift_arg_pages+0xbd/0x1b0
> > [ 9448.325560]  [<ffffffff81181671>] ? mprotect_fixup+0x151/0x290
> > [ 9448.325564]  [<ffffffff811c26cb>] setup_arg_pages+0x1db/0x200
> > [ 9448.325570]  [<ffffffff81213edc>] load_elf_binary+0x41c/0xd80
> > [ 9448.325576]  [<ffffffff8131d503>] ? ima_get_action+0x23/0x30
> > [ 9448.325579]  [<ffffffff8131c7e2>] ? process_measurement+0x82/0x2c0
> > [ 9448.325584]  [<ffffffff811c2f7f>] search_binary_handler+0x8f/0x1b0
> > [ 9448.325587]  [<ffffffff811c44f7>] do_execve_common.isra.22+0x5a7/0x7e0
> > [ 9448.325591]  [<ffffffff811c49c6>] SyS_execve+0x36/0x50
> > [ 9448.325596]  [<ffffffff8172cc99>] stub_execve+0x69/0xa0
> > [ 9448.325599] ---[ end trace 53ac16782e751c0a ]---
> > [ 9448.347994] ------------[ cut here ]------------
> > [ 9448.348004] WARNING: CPU: 1 PID: 19902 at
> > /build/buildd/linux-3.13.0/mm/mmap.c:2736 exit_mmap+0x16b/0x170()
> > ==== more message ignored ====
> >
> >
> > I sent emails to the openstack mailing list and they gave me a few
> > suggestions. At last, Bob Ball suggested I should probably ask this
> > question on xen-devel mailing list. Any help will be highly appreciated!
>
> This looks like the bug already reported by George Dunlap, could you try
> to apply the following patch to qemu-xen and recompile?
>
> http://lists.xenproject.org/archives/html/xen-devel/2014-11/msg01112.html
>
> Roger.
>
>

[-- Attachment #1.2: Type: text/html, Size: 7579 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Thu, 2014-11-13 at 22:54 -0700, Xing Lin wrote:
> I have been blocked by this problem for almost three weeks and as far
> as I know, I could not find any online tutorial on setting up xen
> based compute node for openstack juno. I will document steps I take
> and share them with the community. 

Awesome, thanks!

Having this in the Xen wiki would be awesome. If you would like write
access to our wiki (which is granted manually due to spammers spoiling
it for everyone...) then please:

      * Create an account using the "create account" link at the
        top-right hand side of http://wiki.xen.org/wiki/Main_Page
      * Fill in this form giving your chosen username:
        http://xenproject.org/component/content/article/100-misc/145-request-to-be-made-a-wiki-editor.html

You could alternatively drop me a line privately with the user name for
the second step, but filling in the form might give a quicker response
time due to reaching multiple people in multiple timezones...

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Xing Lin]

[-- Attachment #1.1: Type: text/plain, Size: 1331 bytes --]

Hi,

The wiki page is ready. I am not sure whether I am using the correct format
or not. Please let me know if any changes are need. Thanks,

http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno

-Xing

On Fri, Nov 14, 2014 at 2:01 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:

> On Thu, 2014-11-13 at 22:54 -0700, Xing Lin wrote:
> > I have been blocked by this problem for almost three weeks and as far
> > as I know, I could not find any online tutorial on setting up xen
> > based compute node for openstack juno. I will document steps I take
> > and share them with the community.
>
> Awesome, thanks!
>
> Having this in the Xen wiki would be awesome. If you would like write
> access to our wiki (which is granted manually due to spammers spoiling
> it for everyone...) then please:
>
>       * Create an account using the "create account" link at the
>         top-right hand side of http://wiki.xen.org/wiki/Main_Page
>       * Fill in this form giving your chosen username:
>
> http://xenproject.org/component/content/article/100-misc/145-request-to-be-made-a-wiki-editor.html
>
> You could alternatively drop me a line privately with the user name for
> the second step, but filling in the form might give a quicker response
> time due to reaching multiple people in multiple timezones...
>
> Ian.
>
>

[-- Attachment #1.2: Type: text/html, Size: 2174 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1755 bytes --]

Many thanks for your efforts!

By any chance instead of rebuilding xen from source, have you tried
installing xen-system-amd64 (apt-get install xen-system-amd64), the xen
package for ubuntu?



On Fri, 14 Nov 2014, Xing Lin wrote:
> Hi,
> 
> The wiki page is ready. I am not sure whether I am using the correct format or not. Please let me know if any changes are need.
> Thanks,
> http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno
> 
> -Xing
> 
> On Fri, Nov 14, 2014 at 2:01 AM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>       On Thu, 2014-11-13 at 22:54 -0700, Xing Lin wrote:
>       > I have been blocked by this problem for almost three weeks and as far
>       > as I know, I could not find any online tutorial on setting up xen
>       > based compute node for openstack juno. I will document steps I take
>       > and share them with the community.
> 
>       Awesome, thanks!
> 
>       Having this in the Xen wiki would be awesome. If you would like write
>       access to our wiki (which is granted manually due to spammers spoiling
>       it for everyone...) then please:
> 
>             * Create an account using the "create account" link at the
>               top-right hand side of http://wiki.xen.org/wiki/Main_Page
>             * Fill in this form giving your chosen username:
>               http://xenproject.org/component/content/article/100-misc/145-request-to-be-made-a-wiki-editor.html
> 
>       You could alternatively drop me a line privately with the user name for
>       the second step, but filling in the form might give a quicker response
>       time due to reaching multiple people in multiple timezones...
> 
>       Ian.
> 
> 
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Fri, 2014-11-14 at 21:10 -0700, Xing Lin wrote:
> Hi,
> 
> 
> The wiki page is ready. I am not sure whether I am using the correct
> format or not. Please let me know if any changes are need. Thanks,
> 
> 
> http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno

Thanks for this. WRT the need to install virt manager to avoid the
"cannot open shared object file" issue I expect just running "ldconfig"
would have worked instead.

It would also be good to understand why it is necessary to install from
source. Was it just the lack of the xencommons initscript? Debian and
Ubuntu have their own initscripts and don't reuse the xencommons script.
However it should be fairly easy to add the necessary commands to start
qemu to /etc/init.d/xen instead of rebuilding from source.

I'd expect just adding to the end of the "start)" section of the script
to work. e.g. 

                *) log_end_msg 1; exit ;;
        esac
        log_end_msg 0
+       /usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
+          -monitor /dev/null -serial /dev/null -parallel /dev/null \
+          -pidfile /var/run/qemu-xen-dom0.pid
        ;;
  stop)
        capability_check
        case "$?" in
                0) ;;

(nb, that's not a real patch, I just typed it into my mail client as is)

If you can confirm that this works then I can try and get this fixed in
Debian at least.

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Mon, 2014-11-17 at 12:35 +0000, Stefano Stabellini wrote:
> Many thanks for your efforts!
> 
> By any chance instead of rebuilding xen from source, have you tried
> installing xen-system-amd64 (apt-get install xen-system-amd64), the xen
> package for ubuntu?

From
https://launchpadlibrarian.net/183469597/buildlog_ubuntu-trusty-i386.nova_1%3A2014.2~b2-0ubuntu1~cloud0_UPLOADING.txt.gz the nova-compute-xen  package depends on:
 Depends: nova-compute-libvirt (= 1:2014.2~b2-0ubuntu1~cloud0), xen-system-amd64 | xen-system-i386
so I think this is already happening. I think the issue is that the
Debian/Ubuntu packages don't use xencommons and therefore haven't yet
picked up the stuff to start dom0's qemu (see my other reply).

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 2060 bytes --]

On 17.11.2014 13:39, Ian Campbell wrote:
> On Fri, 2014-11-14 at 21:10 -0700, Xing Lin wrote:
>> Hi,
>>
>>
>> The wiki page is ready. I am not sure whether I am using the correct
>> format or not. Please let me know if any changes are need. Thanks,
>>
>>
>> http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno
> 
> Thanks for this. WRT the need to install virt manager to avoid the
> "cannot open shared object file" issue I expect just running "ldconfig"
> would have worked instead.
> 
> It would also be good to understand why it is necessary to install from
> source. Was it just the lack of the xencommons initscript? Debian and
> Ubuntu have their own initscripts and don't reuse the xencommons script.
> However it should be fairly easy to add the necessary commands to start
> qemu to /etc/init.d/xen instead of rebuilding from source.
> 
> I'd expect just adding to the end of the "start)" section of the script
> to work. e.g. 
> 
>                 *) log_end_msg 1; exit ;;
>         esac
>         log_end_msg 0
> +       /usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
> +          -monitor /dev/null -serial /dev/null -parallel /dev/null \
> +          -pidfile /var/run/qemu-xen-dom0.pid
>         ;;
>   stop)
>         capability_check
>         case "$?" in
>                 0) ;;
> 
> (nb, that's not a real patch, I just typed it into my mail client as is)
> 
> If you can confirm that this works then I can try and get this fixed in
> Debian at least.

I'd be interested in getting this working in Ubuntu, too. So the wiki could drop
the requirement for compiling your own xen. Getting openstack set up is hard
enough. So far it seems like the two things missing would be the patch to qemu
and starting an instance of it for dom0 (maybe), right?

-Stefan

> 
> Ian.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Mon, 2014-11-17 at 15:44 +0100, Stefan Bader wrote:
> So far it seems like the two things missing would be the patch to qemu
> and starting an instance of it for dom0 (maybe), right?

Yes to both AFAICT.

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Stefano Stabellini]

On Mon, 17 Nov 2014, Stefan Bader wrote:
> On 17.11.2014 13:39, Ian Campbell wrote:
> > On Fri, 2014-11-14 at 21:10 -0700, Xing Lin wrote:
> >> Hi,
> >>
> >>
> >> The wiki page is ready. I am not sure whether I am using the correct
> >> format or not. Please let me know if any changes are need. Thanks,
> >>
> >>
> >> http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno
> > 
> > Thanks for this. WRT the need to install virt manager to avoid the
> > "cannot open shared object file" issue I expect just running "ldconfig"
> > would have worked instead.
> > 
> > It would also be good to understand why it is necessary to install from
> > source. Was it just the lack of the xencommons initscript? Debian and
> > Ubuntu have their own initscripts and don't reuse the xencommons script.
> > However it should be fairly easy to add the necessary commands to start
> > qemu to /etc/init.d/xen instead of rebuilding from source.
> > 
> > I'd expect just adding to the end of the "start)" section of the script
> > to work. e.g. 
> > 
> >                 *) log_end_msg 1; exit ;;
> >         esac
> >         log_end_msg 0
> > +       /usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name dom0 -nographic -M xenpv -daemonize \
> > +          -monitor /dev/null -serial /dev/null -parallel /dev/null \
> > +          -pidfile /var/run/qemu-xen-dom0.pid
> >         ;;
> >   stop)
> >         capability_check
> >         case "$?" in
> >                 0) ;;
> > 
> > (nb, that's not a real patch, I just typed it into my mail client as is)
> > 
> > If you can confirm that this works then I can try and get this fixed in
> > Debian at least.
> 
> I'd be interested in getting this working in Ubuntu, too. So the wiki could drop
> the requirement for compiling your own xen. Getting openstack set up is hard
> enough. So far it seems like the two things missing would be the patch to qemu
> and starting an instance of it for dom0 (maybe), right?

Starting an instance from dom0 is done my xencommons, you can take a
look at the script as it is part of a normal Xen installation. The 3
lines Ian wrote are taken from it.

The patch to QEMU is now in upstream QEMU, qemu-xen for the coming 4.5
release and I have also backported it to the stable 4.4 and 4.3 qemu
tree.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Xing Lin]

[-- Attachment #1.1: Type: text/plain, Size: 1938 bytes --]

Hi Ian,

Both of your two points are valid. There is no need to install
virt-manager. And the patch to start a qemu process in /etc/init.d/xen
seems to be enough for launching instances from horizon. I have updated the
wiki page.  Please review it at

http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno

Thanks, Ian!

-Xing

On Mon, Nov 17, 2014 at 5:39 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:

> On Fri, 2014-11-14 at 21:10 -0700, Xing Lin wrote:
> > Hi,
> >
> >
> > The wiki page is ready. I am not sure whether I am using the correct
> > format or not. Please let me know if any changes are need. Thanks,
> >
> >
> > http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno
>
> Thanks for this. WRT the need to install virt manager to avoid the
> "cannot open shared object file" issue I expect just running "ldconfig"
> would have worked instead.
>
> It would also be good to understand why it is necessary to install from
> source. Was it just the lack of the xencommons initscript? Debian and
> Ubuntu have their own initscripts and don't reuse the xencommons script.
> However it should be fairly easy to add the necessary commands to start
> qemu to /etc/init.d/xen instead of rebuilding from source.
>
> I'd expect just adding to the end of the "start)" section of the script
> to work. e.g.
>
>                 *) log_end_msg 1; exit ;;
>         esac
>         log_end_msg 0
> +       /usr/local/bin/qemu-system-i386 -xen-domid 0 -xen-attach -name
> dom0 -nographic -M xenpv -daemonize \
> +          -monitor /dev/null -serial /dev/null -parallel /dev/null \
> +          -pidfile /var/run/qemu-xen-dom0.pid
>         ;;
>   stop)
>         capability_check
>         case "$?" in
>                 0) ;;
>
> (nb, that's not a real patch, I just typed it into my mail client as is)
>
> If you can confirm that this works then I can try and get this fixed in
> Debian at least.
>
> Ian.
>
>

[-- Attachment #1.2: Type: text/html, Size: 3004 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Wed, 2014-11-19 at 11:57 -0700, Xing Lin wrote:
> Hi Ian,
> 
> 
> Both of your two points are valid. There is no need to install
> virt-manager. And the patch to start a qemu process in /etc/init.d/xen
> seems to be enough for launching instances from horizon. I have
> updated the wiki page.  Please review it at 
> 
> http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno

Looks much better, thanks!

I think the need to symlink pygrub to /usr/bin should be considered an
openstack bug. I presume nova specifies the absolute path, when it
should just say "pygrub" by default and let the toolstack find the
default one.

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Xing Lin]

[-- Attachment #1.1: Type: text/plain, Size: 2048 bytes --]

Hi,

I git clone nova-juno from github and searched for 'pygrub'. Here is what i
get.

.//etc/nova/rootwrap.d/compute.filters:104:# nova/virt/xenapi/vm_utils.py:
'pygrub', '-qn', dev_path
.//etc/nova/rootwrap.d/compute.filters:105:pygrub: CommandFilter, pygrub,
root
.//nova/tests/unit/virt/xenapi/test_xenapi.py:667:
 self.assertEqual(self.vm['PV_bootloader'], 'pygrub')
.//nova/virt/xenapi/vm_utils.py:298:            rec['PV_bootloader'] =
'pygrub'

It seems that nova does not specify absolute path for pygrub.

I checked libvirt source code and found the following definition.

.//src/libxl/libxl_conf.h:52:# define LIBXL_BOOTLOADER_PATH BINDIR "/pygrub"

Searched further for difinition of BINDIR revealed the followings.

Xing:libvirt-1.2.2 xing$ grep BINDIR -Rn ./
.//ChangeLog:48645:  SBINDIR "/libvirtd"
.//ChangeLog:48646:  SBINDIR "/libvirtd_dbg"
.//build-aux/ltmain.sh:2408:  -bindir BINDIR    specify path to binaries
directory (for systems where
.//gnulib/lib/Makefile.in:2499:  echo '#define BINDIR "$(bindir)"'; \
.//gnulib/lib/Makefile.in:2500:  echo '#define SBINDIR "$(sbindir)"'; \
.//gnulib/lib/gnulib.mk:303:  echo '#define BINDIR "$(bindir)"'; \
.//gnulib/lib/gnulib.mk:304:  echo '#define SBINDIR "$(sbindir)"'; \

I guess BINDIR was pointed to /usr/bin.

-Xing

On Thu, Nov 20, 2014 at 2:30 AM, Ian Campbell <Ian.Campbell@citrix.com>
wrote:

> On Wed, 2014-11-19 at 11:57 -0700, Xing Lin wrote:
> > Hi Ian,
> >
> >
> > Both of your two points are valid. There is no need to install
> > virt-manager. And the patch to start a qemu process in /etc/init.d/xen
> > seems to be enough for launching instances from horizon. I have
> > updated the wiki page.  Please review it at
> >
> > http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno
>
> Looks much better, thanks!
>
> I think the need to symlink pygrub to /usr/bin should be considered an
> openstack bug. I presume nova specifies the absolute path, when it
> should just say "pygrub" by default and let the toolstack find the
> default one.
>
> Ian.
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 3509 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Thu, 2014-11-20 at 16:53 -0700, Xing Lin wrote:

> I git clone nova-juno from github and searched for 'pygrub'. Here is what i get. 
> 
> 
> .//etc/nova/rootwrap.d/compute.filters:104:# nova/virt/xenapi/vm_utils.py: 'pygrub', '-qn', dev_path
> .//etc/nova/rootwrap.d/compute.filters:105:pygrub: CommandFilter, pygrub, root
> .//nova/tests/unit/virt/xenapi/test_xenapi.py:667:        self.assertEqual(self.vm['PV_bootloader'], 'pygrub')
> .//nova/virt/xenapi/vm_utils.py:298:            rec['PV_bootloader'] = 'pygrub'
> 
> 
> It seems that nova does not specify absolute path for pygrub. 
> 
> 
> I checked libvirt source code and found the following definition.
> 
> 
> .//src/libxl/libxl_conf.h:52:# define LIBXL_BOOTLOADER_PATH BINDIR "/pygrub"

Thanks for investigating.

I think libvirt is wrong to specify an absolute path here, IMHO by
default it should just specify "pygrub" and let libxl figure out the
correct path. Jim, what do you think?

Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Fri, 2014-11-21 at 09:28 +0000, Ian Campbell wrote:
> I think libvirt is wrong to specify an absolute path here, IMHO by
> default it should just specify "pygrub" and let libxl figure out the
> correct path. Jim, what do you think?

e.g. something like the following untested (but pretty obvious[0])
patch.

I'm currently rebuilding Debian's libvirt package with this included so
I can give it a go.

Ian.

[0] famous last words, I know!

8<---------------

>From 4edbcbdc7e28896121832d8e226e7aeccf30633c Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Fri, 21 Nov 2014 14:00:38 +0000
Subject: [PATCH] libxl: Allow libxl to find pygrub binary.

Specifying an explicit path to pygrub (e.g. BINDIR "/pygrub") only works if
Xen and libvirt happen to be installed to the same prefix. A more flexible
approach is to simply specify "pygrub" which will cause libxl to use the
correct path which it knows (since it is built with the same prefix as pygrub).

This is particular problematic in the Debian packaging, since the Debian Xen
package relocates pygrub into a libexec dir, however I think this change makes
sense upstream.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 src/libxl/libxl_conf.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/libxl/libxl_conf.h b/src/libxl/libxl_conf.h
index 25f77ea..3669e68 100644
--- a/src/libxl/libxl_conf.h
+++ b/src/libxl/libxl_conf.h
@@ -53,7 +53,7 @@
 # define LIBXL_LIB_DIR LOCALSTATEDIR "/lib/libvirt/libxl"
 # define LIBXL_SAVE_DIR LIBXL_LIB_DIR "/save"
 # define LIBXL_DUMP_DIR LIBXL_LIB_DIR "/dump"
-# define LIBXL_BOOTLOADER_PATH BINDIR "/pygrub"
+# define "pygrub"
 
 /* libxl interface for setting VCPU affinity changed in 4.5. In fact, a new
  * parameter has been added, representative of 'VCPU soft affinity'. If one
-- 
1.7.10.4

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[David Vrabel]

On 21/11/14 14:05, Ian Campbell wrote:
> On Fri, 2014-11-21 at 09:28 +0000, Ian Campbell wrote:
>> I think libvirt is wrong to specify an absolute path here, IMHO by
>> default it should just specify "pygrub" and let libxl figure out the
>> correct path. Jim, what do you think?
> 
> e.g. something like the following untested (but pretty obvious[0])
> patch.
[...]
> +# define "pygrub"

I'm going to go with pretty obviously broken.  You're missing the macro
name.

David

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Fri, 2014-11-21 at 14:18 +0000, David Vrabel wrote:
> On 21/11/14 14:05, Ian Campbell wrote:
> > On Fri, 2014-11-21 at 09:28 +0000, Ian Campbell wrote:
> >> I think libvirt is wrong to specify an absolute path here, IMHO by
> >> default it should just specify "pygrub" and let libxl figure out the
> >> correct path. Jim, what do you think?
> > 
> > e.g. something like the following untested (but pretty obvious[0])
> > patch.
> [...]
> > +# define "pygrub"
> 
> I'm going to go with pretty obviously broken.  You're missing the macro
> name.

Doh! Indeed...

Lets try that again...

8<----------

>From 9f2d8da8264b426f54b92378e9e00973694193d4 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Fri, 21 Nov 2014 14:00:38 +0000
Subject: [PATCH] libxl: Allow libxl to find pygrub binary.

Specifying an explicit path to pygrub (e.g. BINDIR "/pygrub") only works if
Xen and libvirt happen to be installed to the same prefix. A more flexible
approach is to simply specify "pygrub" which will cause libxl to use the
correct path which it knows (since it is built with the same prefix as pygrub).

This is particular problematic in the Debian packaging, since the Debian Xen
package relocates pygrub into a libexec dir, however I think this change makes
sense upstream.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 .gnulib                |    2 +-
 src/libxl/libxl_conf.h |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/.gnulib b/.gnulib
index 9565c3b..2d28074 160000
--- a/.gnulib
+++ b/.gnulib
@@ -1 +1 @@
-Subproject commit 9565c3be73eb6d76b7b42a21d68d2e00a62abb6d
+Subproject commit 2d280742a9e30088aa169f53353765d5daafe4c0
diff --git a/src/libxl/libxl_conf.h b/src/libxl/libxl_conf.h
index 25f77ea..d7a3971 100644
--- a/src/libxl/libxl_conf.h
+++ b/src/libxl/libxl_conf.h
@@ -53,7 +53,7 @@
 # define LIBXL_LIB_DIR LOCALSTATEDIR "/lib/libvirt/libxl"
 # define LIBXL_SAVE_DIR LIBXL_LIB_DIR "/save"
 # define LIBXL_DUMP_DIR LIBXL_LIB_DIR "/dump"
-# define LIBXL_BOOTLOADER_PATH BINDIR "/pygrub"
+# define LIBXL_BOOTLOADER_PATH "pygrub"
 
 /* libxl interface for setting VCPU affinity changed in 4.5. In fact, a new
  * parameter has been added, representative of 'VCPU soft affinity'. If one
-- 
1.7.10.4

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Wed, 2014-11-19 at 11:57 -0700, Xing Lin wrote:
> Hi Ian,
> 
> 
> Both of your two points are valid. There is no need to install
> virt-manager. And the patch to start a qemu process in /etc/init.d/xen
> seems to be enough for launching instances from horizon. I have
> updated the wiki page.  Please review it at 
> 
> http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno

FYI I've filed a few bugs/patches in Debian based on the issues you've
found here:

http://bugs.debian.org/770456 -- xen: start qemu for dom0 from initscript
http://bugs.debian.org/770468 -- qemu: backport fix for qemu crash
http://bugs.debian.org/770485 -- libvirt: pygrub path thing.


Ian.

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Stefan Bader]

[-- Attachment #1.1: Type: text/plain, Size: 919 bytes --]

On 21.11.2014 17:59, Ian Campbell wrote:
> On Wed, 2014-11-19 at 11:57 -0700, Xing Lin wrote:
>> Hi Ian,
>>
>>
>> Both of your two points are valid. There is no need to install
>> virt-manager. And the patch to start a qemu process in /etc/init.d/xen
>> seems to be enough for launching instances from horizon. I have
>> updated the wiki page.  Please review it at 
>>
>> http://wiki.xenproject.org/wiki/Xen_via_libvirt_for_OpenStack_juno
> 
> FYI I've filed a few bugs/patches in Debian based on the issues you've
> found here:
> 
> http://bugs.debian.org/770456 -- xen: start qemu for dom0 from initscript
> http://bugs.debian.org/770468 -- qemu: backport fix for qemu crash
> http://bugs.debian.org/770485 -- libvirt: pygrub path thing.

Thanks for the links. Will try to shadow those in launchpad next week. For qemu
Xing opened http://bugs.launchpad.net/bugs/1394327

> 
> 
> Ian.
> 



[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Ian Campbell]

On Fri, 2014-11-21 at 14:32 +0000, Ian Campbell wrote:

I've now built and tested this with the 1.2.9 Debian package, works like
a charm...

> From 9f2d8da8264b426f54b92378e9e00973694193d4 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Fri, 21 Nov 2014 14:00:38 +0000
> Subject: [PATCH] libxl: Allow libxl to find pygrub binary.
> 
> Specifying an explicit path to pygrub (e.g. BINDIR "/pygrub") only works if
> Xen and libvirt happen to be installed to the same prefix. A more flexible
> approach is to simply specify "pygrub" which will cause libxl to use the
> correct path which it knows (since it is built with the same prefix as pygrub).
> 
> This is particular problematic in the Debian packaging, since the Debian Xen
> package relocates pygrub into a libexec dir, however I think this change makes
> sense upstream.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>  .gnulib                |    2 +-
>  src/libxl/libxl_conf.h |    2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/.gnulib b/.gnulib
> index 9565c3b..2d28074 160000
> --- a/.gnulib
> +++ b/.gnulib
> @@ -1 +1 @@
> -Subproject commit 9565c3be73eb6d76b7b42a21d68d2e00a62abb6d
> +Subproject commit 2d280742a9e30088aa169f53353765d5daafe4c0
> diff --git a/src/libxl/libxl_conf.h b/src/libxl/libxl_conf.h
> index 25f77ea..d7a3971 100644
> --- a/src/libxl/libxl_conf.h
> +++ b/src/libxl/libxl_conf.h
> @@ -53,7 +53,7 @@
>  # define LIBXL_LIB_DIR LOCALSTATEDIR "/lib/libvirt/libxl"
>  # define LIBXL_SAVE_DIR LIBXL_LIB_DIR "/save"
>  # define LIBXL_DUMP_DIR LIBXL_LIB_DIR "/dump"
> -# define LIBXL_BOOTLOADER_PATH BINDIR "/pygrub"
> +# define LIBXL_BOOTLOADER_PATH "pygrub"
>  
>  /* libxl interface for setting VCPU affinity changed in 4.5. In fact, a new
>   * parameter has been added, representative of 'VCPU soft affinity'. If one

Re: dom0 kenrel crashes for openstack + libvirt + libxl

[Jim Fehlig]

On 11/21/2014 07:32 AM, Ian Campbell wrote:
> >From 9f2d8da8264b426f54b92378e9e00973694193d4 Mon Sep 17 00:00:00 2001
> From: Ian Campbell <ian.campbell@citrix.com>
> Date: Fri, 21 Nov 2014 14:00:38 +0000
> Subject: [PATCH] libxl: Allow libxl to find pygrub binary.
>
> Specifying an explicit path to pygrub (e.g. BINDIR "/pygrub") only works if
> Xen and libvirt happen to be installed to the same prefix. A more flexible
> approach is to simply specify "pygrub" which will cause libxl to use the
> correct path which it knows (since it is built with the same prefix as pygrub).
>
> This is particular problematic in the Debian packaging, since the Debian Xen
> package relocates pygrub into a libexec dir, however I think this change makes
> sense upstream.

Yep agreed.  I forgot about making this change a few weeks back after seeing the 
following in the logs when not explicitly specifying a bootloader

libxl: warning: libxl_bootloader.c:413:bootloader_disk_attached_cb: 
bootloader='/usr/bin/pygrub' is deprecated; use bootloader='pygrub' instead

>
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> ---
>   .gnulib                |    2 +-
>   src/libxl/libxl_conf.h |    2 +-
>   2 files changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/.gnulib b/.gnulib
> index 9565c3b..2d28074 160000
> --- a/.gnulib
> +++ b/.gnulib
> @@ -1 +1 @@
> -Subproject commit 9565c3be73eb6d76b7b42a21d68d2e00a62abb6d
> +Subproject commit 2d280742a9e30088aa169f53353765d5daafe4c0

ACK and pushed after removing this hunk.

Regards,
Jim

> diff --git a/src/libxl/libxl_conf.h b/src/libxl/libxl_conf.h
> index 25f77ea..d7a3971 100644
> --- a/src/libxl/libxl_conf.h
> +++ b/src/libxl/libxl_conf.h
> @@ -53,7 +53,7 @@
>   # define LIBXL_LIB_DIR LOCALSTATEDIR "/lib/libvirt/libxl"
>   # define LIBXL_SAVE_DIR LIBXL_LIB_DIR "/save"
>   # define LIBXL_DUMP_DIR LIBXL_LIB_DIR "/dump"
> -# define LIBXL_BOOTLOADER_PATH BINDIR "/pygrub"
> +# define LIBXL_BOOTLOADER_PATH "pygrub"
>   
>   /* libxl interface for setting VCPU affinity changed in 4.5. In fact, a new
>    * parameter has been added, representative of 'VCPU soft affinity'. If one