Ongoing issue with libvirt-xen driver for Openstack Nova

Ongoing issue with libvirt-xen driver for Openstack Nova

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1414 bytes --]

Could you, please, point to the most  recent commit in 

https://git.openstack.org/cgit/openstack/nova/   

for which procedure described in

http://wiki.xen.org/wiki/OpenStack_via_DevStack

provide an ability to launch nova instances via  horizion or nova CLI.

I am aware of bug https://bugs.launchpad.net/nova/+bug/1425115

and tried to revert commit http://git.openstack.org/cgit/openstack/nova/commit/?id=0696a5cd5f0fdc08951a074961bb8ce0c3310086
right after stack.sh cloned /opt/stack/nova and was able successfully complete devstack install with nova-compute daemon up and running.
However, finally attempting to launch Cirros instance I've got following errors in /var/log/libvirt/libxl/instance-00000003.log :-

libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [-1] exited with error status 1
libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: ip link set vif1.0 name tap7f39f4aa-d1 failed
libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f9e67ee53c0: deregister unregistered
libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable to add nic devices

Another approach to test driver described in details in bug

https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1439645

resulted absolutely same errors errors in same log file. 

Thank you.
Boris Derzhavets.
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 2156 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Anthony PERARD]

On Fri, Apr 03, 2015 at 01:57:16PM -0400, Boris Derzhavets wrote:
> Could you, please, point to the most  recent commit in 
> 
> https://git.openstack.org/cgit/openstack/nova/   
> 
> for which procedure described in
> 
> http://wiki.xen.org/wiki/OpenStack_via_DevStack
> 
> provide an ability to launch nova instances via  horizion or nova CLI.
> 
> I am aware of bug https://bugs.launchpad.net/nova/+bug/1425115
> 
> and tried to revert commit http://git.openstack.org/cgit/openstack/nova/commit/?id=0696a5cd5f0fdc08951a074961bb8ce0c3310086
> right after stack.sh cloned /opt/stack/nova and was able successfully complete devstack install with nova-compute daemon up and running.
> However, finally attempting to launch Cirros instance I've got following errors in /var/log/libvirt/libxl/instance-00000003.log :-
> 
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [-1] exited with error status 1
> libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: ip link set vif1.0 name tap7f39f4aa-d1 failed
> libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f9e67ee53c0: deregister unregistered
> libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable to add nic devices
> 
> Another approach to test driver described in details in bug
> 
> https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1439645
> 
> resulted absolutely same errors errors in same log file. 

Hi Boris,

from the look at the bug 1439645, it looks like your are using Neutron for
the network and this probably were things goes wrong. I think there is a
compatibility issue between Neutron and the script
/etc/xen/scripts/vif-bridge from the XenProject.

Can you look into /var/log/xen/xen-hotplug.log ? There is maybe an error
message that would tell why "ip link set ..." failed.

-- 
Anthony PERARD

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 2620 bytes --]

Anthony ,

Unfortunately, I dropped instance running XEN . In a couple of days,
I will reproduce environment and send /var/log/xen/xen-hotplug.log  to you.
Regarding Libvirt version,  I also tested driver with  libvirt 1.2.14 , manual build
via  libvirt-1.2.14.tar.gz ball  and upgrade on Ubuntu 14.04.2 after stack.sh  completion.
I got same error in  libxl-driver.log

Thank you.
Boris



> Date: Wed, 15 Apr 2015 16:48:54 +0100
> From: anthony.perard@citrix.com
> To: bderzhavets@hotmail.com
> CC: xen-devel@lists.xen.org
> Subject: Re: Ongoing issue with libvirt-xen driver for Openstack Nova
> 
> On Fri, Apr 03, 2015 at 01:57:16PM -0400, Boris Derzhavets wrote:
> > Could you, please, point to the most  recent commit in 
> > 
> > https://git.openstack.org/cgit/openstack/nova/   
> > 
> > for which procedure described in
> > 
> > http://wiki.xen.org/wiki/OpenStack_via_DevStack
> > 
> > provide an ability to launch nova instances via  horizion or nova CLI.
> > 
> > I am aware of bug https://bugs.launchpad.net/nova/+bug/1425115
> > 
> > and tried to revert commit http://git.openstack.org/cgit/openstack/nova/commit/?id=0696a5cd5f0fdc08951a074961bb8ce0c3310086
> > right after stack.sh cloned /opt/stack/nova and was able successfully complete devstack install with nova-compute daemon up and running.
> > However, finally attempting to launch Cirros instance I've got following errors in /var/log/libvirt/libxl/instance-00000003.log :-
> > 
> > libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [-1] exited with error status 1
> > libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: ip link set vif1.0 name tap7f39f4aa-d1 failed
> > libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f9e67ee53c0: deregister unregistered
> > libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable to add nic devices
> > 
> > Another approach to test driver described in details in bug
> > 
> > https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1439645
> > 
> > resulted absolutely same errors errors in same log file. 
> 
> Hi Boris,
> 
> from the look at the bug 1439645, it looks like your are using Neutron for
> the network and this probably were things goes wrong. I think there is a
> compatibility issue between Neutron and the script
> /etc/xen/scripts/vif-bridge from the XenProject.
> 
> Can you look into /var/log/xen/xen-hotplug.log ? There is maybe an error
> message that would tell why "ip link set ..." failed.
> 
> -- 
> Anthony PERARD
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 3212 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 2328 bytes --]

Done

/var/log/xen/xen-hotplug.log:-

RTNETLINK answers: Device or resource busy

Thank you.

Boris.

> Date: Wed, 15 Apr 2015 16:48:54 +0100
> From: anthony.perard@citrix.com
> To: bderzhavets@hotmail.com
> CC: xen-devel@lists.xen.org
> Subject: Re: Ongoing issue with libvirt-xen driver for Openstack Nova
> 
> On Fri, Apr 03, 2015 at 01:57:16PM -0400, Boris Derzhavets wrote:
> > Could you, please, point to the most  recent commit in 
> > 
> > https://git.openstack.org/cgit/openstack/nova/   
> > 
> > for which procedure described in
> > 
> > http://wiki.xen.org/wiki/OpenStack_via_DevStack
> > 
> > provide an ability to launch nova instances via  horizion or nova CLI.
> > 
> > I am aware of bug https://bugs.launchpad.net/nova/+bug/1425115
> > 
> > and tried to revert commit http://git.openstack.org/cgit/openstack/nova/commit/?id=0696a5cd5f0fdc08951a074961bb8ce0c3310086
> > right after stack.sh cloned /opt/stack/nova and was able successfully complete devstack install with nova-compute daemon up and running.
> > However, finally attempting to launch Cirros instance I've got following errors in /var/log/libvirt/libxl/instance-00000003.log :-
> > 
> > libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: /etc/xen/scripts/vif-bridge online [-1] exited with error status 1
> > libxl: error: libxl_device.c:1085:device_hotplug_child_death_cb: script: ip link set vif1.0 name tap7f39f4aa-d1 failed
> > libxl: debug: libxl_event.c:618:libxl__ev_xswatch_deregister: watch w=0x7f9e67ee53c0: deregister unregistered
> > libxl: error: libxl_create.c:1226:domcreate_attach_vtpms: unable to add nic devices
> > 
> > Another approach to test driver described in details in bug
> > 
> > https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1439645
> > 
> > resulted absolutely same errors errors in same log file. 
> 
> Hi Boris,
> 
> from the look at the bug 1439645, it looks like your are using Neutron for
> the network and this probably were things goes wrong. I think there is a
> compatibility issue between Neutron and the script
> /etc/xen/scripts/vif-bridge from the XenProject.
> 
> Can you look into /var/log/xen/xen-hotplug.log ? There is maybe an error
> message that would tell why "ip link set ..." failed.
> 
> -- 
> Anthony PERARD
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 2884 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Ian Campbell]

On Thu, 2015-04-16 at 05:14 -0400, Boris Derzhavets wrote:

Please can you not top post.

> Done
> 
> /var/log/xen/xen-hotplug.log:-
> 
> RTNETLINK answers: Device or resource busy
> 
> Thank you.

My usual technique for debugging these sorts of hotplug script issues is
to insert near the top of vif-bridge:
    exec >> /tmp/hotplug.log 2>&1
    echo `date` running vif-bridge $@
    set -x
I hope I recalled the syntax correctly. The "set -x" makes it quite
noisy but gives pretty good clues.

Ian.

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1818 bytes --]

Done

RTNETLINK answers: Device or resource busy
++ fatal ip link set vif1.0 name 'tapc32a6b48-25 failed'
++ _xenstore_write backend/vif/1/0/hotplug-error 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status error
++ log debug 'Writing backend/vif/1/0/hotplug-error' 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status 'error to xenstore.'
++ local level=debug
++ shift
++ logger -p daemon.debug -- /etc/xen/scripts/vif-bridge: 'Writing backend/vif/1/0/hotplug-error' 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status 'error to xenstore.'
++ xenstore-write backend/vif/1/0/hotplug-error 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status error
++ log err ip link set vif1.0 name 'tapc32a6b48-25 failed'
++ local level=err
++ shift
++ logger -p daemon.err -- /etc/xen/scripts/vif-bridge: ip link set vif1.0 name 'tapc32a6b48-25 failed'
++ exit 1

Boris.



> Subject: Re: [Xen-devel] Ongoing issue with libvirt-xen driver for Openstack Nova
> From: ian.campbell@citrix.com
> To: bderzhavets@hotmail.com
> CC: anthony.perard@citrix.com; xen-devel@lists.xen.org
> Date: Thu, 16 Apr 2015 10:27:09 +0100
> 
> On Thu, 2015-04-16 at 05:14 -0400, Boris Derzhavets wrote:
> 
> Please can you not top post.
> 
> > Done
> > 
> > /var/log/xen/xen-hotplug.log:-
> > 
> > RTNETLINK answers: Device or resource busy
> > 
> > Thank you.
> 
> My usual technique for debugging these sorts of hotplug script issues is
> to insert near the top of vif-bridge:
>     exec >> /tmp/hotplug.log 2>&1
>     echo `date` running vif-bridge $@
>     set -x
> I hope I recalled the syntax correctly. The "set -x" makes it quite
> noisy but gives pretty good clues.
> 
> Ian.
> 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 2260 bytes --]

[-- Attachment #2: xen-hotplug.log.gz --]
[-- Type: application/x-gzip, Size: 1439 bytes --]

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Ian Campbell]

On Thu, 2015-04-16 at 06:26 -0400, Boris Derzhavets wrote:

_Please stop top posting_.

http://www.idallen.com/topposting.html

> Done

Was this just the tail of the log or the complete thing?

> RTNETLINK answers: Device or resource busy
> ++ fatal ip link set vif1.0 name 'tapc32a6b48-25 failed'
> ++ _xenstore_write backend/vif/1/0/hotplug-error 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status error
> ++ log debug 'Writing backend/vif/1/0/hotplug-error' 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status 'error to xenstore.'

So for some reason the hotplug script has been unable to rename the
interface, apparently because it is busy.

Based on some local experiments it seems that busy might be because the
interface has already been brought up. vif-bridge is changing the name
before bringing up the interface, perhaps something else in your system
(some bit of openstack perhaps) is also trying to manage this device and
has configured it?

I'm out of my depth wrt the openstack side of things now I'm afraid.

Ian.

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1506 bytes --]



> Subject: Re: [Xen-devel] Ongoing issue with libvirt-xen driver for Openstack Nova
> From: ian.campbell@citrix.com
> To: bderzhavets@hotmail.com
> CC: anthony.perard@citrix.com; xen-devel@lists.xen.org
> Date: Thu, 16 Apr 2015 11:35:38 +0100
> 
> On Thu, 2015-04-16 at 06:26 -0400, Boris Derzhavets wrote:
> 
> _Please stop top posting_.
> 
> http://www.idallen.com/topposting.html
> 
> > Done
> 
> Was this just the tail of the log or the complete thing?


No it was in the middle . I am attaching xen-hotplug.log.gz


> 
> > RTNETLINK answers: Device or resource busy
> > ++ fatal ip link set vif1.0 name 'tapc32a6b48-25 failed'
> > ++ _xenstore_write backend/vif/1/0/hotplug-error 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status error
> > ++ log debug 'Writing backend/vif/1/0/hotplug-error' 'ip link set vif1.0 name tapc32a6b48-25 failed' backend/vif/1/0/hotplug-status 'error to xenstore.'
> 
> So for some reason the hotplug script has been unable to rename the
> interface, apparently because it is busy.
> 
> Based on some local experiments it seems that busy might be because the
> interface has already been brought up. vif-bridge is changing the name
> before bringing up the interface, perhaps something else in your system
> (some bit of openstack perhaps) is also trying to manage this device and
> has configured it?
> 
> I'm out of my depth wrt the openstack side of things now I'm afraid.
> 
> Ian.
> 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 1933 bytes --]

[-- Attachment #2: xen-hotplug.log.gz --]
[-- Type: application/x-gzip, Size: 1439 bytes --]

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Ian Campbell]

On Thu, 2015-04-16 at 06:46 -0400, Boris Derzhavets wrote:

> > Was this just the tail of the log or the complete thing?
> 
> 
> No it was in the middle . I am attaching xen-hotplug.log.gz

Thanks, I think this shows that as expected our hotplug scripts have not
been responsible for bringing vif1.0 up before renaming, so something
else must be doing it.

If not some openstack component perhaps something like network-manager?
Or some other udev rules perhaps?

I'm afraid I'll have to leave the openstack side of things to you and
others.

Ian.

FW: Ongoing issue with libvirt-xen driver for Openstack Nova

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1131 bytes --]

Anthony,

I am forwarding to you the final feed from Ian Campbell.
Please, view thread "Ongoing issue with libvirt-xen driver for Openstack Nova" 
for details of troubleshooting  from Xen side performed under Ian's supervision.

Thank you.
Boris.

> Subject: Re: [Xen-devel] Ongoing issue with libvirt-xen driver for Openstack Nova
> From: ian.campbell@citrix.com
> To: bderzhavets@hotmail.com
> CC: anthony.perard@citrix.com; xen-devel@lists.xen.org
> Date: Thu, 16 Apr 2015 13:05:44 +0100
> 
> On Thu, 2015-04-16 at 06:46 -0400, Boris Derzhavets wrote:
> 
> > > Was this just the tail of the log or the complete thing?
> > 
> > 
> > No it was in the middle . I am attaching xen-hotplug.log.gz
> 
> Thanks, I think this shows that as expected our hotplug scripts have not
> been responsible for bringing vif1.0 up before renaming, so something
> else must be doing it.
> 
> If not some openstack component perhaps something like network-manager?
> Or some other udev rules perhaps?
> 
> I'm afraid I'll have to leave the openstack side of things to you and
> others.
> 
> Ian.
> 
> 
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 1527 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Anthony PERARD]

On Thu, Apr 16, 2015 at 01:05:44PM +0100, Ian Campbell wrote:
> On Thu, 2015-04-16 at 06:46 -0400, Boris Derzhavets wrote:
> 
> > > Was this just the tail of the log or the complete thing?
> > 
> > 
> > No it was in the middle . I am attaching xen-hotplug.log.gz
> 
> Thanks, I think this shows that as expected our hotplug scripts have not
> been responsible for bringing vif1.0 up before renaming, so something
> else must be doing it.
> 
> If not some openstack component perhaps something like network-manager?
> Or some other udev rules perhaps?

It is likely to be something done in Neutron. You can always try to read
the logs of the different services (q-svc q-agt q-dhcp q-l3 q-meta) to
check what is happening to the network interface (vif1.0). There also could
be something wrong with Nova that will not set network properly in this
case. Or some other compatibility issue...

In anycase, the combo openstack-libvirt-xen works better with the default
set of services runned by devstack (as long as they don't switch yet to
Neutron).

I will try to setup devstack with your local.conf to reproduce the issue.

-- 
Anthony PERARD

Re: Ongoing issue with libvirt-xen driver for Openstack Nova

[Boris Derzhavets]

[-- Attachment #1.1: Type: text/plain, Size: 1926 bytes --]



> Date: Thu, 16 Apr 2015 16:12:49 +0100
> From: anthony.perard@citrix.com
> To: bderzhavets@hotmail.com
> CC: xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] Ongoing issue with libvirt-xen driver for Openstack Nova
> 
> On Thu, Apr 16, 2015 at 01:05:44PM +0100, Ian Campbell wrote:
> > On Thu, 2015-04-16 at 06:46 -0400, Boris Derzhavets wrote:
> > 
> > > > Was this just the tail of the log or the complete thing?
> > > 
> > > 
> > > No it was in the middle . I am attaching xen-hotplug.log.gz
> > 
> > Thanks, I think this shows that as expected our hotplug scripts have not
> > been responsible for bringing vif1.0 up before renaming, so something
> > else must be doing it.
> > 
> > If not some openstack component perhaps something like network-manager?
> > Or some other udev rules perhaps?
> 
> It is likely to be something done in Neutron. You can always try to read
> the logs of the different services (q-svc q-agt q-dhcp q-l3 q-meta) to
> check what is happening to the network interface (vif1.0). There also could
> be something wrong with Nova that will not set network properly in this
> case. Or some other compatibility issue...
> 
> In anycase, the combo openstack-libvirt-xen works better with the default
> set of services runned by devstack (as long as they don't switch yet to
> Neutron).

 My core stuff  is supposed to be RDO Kilo (RH) on Fedoras, like now it is RDO Juno.
 RH deprecated nova-networking a while ago , I guess  in Havana release.
 Devstack&&stack.sh on Ubuntu 14.04  is just a test-bed for the most recent nova. 
 
Thank you  very much for your support and getting in touch with explanation.
I couldn't imagine , that slide show http://www.slideshare.net/xen_com_mgr/openstack-xenfinal
is for nova networking

Boris

> 
> I will try to setup devstack with your local.conf to reproduce the issue.
> 
> -- 
> Anthony PERARD
 		 	   		  

[-- Attachment #1.2: Type: text/html, Size: 2544 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel