[PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

This is required by libvirt for live migration (netcat-traditional
doesn't cut it).

netcat-openbsd has higher update-alternatives priority, so it will be
used if installed.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
Cc: Jim Fehlig <jfehlig@suse.com>
---
 ts-xen-install | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ts-xen-install b/ts-xen-install
index 0f53382..0d097bc 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -53,7 +53,8 @@ my %distpath;
 sub packages () {
     target_install_packages($ho,
                             qw(bridge-utils vncsnapshot libaio1 libpixman-1-0
-                               libsdl1.2debian libglib2.0-0 liblzma5));
+                               libsdl1.2debian libglib2.0-0 liblzma5
+                               netcat-openbsd));
     if ($ho->{Suite} =~ m/jessie/) {
         target_install_packages($ho, 'libnl-route-3-200');
     }
-- 
2.1.4

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

On Wed, 2015-08-12 at 17:09 +0100, Ian Campbell wrote:
> This is required by libvirt for live migration (netcat-traditional
> doesn't cut it).
> 
> netcat-openbsd has higher update-alternatives priority, so it will be
> used if installed.
> 
> Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
> Cc: Jim Fehlig <jfehlig@suse.com>

I ran an adhoc test with this applied which resulted in:

2015-08-12 18:07:26 Z executing ssh ... root@10.80.229.144 virsh migrate --live debian.guest.osstest xen+ssh://lace-bug
error: unable to connect to 'lace-bug.uk.xensource.com:49152': Invalid argument

(note that the domainname for this osstest instance is "xs.citrite.net",
there is such a domain as "uk.xensource.com" which may have come from DHCP
or DNS or something).

So I switched to using $dho->{Fqdn} instead of just $dho->{Name} and tried
again resulting in:

2015-08-12 21:49:22 Z executing ssh ... root@10.80.229.144 virsh migrate --live debian.guest.osstest xen+ssh://lace-bug.xs.citrite.net
error: unable to connect to 'lace-bug.uk.xensource.com:49152': Invalid argument

Which is pretty WTF... Logs for this one are at 
http://xenbits.xen.org/people/ianc/tmp/37828/
http://xenbits.xen.org/people/ianc/tmp/37828/test-amd64-amd64-libvirt-pair/info.html

I'm doing an experiment now with $dho->{Ip} but that seems like it would
just be papering over the issue, whatever it is.

Ian.

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

On Thu, 2015-08-13 at 09:21 +0100, Ian Campbell wrote:
> 
> Which is pretty WTF... Logs for this one are at 
> http://xenbits.xen.org/people/ianc/tmp/37828/
> http://xenbits.xen.org/people/ianc/tmp/37828/test-amd64-amd64-libvirt
> -pair/info.html
> 
> I'm doing an experiment now with $dho->{Ip} but that seems like it would
> just be papering over the issue, whatever it is.

http://xenbits.xen.org/people/ianc/tmp/37831/ =>
http://xenbits.xen.org/people/ianc/tmp/37831/test-amd64-amd64-libvirt-pair/21.ts-guest-migrate.log

2015-08-13 09:42:45 Z executing ssh ... root@10.80.229.144 virsh migrate --live debian.guest.osstest xen+ssh://10.80.228.77
error: unable to connect to 'lace-bug.uk.xensource.com:49152': Invalid argument

Which, just, WTF?

Ian.

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

On Thu, 2015-08-13 at 11:27 +0100, Ian Campbell wrote:
> On Thu, 2015-08-13 at 09:21 +0100, Ian Campbell wrote:
> > 
> > Which is pretty WTF... Logs for this one are at 
> > http://xenbits.xen.org/people/ianc/tmp/37828/
> > http://xenbits.xen.org/people/ianc/tmp/37828/test-amd64-amd64-libvirt
> > -pair/info.html
> > 
> > I'm doing an experiment now with $dho->{Ip} but that seems like it 
> > would
> > just be papering over the issue, whatever it is.
> 
> http://xenbits.xen.org/people/ianc/tmp/37831/ =>
> http://xenbits.xen.org/people/ianc/tmp/37831/test-amd64-amd64-libvirt
> -pair/21.ts-guest-migrate.log
> 
> 2015-08-13 09:42:45 Z executing ssh ... root@10.80.229.144 virsh migrate 
> --live debian.guest.osstest xen+ssh://10.80.228.77
> error: unable to connect to 'lace-bug.uk.xensource.com:49152': Invalid 
> argument
> 
> Which, just, WTF?

http://xenbits.xen.org/people/ianc/tmp/37831/test-amd64-amd64-libvirt-pair/moss-bug---var-log-libvirt-libvirtd.log.gz
is the libvirt log on the send (moss-bug == 10.80.229.144) has the first
mention of migrate as:

2015-08-13 09:42:45.982+0000: 3457: debug : virDomainMigratePerform3Params:5229 : params["migrate_uri"]=(string)tcp://lace-bug.uk.xensource.com:49152

and no mentions of 10.80.228.77 anywhere, so I suppose whatever is going on
is a client side thing? Lets see if "virsh --debug 0" tells me anything
more.

Ian.

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Jackson]

Ian Campbell writes ("[PATCH OSSTEST] ts-xen-install: Install netcat-openbsd"):
> This is required by libvirt for live migration (netcat-traditional
> doesn't cut it).
> 
> netcat-openbsd has higher update-alternatives priority, so it will be
> used if installed.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

On Thu, 2015-08-13 at 11:38 +0100, Ian Campbell wrote:
> Lets see if "virsh --debug 0" tells me anything

http://xenbits.xen.org/people/ianc/tmp/37833/

2015-08-13 13:02:39 Z executing ssh ... root@10.80.229.144 virsh --debug 0 migrate --live debian.guest.osstest xen+ssh://10.80.228.77
migrate: live(bool): (none)
migrate: domain(optdata): debian.guest.osstest
migrate: desturi(optdata): xen+ssh://10.80.228.77
migrate: found option <domain>: debian.guest.osstest
migrate: <domain> trying as domain NAME
migrate: found option <domain>: debian.guest.osstest
migrate: <domain> trying as domain NAME
error: unable to connect to 'lace-bug.uk.xensource.com:49152': Invalid argument

So I'm no wiser...

I grabbed the box in this state and a simple "ssh lace-bug" (AKA
10.80.228.77) just worked.

The host has:

# cat /etc/resolv.conf 
domain uk.xensource.com
search uk.xensource.com. xensource.com. xs.citrite.net. eng.citrite.net. citrite.net.
nameserver 10.80.16.125
nameserver 10.80.16.126
#

Which I guess explains to some extent where lace-bug.uk.xensource.com is
coming from in the xen+ssh://lace-bug case (sort of) but not really for the
xen+ssh://lace-bug.xs.citrite.net nor xen+ssh://10.80.228.77 cases.

Jim, do you have any idea what is going on or who/what is doing this
translation?

This is all on the Cambridge instance of osstest, the XenProject instance
in the colo has only a single domain name, so things may work as they are
today (once the right version of netcat is added), but I'd still like to
get to the bottom of it, since it'll affect other people I'm sure.

Ian.

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

On Thu, 2015-08-13 at 14:59 +0100, Ian Campbell wrote:
> On Thu, 2015-08-13 at 11:38 +0100, Ian Campbell wrote:
> > Lets see if "virsh --debug 0" tells me anything
> 
> http://xenbits.xen.org/people/ianc/tmp/37833/
> 
> 2015-08-13 13:02:39 Z executing ssh ... root@10.80.229.144 virsh --debug 
> 0 migrate --live debian.guest.osstest xen+ssh://10.80.228.77
> migrate: live(bool): (none)
> migrate: domain(optdata): debian.guest.osstest
> migrate: desturi(optdata): xen+ssh://10.80.228.77
> migrate: found option <domain>: debian.guest.osstest
> migrate: <domain> trying as domain NAME
> migrate: found option <domain>: debian.guest.osstest
> migrate: <domain> trying as domain NAME
> error: unable to connect to 'lace-bug.uk.xensource.com:49152': Invalid 
> argument
> 
> So I'm no wiser...
> 
> I grabbed the box in this state and a simple "ssh lace-bug" (AKA
> 10.80.228.77) just worked.
> 
> The host has:
> 
> # cat /etc/resolv.conf 
> domain uk.xensource.com
> search uk.xensource.com. xensource.com. xs.citrite.net. eng.citrite.net. 
> citrite.net.
> nameserver 10.80.16.125
> nameserver 10.80.16.126
> #
> 
> Which I guess explains to some extent where lace-bug.uk.xensource.com is
> coming from in the xen+ssh://lace-bug case (sort of) but not really for 
> the
> xen+ssh://lace-bug.xs.citrite.net nor xen+ssh://10.80.228.77 cases.

I've just spotted (on moss-bug, lace-bug has equivalent):

    # cat /etc/hosts
    127.0.0.1    	    localhost
    127.0.1.1    	    moss-bug.uk.xensource.com    	    moss-bug

    # The following lines are desirable for IPv6 capable hosts
    ::1     localhost ip6-localhost ip6-loopback
    ff02::1 ip6-allnodes
    ff02::2 ip6-allrouters
    # cat /var/lib/dhcp/dhclient.eth0.leases 
    lease {
      interface "eth0";
      fixed-address 10.80.229.144;
      filename "/pxelinux.0";
      option subnet-mask 255.255.240.0;
      option routers 10.80.224.1;
      option dhcp-lease-time 43200;
      option dhcp-message-type 5;
      option domain-name-servers 10.80.16.125,10.80.16.126;
      option dhcp-server-identifier 10.80.16.125;
      option domain-search "uk.xensource.com.", "xensource.com.", "xs.citrite.net.", "eng.citrite.net.", "citrite.net.";
      option ntp-servers 10.80.16.80,10.80.248.8;
      option host-name "moss-bug";
      option domain-name "uk.xensource.com";
      renew 4 2015/08/13 17:38:48;
      rebind 4 2015/08/13 23:19:23;
      expire 5 2015/08/14 00:49:23;
    }
    # domainname 
    (none)
    # hostname -f
    moss-bug.uk.xensource.com

I think /etc/hosts came from the installer getting the same domain-name
option in DHCP as shown above.

So I think that is a local infrastructure bug (which I'll get fixed) but I
also think that virsh's behaviour when given a FQDN or an explicit IP
address is strange.

> 
> Jim, do you have any idea what is going on or who/what is doing this
> translation?
> 
> This is all on the Cambridge instance of osstest, the XenProject instance
> in the colo has only a single domain name, so things may work as they are
> today (once the right version of netcat is added), but I'd still like to
> get to the bottom of it, since it'll affect other people I'm sure.
> 
> Ian.

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

On Thu, 2015-08-13 at 11:46 +0100, Ian Jackson wrote:
> Ian Campbell writes ("[PATCH OSSTEST] ts-xen-install: Install netcat
> -openbsd"):
> > This is required by libvirt for live migration (netcat-traditional
> > doesn't cut it).
> > 
> > netcat-openbsd has higher update-alternatives priority, so it will be
> > used if installed.
> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

I tacked this onto pretest and killed the current flight (60671) which had
only done builds and arm tests thus far and was waiting on machines for all
the x86 tests.

Ian.

Re: [PATCH OSSTEST] ts-xen-install: Install netcat-openbsd

[Ian Campbell]

On Thu, 2015-08-13 at 14:59 +0100, Ian Campbell wrote:
> Jim, do you have any idea what is going on or who/what is doing this
> translation?

I've had a bit of trouble finding the code, but I think what is happening
is that the moss-bug (the source) is doing a ssh to lace-bug (the
destination), which is working fine and lace-bug is then telling moss-bug
to connect to lace-bug.uk.xensource.com (i.e. what it thinks is its FQDN).

Is that consistent with the xen+ssh URI scheme, i.e. that it is for
command/control only and the data goes over a separately established
connection?

Or maybe lace-bug is attempting a loop back connection to itself for some
reason?

So I think in the end this is probably down to the DHCP misconfiguration
here.

I've asked for DHCP to be fixed for the machines here in Cambridge, I think
this will be OK in the colo since its DHCP configuration is not wrong in
this way.

Ian.