[OSSTEST PATCH 1/4] mg-repro-setup: Use new syntax for cs-adjust-flight

[OSSTEST PATCH 1/4] mg-repro-setup: Use new syntax for cs-adjust-flight

[Ian Jackson]

In 497b2c6c933d "Rework runvar-build-set new value handling" we fixed
bugs in cs-adjust-flight which would make mg-repro-setup's attempt to
adjust build jobs sometimes not match at all, but we also changed the
syntax for specifying just a new flight.  I forgot to update the
one in-tree user of that feature.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index 4f2fe54..9663497 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -199,7 +199,7 @@ if $skipcapture; then adjrunvar skip_testids "capture-logs*"; fi
 
 ./cs-adjust-flight $flight						\
 	copy-jobs $example_flight $job					\
-	runvar-build-set . '/buildjob$' "^$flight\\." $example_flight \
+	runvar-build-set . '/buildjob$' "^$flight\\." $example_flight.  \
 	"${adjusts[@]}"
 
 progress "executing ..."
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 2/4] tcl/daemonlib: tolerate no $c(...Host)

[Ian Jackson]

From: osstest service user <osstest@xenbits.xenproject.org>

We are going to want a daemon that bins to * rather than to a known
address.

We achieve this by simply tolerating the lack of the FooHost config
setting; and, in that case, not passing -myaddr to Tcl's socket
command (and adjusting messages accordingly).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 tcl/daemonlib.tcl | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/tcl/daemonlib.tcl b/tcl/daemonlib.tcl
index 1e86d5f..b23caea 100644
--- a/tcl/daemonlib.tcl
+++ b/tcl/daemonlib.tcl
@@ -195,7 +195,7 @@ proc newconn {chan addr port} {
 proc main-daemon {which setup} {
     global c argv
 
-    set host $c(${which}DaemonHost)
+    catch { set host $c(${which}DaemonHost) }
     set port $c(${which}DaemonPort)
 
     foreach arg $argv {
@@ -210,12 +210,21 @@ proc main-daemon {which setup} {
     fconfigure stdout -buffering line
     fconfigure stderr -buffering none
 
-    log "starting $host:$port"
+    set desc $port
+
+    set sockcmd {socket -server newconn}
+    if {[info exists host]} {
+        set desc "$host:$port"
+        lappend sockcmd [list -myaddr $host]
+    }
+    lappend sockcmd $port
+
+    log "starting $desc"
 
     uplevel 1 $setup
 
-    socket -server newconn -myaddr $host $port
-    log "listening $host:$port"
+    eval $sockcmd
+    log "listening $desc"
 
     vwait forever
 }
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 3/4] dhcp leases: Introduce new client/server leases query mechanism

[Ian Jackson]

From: osstest service user <osstest@xenbits.xenproject.org>

osstest needs to know (from the DHCP server) what addresses its guests
have.  Until now, the only way to do this has been to read the DHCP
server's leases file - if necessary, fetching the leases file from the
server.

Of course the size of the leases file is proportional to the size of
the installation, as is the number of simultaneous test jobs.  So the
overall impact of this is O(n^2); currently, the leases file in
Massachusetts is about 300kby and is being fetched fairly frequently.

So, we provide an alternative arrangement.

The ISC DHCP server does not support a standard way of querying it
about leases (eg, SNMP).  It does support an ISC protocol called
"OMAPI".

We don't want to have each osstest test job talk to the DHCP server
via OMAPI because this would involve giving them a password to the
DHCP server.  So we need to provide, effectively, a proxy daemon.

The available OMAPI client libraries are synchronous; the best
available library is written in Python.  We could write the whole
daemon in Python but that would involve doing our own select and line
buffering/reassembly, reimplementing command parsing, etc. - which we
already have implementations of in tcl/daemonlib.tcl.

So we provide, instead:

 * A Python program `ms-leases-omapiproxy' which speaks OMAPI to the
   DHCP server, and a simple synchronous query protocol on its
   stdin/stdout.  OMAPI authentication is based on a shared secret; to
   avoid duplicating it any more than necessary, ms-leases-omapiproxy
   half-arsedly parses the relevant DHCP server config fragment.

 * A Tcl daemon `ms-leasesdaemon' which multiplexes the multiple
   clients, and issues queries to the Python program.  It speaks
   a protocol similar to the other Tcl daemons.

 * A client implementation in Osstest/DhcpWatch/leasesdaemon.pm which
   can be requested by per-host configuration, as the DHCP lease
   method.

Like with the `leases' access method, the hostname of the DHCP server
must be specified in the host property.  The Tcl daemon binds to *, so
does not need to know its hostname.  So the config has only the port,
and the on-DHCP-server settings.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest.pm                        |  4 ++
 Osstest/DhcpWatch/leasesdaemon.pm | 98 +++++++++++++++++++++++++++++++++++++++
 ms-leases-omapiproxy              | 43 +++++++++++++++++
 ms-leasesdaemon                   | 51 ++++++++++++++++++++
 4 files changed, 196 insertions(+)
 create mode 100644 Osstest/DhcpWatch/leasesdaemon.pm
 create mode 100755 ms-leases-omapiproxy
 create mode 100755 ms-leasesdaemon

diff --git a/Osstest.pm b/Osstest.pm
index a78728c..9fdefde 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -104,6 +104,10 @@ our %c = qw(
     HostnameSortSwapWords 0
 
     Timezone UTC
+
+    DhcpServerOmapiKeyFile /etc/dhcp/osstest-omapi-key
+    DhcpServerOmapiPort 7991
+    LeasesDaemonPort 4033
 );
 
 $c{$_}='' foreach qw(
diff --git a/Osstest/DhcpWatch/leasesdaemon.pm b/Osstest/DhcpWatch/leasesdaemon.pm
new file mode 100644
index 0000000..9148880
--- /dev/null
+++ b/Osstest/DhcpWatch/leasesdaemon.pm
@@ -0,0 +1,98 @@
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+package Osstest::DhcpWatch::leasesdaemon;
+
+use strict;
+use warnings;
+
+use POSIX;
+use IO::File;
+use IO::Socket;
+
+use Osstest;
+use Osstest::TestSupport;
+
+BEGIN {
+    use Exporter ();
+    our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);
+    $VERSION     = 1.00;
+    @ISA         = qw(Exporter);
+    @EXPORT      = qw();
+    %EXPORT_TAGS = ( );
+
+    @EXPORT_OK   = qw();
+}
+
+sub new {
+    my ($class, $ho, $meth, $source) = @_;
+    my $mo;
+    if ($source =~ m/:(?=[^:]*)$/) {
+	$mo = {
+            Host => $`,
+            Port => $',
+        };
+    } else {
+	$mo = {
+            Host => $source,
+            Port => $c{LeasesDaemonPort},
+        };
+    }
+    return bless $mo, $class;
+}
+
+sub check_ip ($$) {
+    my ($mo, $gho) = @_;
+
+    my $may_retry = 1;
+    while ($may_retry) {
+        my $rv;
+        eval {
+            my $lserv = $mo->{Conn};
+            if (!$lserv) {
+                logm("leasesdaemon: $gho->{Name}:".
+                     " connecting to $mo->{Host}:$mo->{Port}");
+                $may_retry = 0;
+                $lserv = tcpconnect($mo->{Host}, $mo->{Port});
+                $lserv->autoflush(1);
+                $_ = <$lserv>;
+                defined && m/^OK ms-leasesdaemon\s/ or die "$_ ?";
+                $mo->{Conn} = $lserv;
+            }
+            print $lserv "ether-ip4 $gho->{Ether}\n";
+            $_ = <$lserv>;
+            if (m/^\d+\.\d+\.\d+\.\d+$/) {
+                $gho->{Ip} = $&;
+                report_once($gho, 'leasesdaemon::check_ip', 
+                            "guest $gho->{Name}: $gho->{Ether} $gho->{Ip}");
+                $rv = undef;
+            } elsif (m/^none$/) {
+                $rv = 'no lease';
+            } else {
+                die "got $_ !";
+            }
+        };
+        if (!length $@) {
+            return $rv;
+        }
+        $may_retry = 0;
+        delete $mo->{Conn};
+    }
+    return "error talking to ms-leasesdaemon: $@";
+}
+
+1;
diff --git a/ms-leases-omapiproxy b/ms-leases-omapiproxy
new file mode 100755
index 0000000..a302a85
--- /dev/null
+++ b/ms-leases-omapiproxy
@@ -0,0 +1,43 @@
+#!/usr/bin/python -u
+
+import sys
+import os
+import pypureomapi
+import re as regexp
+
+def getkey(filename):
+    global key_name
+    global secret_key
+
+    kn_re = regexp.compile(r'''\s*key\s+(\w+)\s*\{$''')
+    sk_re = regexp.compile(r'''\s*secret\s*(['"])(.*)\1\s*\;\s*$''')
+    f = open(filename)
+    for l in f:
+        m = kn_re.match(l)
+        if m: key_name = m.group(1)
+        m = sk_re.match(l)
+        if m: secret_key = m.group(2)
+
+    key_name
+    secret_key
+
+def connect(port):
+    global om
+    om = pypureomapi.Omapi('localhost', 7911, key_name, secret_key)
+
+def iterate():
+    while True:
+        l = sys.stdin.readline()
+        if not l: break
+
+        mac = l.rstrip()
+        try:
+            ip = om.lookup_ip(mac)
+        except pypureomapi.OmapiErrorNotFound:
+            ip = 'none'
+        print('%s %s' % (mac, ip))
+
+getkey(sys.argv[2])
+connect(int(sys.argv[1]))
+print('OK')
+iterate()
diff --git a/ms-leasesdaemon b/ms-leasesdaemon
new file mode 100755
index 0000000..e72596c
--- /dev/null
+++ b/ms-leasesdaemon
@@ -0,0 +1,51 @@
+#!/usr/bin/tclsh8.5
+# -*- Tcl -*- 
+# usage: ./ms-ownerdaemon  ... | logger
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+source ./tcl/daemonlib.tcl
+
+proc chan-destroy-stuff {chan} { }
+
+proc cmd/ether-ip4 {chan desc mac} {
+    global omclichan
+    if {![regexp {^(?:[0-9a-f]{2}:){5}[0-9a-f]{2}$} $mac mac]} {
+        chan-error $chan "bad mac address"
+        return
+    }
+    puts-chan $omclichan "$mac"
+    set l [must-gets-chan $omclichan "^$mac "]
+    puts-chan $chan [lindex $l 1]
+}
+
+proc banner {chan} {
+    return "OK ms-leasesdaemon"
+}
+
+main-daemon Leases {
+    global c omclichan chandesc
+
+    set cmd [list ./ms-leases-omapiproxy \
+                 $c(DhcpServerOmapiPort) $c(DhcpServerOmapiKeyFile) \
+                 2>@ stderr]
+    set omclichan [open |$cmd r+]
+    fconfigure $omclichan -buffering line -translation lf
+    set chandesc($omclichan) omcli
+    must-gets-chan $omclichan {^OK}
+}
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH 4/4] production-config: Use the new leases server for all hosts by default

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 production-config | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/production-config b/production-config
index 96831c7..63d6641 100644
--- a/production-config
+++ b/production-config
@@ -94,6 +94,8 @@ TftpNetbootGroup osstest
 TftpDiVersion_wheezy 2016-06-08
 TftpDiVersion_jessie 2017-04-06
 
+HostProp_DhcpWatchMethod leasesdaemon infra
+
 # For ISO installs
 DebianImageVersion_wheezy 7.2.0
 DebianImageVersion_jessie 8.2.0
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 3/4] dhcp leases: Introduce new client/server leases query mechanism

[Ian Jackson]

Ian Jackson writes ("[OSSTEST PATCH 3/4] dhcp leases: Introduce new client/server leases query mechanism"):
>  * A Python program `ms-leases-omapiproxy' which speaks OMAPI to the
>    DHCP server, and a simple synchronous query protocol on its
>    stdin/stdout.  OMAPI authentication is based on a shared secret; to
>    avoid duplicating it any more than necessary, ms-leases-omapiproxy
>    half-arsedly parses the relevant DHCP server config fragment.

This does not work, for this reason:

  https://lists.isc.org/pipermail/dhcp-hackers/2005-August/001399.html

I have backed out this change, and filed a bug with ISC [ISC-Bugs #45398].

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel