xen-devel.lists.xenproject.org archive mirror
 help / color / mirror / Atom feed
* [OSSTEST PATCH 0/7] osstest improvements
@ 2018-12-04 21:37 Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
                   ` (6 more replies)
  0 siblings, 7 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

Things I've learned and improved while setting osstest standalone myself.
First two patches add support for dnsmasq (note: I don't have setup to check if
haven't broken dhcp3 in the process!). Other patches are mostly independent.

Marek Marczykowski-Górecki (7):
  DhcpWatch: extract dhcp3 parsing into separate function
  DhcpWatch: add dnsmasq support
  xen-install: use route with metric 0 to retrieve local network netmask
  Replace with-lock-ex with flock
  Optimize git clone - use --depth=1
  Extend README for standalone mode
  README: add Fedora setup section

 Osstest/DhcpWatch/leases.pm | 169 +++++++++++++++++++++++--------------
 Osstest/Executive.pm        |   2 +-
 Osstest/TestSupport.pm      |  20 ++--
 README                      |  45 ++++++++--
 cr-daily-branch             |   2 +-
 cr-for-branches             |   6 +-
 cr-try-bisect               |   2 +-
 cri-args-hostlists          |   4 +-
 cri-getconfig               |   2 +-
 cri-lock-repos              |   4 +-
 crontab                     |   2 +-
 crontab-cambridge           |   2 +-
 memoise                     |   2 +-
 ts-freebsd-host-install     |   2 +-
 ts-xen-install              |   2 +-
 15 files changed, 175 insertions(+), 91 deletions(-)

base-commit: c65d7eb3f6c424d6c1fe69c5ecfca9c0b6cf4302
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply	[flat|nested] 8+ messages in thread
* [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function
  2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support Marek Marczykowski-Górecki
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

Preparation for dnsmasq support. Keep generic code in check_ip, but
extract format-specific handling into separat function. No intentional
behaviour change (besides slightly different warning reporting).

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 Osstest/DhcpWatch/leases.pm | 131 ++++++++++++++++++++-----------------
 1 file changed, 71 insertions(+), 60 deletions(-)

diff --git a/Osstest/DhcpWatch/leases.pm b/Osstest/DhcpWatch/leases.pm
index 98c22b9..5ec305f 100644
--- a/Osstest/DhcpWatch/leases.pm
+++ b/Osstest/DhcpWatch/leases.pm
@@ -48,68 +48,20 @@ sub new {
     }, $class;
 }
 
-sub check_ip ($$) {
-    my ($mo, $gho) = @_;
-
-    my $leases;
-    my $leasesfn = $mo->{Source};
-
-    if ($leasesfn =~ m,/,) {
-	$leases= new IO::File $leasesfn, 'r';
-	if (!defined $leases) { return "open $leasesfn: $!"; }
-    } else {
-	$leases= new IO::Socket::INET(PeerAddr => $leasesfn);
-	if (!defined $leases) { return "connect to $leasesfn: $!"; }
-    }
-
-    my $lstash= "dhcpleases-$gho->{Guest}";
+sub parse_dhcp_dhcp3($$$$) {
+    my ($gho, $leases, $copy, $leasesfn) = @_;
     my $inlease;
     my $props;
     my $best;
-    my @warns;
-
-    my $copy= new IO::File "$stash/$lstash.new", 'w';
-    $copy or die "$lstash.new $!";
-
-    my $saveas= sub {
-        my ($fn,$keep) = @_;
-
-        while (<$leases>) { print $copy $_ or die $!; }
-        die $! unless $leases->eof;
-
-        my $rename= sub {
-            my ($src,$dst) = @_;
-            rename "$stash/$src", "$stash/$dst"
-                or $!==&ENOENT
-                or die "rename $fn.$keep $!";
-        };
-        while (--$keep>0) {
-            $rename->("$fn.$keep", "$fn.".($keep+1));
-        }
-        if ($keep>=0) {
-            die if $keep;
-            $rename->("$fn", "$fn.$keep");
-        }
-        $copy->close();
-        rename "$stash/$lstash.new", "$stash/$fn" or die "$lstash.new $fn $!";
-        logm("warning: $_") foreach grep { defined } @warns[0..5];
-        # logm("$fn: rotated and stashed current leases");
-    };
-
-    my $badleases= sub {
-        my ($m) = @_;
-        $m= "$leasesfn:$.: unknown syntax";
-        $saveas->("$lstash.bad", 7);
-        return $m;
-    };
 
     while (<$leases>) {
         print $copy $_ or die $!;
 
         chomp; s/^\s+//; s/\s+$//;
         next if m/^\#/;  next unless m/\S/;
+
         if (m/^lease\s+([0-9.]+)\s+\{$/) {
-            return $badleases->("lease inside lease") if defined $inlease;
+            die "lease inside lease" if defined $inlease;
             $inlease= $1;
             $props= { };
             next;
@@ -122,13 +74,13 @@ sub check_ip ($$) {
             s/^( [-a-z0-9]+
                ) \s+//x
                or
-              return $badleases->("unknown syntax");
+              die "unknown syntax";
             my $prop= $1;
-            s/\s*\;$// or return $badleases->("missing semicolon");
+            s/\s*\;$// or die "missing semicolon";
             $props->{$prop}= $_;
             next;
         }
-        return $badleases->("end lease not inside lease")
+        die "end lease not inside lease"
             unless defined $inlease;
 
         $props->{' addr'}= $inlease;
@@ -144,25 +96,84 @@ sub check_ip ($$) {
         my @missing= grep { !defined $props->{$_} }
             ('binding state', 'hardware ethernet', 'ends');
         if (@missing) {
-            push @warns, "$leasesfn:$.: lease without \`$_'"
+            warn "$leasesfn:$.: lease without \`$_'"
                 foreach @missing;
             next;
         }
 
-        # ignore leases for other hosts
-        next unless lc $props->{'hardware ethernet'} eq lc $gho->{Ether};
-
         $props->{' ends'}= $props->{'ends'};
         $props->{' ends'} =~
             s/^[0-6]\s+(\S+)\s+(\d+)\:(\d+\:\d+)$/
                 sprintf "%s %02d:%s", $1,$2,$3 /e
-                or return $badleases->("unexpected syntax for ends");
+                or die "unexpected syntax for ends";
+
+        # ignore leases for other hosts
+        next unless lc $props->{'hardware ethernet'} eq lc $gho->{Ether};
 
         next if $best &&
             $best->{' ends'} gt $props->{' ends'};
         $best= $props;
     }
 
+    return $best;
+}
+
+sub check_ip ($$) {
+    my ($mo, $gho) = @_;
+
+    my $leases;
+    my $leasesfn = $mo->{Source};
+
+    if ($leasesfn =~ m,/,) {
+	$leases= new IO::File $leasesfn, 'r';
+	if (!defined $leases) { return "open $leasesfn: $!"; }
+    } else {
+	$leases= new IO::Socket::INET(PeerAddr => $leasesfn);
+	if (!defined $leases) { return "connect to $leasesfn: $!"; }
+    }
+
+    my $lstash= "dhcpleases-$gho->{Guest}";
+    my $best;
+
+    my $copy= new IO::File "$stash/$lstash.new", 'w';
+    $copy or die "$lstash.new $!";
+
+    my $saveas= sub {
+        my ($fn,$keep) = @_;
+
+        while (<$leases>) { print $copy $_ or die $!; }
+        die $! unless $leases->eof;
+
+        my $rename= sub {
+            my ($src,$dst) = @_;
+            rename "$stash/$src", "$stash/$dst"
+                or $!==&ENOENT
+                or die "rename $fn.$keep $!";
+        };
+        while (--$keep>0) {
+            $rename->("$fn.$keep", "$fn.".($keep+1));
+        }
+        if ($keep>=0) {
+            die if $keep;
+            $rename->("$fn", "$fn.$keep");
+        }
+        $copy->close();
+        rename "$stash/$lstash.new", "$stash/$fn" or die "$lstash.new $fn $!";
+        # logm("$fn: rotated and stashed current leases");
+    };
+
+    my $badleases= sub {
+        my ($m) = @_;
+        $m= "$leasesfn:$.: unknown syntax " . $m;
+        $saveas->("$lstash.bad", 7);
+        return $m;
+    };
+
+    eval { $best = parse_dhcp_dhcp3($gho, $leases, $copy, $leasesfn) };
+    if ($@) {
+        return $badleases->($@);
+    }
+
     if (!$best) {
         $saveas->("$lstash.nolease", 3);
 	if ($leases->error) {
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support
  2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask Marek Marczykowski-Górecki
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

dnsmasq use much simpler format - one lease by line with fields
separated with spaces. Add separate function to parse this format and
plug it info check_ip.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 Osstest/DhcpWatch/leases.pm | 40 +++++++++++++++++++++++++++++++++++---
 README                      |  2 +-
 2 files changed, 38 insertions(+), 4 deletions(-)

diff --git a/Osstest/DhcpWatch/leases.pm b/Osstest/DhcpWatch/leases.pm
index 5ec305f..0c379e9 100644
--- a/Osstest/DhcpWatch/leases.pm
+++ b/Osstest/DhcpWatch/leases.pm
@@ -40,8 +40,8 @@ BEGIN {
 
 sub new {
     my ($class, $ho, $meth, $format, $source) = @_;
-    die "'$format' is missing dhcp3 in it! (args: @_) ?"
-	unless $format eq 'dhcp3';
+    die "'$format' is missing dhcp3/dnsmasq in it! (args: @_) ?"
+	unless $format eq 'dhcp3' or $format eq 'dnsmasq';
     return bless {
 	Format => $format,
 	Source => $source,
@@ -118,6 +118,36 @@ sub parse_dhcp_dhcp3($$$$) {
     return $best;
 }
 
+sub parse_dhcp_dnsmasq($$$$) {
+    my ($gho, $leases, $copy, $leasesfn) = @_;
+
+    my $props;
+    my $best;
+
+    while (<$leases>) {
+        print $copy $_ or die $!;
+
+        chomp; s/^\s+//; s/\s+$//;
+        next if m/^\#/;  next unless m/\S/;
+
+        my ($ends, $ether, $ip, $hostname, $client_id) = split / /;
+        $ends = strftime "%Y/%m/%d %T", localtime($ends);
+        $props = {
+            ' addr' => $ip,
+            ' ends' => $ends,
+            'hardware ethernet' => $ether
+        };
+        # ignore leases for other hosts
+        next unless lc $props->{'hardware ethernet'} eq lc $gho->{Ether};
+
+        next if $best &&
+            $best->{' ends'} gt $props->{' ends'};
+        $best= $props;
+    }
+
+    return $best;
+}
+
 sub check_ip ($$) {
     my ($mo, $gho) = @_;
 
@@ -169,7 +199,11 @@ sub check_ip ($$) {
         return $m;
     };
 
-    eval { $best = parse_dhcp_dhcp3($gho, $leases, $copy, $leasesfn) };
+    if ($mo->{Format} eq 'dnsmasq') {
+        eval { $best = parse_dhcp_dnsmasq($gho, $leases, $copy, $leasesfn) };
+    } elsif ($mo->{Format} eq 'dhcp3') {
+        eval { $best = parse_dhcp_dhcp3($gho, $leases, $copy, $leasesfn) };
+    }
     if ($@) {
         return $badleases->($@);
     }
diff --git a/README b/README
index bb9dc18..4e4066a 100644
--- a/README
+++ b/README
@@ -508,7 +508,7 @@ HostFlags_<host>         flag,^flag,^flag,flag...
 
 HostProp_DhcpWatchMethod
    leases <format> <source>
-      where <format> is dhcp3
+      where <format> is dhcp3 or dnsmasq
             <source> is filename (with slash) or <host>:<port>
 
 AuthorizedKeysFiles
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask
  2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 4/7] Replace with-lock-ex with flock Marek Marczykowski-Górecki
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

Otherwise it will match 169.254.0.0/16 route, which has always /16
netmask, instead of what really is used on the local network.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 ts-xen-install | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ts-xen-install b/ts-xen-install
index 8de94ac..2f0d263 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -349,7 +349,7 @@ END
 
 	my $routes= target_cmd_output_root($ho, "route -n");
 
-	$routes =~ m/^ [0-9.]+ \s+ 0\.0\.0\.0 \s+ ([0-9.]+) \s+ \S*U\S* \s /mxi
+	$routes =~ m/^ [0-9.]+ \s+ 0\.0\.0\.0 \s+ ([0-9.]+) \s+ \S*U\S* \s+ 0 \s/mxi
 	    or die "no own local network in route ?  $routes ";
 	my $netmask= $1;
 
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [OSSTEST PATCH 4/7] Replace with-lock-ex with flock
  2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
                   ` (2 preceding siblings ...)
  2018-12-04 21:37 ` [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1 Marek Marczykowski-Górecki
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

with-lock-ex comes from a Debian-only, not installed by default
chiark-utils-bin package, while flock have equivalent functionality and
is part of standard util-linux package.

This is especially important for apt-get call under the lock
(TestSupport.pm:target_run_pkgmanager_install function), which could be
used to install chiark-utils-bin but will fail because with-lock-ex is
not there.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 Osstest/Executive.pm    | 2 +-
 Osstest/TestSupport.pm  | 2 +-
 cr-daily-branch         | 2 +-
 cr-for-branches         | 6 +++---
 cr-try-bisect           | 2 +-
 cri-args-hostlists      | 4 ++--
 cri-getconfig           | 2 +-
 cri-lock-repos          | 4 ++--
 crontab                 | 2 +-
 crontab-cambridge       | 2 +-
 memoise                 | 2 +-
 ts-freebsd-host-install | 2 +-
 12 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 0be27b6..8a272e7 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -130,7 +130,7 @@ sub grabrepolock_reexec {
     my $repos_locked= $ENV{OSSTEST_REPOS_LOCK_LOCKED};
     unless (defined $repos_locked && $repos_locked eq $repos_lock) {
         $ENV{OSSTEST_REPOS_LOCK_LOCKED}= $repos_lock;
-        exec "with-lock-ex","-w",$repos_lock, $0,@org_argv;
+        exec "flock","-w",$repos_lock, $0,@org_argv;
         die $!;
     }
 }
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 2910842..9515c9b 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -535,7 +535,7 @@ sub target_run_pkgmanager_install ($$;$$) {
         push @cmd, qw(lockf /var/run/osstest-pkg-lock pkg-static install);
     } else {
         push @cmd, qw(DEBIAN_PRIORITY=critical UCF_FORCE_CONFFOLD=y
-                      with-lock-ex -w /var/lock/osstest-apt apt-get);
+                      flock /var/lock/osstest-apt apt-get);
 	push @cmd, qw(-f) if $force;
 	push @cmd, qw(--no-install-recommends) if $norec;
 	push @cmd, qw(-y install);
diff --git a/cr-daily-branch b/cr-daily-branch
index e1528d4..492eff0 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -475,7 +475,7 @@ if grep -xF "$NEW_REVISION" $branch.force-rev; then push=$OSSTEST_PUSH; fi
 if test -f $branch.block; then push=false; fi
 
 if test -e $mrof && test -e $tree_bisect && ! grep '^broken' $mrof; then
-	with-lock-ex -w $tree_bisect/$mrof.lock bash -xec "
+	flock $tree_bisect/$mrof.lock bash -xec "
 		rm -f $tree_bisect/$mrof.in.new
 		cp $mrof $tree_bisect/$mrof.in.new
 		cd $tree_bisect
diff --git a/cr-for-branches b/cr-for-branches
index 2e9717e..93a713a 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -1,6 +1,6 @@
 #!/bin/bash
 # usage: cr-for-branches BRANCHESDIR WLEM "SCRIPT OPTIONS" ARGS...
-# will run   cd BRANCHESDIR && with-lock-ex WLEM SCRIPT OPTIONS BRANCH ARGS...
+# will run   cd BRANCHESDIR && flock WLEM SCRIPT OPTIONS BRANCH ARGS...
 
 # This is part of "osstest", an automated testing framework for Xen.
 # Copyright (C) 2009-2013 Citrix Inc.
@@ -39,7 +39,7 @@ if [ "x$fetchwlem" = x-q ]; then
 	fetchwlem=-f
 fi
 
-with-lock-ex $fetchwlem data-tree-lock bash -ec '
+flock $fetchwlem data-tree-lock bash -ec '
 	exec >>$LOGFILE
 	date
         printf "%s\n" "$FOR_LOGFILE"
@@ -68,7 +68,7 @@ for branch in $BRANCHES; do
 
 	export LOCK_ACQU_START=`date +%s`
 
-	with-lock-ex $wlem data-tree-lock bash -ec '
+	flock $wlem data-tree-lock bash -ec '
 		m="$*"
 
 		mkdir -p tmp
diff --git a/cr-try-bisect b/cr-try-bisect
index d613c34..afe4b86 100755
--- a/cr-try-bisect
+++ b/cr-try-bisect
@@ -31,7 +31,7 @@ startstamp=tmp/bisection-start-stamp
 
 anyflagfile=tmp/bisected-any.$branch
 
-with-lock-ex -w $mrof.lock bash -xec "
+flock $mrof.lock bash -xec "
 	if test -e $mrof.in; then
 		if test -e $mrof; then mv $mrof $mrof.old; fi
                 rm -f $anyflagfile
diff --git a/cri-args-hostlists b/cri-args-hostlists
index 7d23087..a502ac1 100644
--- a/cri-args-hostlists
+++ b/cri-args-hostlists
@@ -114,7 +114,7 @@ start_email () {
 
 	globallockdir=`getconfig GlobalLockDir`
 
-	with-lock-ex -w $globallockdir/report-lock \
+	flock $globallockdir/report-lock \
 	  ./sg-report-job-history --html-dir=$job_html_dir --flight=$flight
 
 	./sg-report-flight --html-dir=$flight_html_dir/$flight/ \
@@ -122,7 +122,7 @@ start_email () {
 		$sgr_args $flight
 
 	mkdir -p $host_html_dir
-	with-lock-ex -w $globallockdir/report-lock \
+	flock $globallockdir/report-lock \
 	  ./sg-report-host-history --html-dir=$host_html_dir flight:$flight
 }
 
diff --git a/cri-getconfig b/cri-getconfig
index f8397c1..928f140 100644
--- a/cri-getconfig
+++ b/cri-getconfig
@@ -57,7 +57,7 @@ get_psql_cmd () {
 	#  and set OSSTEST_PSQL_ONLY_DO to an integer
 	if [ "x$OSSTEST_PSQL_ONLY_DO" != x ]; then
 		local f=t.psql-counter
-		psql_counter=$( with-lock-ex -w $f.lock bash -ec '
+		psql_counter=$( flock $f.lock bash -ec '
 			psql_counter=$(cat '$f' || echo 0)
 			echo $(( $psql_counter + 1 )) >'$f'.tmp
 			mv -f '$f'.tmp '$f'
diff --git a/cri-lock-repos b/cri-lock-repos
index e7e0fa8..b3500f9 100644
--- a/cri-lock-repos
+++ b/cri-lock-repos
@@ -26,8 +26,8 @@ if [ "x$AP_FETCH_PLACEHOLDERS" = xy ] && ! [ -e $repos ]; then
 	echo "AP_FETCH_PLACEHOLDERS, not locking" >&2
 elif [ "x$OSSTEST_REPOS_LOCK_LOCKED" != "x$repos_lock" ]; then
 	OSSTEST_REPOS_LOCK_LOCKED="$repos_lock" \
-	exec with-lock-ex -w "$repos_lock" \
+	exec flock "$repos_lock" \
 	"$0" "$@"
-	echo >&2 "arrgh, exec with-lock-ex failed $?"
+	echo >&2 "arrgh, exec flock failed $?"
 	exit 1
 fi
diff --git a/crontab b/crontab
index e1e798a..9c2489f 100755
--- a/crontab
+++ b/crontab
@@ -16,6 +16,6 @@ MAILTO=osstest-admin@xenproject.org
 34		15	23 * *		cd testing.git && BRANCHES=examine		./cr-for-branches branches -w "./cr-daily-branch --real"
 18		4	* * *		cd testing.git && BRANCHES='linux-3.0 libvirt rumprun' ./cr-for-branches branches -w "./cr-daily-branch --real"
 6-59/15   	*	* * *		cd testing.git && EXTRA_BRANCHES='xen-unstable-smoke linux-3.0 rumprun libvirt freebsd-master' ./cr-for-branches bisects -w "./cr-try-bisect --real"
-#8-59/5		*	* * *		cd bisects/adhoc.git &&	with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
+#8-59/5		*	* * *		cd bisects/adhoc.git &&	flock -n -E 0 data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
 22		8	* * *		cd testing.git && BRANCHES=maintjobs		./cr-for-branches . -w ./cr-all-branch-statuses ''
 3		4	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
diff --git a/crontab-cambridge b/crontab-cambridge
index 86f069a..8c0d346 100755
--- a/crontab-cambridge
+++ b/crontab-cambridge
@@ -21,6 +21,6 @@ MAILTO=ian.jackson@citrix.com
 
 34		15	25 * *		cd testing.git && BRANCHES=examine		./cr-for-branches branches -w "./cr-daily-branch --real"
 
-#8-59/5		*	* * *		cd bisects/adhoc.git && with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
+#8-59/5		*	* * *		cd bisects/adhoc.git && flock -n -E 0 data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
 22		8	* * *		cd testing.git && BRANCHES=maintjobs		./cr-for-branches . -w ./cr-all-branch-statuses ''
 3		4	* * *		savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
diff --git a/memoise b/memoise
index 73f15e6..9a6439d 100755
--- a/memoise
+++ b/memoise
@@ -43,7 +43,7 @@ id=${id%  -}
 f="$datadir/$id"
 
 if ! [ -f "$f.o" ]; then
-	with-lock-ex -w "$f.l" sh -ec '
+	flock "$f.l" sh -ec '
 		f=$1; shift
 		if [ -f "$f.o" ]; then exit 0; fi
 		exec </dev/null >"$f.t"
diff --git a/ts-freebsd-host-install b/ts-freebsd-host-install
index 53daeef..ee1cede 100755
--- a/ts-freebsd-host-install
+++ b/ts-freebsd-host-install
@@ -105,7 +105,7 @@ ln $sharedpath $targetpath
 # Prune old images not used anymore
 find `dirname $sharedpath` -links 1 -ctime +7 -delete
 END
-    my @cmd = ( "with-lock-ex", "-w", "$tftp_freebsd/lock",
+    my @cmd = ( "flock", "$tftp_freebsd/lock",
                 "bash", "-exc", "$script", "x",
                 "$tftp_freebsd", "$image", "by-hash/$hash.img",
                 "$ho->{Tftp}{Path}/$pxeimg" );
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1
  2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
                   ` (3 preceding siblings ...)
  2018-12-04 21:37 ` [OSSTEST PATCH 4/7] Replace with-lock-ex with flock Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 6/7] Extend README for standalone mode Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 7/7] README: add Fedora setup section Marek Marczykowski-Górecki
  6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

Do not clone the whole history to save time, disk space and bandwidth.
This reduce (on my machine) Linux clone from over half an hour to few
minutes.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
This should be fine, unless this function is also used when preparing
for bisection. Is it?
---
 Osstest/TestSupport.pm | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 9515c9b..e51ffc7 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1492,15 +1492,19 @@ END
 END
     } elsif ($vcs eq 'git') {
 
-        target_cmd_build($ho, $timeout, $builddir, <<END.
-            $rm
-            git clone '$tree' $subdir
-            cd $subdir
+        if (length($r{"revision_$which"})) {
+            target_cmd_build($ho, $timeout, $builddir, <<END);
+                $rm
+                git clone --depth=1 -b '$r{"revision_$which"}' '$tree' $subdir
+                cd $subdir
 END
-                         (length($r{"revision_$which"}) ? <<END : ''));
-	    git checkout '$r{"revision_$which"}'
-	    git clean -xdf
+        } else {
+            target_cmd_build($ho, $timeout, $builddir, <<END);
+                $rm
+                git clone --depth=1 '$tree' $subdir
+                cd $subdir
 END
+        }
     } else {
         die "$vcs $which $tree ?";
     }
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [OSSTEST PATCH 6/7] Extend README for standalone mode
  2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
                   ` (4 preceding siblings ...)
  2018-12-04 21:37 ` [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1 Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
  2018-12-04 21:37 ` [OSSTEST PATCH 7/7] README: add Fedora setup section Marek Marczykowski-Górecki
  6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

Things I've learned when setting it up myself.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 README | 29 +++++++++++++++++++++++++----
 1 file changed, 25 insertions(+), 4 deletions(-)

diff --git a/README b/README
index 4e4066a..a4520c8 100644
--- a/README
+++ b/README
@@ -305,6 +305,12 @@ To run osstest in standalone mode:
      ipmitool -- for hosts which use IPMI for power control
      grub-common -- for mg-netgrub-loader-update
 
+ - Setup TFTP server (see 'TftpPath' setting below), copy pxelinux
+   files there (pxelinux.0, ldlinux.c32, libcom32.c32), setup your
+   DHCP server to point PXE clients at it
+
+ - Setup HTTP server (see 'Webspace*' settings below)
+
  - Write a config file
     ~/.xen-osstest/config
    See below.
@@ -338,9 +344,16 @@ To run osstest in standalone mode:
 
    However, as test-amd64-{i386,amd64}-xl and other tests depends on
    some runtime variables generated by build-* jobs, you can run
-   build-* jobs before running actual test jobs. If you don't want to do
-   so you need to insert those missing runvars into standalone.db with
-   sqlite3.
+   build-* jobs before running actual test jobs. The error message will
+   look like this:
+
+       2018-12-04 20:56:52 Z checking builds ...
+       2018-12-04 20:56:52 Z checking buildjob path_dist
+       need path_dist in build-i386 at Osstest/TestSupport.pm line 327.
+                         ^^^^^^^^^^ name of build job to run
+
+   If you don't want to do so you need to insert those missing runvars
+   into standalone.db with sqlite3.
 
  - Don't forget to set the machine to netboot in the BIOS.
 
@@ -407,6 +420,10 @@ HostProp_<testbox>_Ether
    MAC address of the box <testbox>.  Only needed if you want
    to use the osstest host and Xen installer.
 
+HostProp_<testbox>_IpAddr
+   IP address of the box <testbox>.  Only needed if
+   <testbox>.<DnsDomain> does not resolve to the correct IP.
+
 HostProp_<testbox>_Build_Make_Flags
    Extra flags to pass to "make". e.g. "-j<SOMETHING>"
 
@@ -511,6 +528,10 @@ HostProp_DhcpWatchMethod
       where <format> is dhcp3 or dnsmasq
             <source> is filename (with slash) or <host>:<port>
 
+   When using <host>:<port> syntax, connecting to it should return the
+   whole dhcp leases file. For example use socat on the DHCP server host:
+     socat -U TCP-LISTEN:5556,fork,reuseaddr EXEC:'cat /var/lib/dhcpd/dhcpd.leases'
+
 AuthorizedKeysFiles
      :-separated list of files to concatenate into authorized_keys
 AuthorizedKeysAppend
@@ -520,7 +541,7 @@ The keys in ~/.ssh/id_{rsa,dsa}.pub and ~/.ssh/authorized_keys
 
 TestHostKeypairPath
 
-Platforms<Arch>
+Platforms   <Arch>
    List of platforms (i.e. distinct host types) to run a basic test on.
 
 HostProp_GenEtherPrefixBase 5e:36:0e:f5
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread
* [OSSTEST PATCH 7/7] README: add Fedora setup section
  2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
                   ` (5 preceding siblings ...)
  2018-12-04 21:37 ` [OSSTEST PATCH 6/7] Extend README for standalone mode Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
  6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
  To: xen-devel; +Cc: Marek Marczykowski-Górecki

Package names are slightly different. Additionally some parts are
Debian-only.

Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
 README | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/README b/README
index a4520c8..c883728 100644
--- a/README
+++ b/README
@@ -290,6 +290,7 @@ Standalone Mode
 To run osstest in standalone mode:
 
  - You need to install
+   Debian:
      sqlite3
      tcl8.5 tclx8.4 libsqlite3-tcl
      libdbi-perl libdbd-sqlite3-perl
@@ -300,6 +301,19 @@ To run osstest in standalone mode:
      libxml-libxml-perl
      dctrl-tools
      libnet-snmp-perl (if you are going to use Masterswitch PDUs)
+   Fedora:
+     sqlite
+     tcl tclx sqlite-tcl
+     perl-DBI perl-DBD-SQLite
+     spax rsync
+     curl
+     nmap-ncat
+     perl-libxml-perl
+     net-snmp-perl
+
+     Additionally:
+      - there is no dctrl-tools, so you need to set DebianNonfreeFirmware='' in config
+      - replace /usr/bin/tclsh8.5 shebang with /usr/bin/tclsh in scripts
 
  - Optional:
      ipmitool -- for hosts which use IPMI for power control
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

^ permalink raw reply related	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-12-04 21:40 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 4/7] Replace with-lock-ex with flock Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1 Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 6/7] Extend README for standalone mode Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 7/7] README: add Fedora setup section Marek Marczykowski-Górecki

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).