* [OSSTEST PATCH 0/7] osstest improvements
@ 2018-12-04 21:37 Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
` (6 more replies)
0 siblings, 7 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
Things I've learned and improved while setting osstest standalone myself.
First two patches add support for dnsmasq (note: I don't have setup to check if
haven't broken dhcp3 in the process!). Other patches are mostly independent.
Marek Marczykowski-Górecki (7):
DhcpWatch: extract dhcp3 parsing into separate function
DhcpWatch: add dnsmasq support
xen-install: use route with metric 0 to retrieve local network netmask
Replace with-lock-ex with flock
Optimize git clone - use --depth=1
Extend README for standalone mode
README: add Fedora setup section
Osstest/DhcpWatch/leases.pm | 169 +++++++++++++++++++++++--------------
Osstest/Executive.pm | 2 +-
Osstest/TestSupport.pm | 20 ++--
README | 45 ++++++++--
cr-daily-branch | 2 +-
cr-for-branches | 6 +-
cr-try-bisect | 2 +-
cri-args-hostlists | 4 +-
cri-getconfig | 2 +-
cri-lock-repos | 4 +-
crontab | 2 +-
crontab-cambridge | 2 +-
memoise | 2 +-
ts-freebsd-host-install | 2 +-
ts-xen-install | 2 +-
15 files changed, 175 insertions(+), 91 deletions(-)
base-commit: c65d7eb3f6c424d6c1fe69c5ecfca9c0b6cf4302
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply [flat|nested] 8+ messages in thread
* [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support Marek Marczykowski-Górecki
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
Preparation for dnsmasq support. Keep generic code in check_ip, but
extract format-specific handling into separat function. No intentional
behaviour change (besides slightly different warning reporting).
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Osstest/DhcpWatch/leases.pm | 131 ++++++++++++++++++++-----------------
1 file changed, 71 insertions(+), 60 deletions(-)
diff --git a/Osstest/DhcpWatch/leases.pm b/Osstest/DhcpWatch/leases.pm
index 98c22b9..5ec305f 100644
--- a/Osstest/DhcpWatch/leases.pm
+++ b/Osstest/DhcpWatch/leases.pm
@@ -48,68 +48,20 @@ sub new {
}, $class;
}
-sub check_ip ($$) {
- my ($mo, $gho) = @_;
-
- my $leases;
- my $leasesfn = $mo->{Source};
-
- if ($leasesfn =~ m,/,) {
- $leases= new IO::File $leasesfn, 'r';
- if (!defined $leases) { return "open $leasesfn: $!"; }
- } else {
- $leases= new IO::Socket::INET(PeerAddr => $leasesfn);
- if (!defined $leases) { return "connect to $leasesfn: $!"; }
- }
-
- my $lstash= "dhcpleases-$gho->{Guest}";
+sub parse_dhcp_dhcp3($$$$) {
+ my ($gho, $leases, $copy, $leasesfn) = @_;
my $inlease;
my $props;
my $best;
- my @warns;
-
- my $copy= new IO::File "$stash/$lstash.new", 'w';
- $copy or die "$lstash.new $!";
-
- my $saveas= sub {
- my ($fn,$keep) = @_;
-
- while (<$leases>) { print $copy $_ or die $!; }
- die $! unless $leases->eof;
-
- my $rename= sub {
- my ($src,$dst) = @_;
- rename "$stash/$src", "$stash/$dst"
- or $!==&ENOENT
- or die "rename $fn.$keep $!";
- };
- while (--$keep>0) {
- $rename->("$fn.$keep", "$fn.".($keep+1));
- }
- if ($keep>=0) {
- die if $keep;
- $rename->("$fn", "$fn.$keep");
- }
- $copy->close();
- rename "$stash/$lstash.new", "$stash/$fn" or die "$lstash.new $fn $!";
- logm("warning: $_") foreach grep { defined } @warns[0..5];
- # logm("$fn: rotated and stashed current leases");
- };
-
- my $badleases= sub {
- my ($m) = @_;
- $m= "$leasesfn:$.: unknown syntax";
- $saveas->("$lstash.bad", 7);
- return $m;
- };
while (<$leases>) {
print $copy $_ or die $!;
chomp; s/^\s+//; s/\s+$//;
next if m/^\#/; next unless m/\S/;
+
if (m/^lease\s+([0-9.]+)\s+\{$/) {
- return $badleases->("lease inside lease") if defined $inlease;
+ die "lease inside lease" if defined $inlease;
$inlease= $1;
$props= { };
next;
@@ -122,13 +74,13 @@ sub check_ip ($$) {
s/^( [-a-z0-9]+
) \s+//x
or
- return $badleases->("unknown syntax");
+ die "unknown syntax";
my $prop= $1;
- s/\s*\;$// or return $badleases->("missing semicolon");
+ s/\s*\;$// or die "missing semicolon";
$props->{$prop}= $_;
next;
}
- return $badleases->("end lease not inside lease")
+ die "end lease not inside lease"
unless defined $inlease;
$props->{' addr'}= $inlease;
@@ -144,25 +96,84 @@ sub check_ip ($$) {
my @missing= grep { !defined $props->{$_} }
('binding state', 'hardware ethernet', 'ends');
if (@missing) {
- push @warns, "$leasesfn:$.: lease without \`$_'"
+ warn "$leasesfn:$.: lease without \`$_'"
foreach @missing;
next;
}
- # ignore leases for other hosts
- next unless lc $props->{'hardware ethernet'} eq lc $gho->{Ether};
-
$props->{' ends'}= $props->{'ends'};
$props->{' ends'} =~
s/^[0-6]\s+(\S+)\s+(\d+)\:(\d+\:\d+)$/
sprintf "%s %02d:%s", $1,$2,$3 /e
- or return $badleases->("unexpected syntax for ends");
+ or die "unexpected syntax for ends";
+
+ # ignore leases for other hosts
+ next unless lc $props->{'hardware ethernet'} eq lc $gho->{Ether};
next if $best &&
$best->{' ends'} gt $props->{' ends'};
$best= $props;
}
+ return $best;
+}
+
+sub check_ip ($$) {
+ my ($mo, $gho) = @_;
+
+ my $leases;
+ my $leasesfn = $mo->{Source};
+
+ if ($leasesfn =~ m,/,) {
+ $leases= new IO::File $leasesfn, 'r';
+ if (!defined $leases) { return "open $leasesfn: $!"; }
+ } else {
+ $leases= new IO::Socket::INET(PeerAddr => $leasesfn);
+ if (!defined $leases) { return "connect to $leasesfn: $!"; }
+ }
+
+ my $lstash= "dhcpleases-$gho->{Guest}";
+ my $best;
+
+ my $copy= new IO::File "$stash/$lstash.new", 'w';
+ $copy or die "$lstash.new $!";
+
+ my $saveas= sub {
+ my ($fn,$keep) = @_;
+
+ while (<$leases>) { print $copy $_ or die $!; }
+ die $! unless $leases->eof;
+
+ my $rename= sub {
+ my ($src,$dst) = @_;
+ rename "$stash/$src", "$stash/$dst"
+ or $!==&ENOENT
+ or die "rename $fn.$keep $!";
+ };
+ while (--$keep>0) {
+ $rename->("$fn.$keep", "$fn.".($keep+1));
+ }
+ if ($keep>=0) {
+ die if $keep;
+ $rename->("$fn", "$fn.$keep");
+ }
+ $copy->close();
+ rename "$stash/$lstash.new", "$stash/$fn" or die "$lstash.new $fn $!";
+ # logm("$fn: rotated and stashed current leases");
+ };
+
+ my $badleases= sub {
+ my ($m) = @_;
+ $m= "$leasesfn:$.: unknown syntax " . $m;
+ $saveas->("$lstash.bad", 7);
+ return $m;
+ };
+
+ eval { $best = parse_dhcp_dhcp3($gho, $leases, $copy, $leasesfn) };
+ if ($@) {
+ return $badleases->($@);
+ }
+
if (!$best) {
$saveas->("$lstash.nolease", 3);
if ($leases->error) {
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask Marek Marczykowski-Górecki
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
dnsmasq use much simpler format - one lease by line with fields
separated with spaces. Add separate function to parse this format and
plug it info check_ip.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Osstest/DhcpWatch/leases.pm | 40 +++++++++++++++++++++++++++++++++++---
README | 2 +-
2 files changed, 38 insertions(+), 4 deletions(-)
diff --git a/Osstest/DhcpWatch/leases.pm b/Osstest/DhcpWatch/leases.pm
index 5ec305f..0c379e9 100644
--- a/Osstest/DhcpWatch/leases.pm
+++ b/Osstest/DhcpWatch/leases.pm
@@ -40,8 +40,8 @@ BEGIN {
sub new {
my ($class, $ho, $meth, $format, $source) = @_;
- die "'$format' is missing dhcp3 in it! (args: @_) ?"
- unless $format eq 'dhcp3';
+ die "'$format' is missing dhcp3/dnsmasq in it! (args: @_) ?"
+ unless $format eq 'dhcp3' or $format eq 'dnsmasq';
return bless {
Format => $format,
Source => $source,
@@ -118,6 +118,36 @@ sub parse_dhcp_dhcp3($$$$) {
return $best;
}
+sub parse_dhcp_dnsmasq($$$$) {
+ my ($gho, $leases, $copy, $leasesfn) = @_;
+
+ my $props;
+ my $best;
+
+ while (<$leases>) {
+ print $copy $_ or die $!;
+
+ chomp; s/^\s+//; s/\s+$//;
+ next if m/^\#/; next unless m/\S/;
+
+ my ($ends, $ether, $ip, $hostname, $client_id) = split / /;
+ $ends = strftime "%Y/%m/%d %T", localtime($ends);
+ $props = {
+ ' addr' => $ip,
+ ' ends' => $ends,
+ 'hardware ethernet' => $ether
+ };
+ # ignore leases for other hosts
+ next unless lc $props->{'hardware ethernet'} eq lc $gho->{Ether};
+
+ next if $best &&
+ $best->{' ends'} gt $props->{' ends'};
+ $best= $props;
+ }
+
+ return $best;
+}
+
sub check_ip ($$) {
my ($mo, $gho) = @_;
@@ -169,7 +199,11 @@ sub check_ip ($$) {
return $m;
};
- eval { $best = parse_dhcp_dhcp3($gho, $leases, $copy, $leasesfn) };
+ if ($mo->{Format} eq 'dnsmasq') {
+ eval { $best = parse_dhcp_dnsmasq($gho, $leases, $copy, $leasesfn) };
+ } elsif ($mo->{Format} eq 'dhcp3') {
+ eval { $best = parse_dhcp_dhcp3($gho, $leases, $copy, $leasesfn) };
+ }
if ($@) {
return $badleases->($@);
}
diff --git a/README b/README
index bb9dc18..4e4066a 100644
--- a/README
+++ b/README
@@ -508,7 +508,7 @@ HostFlags_<host> flag,^flag,^flag,flag...
HostProp_DhcpWatchMethod
leases <format> <source>
- where <format> is dhcp3
+ where <format> is dhcp3 or dnsmasq
<source> is filename (with slash) or <host>:<port>
AuthorizedKeysFiles
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 4/7] Replace with-lock-ex with flock Marek Marczykowski-Górecki
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
Otherwise it will match 169.254.0.0/16 route, which has always /16
netmask, instead of what really is used on the local network.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
ts-xen-install | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ts-xen-install b/ts-xen-install
index 8de94ac..2f0d263 100755
--- a/ts-xen-install
+++ b/ts-xen-install
@@ -349,7 +349,7 @@ END
my $routes= target_cmd_output_root($ho, "route -n");
- $routes =~ m/^ [0-9.]+ \s+ 0\.0\.0\.0 \s+ ([0-9.]+) \s+ \S*U\S* \s /mxi
+ $routes =~ m/^ [0-9.]+ \s+ 0\.0\.0\.0 \s+ ([0-9.]+) \s+ \S*U\S* \s+ 0 \s/mxi
or die "no own local network in route ? $routes ";
my $netmask= $1;
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OSSTEST PATCH 4/7] Replace with-lock-ex with flock
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
` (2 preceding siblings ...)
2018-12-04 21:37 ` [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1 Marek Marczykowski-Górecki
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
with-lock-ex comes from a Debian-only, not installed by default
chiark-utils-bin package, while flock have equivalent functionality and
is part of standard util-linux package.
This is especially important for apt-get call under the lock
(TestSupport.pm:target_run_pkgmanager_install function), which could be
used to install chiark-utils-bin but will fail because with-lock-ex is
not there.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
Osstest/Executive.pm | 2 +-
Osstest/TestSupport.pm | 2 +-
cr-daily-branch | 2 +-
cr-for-branches | 6 +++---
cr-try-bisect | 2 +-
cri-args-hostlists | 4 ++--
cri-getconfig | 2 +-
cri-lock-repos | 4 ++--
crontab | 2 +-
crontab-cambridge | 2 +-
memoise | 2 +-
ts-freebsd-host-install | 2 +-
12 files changed, 16 insertions(+), 16 deletions(-)
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 0be27b6..8a272e7 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -130,7 +130,7 @@ sub grabrepolock_reexec {
my $repos_locked= $ENV{OSSTEST_REPOS_LOCK_LOCKED};
unless (defined $repos_locked && $repos_locked eq $repos_lock) {
$ENV{OSSTEST_REPOS_LOCK_LOCKED}= $repos_lock;
- exec "with-lock-ex","-w",$repos_lock, $0,@org_argv;
+ exec "flock","-w",$repos_lock, $0,@org_argv;
die $!;
}
}
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 2910842..9515c9b 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -535,7 +535,7 @@ sub target_run_pkgmanager_install ($$;$$) {
push @cmd, qw(lockf /var/run/osstest-pkg-lock pkg-static install);
} else {
push @cmd, qw(DEBIAN_PRIORITY=critical UCF_FORCE_CONFFOLD=y
- with-lock-ex -w /var/lock/osstest-apt apt-get);
+ flock /var/lock/osstest-apt apt-get);
push @cmd, qw(-f) if $force;
push @cmd, qw(--no-install-recommends) if $norec;
push @cmd, qw(-y install);
diff --git a/cr-daily-branch b/cr-daily-branch
index e1528d4..492eff0 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -475,7 +475,7 @@ if grep -xF "$NEW_REVISION" $branch.force-rev; then push=$OSSTEST_PUSH; fi
if test -f $branch.block; then push=false; fi
if test -e $mrof && test -e $tree_bisect && ! grep '^broken' $mrof; then
- with-lock-ex -w $tree_bisect/$mrof.lock bash -xec "
+ flock $tree_bisect/$mrof.lock bash -xec "
rm -f $tree_bisect/$mrof.in.new
cp $mrof $tree_bisect/$mrof.in.new
cd $tree_bisect
diff --git a/cr-for-branches b/cr-for-branches
index 2e9717e..93a713a 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -1,6 +1,6 @@
#!/bin/bash
# usage: cr-for-branches BRANCHESDIR WLEM "SCRIPT OPTIONS" ARGS...
-# will run cd BRANCHESDIR && with-lock-ex WLEM SCRIPT OPTIONS BRANCH ARGS...
+# will run cd BRANCHESDIR && flock WLEM SCRIPT OPTIONS BRANCH ARGS...
# This is part of "osstest", an automated testing framework for Xen.
# Copyright (C) 2009-2013 Citrix Inc.
@@ -39,7 +39,7 @@ if [ "x$fetchwlem" = x-q ]; then
fetchwlem=-f
fi
-with-lock-ex $fetchwlem data-tree-lock bash -ec '
+flock $fetchwlem data-tree-lock bash -ec '
exec >>$LOGFILE
date
printf "%s\n" "$FOR_LOGFILE"
@@ -68,7 +68,7 @@ for branch in $BRANCHES; do
export LOCK_ACQU_START=`date +%s`
- with-lock-ex $wlem data-tree-lock bash -ec '
+ flock $wlem data-tree-lock bash -ec '
m="$*"
mkdir -p tmp
diff --git a/cr-try-bisect b/cr-try-bisect
index d613c34..afe4b86 100755
--- a/cr-try-bisect
+++ b/cr-try-bisect
@@ -31,7 +31,7 @@ startstamp=tmp/bisection-start-stamp
anyflagfile=tmp/bisected-any.$branch
-with-lock-ex -w $mrof.lock bash -xec "
+flock $mrof.lock bash -xec "
if test -e $mrof.in; then
if test -e $mrof; then mv $mrof $mrof.old; fi
rm -f $anyflagfile
diff --git a/cri-args-hostlists b/cri-args-hostlists
index 7d23087..a502ac1 100644
--- a/cri-args-hostlists
+++ b/cri-args-hostlists
@@ -114,7 +114,7 @@ start_email () {
globallockdir=`getconfig GlobalLockDir`
- with-lock-ex -w $globallockdir/report-lock \
+ flock $globallockdir/report-lock \
./sg-report-job-history --html-dir=$job_html_dir --flight=$flight
./sg-report-flight --html-dir=$flight_html_dir/$flight/ \
@@ -122,7 +122,7 @@ start_email () {
$sgr_args $flight
mkdir -p $host_html_dir
- with-lock-ex -w $globallockdir/report-lock \
+ flock $globallockdir/report-lock \
./sg-report-host-history --html-dir=$host_html_dir flight:$flight
}
diff --git a/cri-getconfig b/cri-getconfig
index f8397c1..928f140 100644
--- a/cri-getconfig
+++ b/cri-getconfig
@@ -57,7 +57,7 @@ get_psql_cmd () {
# and set OSSTEST_PSQL_ONLY_DO to an integer
if [ "x$OSSTEST_PSQL_ONLY_DO" != x ]; then
local f=t.psql-counter
- psql_counter=$( with-lock-ex -w $f.lock bash -ec '
+ psql_counter=$( flock $f.lock bash -ec '
psql_counter=$(cat '$f' || echo 0)
echo $(( $psql_counter + 1 )) >'$f'.tmp
mv -f '$f'.tmp '$f'
diff --git a/cri-lock-repos b/cri-lock-repos
index e7e0fa8..b3500f9 100644
--- a/cri-lock-repos
+++ b/cri-lock-repos
@@ -26,8 +26,8 @@ if [ "x$AP_FETCH_PLACEHOLDERS" = xy ] && ! [ -e $repos ]; then
echo "AP_FETCH_PLACEHOLDERS, not locking" >&2
elif [ "x$OSSTEST_REPOS_LOCK_LOCKED" != "x$repos_lock" ]; then
OSSTEST_REPOS_LOCK_LOCKED="$repos_lock" \
- exec with-lock-ex -w "$repos_lock" \
+ exec flock "$repos_lock" \
"$0" "$@"
- echo >&2 "arrgh, exec with-lock-ex failed $?"
+ echo >&2 "arrgh, exec flock failed $?"
exit 1
fi
diff --git a/crontab b/crontab
index e1e798a..9c2489f 100755
--- a/crontab
+++ b/crontab
@@ -16,6 +16,6 @@ MAILTO=osstest-admin@xenproject.org
34 15 23 * * cd testing.git && BRANCHES=examine ./cr-for-branches branches -w "./cr-daily-branch --real"
18 4 * * * cd testing.git && BRANCHES='linux-3.0 libvirt rumprun' ./cr-for-branches branches -w "./cr-daily-branch --real"
6-59/15 * * * * cd testing.git && EXTRA_BRANCHES='xen-unstable-smoke linux-3.0 rumprun libvirt freebsd-master' ./cr-for-branches bisects -w "./cr-try-bisect --real"
-#8-59/5 * * * * cd bisects/adhoc.git && with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
+#8-59/5 * * * * cd bisects/adhoc.git && flock -n -E 0 data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
22 8 * * * cd testing.git && BRANCHES=maintjobs ./cr-for-branches . -w ./cr-all-branch-statuses ''
3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
diff --git a/crontab-cambridge b/crontab-cambridge
index 86f069a..8c0d346 100755
--- a/crontab-cambridge
+++ b/crontab-cambridge
@@ -21,6 +21,6 @@ MAILTO=ian.jackson@citrix.com
34 15 25 * * cd testing.git && BRANCHES=examine ./cr-for-branches branches -w "./cr-daily-branch --real"
-#8-59/5 * * * * cd bisects/adhoc.git && with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
+#8-59/5 * * * * cd bisects/adhoc.git && flock -n -E 0 data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
22 8 * * * cd testing.git && BRANCHES=maintjobs ./cr-for-branches . -w ./cr-all-branch-statuses ''
3 4 * * * savelog -c28 testing.git/tmp/cr-for-branches.log >/dev/null
diff --git a/memoise b/memoise
index 73f15e6..9a6439d 100755
--- a/memoise
+++ b/memoise
@@ -43,7 +43,7 @@ id=${id% -}
f="$datadir/$id"
if ! [ -f "$f.o" ]; then
- with-lock-ex -w "$f.l" sh -ec '
+ flock "$f.l" sh -ec '
f=$1; shift
if [ -f "$f.o" ]; then exit 0; fi
exec </dev/null >"$f.t"
diff --git a/ts-freebsd-host-install b/ts-freebsd-host-install
index 53daeef..ee1cede 100755
--- a/ts-freebsd-host-install
+++ b/ts-freebsd-host-install
@@ -105,7 +105,7 @@ ln $sharedpath $targetpath
# Prune old images not used anymore
find `dirname $sharedpath` -links 1 -ctime +7 -delete
END
- my @cmd = ( "with-lock-ex", "-w", "$tftp_freebsd/lock",
+ my @cmd = ( "flock", "$tftp_freebsd/lock",
"bash", "-exc", "$script", "x",
"$tftp_freebsd", "$image", "by-hash/$hash.img",
"$ho->{Tftp}{Path}/$pxeimg" );
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
` (3 preceding siblings ...)
2018-12-04 21:37 ` [OSSTEST PATCH 4/7] Replace with-lock-ex with flock Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 6/7] Extend README for standalone mode Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 7/7] README: add Fedora setup section Marek Marczykowski-Górecki
6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
Do not clone the whole history to save time, disk space and bandwidth.
This reduce (on my machine) Linux clone from over half an hour to few
minutes.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
This should be fine, unless this function is also used when preparing
for bisection. Is it?
---
Osstest/TestSupport.pm | 18 +++++++++++-------
1 file changed, 11 insertions(+), 7 deletions(-)
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 9515c9b..e51ffc7 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -1492,15 +1492,19 @@ END
END
} elsif ($vcs eq 'git') {
- target_cmd_build($ho, $timeout, $builddir, <<END.
- $rm
- git clone '$tree' $subdir
- cd $subdir
+ if (length($r{"revision_$which"})) {
+ target_cmd_build($ho, $timeout, $builddir, <<END);
+ $rm
+ git clone --depth=1 -b '$r{"revision_$which"}' '$tree' $subdir
+ cd $subdir
END
- (length($r{"revision_$which"}) ? <<END : ''));
- git checkout '$r{"revision_$which"}'
- git clean -xdf
+ } else {
+ target_cmd_build($ho, $timeout, $builddir, <<END);
+ $rm
+ git clone --depth=1 '$tree' $subdir
+ cd $subdir
END
+ }
} else {
die "$vcs $which $tree ?";
}
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OSSTEST PATCH 6/7] Extend README for standalone mode
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
` (4 preceding siblings ...)
2018-12-04 21:37 ` [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1 Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 7/7] README: add Fedora setup section Marek Marczykowski-Górecki
6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
Things I've learned when setting it up myself.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
README | 29 +++++++++++++++++++++++++----
1 file changed, 25 insertions(+), 4 deletions(-)
diff --git a/README b/README
index 4e4066a..a4520c8 100644
--- a/README
+++ b/README
@@ -305,6 +305,12 @@ To run osstest in standalone mode:
ipmitool -- for hosts which use IPMI for power control
grub-common -- for mg-netgrub-loader-update
+ - Setup TFTP server (see 'TftpPath' setting below), copy pxelinux
+ files there (pxelinux.0, ldlinux.c32, libcom32.c32), setup your
+ DHCP server to point PXE clients at it
+
+ - Setup HTTP server (see 'Webspace*' settings below)
+
- Write a config file
~/.xen-osstest/config
See below.
@@ -338,9 +344,16 @@ To run osstest in standalone mode:
However, as test-amd64-{i386,amd64}-xl and other tests depends on
some runtime variables generated by build-* jobs, you can run
- build-* jobs before running actual test jobs. If you don't want to do
- so you need to insert those missing runvars into standalone.db with
- sqlite3.
+ build-* jobs before running actual test jobs. The error message will
+ look like this:
+
+ 2018-12-04 20:56:52 Z checking builds ...
+ 2018-12-04 20:56:52 Z checking buildjob path_dist
+ need path_dist in build-i386 at Osstest/TestSupport.pm line 327.
+ ^^^^^^^^^^ name of build job to run
+
+ If you don't want to do so you need to insert those missing runvars
+ into standalone.db with sqlite3.
- Don't forget to set the machine to netboot in the BIOS.
@@ -407,6 +420,10 @@ HostProp_<testbox>_Ether
MAC address of the box <testbox>. Only needed if you want
to use the osstest host and Xen installer.
+HostProp_<testbox>_IpAddr
+ IP address of the box <testbox>. Only needed if
+ <testbox>.<DnsDomain> does not resolve to the correct IP.
+
HostProp_<testbox>_Build_Make_Flags
Extra flags to pass to "make". e.g. "-j<SOMETHING>"
@@ -511,6 +528,10 @@ HostProp_DhcpWatchMethod
where <format> is dhcp3 or dnsmasq
<source> is filename (with slash) or <host>:<port>
+ When using <host>:<port> syntax, connecting to it should return the
+ whole dhcp leases file. For example use socat on the DHCP server host:
+ socat -U TCP-LISTEN:5556,fork,reuseaddr EXEC:'cat /var/lib/dhcpd/dhcpd.leases'
+
AuthorizedKeysFiles
:-separated list of files to concatenate into authorized_keys
AuthorizedKeysAppend
@@ -520,7 +541,7 @@ The keys in ~/.ssh/id_{rsa,dsa}.pub and ~/.ssh/authorized_keys
TestHostKeypairPath
-Platforms<Arch>
+Platforms <Arch>
List of platforms (i.e. distinct host types) to run a basic test on.
HostProp_GenEtherPrefixBase 5e:36:0e:f5
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
* [OSSTEST PATCH 7/7] README: add Fedora setup section
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
` (5 preceding siblings ...)
2018-12-04 21:37 ` [OSSTEST PATCH 6/7] Extend README for standalone mode Marek Marczykowski-Górecki
@ 2018-12-04 21:37 ` Marek Marczykowski-Górecki
6 siblings, 0 replies; 8+ messages in thread
From: Marek Marczykowski-Górecki @ 2018-12-04 21:37 UTC (permalink / raw)
To: xen-devel; +Cc: Marek Marczykowski-Górecki
Package names are slightly different. Additionally some parts are
Debian-only.
Signed-off-by: Marek Marczykowski-Górecki <marmarek@invisiblethingslab.com>
---
README | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/README b/README
index a4520c8..c883728 100644
--- a/README
+++ b/README
@@ -290,6 +290,7 @@ Standalone Mode
To run osstest in standalone mode:
- You need to install
+ Debian:
sqlite3
tcl8.5 tclx8.4 libsqlite3-tcl
libdbi-perl libdbd-sqlite3-perl
@@ -300,6 +301,19 @@ To run osstest in standalone mode:
libxml-libxml-perl
dctrl-tools
libnet-snmp-perl (if you are going to use Masterswitch PDUs)
+ Fedora:
+ sqlite
+ tcl tclx sqlite-tcl
+ perl-DBI perl-DBD-SQLite
+ spax rsync
+ curl
+ nmap-ncat
+ perl-libxml-perl
+ net-snmp-perl
+
+ Additionally:
+ - there is no dctrl-tools, so you need to set DebianNonfreeFirmware='' in config
+ - replace /usr/bin/tclsh8.5 shebang with /usr/bin/tclsh in scripts
- Optional:
ipmitool -- for hosts which use IPMI for power control
--
git-series 0.9.1
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-12-04 21:40 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-04 21:37 [OSSTEST PATCH 0/7] osstest improvements Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 1/7] DhcpWatch: extract dhcp3 parsing into separate function Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 2/7] DhcpWatch: add dnsmasq support Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 3/7] xen-install: use route with metric 0 to retrieve local network netmask Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 4/7] Replace with-lock-ex with flock Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 5/7] Optimize git clone - use --depth=1 Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 6/7] Extend README for standalone mode Marek Marczykowski-Górecki
2018-12-04 21:37 ` [OSSTEST PATCH 7/7] README: add Fedora setup section Marek Marczykowski-Górecki
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).