[Xen-devel] [OSSTEST PATCH v2 0/8] mg-repro-* improvements

[Xen-devel] [OSSTEST PATCH v2 0/8] mg-repro-* improvements

[Ian Jackson]

This fixes bugs I found trying to use the last version of this series.
This version has been used successfully.

Ian Jackson (8):
  mg-transient-task: New utility
  mg-transient-task: Put the ownd fd on a high fd, say, 114
  mg-repro-setup: Do all builds in their own tasks, regardless
  mg-repro-setup: Do not imply that a --rebuild needs a +TREE
  mg-repro-setup: Introduce `statictask' variable
  mg-repro-setup: New mode --autoalloc
  mg-repro-setup: actually accept --rebuild -rVAR=VALUE
  mg-repro-setup: Do not wrongly reject multiple --rebuild

 cs-adjust-flight        |  2 +-
 mg-repro-setup          | 53 ++++++++++++++++++++++++++++++++++---------------
 mg-transient-task       | 35 ++++++++++++++++++++++++++++++++
 tcl/JobDB-Executive.tcl |  1 +
 4 files changed, 74 insertions(+), 17 deletions(-)
 create mode 100755 mg-transient-task

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 1/8] mg-transient-task: New utility

[Ian Jackson]

Will be useful in combination with new mode(s) of mg-repro-setup.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-transient-task | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100755 mg-transient-task

diff --git a/mg-transient-task b/mg-transient-task
new file mode 100755
index 00000000..2b3b315e
--- /dev/null
+++ b/mg-transient-task
@@ -0,0 +1,32 @@
+#!/usr/bin/tclsh8.5
+# -*- Tcl -*- 
+# usage: ./mg-transient-task PROGRAM [ARGS...]
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2019 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package require Tclx
+
+source ./tcl/daemonlib.tcl
+
+jobdb::become-task [string range "$argv" 0 50]
+exec 2>@ stderr perl -I. -e {
+        use Osstest;
+        use Osstest::Executive;
+        csreadconfig();
+        findtask()
+}
+execl [lindex $argv 0] [lrange $argv 1 end]
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 2/8] mg-transient-task: Put the ownd fd on a high fd, say, 114

[Ian Jackson]

This avoids clashes with other shell scripts' etc. fds.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch
---
 mg-transient-task       | 5 ++++-
 tcl/JobDB-Executive.tcl | 1 +
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/mg-transient-task b/mg-transient-task
index 2b3b315e..ce5180ff 100755
--- a/mg-transient-task
+++ b/mg-transient-task
@@ -22,7 +22,10 @@ package require Tclx
 
 source ./tcl/daemonlib.tcl
 
-jobdb::become-task [string range "$argv" 0 50]
+set ownerqueue [jobdb::become-task [string range "$argv" 0 50]]
+dup $ownerqueue file114
+close $ownerqueue
+
 exec 2>@ stderr perl -I. -e {
         use Osstest;
         use Osstest::Executive;
diff --git a/tcl/JobDB-Executive.tcl b/tcl/JobDB-Executive.tcl
index 4f3409a9..56b61825 100644
--- a/tcl/JobDB-Executive.tcl
+++ b/tcl/JobDB-Executive.tcl
@@ -435,6 +435,7 @@ proc become-task {comment} {
                AND refkey = [pg_quote [lindex $refinfo 1]]
         "
     }
+    return $ownerqueue
 }
 
 proc preserve-task {seconds} {
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 3/8] mg-repro-setup: Do all builds in their own tasks, regardless

[Ian Jackson]

In case OSSTEST_TASK was set by the caller, unset it.  Unsetting it
will cause each sg-run-job (inside mg-execute-task) to become its own
task.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index dc6c5cbb..5a52e617 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -247,7 +247,8 @@ progress "new flight is $flight"
 
 if [ "x$rebuilds_flight" != x ]; then
 	progress "running build(s) $rebuilds_flight"
-	./mg-execute-flight -B$rebuilds_blessing --progress-fd=2 \
+	env -u OSSTEST_TASK \
+                ./mg-execute-flight -B$rebuilds_blessing --progress-fd=2 \
 		"${mgexecflags[@]}" $rebuilds_flight
 
 	mro=tmp/$rebuilds_flight.mro
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 4/8] mg-repro-setup: Do not imply that a --rebuild needs a +TREE

[Ian Jackson]

The +TREEs and other specifications are convolved.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index 5a52e617..9a81c565 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -45,8 +45,7 @@ usage () { cat <<END
 
  REBUILD is
    --rebuild [-B<blessing>] [-r...]
-             [.BUILDJOB | =BUILDJOBRUNVAR...]
-             +TREE=URL#REVISION...
+             [.BUILDJOB | =BUILDJOBRUNVAR | +TREE=URL#REVISION ...]
        Also use a different build.  Specifically, use URL and REVISION
        for certain builds which mention TREE and which are referenced 
        (directly) in the job JOB in EXAMPLE-FLIGHT.  The affected builds
@@ -55,7 +54,7 @@ usage () { cat <<END
        name is any of the BUILDJOB (in which case all references to that
        job are adjusted).  If no .BUILDJOB and no =BUILDJOBRUNVAR are
        specified, then all jobs referred to from the example JOB
-       which mention TREE are affected.
+       which mention a TREE are affected.
 
        For example,
            --rebuild =xenbuildjob \
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 5/8] mg-repro-setup: Introduce `statictask' variable

[Ian Jackson]

We are going to make a mode where we don't set OSSTEST_TASK.  The
result is that our subprocesses will do whatever they usually do.

Those are mg-allocate (which would allocate for our static task) and
mg-execute-flight which will make a dynamic task.  We must therefore
prevent mg-allocate from running since the allocations would not be
useable for the flight execution.

No functional change yet, since nothing sets statictask=false and
therefore OSSTEST_TASK would always be set.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index 9a81c565..45c56f6a 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -102,6 +102,7 @@ duration=28d
 blessing=play
 skipcapture=true
 autoalloc=false
+statictask=true
 
 while true; do
 	case "$1" in
@@ -116,7 +117,7 @@ while true; do
 	-t?*)		duration=${arg#-t}		;;
 	--rogue)	duration=''			;;
 	--capture)	skipcapture=false		;;
-	--autoalloc-nofree) autoalloc=true		;;
+	--autoalloc-nofree) autoalloc=true; statictask=true ;;
 	-l*)		logfile=${arg#-l}		;;
 	-r^*|-r!*)	adjustsets+=("${arg#-r}")	;;
 	# ^ two patterns because there's no way in bash to write
@@ -257,16 +258,25 @@ if [ "x$rebuilds_flight" != x ]; then
 	fi
 fi
 
-OSSTEST_TASK=$(perl -e '
-	use Osstest;
-	use Osstest::Executive;
-	csreadconfig();
-	findtask();
-	printf "%s\n", $ENV{'OSSTEST_TASK'} or die $!;
-')
-export OSSTEST_TASK
+if $statictask; then
+        OSSTEST_TASK=$(perl -e '
+        	use Osstest;
+        	use Osstest::Executive;
+        	csreadconfig();
+        	findtask();
+        	printf "%s\n", $ENV{'OSSTEST_TASK'} or die $!;
+        ')
+        export OSSTEST_TASK
+fi
 
 if [ "${alloc_idents[*]}" ]; then
+        if [ x"$OSSTEST_TASK" = x ]; then
+                # We would have to make a dynamic task and hold the
+                #  fd for it in this script.  Would be quite awkward.
+                echo >&2 'cannot hand-allocate in a way that will free'
+                exit 1
+        fi
+
 	progress "allocating ${alloc_idents[*]} ..."
 	alloc_output=tmp/$flight.allocations
 	./mg-allocate \
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 6/8] mg-repro-setup: New mode --autoalloc

[Ian Jackson]

This just involves turning autoalloc on and statictask off.
It is most useful with mg-transient-task, as documented.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index 45c56f6a..b55f021e 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -28,7 +28,7 @@ usage () { cat <<END
  HOSTSPEC is
    [<ident>=][reuse:]<hostname>    host must be allocated, skip host-install
    [<ident>=]wipe:<hostname>       host must be allocated, wipe it
-   [<ident>=]alloc:<resource-spec> \`mg-allocate <resource-spec>', and wipe
+   [<ident>=]alloc:<resource-spec> \`mg-allocate <resource-spec>', and wipe[*]
    none:                           no hosts (should be only HOSTPSEC)
    (default <ident> is \`host')
 
@@ -42,6 +42,13 @@ usage () { cat <<END
    -E... -f... -P       as for mg-execute-flight
    --autoalloc-nofree   allocate hosts as for production, but keep them
                          (specify no HOSTSPECS; remember to deallocate later)
+   --autoalloc          use hosts as for production (specify no HOSTSPECS)
+                         and free them afterwards[*]
+
+ [*] If \$OSSTEST_TASK is set on entry, it will be used for these
+     allocations.  So, in particular,
+        ./mg-transient-task ./mg-repro-setup .... alloc:...
+     will allocate as specified, but discard the hosts when done.
 
  REBUILD is
    --rebuild [-B<blessing>] [-r...]
@@ -104,6 +111,8 @@ skipcapture=true
 autoalloc=false
 statictask=true
 
+if [ "x$OSSTEST_TASK" != x ]; then statictask=false; fi
+
 while true; do
 	case "$1" in
 	-*)			;;
@@ -117,6 +126,7 @@ while true; do
 	-t?*)		duration=${arg#-t}		;;
 	--rogue)	duration=''			;;
 	--capture)	skipcapture=false		;;
+ 	--autoalloc)        autoalloc=true; statictask=false ;;
 	--autoalloc-nofree) autoalloc=true; statictask=true ;;
 	-l*)		logfile=${arg#-l}		;;
 	-r^*|-r!*)	adjustsets+=("${arg#-r}")	;;
@@ -297,7 +307,7 @@ if [ "${alloc_idents[*]}" ]; then
 	progressf "\n"
 	if $autoalloc; then
 		echo >&2 \
- 'warning: --autoalloc-nofree and HOSTSPECs are do not work well together'
+ 'warning: --autoalloc[-nofree] and HOSTSPECs are do not work well together'
 	fi
 fi
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 7/8] mg-repro-setup: actually accept --rebuild -rVAR=VALUE

[Ian Jackson]

The glob syntax here was wrong, and the code cs-adjust-flight did not
handle it properly either.  So --rebuild -r has not worked since it
first appeared in:
   a1e0e5846f7bb7d82a5db1d7cd643b9f5ca1b9a9
   mg-repro-flight: Provide --rebuild to make variant build jobs

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch
---
 cs-adjust-flight | 2 +-
 mg-repro-setup   | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/cs-adjust-flight b/cs-adjust-flight
index 5be5af38..ae342506 100755
--- a/cs-adjust-flight
+++ b/cs-adjust-flight
@@ -485,7 +485,7 @@ sub change__repro_buildjobs {
     my @refspecs;
     my @varspecs;
     my %treespecs;
-    while (@changes && $changes[0] =~ m/^[.=+]/) {
+    while (@changes && $changes[0] =~ m/^[.=+]|^-r/) {
 	local $_ = shift @changes;
 	if (m/^[.=]/) {
 	    push @refspecs, $_;
diff --git a/mg-repro-setup b/mg-repro-setup
index b55f021e..3ceb7032 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -187,7 +187,7 @@ while [ $# -ne 0 ]; do
 		rebuild_specs=()
 		while true; do
 			case "$1" in
-			[.=+]*|-r) rebuild_specs+=("$1");       shift ;;
+			[.=+]*|-r*) rebuild_specs+=("$1");      shift ;;
 		        -B?*)	rebuilds_blessing=${1#-B};      shift ;;
 			-*)	badusage ': bad --rebuild option'     ;;
                         *)      break                                 ;;
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH v2 8/8] mg-repro-setup: Do not wrongly reject multiple --rebuild

[Ian Jackson]

--rebuild ends the current --rebuild specification.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: New patch
---
 mg-repro-setup | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mg-repro-setup b/mg-repro-setup
index 3ceb7032..374176f0 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -189,6 +189,7 @@ while [ $# -ne 0 ]; do
 			case "$1" in
 			[.=+]*|-r*) rebuild_specs+=("$1");      shift ;;
 		        -B?*)	rebuilds_blessing=${1#-B};      shift ;;
+			--rebuild) break                              ;;
 			-*)	badusage ': bad --rebuild option'     ;;
                         *)      break                                 ;;
 			esac
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel