[OSSTEST PATCH 0/4] mg-*: easier transient autoalloc repro

[OSSTEST PATCH 0/4] mg-*: easier transient autoalloc repro

[Ian Jackson]

Ian Jackson (4):
  mg-repro-setup: Do all builds in their own tasks, regardless
  mg-transient-task: New utility
  mg-repro-setup: Introduce `statictask' variable
  mg-repro-setup: New mode --autoalloc

 mg-repro-setup    | 45 +++++++++++++++++++++++++++++++++------------
 mg-transient-task | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 12 deletions(-)
 create mode 100755 mg-transient-task

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 0/4] mg-*: easier transient autoalloc repro

[Ian Jackson]

Ian Jackson (4):
  mg-repro-setup: Do all builds in their own tasks, regardless
  mg-transient-task: New utility
  mg-repro-setup: Introduce `statictask' variable
  mg-repro-setup: New mode --autoalloc

 mg-repro-setup    | 45 +++++++++++++++++++++++++++++++++------------
 mg-transient-task | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 12 deletions(-)
 create mode 100755 mg-transient-task

-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 1/4] mg-repro-setup: Do all builds in their own tasks, regardless

[Ian Jackson]

In case OSSTEST_TASK was set by the caller, unset it.  Unsetting it
will cause each sg-run-job (inside mg-execute-task) to become its own
task.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index dc6c5cbb..5a52e617 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -247,7 +247,8 @@ progress "new flight is $flight"
 
 if [ "x$rebuilds_flight" != x ]; then
 	progress "running build(s) $rebuilds_flight"
-	./mg-execute-flight -B$rebuilds_blessing --progress-fd=2 \
+	env -u OSSTEST_TASK \
+                ./mg-execute-flight -B$rebuilds_blessing --progress-fd=2 \
 		"${mgexecflags[@]}" $rebuilds_flight
 
 	mro=tmp/$rebuilds_flight.mro
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 1/4] mg-repro-setup: Do all builds in their own tasks, regardless

[Ian Jackson]

In case OSSTEST_TASK was set by the caller, unset it.  Unsetting it
will cause each sg-run-job (inside mg-execute-task) to become its own
task.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index dc6c5cbb..5a52e617 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -247,7 +247,8 @@ progress "new flight is $flight"
 
 if [ "x$rebuilds_flight" != x ]; then
 	progress "running build(s) $rebuilds_flight"
-	./mg-execute-flight -B$rebuilds_blessing --progress-fd=2 \
+	env -u OSSTEST_TASK \
+                ./mg-execute-flight -B$rebuilds_blessing --progress-fd=2 \
 		"${mgexecflags[@]}" $rebuilds_flight
 
 	mro=tmp/$rebuilds_flight.mro
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 2/4] mg-transient-task: New utility

[Ian Jackson]

Will be useful in combination with new mode(s) of mg-repro-setup.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-transient-task | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100755 mg-transient-task

diff --git a/mg-transient-task b/mg-transient-task
new file mode 100755
index 00000000..2b3b315e
--- /dev/null
+++ b/mg-transient-task
@@ -0,0 +1,32 @@
+#!/usr/bin/tclsh8.5
+# -*- Tcl -*- 
+# usage: ./mg-transient-task PROGRAM [ARGS...]
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2019 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package require Tclx
+
+source ./tcl/daemonlib.tcl
+
+jobdb::become-task [string range "$argv" 0 50]
+exec 2>@ stderr perl -I. -e {
+        use Osstest;
+        use Osstest::Executive;
+        csreadconfig();
+        findtask()
+}
+execl [lindex $argv 0] [lrange $argv 1 end]
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 2/4] mg-transient-task: New utility

[Ian Jackson]

Will be useful in combination with new mode(s) of mg-repro-setup.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-transient-task | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100755 mg-transient-task

diff --git a/mg-transient-task b/mg-transient-task
new file mode 100755
index 00000000..2b3b315e
--- /dev/null
+++ b/mg-transient-task
@@ -0,0 +1,32 @@
+#!/usr/bin/tclsh8.5
+# -*- Tcl -*- 
+# usage: ./mg-transient-task PROGRAM [ARGS...]
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2019 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+package require Tclx
+
+source ./tcl/daemonlib.tcl
+
+jobdb::become-task [string range "$argv" 0 50]
+exec 2>@ stderr perl -I. -e {
+        use Osstest;
+        use Osstest::Executive;
+        csreadconfig();
+        findtask()
+}
+execl [lindex $argv 0] [lrange $argv 1 end]
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 3/4] mg-repro-setup: Introduce `statictask' variable

[Ian Jackson]

We are going to make a mode where we don't set OSSTEST_TASK.  The
result is that our subprocesses will do whatever they usually do.

Those are mg-allocate (which would allocate for our static task) and
mg-execute-flight which will make a dynamic task.  We must therefore
prevent mg-allocate from running since the allocations would not be
useable for the flight execution.

No functional change yet, since nothing sets statictask=false and
therefore OSSTEST_TASK would always be set.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index 5a52e617..b60389d8 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -103,6 +103,7 @@ duration=28d
 blessing=play
 skipcapture=true
 autoalloc=false
+statictask=true
 
 while true; do
 	case "$1" in
@@ -117,7 +118,7 @@ while true; do
 	-t?*)		duration=${arg#-t}		;;
 	--rogue)	duration=''			;;
 	--capture)	skipcapture=false		;;
-	--autoalloc-nofree) autoalloc=true		;;
+	--autoalloc-nofree) autoalloc=true; statictask=true ;;
 	-l*)		logfile=${arg#-l}		;;
 	-r^*|-r!*)	adjustsets+=("${arg#-r}")	;;
 	# ^ two patterns because there's no way in bash to write
@@ -258,16 +259,25 @@ if [ "x$rebuilds_flight" != x ]; then
 	fi
 fi
 
-OSSTEST_TASK=$(perl -e '
-	use Osstest;
-	use Osstest::Executive;
-	csreadconfig();
-	findtask();
-	printf "%s\n", $ENV{'OSSTEST_TASK'} or die $!;
-')
-export OSSTEST_TASK
+if $statictask; then
+        OSSTEST_TASK=$(perl -e '
+        	use Osstest;
+        	use Osstest::Executive;
+        	csreadconfig();
+        	findtask();
+        	printf "%s\n", $ENV{'OSSTEST_TASK'} or die $!;
+        ')
+        export OSSTEST_TASK
+fi
 
 if [ "${alloc_idents[*]}" ]; then
+        if [ x"$OSSTEST_TASK" = x ]; then
+                # We would have to make a dynamic task and hold the
+                #  fd for it in this script.  Would be quite awkward.
+                echo >&2 'cannot hand-allocate in a way that will free'
+                exit 1
+        fi
+
 	progress "allocating ${alloc_idents[*]} ..."
 	alloc_output=tmp/$flight.allocations
 	./mg-allocate \
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 3/4] mg-repro-setup: Introduce `statictask' variable

[Ian Jackson]

We are going to make a mode where we don't set OSSTEST_TASK.  The
result is that our subprocesses will do whatever they usually do.

Those are mg-allocate (which would allocate for our static task) and
mg-execute-flight which will make a dynamic task.  We must therefore
prevent mg-allocate from running since the allocations would not be
useable for the flight execution.

No functional change yet, since nothing sets statictask=false and
therefore OSSTEST_TASK would always be set.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 28 +++++++++++++++++++---------
 1 file changed, 19 insertions(+), 9 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index 5a52e617..b60389d8 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -103,6 +103,7 @@ duration=28d
 blessing=play
 skipcapture=true
 autoalloc=false
+statictask=true
 
 while true; do
 	case "$1" in
@@ -117,7 +118,7 @@ while true; do
 	-t?*)		duration=${arg#-t}		;;
 	--rogue)	duration=''			;;
 	--capture)	skipcapture=false		;;
-	--autoalloc-nofree) autoalloc=true		;;
+	--autoalloc-nofree) autoalloc=true; statictask=true ;;
 	-l*)		logfile=${arg#-l}		;;
 	-r^*|-r!*)	adjustsets+=("${arg#-r}")	;;
 	# ^ two patterns because there's no way in bash to write
@@ -258,16 +259,25 @@ if [ "x$rebuilds_flight" != x ]; then
 	fi
 fi
 
-OSSTEST_TASK=$(perl -e '
-	use Osstest;
-	use Osstest::Executive;
-	csreadconfig();
-	findtask();
-	printf "%s\n", $ENV{'OSSTEST_TASK'} or die $!;
-')
-export OSSTEST_TASK
+if $statictask; then
+        OSSTEST_TASK=$(perl -e '
+        	use Osstest;
+        	use Osstest::Executive;
+        	csreadconfig();
+        	findtask();
+        	printf "%s\n", $ENV{'OSSTEST_TASK'} or die $!;
+        ')
+        export OSSTEST_TASK
+fi
 
 if [ "${alloc_idents[*]}" ]; then
+        if [ x"$OSSTEST_TASK" = x ]; then
+                # We would have to make a dynamic task and hold the
+                #  fd for it in this script.  Would be quite awkward.
+                echo >&2 'cannot hand-allocate in a way that will free'
+                exit 1
+        fi
+
 	progress "allocating ${alloc_idents[*]} ..."
 	alloc_output=tmp/$flight.allocations
 	./mg-allocate \
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[OSSTEST PATCH 4/4] mg-repro-setup: New mode --autoalloc

[Ian Jackson]

This just involves turning autoalloc on and statictask off.
It is most useful with mg-transient-task, as documented.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index b60389d8..e6265b32 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -28,7 +28,7 @@ usage () { cat <<END
  HOSTSPEC is
    [<ident>=][reuse:]<hostname>    host must be allocated, skip host-install
    [<ident>=]wipe:<hostname>       host must be allocated, wipe it
-   [<ident>=]alloc:<resource-spec> \`mg-allocate <resource-spec>', and wipe
+   [<ident>=]alloc:<resource-spec> \`mg-allocate <resource-spec>', and wipe[*]
    none:                           no hosts (should be only HOSTPSEC)
    (default <ident> is \`host')
 
@@ -42,6 +42,13 @@ usage () { cat <<END
    -E... -f... -P       as for mg-execute-flight
    --autoalloc-nofree   allocate hosts as for production, but keep them
                          (specify no HOSTSPECS; remember to deallocate later)
+   --autoalloc          use hosts as for production (specify no HOSTSPECS)
+                         and free them afterwards[*]
+
+ [*] If \$OSSTEST_TASK is set on entry, it will be used for these
+     allocations.  So, in particular,
+        ./mg-transient-task ./mg-repro-setup .... alloc:...
+     will allocate as specified, but discard the hosts when done.
 
  REBUILD is
    --rebuild [-B<blessing>] [-r...]
@@ -105,6 +112,8 @@ skipcapture=true
 autoalloc=false
 statictask=true
 
+if [ "x$OSSTEST_TASK" != x ]; then statictask=false; fi
+
 while true; do
 	case "$1" in
 	-*)			;;
@@ -118,6 +127,7 @@ while true; do
 	-t?*)		duration=${arg#-t}		;;
 	--rogue)	duration=''			;;
 	--capture)	skipcapture=false		;;
+ 	--autoalloc)        autoalloc=true; statictask=false ;;
 	--autoalloc-nofree) autoalloc=true; statictask=true ;;
 	-l*)		logfile=${arg#-l}		;;
 	-r^*|-r!*)	adjustsets+=("${arg#-r}")	;;
@@ -298,7 +308,7 @@ if [ "${alloc_idents[*]}" ]; then
 	progressf "\n"
 	if $autoalloc; then
 		echo >&2 \
- 'warning: --autoalloc-nofree and HOSTSPECs are do not work well together'
+ 'warning: --autoalloc[-nofree] and HOSTSPECs are do not work well together'
 	fi
 fi
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [OSSTEST PATCH 4/4] mg-repro-setup: New mode --autoalloc

[Ian Jackson]

This just involves turning autoalloc on and statictask off.
It is most useful with mg-transient-task, as documented.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 mg-repro-setup | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/mg-repro-setup b/mg-repro-setup
index b60389d8..e6265b32 100755
--- a/mg-repro-setup
+++ b/mg-repro-setup
@@ -28,7 +28,7 @@ usage () { cat <<END
  HOSTSPEC is
    [<ident>=][reuse:]<hostname>    host must be allocated, skip host-install
    [<ident>=]wipe:<hostname>       host must be allocated, wipe it
-   [<ident>=]alloc:<resource-spec> \`mg-allocate <resource-spec>', and wipe
+   [<ident>=]alloc:<resource-spec> \`mg-allocate <resource-spec>', and wipe[*]
    none:                           no hosts (should be only HOSTPSEC)
    (default <ident> is \`host')
 
@@ -42,6 +42,13 @@ usage () { cat <<END
    -E... -f... -P       as for mg-execute-flight
    --autoalloc-nofree   allocate hosts as for production, but keep them
                          (specify no HOSTSPECS; remember to deallocate later)
+   --autoalloc          use hosts as for production (specify no HOSTSPECS)
+                         and free them afterwards[*]
+
+ [*] If \$OSSTEST_TASK is set on entry, it will be used for these
+     allocations.  So, in particular,
+        ./mg-transient-task ./mg-repro-setup .... alloc:...
+     will allocate as specified, but discard the hosts when done.
 
  REBUILD is
    --rebuild [-B<blessing>] [-r...]
@@ -105,6 +112,8 @@ skipcapture=true
 autoalloc=false
 statictask=true
 
+if [ "x$OSSTEST_TASK" != x ]; then statictask=false; fi
+
 while true; do
 	case "$1" in
 	-*)			;;
@@ -118,6 +127,7 @@ while true; do
 	-t?*)		duration=${arg#-t}		;;
 	--rogue)	duration=''			;;
 	--capture)	skipcapture=false		;;
+ 	--autoalloc)        autoalloc=true; statictask=false ;;
 	--autoalloc-nofree) autoalloc=true; statictask=true ;;
 	-l*)		logfile=${arg#-l}		;;
 	-r^*|-r!*)	adjustsets+=("${arg#-r}")	;;
@@ -298,7 +308,7 @@ if [ "${alloc_idents[*]}" ]; then
 	progressf "\n"
 	if $autoalloc; then
 		echo >&2 \
- 'warning: --autoalloc-nofree and HOSTSPECs are do not work well together'
+ 'warning: --autoalloc[-nofree] and HOSTSPECs are do not work well together'
 	fi
 fi
 
-- 
2.11.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel