(no subject)

(no subject)

[Ian Jackson]

I'm intending to be able to do database schema updates.  But I don't
want to play around with such code on a live database.  So I need a
test database.

Thus, a yak: arrangements to make a test database.

As I say, I have tested this and it now does the right things in,
apparently, the right order, and seems to leave things in a good state
even when it collapses halfway or is ^C'd.

In v2 I have addressed the comments, and added a couple of new safety
catches.  For ease of review these are mostly as followup patches to
the mg-schema-test-database script, rather than folded in.

[OSSTEST PATCH 01/16] tcl daemons: log host and port number we bind to, at startup

[Ian Jackson]

If the socket setup fails, this makes it easier to see what the
program was trying to do.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 tcl/daemonlib.tcl |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tcl/daemonlib.tcl b/tcl/daemonlib.tcl
index 972b5e2..b76ff12 100644
--- a/tcl/daemonlib.tcl
+++ b/tcl/daemonlib.tcl
@@ -210,7 +210,7 @@ proc main-daemon {which setup} {
     fconfigure stdout -buffering line
     fconfigure stderr -buffering none
 
-    log "starting"
+    log "starting $host:$port"
 
     uplevel 1 $setup
 
-- 
1.7.10.4

[OSSTEST PATCH 02/16] cri-getconfig: Break out exec_resetting_sigint.

[Ian Jackson]

Move this oddity (and the associated comment) from
standalone-generate-dump-flight-runvars to cri-getconfig.  We are
going to want it elsewhere.

We put this in cri-getconfig because that is the one library of
generic shell functions which everything includes.  Perhaps this file
is misnamed.

No overall functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 cri-getconfig                           |   30 ++++++++++++++++++++++++++++++
 standalone-generate-dump-flight-runvars |   28 +---------------------------
 2 files changed, 31 insertions(+), 27 deletions(-)

diff --git a/cri-getconfig b/cri-getconfig
index 0589bf0..ee1cc40 100644
--- a/cri-getconfig
+++ b/cri-getconfig
@@ -39,3 +39,33 @@ getrepos() {
 	fi
 	echo $repos
 }
+
+# Good grief, handling background proceesses from shell is a pain.
+#
+# For stupid historical reasons, background processes start with
+# SIGINT (and QUIT) ignored (SuSv3 2.11).  bash does not currently
+# offer a way to ask it not to do this; nor is there a reliable way to
+# put the SIGINT handling back to normal.
+#
+# "trap - INT" can be used to put the handling back in recent versions
+# of bash (eg Debian jessie), but earlier versions are buggy, so
+# we use perl.
+#
+# However, there is still a race: if the signal arrives just after the
+# fork, after the shell has (in the child) set it to to IGN, but
+# before Perl has put it back, the child might still escape.
+# There is no reasonable way to deal with this race.  So the result
+# may still be slightly racy in the case that s-g-d-f-r is ^C'd right
+# after starting.
+#
+# Hopefully in the future we can fix this with something like
+# "shopt -s no_async_sig_ignore".  See
+# https://lists.gnu.org/archive/html/bug-bash/2015-10/msg00077.html
+#
+exec_resetting_sigint () {
+	exec perl -e '
+	        $SIG{$_}=DFL foreach qw(INT QUIT HUP);
+		kill 1, $$ unless getppid=='$$';
+		exec @ARGV or die $!;
+   	' "$@"
+}
diff --git a/standalone-generate-dump-flight-runvars b/standalone-generate-dump-flight-runvars
index e91026a..3b20c62 100755
--- a/standalone-generate-dump-flight-runvars
+++ b/standalone-generate-dump-flight-runvars
@@ -58,35 +58,9 @@ perbranch () {
     flight=check_${branch//[-._]/_}
 }
 
-# Good grief, handling background proceesses from shell is a pain.
-#
-# For stupid historical reasons, background processes start with
-# SIGINT (and QUIT) ignored (SuSv3 2.11).  bash does not currently
-# offer a way to ask it not to do this; nor is there a reliable way to
-# put the SIGINT handling back to normal.
-#
-# "trap - INT" can be used to put the handling back in recent versions
-# of bash (eg Debian jessie), but earlier versions are buggy, so
-# we use perl.
-#
-# However, there is still a race: if the signal arrives just after the
-# fork, after the shell has (in the child) set it to to IGN, but
-# before Perl has put it back, the child might still escape.
-# There is no reasonable way to deal with this race.  So the result
-# may still be slightly racy in the case that s-g-d-f-r is ^C'd right
-# after starting.
-#
-# Hopefully in the future we can fix this with something like
-# "shopt -s no_async_sig_ignore".  See
-# https://lists.gnu.org/archive/html/bug-bash/2015-10/msg00077.html
-
 for branch in $@; do
     perbranch
-    perl -e '
-        $SIG{$_}=DFL foreach qw(INT QUIT HUP);
-	kill 1, $$ unless getppid=='$$';
-	exec @ARGV or die $!;
-    ' \
+    exec_resetting_sigint \
     ./standalone make-flight -f $flight $branch >$log 2>&1 &
     procs+=" $branch=$!"
 done
-- 
1.7.10.4

[OSSTEST PATCH 03/16] Configuration: No longer set password=<~/.xen-osstest/db-password>

[Ian Jackson]

Instead, expect the user to provide ~/.pgpass.

This is a good idea because we don't really want to be handling
passwords ourselves if we can help it.  And, we are shortly going to
want to do some exciting mangling of the database access
configuration, which would be complicated by the presence of this
password expansion.

This may break for some users of existing Executive (non-standalone)
setups which are using production-config-cambridge or the default
built-in configuration.

DEPLOYMENT NOTE: After this passes the push gate in Cambridge,
/export/home/osstest/.{xen-,}osstest/db-password should be deleted to
avoid confusion in the future.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest.pm                  |    3 +--
 production-config-cambridge |    2 +-
 2 files changed, 2 insertions(+), 3 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index ec50d60..95e4d46 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -191,8 +191,7 @@ sub readglobalconfig () {
 
     # dynamic default config settings
     $c{ExecutiveDbnamePat} ||= "dbname=<dbname>;user=<whoami>;".
-	"host=<dbname>.db.$c{DnsDomain};".
-	"password=<~/.xen-osstest/db-password>"
+	"host=<dbname>.db.$c{DnsDomain}"
 	if defined $c{DnsDomain};
     # 1. <\w+> is replaced with variables:
     #         <dbname>    database name
diff --git a/production-config-cambridge b/production-config-cambridge
index f801303..412766c 100644
--- a/production-config-cambridge
+++ b/production-config-cambridge
@@ -23,7 +23,7 @@ HostDB_Executive_NoConfigDB 1
 
 OwnerDaemonHost owner.daemon.osstest.xs.citrite.net
 QueueDaemonHost queue.daemon.osstest.xs.citrite.net
-ExecutiveDbnamePat dbname=<dbname>;user=<whoami>;host=osstestdb.xs.citrite.net;password=<~/.xen-osstest/db-password>
+ExecutiveDbnamePat dbname=<dbname>;user=<whoami>;host=osstestdb.xs.citrite.net
 
 HostnameSortSwapWords 1
 
-- 
1.7.10.4

[OSSTEST PATCH 04/16] mg-debug-fail: New utility script for debugging

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Use "egrep ''" rather than "egrep .".  Both sanitise
     missing-final-newline but "egrep ''" will print blank lines,
     which is desirable here.
---
 mg-debug-fail |   13 +++++++++++++
 1 file changed, 13 insertions(+)
 create mode 100755 mg-debug-fail

diff --git a/mg-debug-fail b/mg-debug-fail
new file mode 100755
index 0000000..a163aef
--- /dev/null
+++ b/mg-debug-fail
@@ -0,0 +1,13 @@
+#!/bin/sh
+#
+# This script can be provided anywhere an executable or command name is
+# wanted.  It prints its arguments, and its stdin, to its stderr, and
+# then exits nonzero.
+#
+# When using this it may be useful to provide </dev/null as a
+# redirection for the whole program under test.  Otherwise things
+# can mysteriously hang.
+
+bash -xc ': mg-debug-fail "$@"' x "$@"
+egrep '' >&2
+exit 127
-- 
1.7.10.4

[OSSTEST PATCH 05/16] mg-debug-fail: Catch attempts to read from a tty

[Ian Jackson]

When stdin is a tty, do not try to dump it.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debug-fail |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/mg-debug-fail b/mg-debug-fail
index a163aef..47ad68a 100755
--- a/mg-debug-fail
+++ b/mg-debug-fail
@@ -3,10 +3,10 @@
 # This script can be provided anywhere an executable or command name is
 # wanted.  It prints its arguments, and its stdin, to its stderr, and
 # then exits nonzero.
-#
-# When using this it may be useful to provide </dev/null as a
-# redirection for the whole program under test.  Otherwise things
-# can mysteriously hang.
+
+if tty >/dev/null 2>&1; then
+	exec </dev/null
+fi
 
 bash -xc ': mg-debug-fail "$@"' x "$@"
 egrep '' >&2
-- 
1.7.10.4

[OSSTEST PATCH 06/16] cri-getconfig: Provide get_psql_cmd and get_pgdump_cmd

[Ian Jackson]

This is for (non-standalone-mode) shell scripts which want to access
the postgresql database.

get_psql_command provides `-v ON_ERROR_STOP' because it is not the
default (!) and no sane caller would not want it.

No callers as yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
v2: Fix typo in comment.
---
 cri-getconfig |   33 ++++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)

diff --git a/cri-getconfig b/cri-getconfig
index ee1cc40..973f1c0 100644
--- a/cri-getconfig
+++ b/cri-getconfig
@@ -40,7 +40,38 @@ getrepos() {
 	echo $repos
 }
 
-# Good grief, handling background proceesses from shell is a pain.
+get_psql_cmd () {
+	perl -we '
+	use Osstest;
+	use Osstest::Executive;
+	use DBI;
+	csreadconfig();
+	print "psql",
+	     " -d ", $dbh_tests->{pg_db},
+	     " -h ", $dbh_tests->{pg_host},
+	     " -p ", $dbh_tests->{pg_port},
+	     " -U ", $dbh_tests->{pg_user},
+	     " -v ON_ERROR_STOP=1\n"
+	     or die $!;
+'
+}
+
+get_pgdump_cmd () {
+	perl -we '
+	use Osstest;
+	use Osstest::Executive;
+	use DBI;
+	csreadconfig();
+	print "pg_dump",
+	     " -h ", $dbh_tests->{pg_host},
+	     " -p ", $dbh_tests->{pg_port},
+	     " -U ", $dbh_tests->{pg_user},
+	     " ",    $dbh_tests->{pg_db}, "\n"
+	     or die $!;
+'
+}
+
+# Good grief, handling background processes from shell is a pain.
 #
 # For stupid historical reasons, background processes start with
 # SIGINT (and QUIT) ignored (SuSv3 2.11).  bash does not currently
-- 
1.7.10.4

[OSSTEST PATCH 07/16] cri-getconfig: Provide debugging for get_psql_cmd

[Ian Jackson]

This allows us to execute only the first <some number> SQL
invocations.  The first non-executed one is dumped, instead, by having
get_psql_command print a rune involving ./mg-debug-fail (which the
caller will then execute).

The locking makes things work roughly-correctly if get_psql_cmd is run
in multiple processes at once: it is not defined exactly which
invocations get which counter values, but they will all work properly
and get exactly one counter value each.

If set -x is in force, turn it off for get_psql_cmd: our perl rune is
uninteresting to see repeated ad infinitum in debugging output.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 cri-getconfig |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/cri-getconfig b/cri-getconfig
index 973f1c0..7b75700 100644
--- a/cri-getconfig
+++ b/cri-getconfig
@@ -41,6 +41,22 @@ getrepos() {
 }
 
 get_psql_cmd () {
+	# To use this:
+	#  on each test run, rm -f t.psql-counter
+	#  and set OSSTEST_PSQL_ONLY_DO to an integer
+	if [ "x$OSSTEST_PSQL_ONLY_DO" != x ]; then
+		local f=t.psql-counter
+		psql_counter=$( with-lock-ex -w $f.lock bash -ec '
+			psql_counter=$(cat '$f' || echo 0)
+			echo $(( $psql_counter + 1 )) >'$f'.tmp
+			mv -f '$f'.tmp '$f'
+			echo $psql_counter' )
+		if ! [ $psql_counter -lt $OSSTEST_PSQL_ONLY_DO ]; then
+			printf './mg-debug-fail '
+		fi
+	fi
+
+	set +x
 	perl -we '
 	use Osstest;
 	use Osstest::Executive;
-- 
1.7.10.4

[OSSTEST PATCH 08/16] Osstest.pm: Break out and export globalconfigfiles

[Ian Jackson]

No functional change; no callers as yet.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 Osstest.pm |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 95e4d46..20b8f62 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -30,7 +30,7 @@ BEGIN {
     @ISA         = qw(Exporter);
     @EXPORT      = qw(
                       readglobalconfig %c $mjobdb $mhostdb
-                      augmentconfigdefaults
+                      augmentconfigdefaults globalconfigfiles
                       csreadconfig
                       getmethod
                       postfork
@@ -124,6 +124,10 @@ sub getmethod {
     return $r;
 }
 
+sub globalconfigfiles () {
+	$ENV{'OSSTEST_CONFIG'} || "$ENV{'HOME'}/.xen-osstest/config";
+}
+
 sub readglobalconfig () {
     our $readglobalconfig_done;
     return if $readglobalconfig_done;
@@ -135,7 +139,7 @@ sub readglobalconfig () {
     $c{AuthorizedKeysFiles} = '';
     $c{AuthorizedKeysAppend} = '';
 
-    my $cfgfiles = $ENV{'OSSTEST_CONFIG'} || "$ENV{'HOME'}/.xen-osstest/config";
+    my $cfgfiles = globalconfigfiles();
 
     my $readcfg;
     $readcfg = sub ($$) {
-- 
1.7.10.4

[OSSTEST PATCH 09/16] mg-schema-test-database: New script

[Ian Jackson]

This allows a user in non-standalone mode to make a whole new test
database, which is largely a clone of the original database.

The new db refers to the same resources (hosts), and more-or-less
safely borrows some of those hosts.

Currently we don't do anything about the queue and owner daemons.
This means that queue-daemon-based resource allocation is broken when
clients are pointed at the test db.  But non-queue-based allocation
(eg, ./mg-allocate without -U) works, and the test db can be used for
db-related experiments and even support individual ts-* scripts (other
than ts-hosts-allocate of course).

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: Do not set *Daemon{Host,Port} - move this chunk to a later patch
---
 mg-schema-test-database |  452 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 452 insertions(+)
 create mode 100755 mg-schema-test-database

diff --git a/mg-schema-test-database b/mg-schema-test-database
new file mode 100755
index 0000000..1226761
--- /dev/null
+++ b/mg-schema-test-database
@@ -0,0 +1,452 @@
+#!/bin/bash
+#
+# usages:
+#
+#
+#  ./mg-schema-test-database create [_SUFFIX] [TASK...] \
+#		[-fMINFLIGHT | -f-NUMFLIGHTS]
+#
+# does `drop' and then creates
+#   - the database    osstestdb_test_SUFFIX
+#   - a file          local-config.test-database_SUFFIX
+#
+# default for SUFFIX is your local username
+#
+# Resources owned in the main db by a task in the list of specified
+# TASKs become idle in the test copy.  Others become allocated to
+# a specially-created `owned by someone in real db' task.
+#
+#
+#  ./mg-schema-test-database drop [_SUFFIX]
+#
+# deletes your test database and removes the local-config file
+#
+#
+
+set -e -o posix ${OSSTEST_DEBUG:+-x}
+
+. ./cri-getconfig
+. ./mgi-common
+
+if [ $# -lt 1 ]; then fail "need operation"; fi
+
+cmd="$1"; shift
+
+localconfig=local-config.test-database
+
+maindbname=$(perl -we '
+	use Osstest;
+	use Osstest::Executive;
+	use DBI;
+	csreadconfig();
+	print $dbh_tests->{pg_db},"\n"
+	     or die $!;
+')
+
+parse_only_suffix () {
+	for arg in "$@"; do
+		case "$arg" in
+		_*)	suffix="$arg" ;;
+		*)	fail 'bad usage' ;;
+		esac
+	done
+}
+
+dbname () {
+	dbname="${maindbname}_test$suffix"
+	t="tmp/testdb$suffix"
+	tcfg=local-config.test-database$suffix
+}
+
+psql_query_internal () {
+	$(get_psql_cmd) -At -R' ' -f- "$@"
+}
+
+psql_query () {
+        if [ x$OSSTEST_DEBUG != x ]; then
+		tee /dev/stderr | psql_query_internal "$@"
+	else
+		psql_query_internal "$@"
+	fi
+}
+
+psql_do_cmd () {
+	echo "$(get_psql_cmd) ${OSSTEST_DEBUG:+-e -a}" 
+}
+
+psql_do () {
+	$(psql_do_cmd) -q -f- "$@"
+}
+
+moretasks () {
+	local ifnone="$1"; shift
+	local where="$1"; shift
+	local r
+	r=$(
+		psql_query "$@" <<END
+			SELECT taskid
+			  FROM tasks
+			 $where
+END
+	)
+	if [ "x$r" = x ]; then
+		local m="no tasks matched \`$arg'"
+		case $ifnone in
+		error)
+			fail "error: $m"
+			;;
+		warning)
+			echo >&2 "warning: $m"
+			;;
+		*)
+			fail-bad-moretasks-ifnone-"$ifnone"
+			;;
+		esac
+	fi
+
+	tasks+=" $r"
+}
+
+withtest () {
+	OSSTEST_CONFIG="$test_cfg_setting" "$@"
+}
+
+each_copy_table () {
+	local tab=$1; shift
+	local cond=$1; shift
+
+	p=$t.tabledata.$tab
+	rm -f $p
+
+	cat <<END >>$t.export
+		\\COPY (SELECT * FROM $tab WHERE $cond) TO $p $copyhow
+END
+	cat <<END >>$t.import
+		\\COPY $tab FROM $p $copyhow
+END
+}
+
+make_xdbref_task () {
+	local refkey=$1; shift
+	local comment=$1; shift
+	local refinfo=$1; shift
+	echo "
+		INSERT INTO tasks
+			(type, refkey, username, comment, live, refinfo)
+		  VALUES ('xdbref','$refkey','$username@$nodename',
+			  '$comment','t','$refinfo');
+	"
+}
+
+taskid () {
+	local type=$1; shift
+	local refkey=$1; shift
+	local xcond=$1
+	echo "(SELECT taskid FROM tasks WHERE
+		type='$type' AND refkey='$refkey' $xcond)"
+}
+borrowtaskid () {
+	local bt=$1
+	taskid xdbref $dbname "
+		AND live AND refinfo IS NOT NULL AND refinfo='$bt'
+	"
+}
+
+username=`whoami`
+nodename=`uname -n`
+suffix=_$username
+invocation_now=`date +%s`
+
+case "$cmd" in
+
+#========== CREATE ==========
+
+create)
+	#---------- argument parsing ----------
+
+	tasks=''
+	minflight=-1000
+	for arg in "$@"; do
+		case "$arg" in
+		*@*)
+			moretasks warning			\
+				"WHERE type = 'static'
+				   AND refkey LIKE :'pattern'"	\
+				-v pattern="${arg//\*/%}"	\
+			;;
+		*" "*" "*)
+			local rhs="${arg#* }"
+			moretasks error				\
+				"WHERE taskid = :'taskid'
+				   AND type = :'type'
+				   AND refkey = :'refkey'"	\
+				-v taskid="${arg%% *}"		\
+				-v type="${rhs%% *}"		\
+				-v refkey="${rhs#* }"
+			;;
+		_)	suffix="$arg"
+			;;
+		-f*)	minflight="${arg#-f}"
+			;;
+		*)	fail "bad arg to create"
+			;;
+		esac
+	done
+
+	if [ "x$tasks" = x ]; then
+		moretasks error					\
+			"WHERE type = 'static'
+			   AND refkey = :'refkey'"		\
+			-v refkey="$(whoami)@$(uname -n)"
+	fi
+
+	tasks_cond=${tasks// / OR T=}
+	tasks_cond=${tasks_cond# OR }
+
+	case "$minflight" in
+	-*)
+		minflight=$( psql_query <<END
+                        SELECT flight FROM
+                                (SELECT flight FROM flights
+                                        ORDER BY flight DESC
+                                        LIMIT ${minflight#-})
+				AS last
+                                ORDER BY FLIGHT ASC
+                                LIMIT 1
+END
+		)
+		;;
+	esac
+
+	#---------- preparation and data-gathering ----------
+
+	bad=$(	psql_query <<END
+		SELECT * FROM tasks WHERE (${tasks_cond//T/taskid})
+				AND NOT live
+END
+	)
+	case "$bad" in
+	*[0-9]*)
+		fail "Borrowing from NON-LIVE TASKS $bad"
+		;;
+	esac
+
+	# drop any previous test db
+	"$0" drop $suffix
+
+	dbname
+
+	printf "Setting up %s (minflight=%d, tasks=%s)...\n" \
+		$dbname "$minflight" "${tasks# }"
+
+	# create the config overlay
+	perl >$tcfg.tmp -we '
+		use Osstest;
+		use Osstest::Executive;
+		use DBI;
+		csreadconfig();
+		print "ExecutiveDbname_osstestdb ".
+			"dbname='$dbname';".
+			"host=$dbh_tests->{pg_host};".
+			"user=$dbh_tests->{pg_user};".
+			"port=$dbh_tests->{pg_port}\n"
+			or die $!;
+	'
+	mv -f $tcfg.tmp $tcfg
+
+	# this makes `withtest' work
+	test_cfg_setting="$(perl -we '
+			use Osstest;
+			print globalconfigfiles() or die $!;
+		'):$tcfg"
+
+	# Extract the schema for reference
+	$(get_pgdump_cmd) -s -O -x >$t.schema
+
+	# Keep a copy as it came from dump, for comparison
+	cp $t.schema $t.schema.orig
+
+	# http://www.postgresql.org/message-id/26790.1306355327@sss.pgh.pa.us
+	perl -i~ -pe '
+		s/^/--/ if
+			m/^CREATE EXTENSION IF NOT EXISTS plpgsql / ||
+			m/^COMMENT ON EXTENSION plpgsql /;
+	' $t.schema
+
+	printf "Tables:"
+
+	# What tables are there ?
+	perl -ne <$t.schema >$t.tablevars '
+		if (m/^CREATE SEQUENCE (\w+)/) {
+			print "sequences+=\" $1\"\n";
+		} elsif (m/^CREATE TABLE (\w+)/) {
+			$table=$1;
+		} elsif (m/^\s*flight\s+integer/) {
+			print "ftables+=\" $table\"\n";
+		} elsif ($table && m/^\)\;$/) {
+			print "tables+=\" $table\"\n";
+		}
+	'
+
+	. $t.tablevars
+
+	>$t.tablesortlist
+
+	for table in $tables; do
+		LC_MESSAGES=C $(get_psql_cmd) <<END >$t.display.$table
+			\d $table
+END
+		echo >>$t.tablesortlist "$table $table"
+		perl -ne <$t.display.$table >>$t.tablesortlist '
+			next unless m/^Foreign-key constraints:/ ... m/^\S/;
+			next if m/DEFERRABLE/;
+			next unless m/FOREIGN KEY.*REFERENCES (\w+)/;
+			print "$1 '"$table"'\n" or die $!;
+		'
+		printf " $table"
+	done
+
+	tables=$(tsort <$t.tablesortlist)
+
+	# We don't want to set the permissions
+	perl <executive-postgresql-schema >$t.new-schema -pe '
+		s/^/--/ if
+			m/^ALTER TABLE .* OWNER TO / ||
+			m/^GRANT |^REVOKE /
+	'
+
+	#---------- create test db ----------
+
+	psql_do <<END
+		CREATE DATABASE $dbname;
+END
+	$(withtest get_psql_cmd) -q -f $t.new-schema
+
+	printf ".\n"
+
+	# Schema should now be identical to main DB
+	$(withtest get_pgdump_cmd) -s -O -x >$t.schema.created
+	diff -u $t.schema.orig $t.schema.created
+
+	#---------- mark resources that we are going to borrow ----------
+
+	for task in $tasks; do
+		psql_do <<END
+			BEGIN;
+			$(make_xdbref_task $dbname 'borrowed for test db' $task)
+			UPDATE resources SET owntaskid = $(borrowtaskid $task)
+				WHERE owntaskid=$task;
+			COMMIT;
+END
+	done
+
+	#---------- copy data from live to test db ----------
+
+	copyhow="CSV HEADER NULL e'\\\\n'"
+
+	rm -f $t.import $t.export
+
+	cat >>$t.import <<END
+		\o $t.import-output
+		BEGIN;
+		SET CONSTRAINTS ALL DEFERRED;
+END
+
+	$(get_pgdump_cmd) -a -O -x ${sequences// / -t } >$t.sequences-import
+	perl <$t.sequences-import >>$t.import -ne '
+		next if m/^--/;
+		next if m/^SET /;
+		next unless m/\S/;
+		print or die $!;
+	'
+
+	for table in $tables; do
+		case " $ftables " in
+		*" $table "*)	condition="flight >= $minflight" ;;
+		*)		condition="1=1" ;;
+		esac
+		each_copy_table $table "$condition"
+	done
+
+	# As we copy, we note everything we're not borrowing as
+	# belonging to the parent db.
+	cat >>$t.import <<END
+		$(make_xdbref_task $maindbname 'not borrowed' '')
+		UPDATE resources
+			SET owntaskid = $(taskid xdbref $maindbname)
+			WHERE owntaskid != $(borrowtaskid $task);
+		COMMIT;
+END
+
+	printf "Copy..."
+
+	printf "export..."
+	$(psql_do_cmd) -f $t.export
+
+	printf "import..."
+	$(withtest psql_do_cmd) -f $t.import
+
+	rm -f $t.tabledata.*
+
+	#---------- actually borrow resources ----------
+
+	printf "borrow..."
+
+	for task in $tasks; do
+		withtest psql_do <<END
+			BEGIN;
+			UPDATE resources
+				SET owntaskid = $(taskid magic idle)
+				WHERE owntaskid = $(borrowtaskid $task);
+			COMMIT;
+END
+	done
+	withtest psql_do <<END
+		DELETE FROM tasks
+			WHERE type='xdbref' AND refkey='$dbname';
+END
+
+	printf "\n"
+
+	cat <<END
+Test database $dbname now set up.
+export OSSTEST_CONFIG=$test_cfg_setting
+END
+	;;
+
+#========== DROP ==========
+
+drop)
+	parse_only_suffix "$@"
+
+	dbname
+
+	printf "Dropping %s.\n" "$dbname"
+
+	psql_do <<END
+                SET client_min_messages = WARNING;
+		DROP DATABASE IF EXISTS $dbname;
+		UPDATE resources
+			SET owntaskid = CAST(tasks.refinfo AS INTEGER)
+			FROM tasks
+			WHERE resources.owntaskid = tasks.taskid
+                          AND tasks.type = 'xdbref'
+			  AND tasks.refkey = '$dbname'
+			  AND tasks.live
+			  AND tasks.refinfo IS NOT NULL;
+		UPDATE tasks
+			SET live = 'f'
+			WHERE tasks.type = 'xdbref'
+			  AND tasks.refkey = '$dbname';
+END
+
+	rm -f $tcfg
+
+	;;
+
+#========== EPILOGUE ==========
+
+*)
+	fail "unknown operation \`$cmd'"
+	;;
+esac
-- 
1.7.10.4

Re: [OSSTEST PATCH 09/16] mg-schema-test-database: New script

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> This allows a user in non-standalone mode to make a whole new test
> database, which is largely a clone of the original database.
> 
> The new db refers to the same resources (hosts), and more-or-less
> safely borrows some of those hosts.
> 
> Currently we don't do anything about the queue and owner daemons.
> This means that queue-daemon-based resource allocation is broken when
> clients are pointed at the test db.  But non-queue-based allocation
> (eg, ./mg-allocate without -U) works, and the test db can be used for
> db-related experiments and even support individual ts-* scripts (other
> than ts-hosts-allocate of course).
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 10/16] mg-schema-test-database: Move setting of test_cfg_setting to dbname

[Ian Jackson]

This will makes it available to a wider subset of the script, which is
going to be important in a moment.

No functional change.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-schema-test-database |   12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index 1226761..a1eb49c 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -56,6 +56,12 @@ dbname () {
 	dbname="${maindbname}_test$suffix"
 	t="tmp/testdb$suffix"
 	tcfg=local-config.test-database$suffix
+
+	test_cfg_setting="$(perl -we '
+			use Osstest;
+			print globalconfigfiles() or die $!;
+		'):$tcfg"
+
 }
 
 psql_query_internal () {
@@ -254,12 +260,6 @@ END
 	'
 	mv -f $tcfg.tmp $tcfg
 
-	# this makes `withtest' work
-	test_cfg_setting="$(perl -we '
-			use Osstest;
-			print globalconfigfiles() or die $!;
-		'):$tcfg"
-
 	# Extract the schema for reference
 	$(get_pgdump_cmd) -s -O -x >$t.schema
 
-- 
1.7.10.4

[OSSTEST PATCH 11/16] mg-schema-test-database: Sort out daemons; provide `daemons' subcommand

[Ian Jackson]

We arrange for the test configuration to look for the daemons on a
different host and port, and we provide a convenient way to run such a
pair of daemons.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
v2: Moved setting of *Daemon{Host,Port} to this patch (was
     previously in `mg-schema-test-database: New script')
---
 mg-schema-test-database |   63 ++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 62 insertions(+), 1 deletion(-)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index a1eb49c..73d92f3 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -4,7 +4,8 @@
 #
 #
 #  ./mg-schema-test-database create [_SUFFIX] [TASK...] \
-#		[-fMINFLIGHT | -f-NUMFLIGHTS]
+#		[-fMINFLIGHT | -f-NUMFLIGHTS] \
+#		[-hCTRL_DAEMONS_HOST] [-fOWNER_D_PORT[,QUEUE_D_PORT]]
 #
 # does `drop' and then creates
 #   - the database    osstestdb_test_SUFFIX
@@ -16,12 +17,24 @@
 # TASKs become idle in the test copy.  Others become allocated to
 # a specially-created `owned by someone in real db' task.
 #
+# Default for CTRL_DAEMONS_HOST is localhost; if no port is supplied,
+# we use the corresponding production port + 2; if only one port is
+# supplied we use that and the next port number.
+#
 #
 #  ./mg-schema-test-database drop [_SUFFIX]
 #
 # deletes your test database and removes the local-config file
 #
 #
+#  ./mg-schema-test-database daemons [_SUFFIX]
+#
+# synchronously runs owner and queue daemons for your test database
+#
+# NB that you can't drop a test database with these daemons running,
+# because Postgres will refuse to drop a database that anyone is
+# connected to.
+
 
 set -e -o posix ${OSSTEST_DEBUG:+-x}
 
@@ -114,6 +127,9 @@ END
 }
 
 withtest () {
+	if ! [ -e "$tcfg" ]; then
+		fail "test $dbname not set up ($tcfg does not exist)"
+	fi
 	OSSTEST_CONFIG="$test_cfg_setting" "$@"
 }
 
@@ -194,6 +210,10 @@ create)
 			;;
 		-f*)	minflight="${arg#-f}"
 			;;
+		-h*)	ctrlhost="${arg#-h}"
+			;;
+		-p*)	ctrlports="${arg#-p}"
+			;;
 		*)	fail "bad arg to create"
 			;;
 		esac
@@ -224,6 +244,18 @@ END
 		;;
 	esac
 
+	if [ "x$ctrlhost" = x ]; then
+		ctrlhost=localhost
+	fi
+	case "$ctrlports" in
+		*,*)	;;
+		?*)	ctrlports+=,$(( $ctrlports + 1 )) ;;
+		'')
+ ctrlports=$(( $(getconfig OwnerDaemonPort) + 2)),$((
+ 		$(getconfig QueueDaemonPort) + 2))
+ 			;;
+	esac
+
 	#---------- preparation and data-gathering ----------
 
 	bad=$(	psql_query <<END
@@ -244,6 +276,8 @@ END
 
 	printf "Setting up %s (minflight=%d, tasks=%s)...\n" \
 		$dbname "$minflight" "${tasks# }"
+	printf "Configuring for any daemons to be on %s:%s.\n" \
+		$ctrlhost $ctrlports
 
 	# create the config overlay
 	perl >$tcfg.tmp -we '
@@ -258,6 +292,12 @@ END
 			"port=$dbh_tests->{pg_port}\n"
 			or die $!;
 	'
+	cat >>$tcfg.tmp <<END
+OwnerDaemonHost $ctrlhost
+QueueDaemonHost $ctrlhost
+OwnerDaemonPort ${ctrlports%,*}
+QueueDaemonPort ${ctrlports#*,}
+END
 	mv -f $tcfg.tmp $tcfg
 
 	# Extract the schema for reference
@@ -444,6 +484,27 @@ END
 
 	;;
 
+#========== DAEMONS ==========
+
+daemons)
+	parse_only_suffix "$@"
+
+	dbname
+
+	printf "Running daemons for %s....\n" "$dbname"
+
+	withtest \
+	exec_resetting_sigint ./ms-ownerdaemon &
+
+	sleep 1
+
+	withtest \
+	exec_resetting_sigint ./ms-queuedaemon &
+
+	wait
+
+	;;
+
 #========== EPILOGUE ==========
 
 *)
-- 
1.7.10.4

Re: [OSSTEST PATCH 11/16] mg-schema-test-database: Sort out daemons; provide `daemons' subcommand

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> We arrange for the test configuration to look for the daemons on a
> different host and port, and we provide a convenient way to run such a
> pair of daemons.
> 
> Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 12/16] Configuration: Introduce $c{Username}

[Ian Jackson]

This makes it easier to share the output of whoami.  As a beneficial
side effect it can now be overridden.

Replace many open-coded calls to `whoami` etc. with references to
$c{Username}.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch
---
 Osstest.pm              |    5 +++--
 Osstest/Executive.pm    |   15 ++++-----------
 README                  |    4 ++++
 mg-crontab-install      |    2 +-
 mg-execute-flight       |    2 +-
 mg-schema-test-database |    4 ++--
 6 files changed, 15 insertions(+), 17 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 20b8f62..d4ddda7 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -209,6 +209,7 @@ sub readglobalconfig () {
 
     my $whoami = `whoami` or die $!;
     chomp($whoami) or die;
+    $c{Username} ||= $whoami;
 
     my $nodename = `uname -n` or die $!;
     chomp($nodename) or die;
@@ -219,7 +220,7 @@ sub readglobalconfig () {
     $c{TftpPath} ||= "/tftpboot/";
     $c{TftpPxeDir} ||= "pxelinux.cfg/";
     $c{TftpPxeTemplates} ||= '%ipaddrhex% 01-%etherhyph%';
-    $c{TftpPlayDir} ||= "$whoami/osstest/";
+    $c{TftpPlayDir} ||= "$c{Username}/osstest/";
     $c{TftpTmpDir} ||= "$c{TftpPlayDir}tmp/";
 
     $c{TftpDiBase} ||= "$c{TftpPlayDir}debian-installer";
@@ -229,7 +230,7 @@ sub readglobalconfig () {
     $c{TftpGrubVersion} ||= 'current';
 
     $c{WebspaceFile} ||= "$ENV{'HOME'}/public_html/";
-    $c{WebspaceUrl} ||= "http://$myfqdn/~$whoami/";
+    $c{WebspaceUrl} ||= "http://$myfqdn/~$c{Username}/";
     $c{WebspaceCommon} ||= 'osstest/';
     $c{WebspaceLog} ||= '/var/log/apache2/access.log';
 
diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm
index 2214084..fcef83f 100644
--- a/Osstest/Executive.pm
+++ b/Osstest/Executive.pm
@@ -134,19 +134,14 @@ sub opendb_state () {
     return opendb('statedb');
 }
 
-our $whoami;
-
 sub db_pg_dsn ($) {
     my ($dbname) = @_;
     my $pg= $c{"ExecutiveDbname_$dbname"};
 
     if (!defined $pg) {
-	if (!defined $whoami) {
-	    $whoami = `whoami`;  die if $?;  chomp $whoami;
-	}
         my $pat= $c{ExecutiveDbnamePat};
         my %vars= ('dbname' => $dbname,
-                   'whoami' => $whoami);
+                   'whoami' => $c{Username});
         $pat =~ s#\<(\w+)\>#
             my $val=$vars{$1};  defined $val or die "$pat $1 ?";
             $val;
@@ -449,10 +444,9 @@ sub findtask () {
     my $q;
     my $what;
     if (!defined $spec) {
-        $!=0; $?=0; my $whoami= `whoami`;   defined $whoami or die "$? $!";
         $!=0; $?=0; my $node=   `uname -n`; defined $node   or die "$? $!";
-        chomp($whoami); chomp($node); $node =~ s/\..*//;
-        my $refkey= "$whoami\@$node";
+        chomp($node); $node =~ s/\..*//;
+        my $refkey= "$c{Username}\@$node";
         $what= "static $refkey";
         $q= $dbh_tests->prepare(<<END);
             SELECT * FROM tasks
@@ -489,9 +483,8 @@ END
 }        
 
 sub manual_allocation_base_jobinfo () {
-    my $whoami = `whoami`; chomp $whoami;
     my $hostname = `uname -n`; chomp $hostname;
-    my $info = "$whoami\@$hostname";
+    my $info = "$c{Username}\@$hostname";
     my $tty = `tty 2>/dev/null`; chomp $tty;
     $info .= " ($tty)" unless $?;
     return $info;
diff --git a/README b/README
index 3e64a13..5740ac0 100644
--- a/README
+++ b/README
@@ -535,6 +535,10 @@ TftpPxeTemplatesReal
 Timezone
     Olson TZ name, used by host and guest installers
 
+Username
+    User name on this host.  Defaults to the output of `whoami'.
+    Should not normally be changed.
+
 ========================================
 
 Host-specific config settigs
diff --git a/mg-crontab-install b/mg-crontab-install
index 5a59d45..4df8e6e 100755
--- a/mg-crontab-install
+++ b/mg-crontab-install
@@ -12,7 +12,7 @@ case "$#.$1" in
 esac
 
 expect="#@@ "
-expect+="`whoami`"
+expect+="`whoami`" # not $c{Username}
 expect+="@"
 expect+="`hostname -f`"
 
diff --git a/mg-execute-flight b/mg-execute-flight
index 8b2c21c..ec534da 100755
--- a/mg-execute-flight
+++ b/mg-execute-flight
@@ -53,7 +53,7 @@ done
 if [ x"$flight" = x ]; then badusage; fi
 
 : ${blessing:=play}
-: ${email:=`whoami`}
+: ${email:=`getconfig Username`}
 
 set +e
 tty=`exec 2>/dev/null; tty`
diff --git a/mg-schema-test-database b/mg-schema-test-database
index 73d92f3..b1b5e60 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -174,7 +174,7 @@ borrowtaskid () {
 	"
 }
 
-username=`whoami`
+username=`getconfig Username`
 nodename=`uname -n`
 suffix=_$username
 invocation_now=`date +%s`
@@ -223,7 +223,7 @@ create)
 		moretasks error					\
 			"WHERE type = 'static'
 			   AND refkey = :'refkey'"		\
-			-v refkey="$(whoami)@$(uname -n)"
+			-v refkey="${username}@$(uname -n)"
 	fi
 
 	tasks_cond=${tasks// / OR T=}
-- 
1.7.10.4

Re: [OSSTEST PATCH 12/16] Configuration: Introduce $c{Username}

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> This makes it easier to share the output of whoami.  As a beneficial
> side effect it can now be overridden.
> 
> Replace many open-coded calls to `whoami` etc. with references to
> $c{Username}.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 12/16] Configuration: Introduce $c{Username}

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> This makes it easier to share the output of whoami.  As a beneficial
> side effect it can now be overridden.
> 
> Replace many open-coded calls to `whoami` etc. with references to
> $c{Username}.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[OSSTEST PATCH 13/16] mg-schema-test-database: Change username for back-to-main-db xref

[Ian Jackson]

The `username' of the xdbref task in the test db, referring to the
main db, is changed to `PARENT' (from `<username>@<nodename>').

Currently this is purely cosmetic, but it is going to be useful to
distinguish the two cases:
 * This is a test DB and contains references to a parent
 * This is a parent DB (probably the main DB) which contains
   references to child test DBS.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch
---
 mg-schema-test-database |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index b1b5e60..78f26db 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -152,10 +152,11 @@ make_xdbref_task () {
 	local refkey=$1; shift
 	local comment=$1; shift
 	local refinfo=$1; shift
+	local taskusername=$1; shift
 	echo "
 		INSERT INTO tasks
 			(type, refkey, username, comment, live, refinfo)
-		  VALUES ('xdbref','$refkey','$username@$nodename',
+		  VALUES ('xdbref','$refkey','$taskusername',
 			  '$comment','t','$refinfo');
 	"
 }
@@ -373,7 +374,8 @@ END
 	for task in $tasks; do
 		psql_do <<END
 			BEGIN;
-			$(make_xdbref_task $dbname 'borrowed for test db' $task)
+			$(make_xdbref_task $dbname 'borrowed for test db' \
+				$task "$username@$nodename")
 			UPDATE resources SET owntaskid = $(borrowtaskid $task)
 				WHERE owntaskid=$task;
 			COMMIT;
@@ -411,7 +413,7 @@ END
 	# As we copy, we note everything we're not borrowing as
 	# belonging to the parent db.
 	cat >>$t.import <<END
-		$(make_xdbref_task $maindbname 'not borrowed' '')
+		$(make_xdbref_task $maindbname 'not borrowed' '' PARENT)
 		UPDATE resources
 			SET owntaskid = $(taskid xdbref $maindbname)
 			WHERE owntaskid != $(borrowtaskid $task);
-- 
1.7.10.4

Re: [OSSTEST PATCH 13/16] mg-schema-test-database: Change username for back-to-main-db xref

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> The `username' of the xdbref task in the test db, referring to the
> main db, is changed to `PARENT' (from `<username>@<nodename>').
> 
> Currently this is purely cosmetic, but it is going to be useful to
> distinguish the two cases:
>  * This is a test DB and contains references to a parent
>  * This is a parent DB (probably the main DB) which contains
>    references to child test DBS.

Did you mean "DBS" or just "DB"?

> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 13/16] mg-schema-test-database: Change username for back-to-main-db xref

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 13/16] mg-schema-test-database: Change username for back-to-main-db xref"):
> On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> > Currently this is purely cosmetic, but it is going to be useful to
> > distinguish the two cases:
> >  * This is a test DB and contains references to a parent
> >  * This is a parent DB (probably the main DB) which contains
> >    references to child test DBS.
> 
> Did you mean "DBS" or just "DB"?

I meant `DB(s)' (possibly several).  Fixed.

> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks,
Ian.

[OSSTEST PATCH 14/16] mg-schema-test-database: Bump flight sequence number in test DB

[Ian Jackson]

This makes test flights have different numbers to those currently in
production, which will help avoid accidents.

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch
---
 mg-schema-test-database |   13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index 78f26db..b41a99b 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -430,6 +430,19 @@ END
 
 	rm -f $t.tabledata.*
 
+	printf "flightseq..."
+
+	lastflight=$(psql_query <<END
+		SELECT last_value FROM flights_flight_seq
+END
+		)
+	newlastflight=$(( 10000 * (2 + $lastflight / 10000) ))
+
+	withtest psql_do <<END
+		ALTER SEQUENCE flights_flight_seq
+			RESTART WITH $newlastflight;
+END
+
 	#---------- actually borrow resources ----------
 
 	printf "borrow..."
-- 
1.7.10.4

Re: [OSSTEST PATCH 14/16] mg-schema-test-database: Bump flight sequence number in test DB

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> This makes test flights have different numbers to those currently in
> production, which will help avoid accidents.
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST PATCH 15/16] mg-schema-test-database: Safety catch in JobDB database open

[Ian Jackson]

When we open the `osstest' database, see whether this is a parent DB
(main DB) from which a test DB has been spawned by this user.

If it has, bomb out, unless the user has specified a suitable regexp
matching the DB name in the env var
  OSSTEST_DB_USEREAL_IGNORETEST

This means that when a test database is in play, the user who created
it cannot accidentally operate on the real DB.

The safety catch does not affect Tcl programs, which get the DB config
directly, but in general that just means sg-execute-flight and
sg-run-job which already have a fair amount of safety catch because
they demand flight numbers.

mg-schema-test-database hits this feature over the head.  We assume
that the caller of mg-schema-test-database knows what they are doing;
particularly, that if they create nested test DBs (!), they do not
need the assitance of this feature to stop themselves operating
mg-schema-test-database incorrectly.  Anyone who creates nested test
DBs will hopefully recognise the potential for confusion!

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
 Osstest/JobDB/Executive.pm |   33 ++++++++++++++++++++++++++++++++-
 mg-schema-test-database    |    2 ++
 2 files changed, 34 insertions(+), 1 deletion(-)

diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
index 2572f5f..29dbc6f 100644
--- a/Osstest/JobDB/Executive.pm
+++ b/Osstest/JobDB/Executive.pm
@@ -51,8 +51,39 @@ sub current_flight ($) { #method
     return $ENV{'OSSTEST_FLIGHT'};
 }
 
+sub _check_testdbs ($) {
+    my ($dbh) = @_;
+
+    my $re = $ENV{OSSTEST_DB_USEREAL_IGNORETEST} // '';
+    return if $re eq '.*'; # needed by mg-schema-test-database during setup
+
+    # mg-schema-test-database creates a task
+    #   xdbref/DBNAME with username ${Username}@
+    my $sth = $dbh->prepare(<<END);
+        SELECT refkey AS dbname,
+               username, comment
+            FROM tasks
+	    WHERE type = 'xdbref'
+	      AND live
+	      AND username LIKE (? || '@%')
+END
+    $sth->execute($c{Username});
+    my $allok = 1;
+    while (my $row = $sth->fetchrow_hashref()) {
+	next if $row->{dbname} =~ m/^$re$/o;
+	$allok = 0;
+	print STDERR <<END;
+Test DB exists: $row->{dbname} ($row->{username}, "$row->{comment}")
+END
+    }
+    die "Test dbs exist.  Set OSSTEST_DB_USEREAL_IGNORETEST to regexp ?\n"
+	unless $allok;
+}
+
 sub open ($) { #method
-    return opendb('osstestdb');
+    my $dbh = opendb('osstestdb');
+    _check_testdbs($dbh);
+    return $dbh;
 }
 
 sub dbfl_check ($$) { #method
diff --git a/mg-schema-test-database b/mg-schema-test-database
index b41a99b..90a361b 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -47,6 +47,8 @@ cmd="$1"; shift
 
 localconfig=local-config.test-database
 
+export OSSTEST_DB_USEREAL_IGNORETEST='.*'
+
 maindbname=$(perl -we '
 	use Osstest;
 	use Osstest::Executive;
-- 
1.7.10.4

Re: [OSSTEST PATCH 15/16] mg-schema-test-database: Safety catch in JobDB database open

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> When we open the `osstest' database, see whether this is a parent DB
> (main DB) from which a test DB has been spawned by this user.
> 
> If it has, bomb out, unless the user has specified a suitable regexp
> matching the DB name in the env var
>   OSSTEST_DB_USEREAL_IGNORETEST
> 
> This means that when a test database is in play, the user who created
> it cannot accidentally operate on the real DB.
> 
> The safety catch does not affect Tcl programs, which get the DB config
> directly, but in general that just means sg-execute-flight and
> sg-run-job which already have a fair amount of safety catch because
> they demand flight numbers.
> 
> mg-schema-test-database hits this feature over the head.  We assume
> that the caller of mg-schema-test-database knows what they are doing;
> particularly, that if they create nested test DBs (!), they do not
> need the assitance of this feature to stop themselves operating
> mg-schema-test-database incorrectly.  Anyone who creates nested test
> DBs will hopefully recognise the potential for confusion!
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
> ---
>  Osstest/JobDB/Executive.pm |   33 ++++++++++++++++++++++++++++++++-
>  mg-schema-test-database    |    2 ++
>  2 files changed, 34 insertions(+), 1 deletion(-)
> 
> diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm
> index 2572f5f..29dbc6f 100644
> --- a/Osstest/JobDB/Executive.pm
> +++ b/Osstest/JobDB/Executive.pm
> @@ -51,8 +51,39 @@ sub current_flight ($) { #method
>      return $ENV{'OSSTEST_FLIGHT'};
>  }
>  
> +sub _check_testdbs ($) {
> +    my ($dbh) = @_;
> +
> +    my $re = $ENV{OSSTEST_DB_USEREAL_IGNORETEST} // '';
> +    return if $re eq '.*'; # needed by mg-schema-test-database during
> setup
> +
> +    # mg-schema-test-database creates a task
> +    #   xdbref/DBNAME with username ${Username}@

Was there intended to be a wildcard or pattern of some sort after the @ in
this comment?

> +    my $sth = $dbh->prepare(<<END);
> +        SELECT refkey AS dbname,
> +               username, comment
> +            FROM tasks
> +	    WHERE type = 'xdbref'
> +	      AND live
> +	      AND username LIKE (? || '@%')

Nice of them to pick "||" as the string concatenation operator. It's almost
like they hate us C programmers...

Other than the comment thing above:
Acked-by: Ian Campbell <ian.campbell@citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [OSSTEST PATCH 15/16] mg-schema-test-database: Safety catch in JobDB database open

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 15/16] mg-schema-test-database: Safety catch in JobDB database open"):
> On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> > +    # mg-schema-test-database creates a task
> > +    #   xdbref/DBNAME with username ${Username}@
> 
> Was there intended to be a wildcard or pattern of some sort after the @ in
> this comment?

Yes.  I have changed this to read:

    #   xdbref/DBNAME with username ${Username}@NODENAME

> Other than the comment thing above:
> Acked-by: Ian Campbell <ian.campbell@citrix.com>

Thanks,
Ian.

[OSSTEST PATCH 16/16] mg-schema-test-database: Add workflow doc comment

[Ian Jackson]

Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v2: New patch
---
 mg-schema-test-database |   37 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/mg-schema-test-database b/mg-schema-test-database
index 90a361b..d704950 100755
--- a/mg-schema-test-database
+++ b/mg-schema-test-database
@@ -34,7 +34,42 @@
 # NB that you can't drop a test database with these daemons running,
 # because Postgres will refuse to drop a database that anyone is
 # connected to.
-
+#
+#
+# Expected usage workflow:
+#
+#  1. Optionally, arrange for some hosts to be allocated to a suitable task
+#    OSSTEST_TASK=iwj@testing ./mg-allocate [-U ...] a-host
+#
+#  2. Create the test DB using the script
+#    ./mg-schema-test-database create [_SUFFIX] iwj@testing
+#
+#  3. Using the `export OSSTEST_CONFIG=' line printed by the above,
+#     play about in the test DB.  `a-host' will be `idle' in the test DB:
+#    OSSTEST_CONFIG=\
+#        /u/iwj/.xen-osstest/config:local-config.test-database_iwj \
+#     ./mg-allocate a-host
+#
+#  3a. Maybe run ./mg-schema-test-database daemons (if full-on queue
+#     running is desired).
+#
+#  3b. Meanwhile from the point of view of other users of the main DB,
+#     `a-host' is borrowed by a special task, but the rest of the main
+#     DB and its hosts can carry on.
+#
+#  3c. When talking to the real database, while you have a test DB set
+#     up, you have to set OSSTEST_DB_USEREAL_IGNORETEST eg
+#    OSSTEST_DB_USEREAL_IGNORETEST=osstest_test_iwj ./sg-run-job etc. etc.
+#     This is just to stop you operating on the real DB by mistake.
+#
+#  4. When you are done,
+#    ./mg-schema-test-database drop [_SUFFIX]
+#   This will throw away all of the information in the test DB.
+#
+#  5. OSSTEST_TASK=iwj@testing ./mg-allocate !a-host
+#   Hosts that were marked in the main DB as borrowed, are returned by
+#   mg-schema-test-database to the main DB task that previously owned
+#   them, but not freed.  So you need to explicitly free them.
 
 set -e -o posix ${OSSTEST_DEBUG:+-x}
 
-- 
1.7.10.4

Re: [OSSTEST PATCH 16/16] mg-schema-test-database: Add workflow doc comment

[Ian Campbell]

On Mon, 2015-12-07 at 17:27 +0000, Ian Jackson wrote:
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>