[OSSTEST PATCH RFC 00/14] Integrate XTF into OSSTest

[OSSTEST PATCH RFC 00/14] Integrate XTF into OSSTest

[Wei Liu]

Hi all

This patch series integrates XTF into OSSTest. It still depends on
quite a few things (listed below) but I think it has gotten to point
that it can be posted for review.

It depends on having a canonical location for xtf.git. Currently this
series contains a patch to point to my own xtf.git tree.

It depends on Ian's work:

1. Add a "skip" state to OSSTest.
2. Add diverse flag.

To make the XTf jobs reliable, we also need to:

1. Fix synchronisation issues in xenconsole related code in Xen.
2. Fix synchronisation issue in xtf-runner.

The effect of  synchronisations can be seen in [0]. Test 1 and 5
failed. I've posted patches for them ([2], [3]).

I created a dummy test case that always fails. We expect ts-xtf-run to
continue running after that. The result is at [1].

Wei.


[0] http://osstest.xs.citrite.net/~osstest/testlogs/logs/66910/
[1] http://osstest.xs.citrite.net/~osstest/testlogs/logs/66916/
[2] <1470057385-31599-1-git-send-email-wei.liu2@citrix.com>
[3] <1470060258-20084-1-git-send-email-wei.liu2@citrix.com>

* Currently all logs are only accessible from within Citrix. Sorry. :-/

Wei Liu (14):
  ts-xen-build: always compile in FEP support
  TestSupport: factor out target_jobdir_subdir
  DO NOT APPLY ts-leak-check: sleep 5 seconds before collecting stuff
  ap-common: add xtf tree
  DO NOT APPLY point xtf to my personal tree
  Introduce ts-xtf-build
  sg-run-job: create xtf build recipe
  Introduce ts-xtf-install
  mfi-common: create xtf build job for 4.8 onwards
  Introduce ts-xtf-fep
  Introduce ts-xtf-run
  sg-run-job: test-xtf recipe
  make-flight: create 5 xtf jobs
  Create XTF branch

 Osstest/TestSupport.pm |  12 +++--
 ap-common              |   4 ++
 ap-fetch-version       |   4 ++
 ap-print-url           |   3 ++
 ap-push                |   5 ++
 cr-daily-branch        |   8 +++
 cr-for-branches        |   2 +-
 cri-common             |   1 +
 make-flight            |  40 ++++++++++++++
 mfi-common             |  29 +++++++++++
 sg-run-job             |  12 +++++
 ts-leak-check          |   3 ++
 ts-xen-build           |   7 ++-
 ts-xtf-build           |  66 +++++++++++++++++++++++
 ts-xtf-fep             |  39 ++++++++++++++
 ts-xtf-install         |  41 +++++++++++++++
 ts-xtf-run             | 139 +++++++++++++++++++++++++++++++++++++++++++++++++
 17 files changed, 410 insertions(+), 5 deletions(-)
 create mode 100755 ts-xtf-build
 create mode 100755 ts-xtf-fep
 create mode 100755 ts-xtf-install
 create mode 100755 ts-xtf-run

-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Wei Liu]

By default FEP depends on debug flag. When we are near release the debug
flag will be turned off. In order to test a release build, we explicitly
enable FEP in build configuration.

Since we target Xen versions that already have Kconfig support, only a
Kconfig option is created for now.

We can easily add config option for older Xen when necessary.

Note that this only compiles in FEP support. To enable it a user needs
to explicitly specify fep=1 in hypervisor command line.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xen-build | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/ts-xen-build b/ts-xen-build
index 5933dd4..58670f1 100755
--- a/ts-xen-build
+++ b/ts-xen-build
@@ -84,9 +84,14 @@ END
                (nonempty($earlyprintk) ? <<END : '').
 	echo >>.config CONFIG_EARLY_PRINTK=$earlyprintk
 END
-               ($ho->{Suite} =~ m/squeeze|wheezy/ ? <<END : '') #Debian #693721
+               ($ho->{Suite} =~ m/squeeze|wheezy/ ? <<END : ''). #Debian #693721
 	echo >>.config PYTHON_PREFIX_ARG=
 END
+        <<END
+	if test -f xen/Kconfig; then
+		echo >>xen/.config CONFIG_HVM_FEP=y
+	fi
+END
                );
 }
 
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 02/14] TestSupport: factor out target_jobdir_subdir

[Wei Liu]

Returns a path inside job-specific directory with leafname subdir.

It will be used to return the distribution location of xtf.

No functional change.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 Osstest/TestSupport.pm | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 39d1c08..4f12ea1 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -64,7 +64,7 @@ BEGIN {
                       target_run_apt
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
-                      target_extract_jobdistpath
+                      target_extract_jobdistpath target_jobdir_subdir
                       lv_create lv_dev_mapper
 
                       poll_loop tcpconnect await_tcp
@@ -2175,14 +2175,20 @@ sub target_jobdir ($) {
     return $jobdir;
 }
 
+# returns a path inside job-specific dir with leafname $subdir on $ho
+sub target_jobdir_subdir($$) {
+    my ($ho, $subdir) = @_;
+    my $jobdir = target_jobdir($ho);
+    return "$jobdir/$subdir";
+}
+
 sub target_extract_jobdistpath_subdir ($$$$) {
     my ($ho, $subdir, $distpart, $buildjob) = @_;
     # Extracts dist part $distpart from job $buildjob into an
     # job-specific directory with leafname $subdir on $ho, and returns
     # the directory's pathname.  Does not need root.
-
     my $jobdir = target_jobdir($ho);
-    my $distdir = "$jobdir/$subdir";
+    my $distdir = target_jobdir_subdir($ho, $subdir);
     target_cmd($ho,"rm -rf $distdir && mkdir $distdir",60);
 
     my $path = get_stashed("path_${distpart}dist", $buildjob);
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 03/14] DO NOT APPLY ts-leak-check: sleep 5 seconds before collecting stuff

[Wei Liu]

The system could be in the process of freeing up resources. Give it some
time to finish.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
This won't be necessary once we fix all synchronisation issues in
xtf-runner.
---
 ts-leak-check | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/ts-leak-check b/ts-leak-check
index 8a97971..1f6d5b0 100755
--- a/ts-leak-check
+++ b/ts-leak-check
@@ -170,6 +170,9 @@ sub inventory () {
 }
 
 if (!eval {
+    # Sleep 5 seconds before colleting stuff in case some resources are
+    # being freed.
+    sleep 5;
     &{ "start_$mode" }();
     inventory();
     &{ "finish_$mode" }();
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ap-common | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/ap-common b/ap-common
index 19c7580..46523f3 100644
--- a/ap-common
+++ b/ap-common
@@ -33,6 +33,10 @@
 
 : ${TREEVCS_LINUX:=git}
 
+: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
+: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
+: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}
+
 : ${TREE_LIBVIRT:=git://libvirt.org/libvirt.git}
 : ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git}
 : ${BASE_TREE_LIBVIRT:=git://xenbits.xen.org/libvirt.git}
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 05/14] DO NOT APPLY point xtf to my personal tree

[Wei Liu]

---
 ap-common | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/ap-common b/ap-common
index 46523f3..e544605 100644
--- a/ap-common
+++ b/ap-common
@@ -33,9 +33,9 @@
 
 : ${TREEVCS_LINUX:=git}
 
-: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
-: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
-: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}
+: ${TREE_XTF:=git://xenbits.xen.org/people/liuw/xtf.git}
+: ${PUSH_TREE_XTF:=/export/home/weil/xtf.git/.git}
+: ${BASE_TREE_XTF:=git://xenbits.xen.org/people/liuw/xtf.git}
 
 : ${TREE_LIBVIRT:=git://libvirt.org/libvirt.git}
 : ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git}
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build

[Wei Liu]

Clone, build and package XTF for later use.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xtf-build | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100755 ts-xtf-build

diff --git a/ts-xtf-build b/ts-xtf-build
new file mode 100755
index 0000000..5c4c3ec
--- /dev/null
+++ b/ts-xtf-build
@@ -0,0 +1,66 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2016 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use File::Path;
+use POSIX;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+selectbuildhost(\@ARGV);
+# remaining arguments are passed as targets to "make"
+builddirsprops();
+
+sub checkout () {
+    prepbuilddirs();
+
+    build_clone($ho, 'xtf', $builddir, 'xtf');
+}
+
+sub build_and_install () {
+    # Generate a job-specific directory. XTF will be installed to and run from
+    # that directory on the test host.
+    my $xtfdir = target_jobdir_subdir($ho, 'xtf');
+
+    target_cmd_build($ho, 300, $builddir, <<END);
+        cd xtf
+        rm -f ../xtf-build-ok-stamp
+        ( make xtfdir=$xtfdir 2>&1 && touch ../xtf-build-ok-stamp
+        ) |tee ../xtf-build-log
+        test -f ../xtf-build-ok-stamp
+        echo ok.
+END
+#/;
+    target_cmd_build($ho, 300, $builddir, <<END);
+        cd xtf
+        make xtfdir=$xtfdir DESTDIR=$builddir/xtfdist install 2>&1
+        echo ok.
+END
+#/;
+    store_runvar("xtfdir", $xtfdir);
+}
+
+sub stash () {
+    built_stash($ho, $builddir, "xtfdist", "xtfdist");
+}
+
+checkout();
+build_and_install();
+stash();
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 07/14] sg-run-job: create xtf build recipe

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 sg-run-job | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index 259fc3b..240b265 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -480,6 +480,7 @@ proc need-hosts/build {} { return BUILD }
 proc need-hosts/build-kern {} { return BUILD }
 proc need-hosts/build-libvirt {} { return BUILD }
 proc need-hosts/build-rumpuserxen {} { return BUILD }
+proc need-hosts/build-xtf {} { return BUILD }
 
 proc run-job/build {} {
     run-ts . = ts-xen-build
@@ -498,6 +499,10 @@ proc run-job/build-rumpuserxen {} {
     run-ts . = ts-xen-build + host tools
 }
 
+proc run-job/build-xtf {} {
+    run-ts . = ts-xtf-build
+}
+
 proc prepare-build-host {} {
     global jobinfo
     run-ts broken = ts-hosts-allocate + host
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 08/14] Introduce ts-xtf-install

[Wei Liu]

Extract XTF to the desire location.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xtf-install | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100755 ts-xtf-install

diff --git a/ts-xtf-install b/ts-xtf-install
new file mode 100755
index 0000000..81e5864
--- /dev/null
+++ b/ts-xtf-install
@@ -0,0 +1,41 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2016 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+
+our $ho= selecthost($whhost);
+
+sub extract () {
+    my $xtfdir;
+    my %distpath;
+
+    $xtfdir = get_runvar('xtfdir', $r{xtfbuildjob});
+    die "xtfdir not set?" if !$xtfdir;
+
+    target_extract_jobdistpath($ho, "xtf", "path_xtfdist",
+                               $r{xtfbuildjob}, \%distpath);
+}
+
+extract();
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 09/14] mfi-common: create xtf build job for 4.8 onwards

[Wei Liu]

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 mfi-common | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/mfi-common b/mfi-common
index 971ded3..b1e8dab 100644
--- a/mfi-common
+++ b/mfi-common
@@ -67,6 +67,21 @@ xenbranch_xsm_variants () {
     esac
 }
 
+xenbranch_wants_xtf_tests () {
+    case "$xenbranch" in
+    xen-3.*-testing) return 1;;
+    xen-4.0-testing) return 1;;
+    xen-4.1-testing) return 1;;
+    xen-4.2-testing) return 1;;
+    xen-4.3-testing) return 1;;
+    xen-4.4-testing) return 1;;
+    xen-4.5-testing) return 1;;
+    xen-4.6-testing) return 1;;
+    xen-4.7-testing) return 1;;
+    *) return 0;;
+    esac
+}
+
 job_create_build () {
   job_create_build_filter_callback "$@" || return 0
 
@@ -265,6 +280,20 @@ create_build_jobs () {
 
     fi
 
+    if xenbranch_wants_xtf_tests; then
+        # Only x86, build once for amd64 and use the same result for
+        # both amd64 and i386
+        if [ x$arch = xamd64 ] ; then
+        job_create_build build-$arch-xtf build-xtf                           \
+                arch=$arch                                                   \
+                $RUNVARS $BUILD_RUNVARS $BUILD_XTF_RUNVARS $arch_runvars     \
+                $hostos_runvars                                              \
+                host_hostflags=$build_hostflags                              \
+                tree_xtf=$TREE_XTF                                           \
+                revision_xtf=$REVISION_XTF
+        fi
+    fi
+
     if branch_wants_rumpkernel_tests; then
 
     case $arch in
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 10/14] Introduce ts-xtf-fep

[Wei Liu]

Test the availability of FEP during runtime.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xtf-fep | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100755 ts-xtf-fep

diff --git a/ts-xtf-fep b/ts-xtf-fep
new file mode 100755
index 0000000..e3884ac
--- /dev/null
+++ b/ts-xtf-fep
@@ -0,0 +1,39 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2016 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use Osstest;
+use POSIX;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our ($expect) = @ARGV;
+our $ho = selecthost('host');
+
+sub fep_test () {
+    my $xtfdir = get_runvar('xtfdir', $r{xtfbuildjob});
+
+    my $ret = target_cmd_root($ho, <<END);
+    $xtfdir/xtf-runner fep
+END
+    die "expecting $expect but fep test returned $ret" if $ret != $expect;
+
+    return 0;
+}
+
+exit fep_test();
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 11/14] Introduce ts-xtf-run

[Wei Liu]

This is the main script for running XTF.  It will first perform
selftest, and then run each XTF test case as a substep.

It does the following things:

1. Run self tests for individual environment and record the result.
2. Collect tests according to available environments.
3. Run the collected tests one by one.

The script may exit early if it detects the test host is down or
xtf-runner returns non-recognisable exit code.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 ts-xtf-run | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 139 insertions(+)
 create mode 100755 ts-xtf-run

diff --git a/ts-xtf-run b/ts-xtf-run
new file mode 100755
index 0000000..596ebc4
--- /dev/null
+++ b/ts-xtf-run
@@ -0,0 +1,139 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2016 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use Osstest;
+use POSIX;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our $ho = selecthost('host');
+
+our $runner;
+our @tests;
+our @all_environments;
+our @environments;
+
+# XTF results (runner returned numeric values) and OSStest results:
+#
+# SUCCESS(0)  -> pass
+# SKIP(3)     -> skip
+# ERROR(4)    -> fail
+# FAILURE(5)  -> fail
+#
+# All other return codes are mapped to fail.
+
+sub xtf_result_to_osstest_result ($) {
+    my ($xret) = @_;
+
+    return "pass" if $xret == 0;
+    return "skip" if $xret == 3;
+    return "fail";
+}
+
+# Check if the xtf result is within pre-defined range
+# See xtf-runner --help
+sub xtf_result_defined ($) {
+    my ($xret) = @_;
+    return $xret >= 0 && $xret <= 5;
+}
+
+sub prep () {
+    my $xtfdir = get_runvar('xtfdir', $r{xtfbuildjob});
+    $runner = "$xtfdir/xtf-runner";
+}
+
+sub get_environments () {
+    my $output = target_cmd_output_root($ho, <<END);
+        $runner --list --environments
+END
+    foreach my $e (split('\n', $output)) {
+        push @all_environments, $e
+    }
+}
+
+# Call the runner on one test case, generate a substep for it in final test
+# report. Return runner exit code to caller. May exit the script early if
+# things go really bad.
+sub do_one_test ($) {
+    my ($name) = @_;
+    my $tid = "xtf/$name";
+    my $cmd = "xtf-runner $name";
+
+    substep_start($tid, $cmd);
+    my $output = target_cmd_output_root($ho, <<END, 600);
+        $runner $name 1>&2; echo \$\?
+END
+    my ($ret) = $output =~ /(\d+)/;
+
+    # If the host doesn't respond after a test case, always make this substep
+    # fail and exit the script early with 0
+    my $msg = target_ping_check_up($ho);
+    if ($msg) {
+        logm("$msg");
+        substep_finish($tid, "fail");
+        exit 0;
+    }
+
+    # If xtf result can't be recognised, always make this substep fail and exit
+    # the script early with status 0.
+    if (!xtf_result_defined($ret)) {
+        substep_finish($tid, "fail");
+        exit 0;
+    }
+
+    substep_finish($tid, xtf_result_to_osstest_result($ret));
+
+    return $ret;
+}
+
+# Run selftest for each environment, record the ones that are
+# funtional to get maximum coverage.
+sub do_selftest () {
+    foreach my $e (sort @all_environments) {
+        my $output = target_cmd_output_root($ho, <<END);
+            $runner --list $e selftest
+END
+        my $ret = do_one_test($output);
+        push @environments, $e if $ret == 0;
+    }
+}
+
+sub get_tests_list () {
+    foreach my $e (sort @environments) {
+        my $output = target_cmd_output_root($ho, <<END);
+            $runner --list $e --all --host
+END
+        foreach my $t (split('\n', $output)) {
+            push @tests, $t
+        }
+    }
+}
+
+sub do_all_tests () {
+    foreach my $t (sort @tests) {
+        do_one_test($t);
+    }
+}
+
+prep();
+get_environments();
+do_selftest();
+get_tests_list();
+do_all_tests();
+exit 0;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 12/14] sg-run-job: test-xtf recipe

[Wei Liu]

Install XTF, run FEP test and then run all available tests.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 sg-run-job | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index 240b265..960fce4 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -411,6 +411,13 @@ proc run-job/test-nested {} {
     run-ts . = ts-guest-stop l1 l2
 }
 
+proc need-hosts/test-xtf {} { return host }
+proc run-job/test-xtf {} {
+    run-ts . = ts-xtf-install
+    run-ts . = ts-xtf-fep + 0
+    run-ts . = ts-xtf-run
+}
+
 proc test-guest-migr {g} {
     set to_reap [spawn-ts . = ts-migrate-support-check + host $g 1]
     set can_migrate [reap-ts $to_reap]
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 13/14] make-flight: create 5 xtf jobs

[Wei Liu]

Create jobs only for x86 and set host flag diverse-xtf-x86.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
 make-flight | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/make-flight b/make-flight
index 18e5bc3..613b3d0 100755
--- a/make-flight
+++ b/make-flight
@@ -422,6 +422,26 @@ do_rtds_tests () {
             $debian_runvars all_hostflags=$most_hostflags
 }
 
+do_xtf_tests () {
+  if ! xenbranch_wants_xtf_tests; then
+      return
+  fi
+
+  # xtf is only meaningful to x86. And amd64 and i386 should be the same,
+  # create only for one subarch is good enough.
+  if [ x$xenarch != xamd64 -o x$dom0arch != xamd64 ]; then
+      return;
+  fi
+
+  for i in `seq 1 5`; do
+      job_create_test test-xtf-$xenarch-$dom0arch-$i                 \
+       test-xtf xl $xenarch $dom0arch                                \
+            xtfbuildjob=build-$xenarch-xtf                           \
+            xen_boot_append='hvm_fep=1'                              \
+            all_hostflags=$most_hostflags,diverse-xtf-x86
+  done
+}
+
 do_multivcpu_tests () {
   if [ $xenarch != $dom0arch ]; then
     return
@@ -705,6 +725,8 @@ test_matrix_do_one () {
 
   do_pygrub_tests
   do_pvgrub_tests
+
+  do_xtf_tests
 }
 
 if [ x$buildflight = x ]; then
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH RFC 14/14] Create XTF branch

[Wei Liu]

This branch contains Xen and Linux build jobs and all jobs which contain
xtf in their name.

Signed-off-by: Wei Liu <wei.liu2@citrix.com>
---
I'm not sure if everything is ok because I can't test all changes.
---
 ap-fetch-version |  4 ++++
 ap-print-url     |  3 +++
 ap-push          |  5 +++++
 cr-daily-branch  |  8 ++++++++
 cr-for-branches  |  2 +-
 cri-common       |  1 +
 make-flight      | 18 ++++++++++++++++++
 7 files changed, 40 insertions(+), 1 deletion(-)

diff --git a/ap-fetch-version b/ap-fetch-version
index f26d60a..fa78d51 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -98,6 +98,10 @@ seabios)
 	repo_tree_rev_fetch_git seabios \
 		$TREE_SEABIOS_UPSTREAM master $LOCALREV_SEABIOS
 	;;
+xtf)
+	repo_tree_rev_fetch_git xtf \
+		$TREE_XTF_UPSTREAM master $LOCALREV_XTF
+	;;
 ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
diff --git a/ap-print-url b/ap-print-url
index 4088852..8c5e604 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -58,6 +58,9 @@ rumpuserxen)
 seabios)
 	echo $TREE_SEABIOS_UPSTREAM
 	;;
+xtf)
+	echo $TREE_XTF_UPSTREAM
+	;;
 ovmf)
 	echo $TREE_OVMF_UPSTREAM
 	;;
diff --git a/ap-push b/ap-push
index e33a803..63b7f80 100755
--- a/ap-push
+++ b/ap-push
@@ -40,6 +40,7 @@ TREE_LIBVIRT=$PUSH_TREE_LIBVIRT
 TREE_RUMPUSERXEN=$PUSH_TREE_RUMPUSERXEN
 TREE_SEABIOS=$PUSH_TREE_SEABIOS
 TREE_OVMF=$PUSH_TREE_OVMF
+TREE_XTF=$PUSH_TREE_XTF
 
 if info_linux_tree "$branch"; then
 	cd $repos/linux
@@ -120,6 +121,10 @@ seabios)
 	cd $repos/seabios
 	git push $TREE_SEABIOS $revision:refs/heads/xen-tested-master
 	;;
+xtf)
+	cd $repos/xtf
+	git push $TREE_XTF $revision:refs/heads/xen-tested-master
+	;;
 ovmf)
 	cd $repos/ovmf
 	git push $TREE_OVMF $revision:refs/heads/xen-tested-master
diff --git a/cr-daily-branch b/cr-daily-branch
index 21780b8..3f74e2e 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -195,6 +195,10 @@ if [ "x$REVISION_LIBVIRT" = x ]; then
 	determine_version REVISION_LIBVIRT libvirt LIBVIRT
 	export REVISION_LIBVIRT
 fi
+if [ "x$REVISION_XTF" = x ]; then
+	determine_version REVISION_XTF xtf XTF
+	export REVISION_XTF
+fi
 if [ "x$REVISION_RUMPUSERXEN" = x ]; then
 	determine_version REVISION_RUMPUSERXEN rumpuserxen RUMPUSERXEN
 	export REVISION_RUMPUSERXEN
@@ -248,6 +252,10 @@ seabios)
 	realtree=seabios
 	NEW_REVISION=$REVISION_SEABIOS
 	;;
+xtf)
+	realtree=xtf
+	NEW_REVISION=$REVISION_XTF
+	;;
 ovmf)
 	realtree=ovmf
 	NEW_REVISION=$REVISION_OVMF
diff --git a/cr-for-branches b/cr-for-branches
index 6a161d4..5e8b1a4 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -31,7 +31,7 @@ scriptoptions="$1"; shift
 LOGFILE=tmp/cr-for-branches.log
 export LOGFILE
 
-: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf ${EXTRA_BRANCHES}}
+: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf ${EXTRA_BRANCHES}}
 export BRANCHES
 
 fetchwlem=$wlem
diff --git a/cri-common b/cri-common
index cdee48d..6d5545c 100644
--- a/cri-common
+++ b/cri-common
@@ -78,6 +78,7 @@ select_xenbranch () {
 	libvirt)		tree=libvirt;	xenbranch=xen-unstable ;;
 	rumpuserxen)	      tree=rumpuserxen; xenbranch=xen-unstable ;;
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
+	xtf)			tree=xtf;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
 	distros-*)		tree=none;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
diff --git a/make-flight b/make-flight
index 613b3d0..8aec1a7 100755
--- a/make-flight
+++ b/make-flight
@@ -75,6 +75,14 @@ job_create_build_filter_callback () {
         *)			return 1 ;;
       esac
     ;;
+    xtf)
+      case "$job" in
+        build-amd64)		;;
+        build-amd64-pvops)	;;
+        build-*-xtf)		;;
+        *)			return 1 ;;
+      esac
+    ;;
   esac
   return 0
 }
@@ -116,6 +124,16 @@ job_create_test_filter_callback () {
           return 1;;
       esac
       ;;
+    xtf)
+      case "$xenarch:$dom0arch" in
+          amd64:amd64) ;;
+          *) return 1;;
+      esac
+      case $job in
+          *-xtf-*) ;;
+          *) return 1;;
+      esac
+      ;;
     seabios)
       case $xenarch in
           amd64|i386) ;;
-- 
2.1.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
> By default FEP depends on debug flag. When we are near release the debug
> flag will be turned off. In order to test a release build, we explicitly
> enable FEP in build configuration.
> 
> Since we target Xen versions that already have Kconfig support, only a
> Kconfig option is created for now.

What do you mean by "we target" ?  osstest needs to support all
versions of Xen which are still in security support, which goes back
to 4.4 right now.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 02/14] TestSupport: factor out target_jobdir_subdir

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 02/14] TestSupport: factor out target_jobdir_subdir"):
> Returns a path inside job-specific directory with leafname subdir.
> 
> It will be used to return the distribution location of xtf.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
...
> +: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
> +: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
> +: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}

We should probably have the osstest tested version in the same repo.

Andrew, do you want to have osstest run its xtf push gate between
`staging' and `master' branches of xtf, or do you want to just push to
xtf master, and have a separate `osstest/tested' or some such branch
for the osstest push gate output ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Wei Liu]

On Thu, Aug 04, 2016 at 12:46:30PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
> > By default FEP depends on debug flag. When we are near release the debug
> > flag will be turned off. In order to test a release build, we explicitly
> > enable FEP in build configuration.
> > 
> > Since we target Xen versions that already have Kconfig support, only a
> > Kconfig option is created for now.
> 
> What do you mean by "we target" ?  osstest needs to support all
> versions of Xen which are still in security support, which goes back
> to 4.4 right now.
> 

I didn't want to enable fep for older versions of xen until xtf-runner
is capable of running on them.

In any case, I can add that for older versions of xen as well. It's
trivial.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build"):
> Clone, build and package XTF for later use.
...
> diff --git a/ts-xtf-build b/ts-xtf-build
> new file mode 100755
> index 0000000..5c4c3ec
> --- /dev/null
> +++ b/ts-xtf-build
...
> +selectbuildhost(\@ARGV);
> +# remaining arguments are passed as targets to "make"

This doesn't seem to be actually true.

> +    target_cmd_build($ho, 300, $builddir, <<END);
> +        cd xtf
> +        rm -f ../xtf-build-ok-stamp
> +        ( make xtfdir=$xtfdir 2>&1 && touch ../xtf-build-ok-stamp
> +        ) |tee ../xtf-build-log
> +        test -f ../xtf-build-ok-stamp

I think you should lift buildcmd_stamped_logged out of ts-xen-build.

> +#/;

These are for the benefit of some editor's indenting ?

> +sub stash () {
> +    built_stash($ho, $builddir, "xtfdist", "xtfdist");
> +}

Can you post a tar zvvtf of a sample xtfdist tarball ?

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 07/14] sg-run-job: create xtf build recipe

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 07/14] sg-run-job: create xtf build recipe"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Andrew Cooper]

On 04/08/16 12:46, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
>> By default FEP depends on debug flag. When we are near release the debug
>> flag will be turned off. In order to test a release build, we explicitly
>> enable FEP in build configuration.
>>
>> Since we target Xen versions that already have Kconfig support, only a
>> Kconfig option is created for now.
> What do you mean by "we target" ?  osstest needs to support all
> versions of Xen which are still in security support, which goes back
> to 4.4 right now.

Xen 4.8 will have CONFIG_HVM_FEP available, and the hvm_fep command line
option is available irrespective of whether it is a debug build of the
hypervisor.

Xen 4.5-4.7 have hvm_fep available on the command line, strictly for
debug builds of Xen only.

Older versions have no FEP support at all.


XTF tests which require FEP to function will probe and SKIP if support
is not available.  This does mean that some tests can only sensibly work
against newer hypervisors.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 08/14] Introduce ts-xtf-install

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 08/14] Introduce ts-xtf-install"):
> Extract XTF to the desire location.
...                           ^
> +    $xtfdir = get_runvar('xtfdir', $r{xtfbuildjob});
> +    die "xtfdir not set?" if !$xtfdir;

I don't understand the purpose of this xtfdir variable.

Surely the extracting job should just put it wherever it wants.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build

[Wei Liu]

On Thu, Aug 04, 2016 at 12:52:26PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build"):
> > Clone, build and package XTF for later use.
> ...
> > diff --git a/ts-xtf-build b/ts-xtf-build
> > new file mode 100755
> > index 0000000..5c4c3ec
> > --- /dev/null
> > +++ b/ts-xtf-build
> ...
> > +selectbuildhost(\@ARGV);
> > +# remaining arguments are passed as targets to "make"
> 
> This doesn't seem to be actually true.
> 

Right, I will delete this.

> > +    target_cmd_build($ho, 300, $builddir, <<END);
> > +        cd xtf
> > +        rm -f ../xtf-build-ok-stamp
> > +        ( make xtfdir=$xtfdir 2>&1 && touch ../xtf-build-ok-stamp
> > +        ) |tee ../xtf-build-log
> > +        test -f ../xtf-build-ok-stamp
> 
> I think you should lift buildcmd_stamped_logged out of ts-xen-build.
> 

NP.

> > +#/;
> 
> These are for the benefit of some editor's indenting ?
> 

Yes. I think so. Copied and pasted from ts-xen-build, actually.


> > +sub stash () {
> > +    built_stash($ho, $builddir, "xtfdist", "xtfdist");
> > +}
> 
> Can you post a tar zvvtf of a sample xtfdist tarball ?

drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/
drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/osstest/
drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/
drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/
-rwxr-xr-x osstest/osstest 20124 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/xtf-runner
drwxr-xr-x osstest/osstest     0 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/
drwxr-xr-x osstest/osstest     0 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-168/
-rwxr-xr-x osstest/osstest 1179952 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-168/test-hvm64-xsa-168
-rw-r--r-- osstest/osstest     563 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-168/test-hvm64-xsa-168.cfg
-rw-r--r-- osstest/osstest     116 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-168/info.json
drwxr-xr-x osstest/osstest       0 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-173/
-rw-r--r-- osstest/osstest     528 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-173/test-hvm64-xsa-173.cfg

[repetitive output truncated]

> 
> Thanks,
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 08/14] Introduce ts-xtf-install

[Wei Liu]

On Thu, Aug 04, 2016 at 12:54:41PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 08/14] Introduce ts-xtf-install"):
> > Extract XTF to the desire location.
> ...                           ^
> > +    $xtfdir = get_runvar('xtfdir', $r{xtfbuildjob});
> > +    die "xtfdir not set?" if !$xtfdir;
> 
> I don't understand the purpose of this xtfdir variable.
> 
> Surely the extracting job should just put it wherever it wants.
> 

No, it can't.

It has to be in the location specified during build because there are
hard-coded paths inside test case config files.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 09/14] mfi-common: create xtf build job for 4.8 onwards

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 09/14] mfi-common: create xtf build job for 4.8 onwards"):
> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
...
> +xenbranch_wants_xtf_tests () {
> +    case "$xenbranch" in
> +    xen-3.*-testing) return 1;;
> +    xen-4.0-testing) return 1;;
> +    xen-4.1-testing) return 1;;
> +    xen-4.2-testing) return 1;;
> +    xen-4.3-testing) return 1;;
> +    xen-4.4-testing) return 1;;
> +    xen-4.5-testing) return 1;;
> +    xen-4.6-testing) return 1;;
> +    xen-4.7-testing) return 1;;

Is there some reason we can't run XTF for older Xen versions too ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 10/14] Introduce ts-xtf-fep

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 10/14] Introduce ts-xtf-fep"):
> Test the availability of FEP during runtime.
...
> +    my $ret = target_cmd_root($ho, <<END);
> +    $xtfdir/xtf-runner fep
> +END
> +    die "expecting $expect but fep test returned $ret" if $ret != $expect;

target_cmd_root does not return a useful return value.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 10/14] Introduce ts-xtf-fep

[Wei Liu]

On Thu, Aug 04, 2016 at 01:00:13PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 10/14] Introduce ts-xtf-fep"):
> > Test the availability of FEP during runtime.
> ...
> > +    my $ret = target_cmd_root($ho, <<END);
> > +    $xtfdir/xtf-runner fep
> > +END
> > +    die "expecting $expect but fep test returned $ret" if $ret != $expect;
> 
> target_cmd_root does not return a useful return value.
> 

Right. I should use the trick we discussed yesterday here.

  my $ret = target_cmd_output_root($ho, <<END);
       $xtfdir/xtf-runner fep ; echo \$\?
END


> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Wei Liu]

On Thu, Aug 04, 2016 at 12:53:33PM +0100, Andrew Cooper wrote:
> On 04/08/16 12:46, Ian Jackson wrote:
> > Wei Liu writes ("[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
> >> By default FEP depends on debug flag. When we are near release the debug
> >> flag will be turned off. In order to test a release build, we explicitly
> >> enable FEP in build configuration.
> >>
> >> Since we target Xen versions that already have Kconfig support, only a
> >> Kconfig option is created for now.
> > What do you mean by "we target" ?  osstest needs to support all
> > versions of Xen which are still in security support, which goes back
> > to 4.4 right now.
> 
> Xen 4.8 will have CONFIG_HVM_FEP available, and the hvm_fep command line
> option is available irrespective of whether it is a debug build of the
> hypervisor.
> 
> Xen 4.5-4.7 have hvm_fep available on the command line, strictly for
> debug builds of Xen only.
> 
> Older versions have no FEP support at all.
> 
> 
> XTF tests which require FEP to function will probe and SKIP if support
> is not available.  This does mean that some tests can only sensibly work
> against newer hypervisors.
> 

In that case I think I would just abolish the fep test? Currently if
hypervisor doesn't support fep, the whole job is aborted.


> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 11/14] Introduce ts-xtf-run

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 11/14] Introduce ts-xtf-run"):
> This is the main script for running XTF.  It will first perform
> selftest, and then run each XTF test case as a substep.
> 
> It does the following things:
> 
> 1. Run self tests for individual environment and record the result.
> 2. Collect tests according to available environments.
> 3. Run the collected tests one by one.
> 
> The script may exit early if it detects the test host is down or
> xtf-runner returns non-recognisable exit code.
...
> +    foreach my $e (split('\n', $output)) {
> +        push @all_environments, $e
> +    }

Why not
   push @all_environments, split /\n/, $output;
?

(NB split takes a pattern - a regexp - not a string.  Your code
provides a literal string which is then used as a regexp.)

(Another occurrence of this pattern, later.)

> +# Call the runner on one test case, generate a substep for it in final test
> +# report. Return runner exit code to caller. May exit the script early if
> +# things go really bad.
> +sub do_one_test ($) {
> +    my ($name) = @_;
> +    my $tid = "xtf/$name";
> +    my $cmd = "xtf-runner $name";
> +
> +    substep_start($tid, $cmd);
> +    my $output = target_cmd_output_root($ho, <<END, 600);
> +        $runner $name 1>&2; echo \$\?
> +END
                                      ^ this second \ is superfluous
> +    my ($ret) = $output =~ /(\d+)/;

die unless the pattern matches.

This code seems to be lacking the error handling we discussed.  In
particular, if target_cmd_output_root fails (because the dom0
crashed), target_cmd_output_root will die.  The script will exit
nonzero and the step will be left `running'.

> +    # If the host doesn't respond after a test case, always make this substep
> +    # fail and exit the script early with 0
> +    my $msg = target_ping_check_up($ho);

I think it would be better to use
   target_cmd_output($ho, 'echo ok.');
than target_ping_check_up.

There are some kinds of failure which leave the host responding to
ping but not actually usefully alive.

> +    # If xtf result can't be recognised, always make this substep fail and exit
> +    # the script early with status 0.
> +    if (!xtf_result_defined($ret)) {
> +        substep_finish($tid, "fail");
> +        exit 0;

The lack of use of `eval' (and appropriate use of `die') has resulted
in a lot of explicit repetition of this error path.

Please arrange that all problems which ought to cause "record step as
`fail' and run no more tests" are handled by:
   
    - having the code which detects the problem calling die
    - that die being caught by a single eval instance
    - the code after the eval handling all exceptions that way

Also there is a latent bug here.  You have xtf_result_defined accept
exit statuses 1 and 2, but those are actually not defined exit
statuses for the xtf runner.

I think that this would be best fixed by using something like this:

   +sub xtf_result_to_osstest_result ($) {
   +    my ($xret) = @_;
   +
   +    return "pass" if $xret == 0;
   +    return "skip" if $xret == 3;
   +    return "fail" if $xret == 4;
   +    return "fail" if $xret == 5;
   +    die "xtf runner gave unexpected exit status $xret";
   +}

(And, obviously, calling it within the eval.)

Then you can abolish xtf_result_defined.

And another thing: AFAICT there is nothing that prints the XTF exit
status.  You need at least to report the numerical exit status;
ideally you would print some human-readable interpretation of it.

The person reading the logs may not be familiar with osstest or xtf.
They ought to be told the xtf name for the exit status as well as the
osstest mapping of it.

> +# Run selftest for each environment, record the ones that are
> +# funtional to get maximum coverage.
        ^c

> +sub get_tests_list () {
> +    foreach my $e (sort @environments) {
> +        my $output = target_cmd_output_root($ho, <<END);
> +            $runner --list $e --all --host
> +END
> +        foreach my $t (split('\n', $output)) {
> +            push @tests, $t
> +        }

It might be worth recording the environment for each test, for the
log, unless the xtf runner prints that.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 12/14] sg-run-job: test-xtf recipe

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 12/14] sg-run-job: test-xtf recipe"):
> Install XTF, run FEP test and then run all available tests.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 13/14] make-flight: create 5 xtf jobs

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 13/14] make-flight: create 5 xtf jobs"):
> Create jobs only for x86 and set host flag diverse-xtf-x86.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Andrew Cooper]

On 04/08/16 13:03, Wei Liu wrote:
> On Thu, Aug 04, 2016 at 12:53:33PM +0100, Andrew Cooper wrote:
>> On 04/08/16 12:46, Ian Jackson wrote:
>>> Wei Liu writes ("[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
>>>> By default FEP depends on debug flag. When we are near release the debug
>>>> flag will be turned off. In order to test a release build, we explicitly
>>>> enable FEP in build configuration.
>>>>
>>>> Since we target Xen versions that already have Kconfig support, only a
>>>> Kconfig option is created for now.
>>> What do you mean by "we target" ?  osstest needs to support all
>>> versions of Xen which are still in security support, which goes back
>>> to 4.4 right now.
>> Xen 4.8 will have CONFIG_HVM_FEP available, and the hvm_fep command line
>> option is available irrespective of whether it is a debug build of the
>> hypervisor.
>>
>> Xen 4.5-4.7 have hvm_fep available on the command line, strictly for
>> debug builds of Xen only.
>>
>> Older versions have no FEP support at all.
>>
>>
>> XTF tests which require FEP to function will probe and SKIP if support
>> is not available.  This does mean that some tests can only sensibly work
>> against newer hypervisors.
>>
> In that case I think I would just abolish the fep test? Currently if
> hypervisor doesn't support fep, the whole job is aborted.

For just running tests normally, you shouldn't need to test for fep.

Some of the functional tests which compare Xen's emulated behaviour to
real hardware have an algorithm which runs first against real hardware,
then emulated (usually via fep).  For these tests, the first half will
always run (and may fail if the test is broken), and then either skip or
run the fep part of the test.

These are still worth running even if fep is unavailable, as it still
runs the first half against real hardware.


However, it might be useful to confirm that fep doesn't accidentally
disappear, or a load of tests which were previously passing will turn
into skip, which I presume OSSTest isn't going to be overly concerned with.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Andrew Cooper]

On 04/08/16 12:49, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
>> Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ...
>> +: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
>> +: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
>> +: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}
> We should probably have the osstest tested version in the same repo.
>
> Andrew, do you want to have osstest run its xtf push gate between
> `staging' and `master' branches of xtf, or do you want to just push to
> xtf master, and have a separate `osstest/tested' or some such branch
> for the osstest push gate output ?

The latter please.

We should also clarify the force push criteria.  It is moderately likely
that we get a fix or extension to an existing test which starts showing
up a new bug in the code under test.

In this case, OSSTest will identify a regression, but it isn't a
regression isn't in XTF, nor is it reasonable to call it a regression in
Xen.  We would absolutely want to fix it in upstream, but there are no
guarentees that the bugfix would be suitable for backport to older trees.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Wei Liu]

On Thu, Aug 04, 2016 at 02:28:01PM +0100, Andrew Cooper wrote:
> On 04/08/16 13:03, Wei Liu wrote:
> > On Thu, Aug 04, 2016 at 12:53:33PM +0100, Andrew Cooper wrote:
> >> On 04/08/16 12:46, Ian Jackson wrote:
> >>> Wei Liu writes ("[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
> >>>> By default FEP depends on debug flag. When we are near release the debug
> >>>> flag will be turned off. In order to test a release build, we explicitly
> >>>> enable FEP in build configuration.
> >>>>
> >>>> Since we target Xen versions that already have Kconfig support, only a
> >>>> Kconfig option is created for now.
> >>> What do you mean by "we target" ?  osstest needs to support all
> >>> versions of Xen which are still in security support, which goes back
> >>> to 4.4 right now.
> >> Xen 4.8 will have CONFIG_HVM_FEP available, and the hvm_fep command line
> >> option is available irrespective of whether it is a debug build of the
> >> hypervisor.
> >>
> >> Xen 4.5-4.7 have hvm_fep available on the command line, strictly for
> >> debug builds of Xen only.
> >>
> >> Older versions have no FEP support at all.
> >>
> >>
> >> XTF tests which require FEP to function will probe and SKIP if support
> >> is not available.  This does mean that some tests can only sensibly work
> >> against newer hypervisors.
> >>
> > In that case I think I would just abolish the fep test? Currently if
> > hypervisor doesn't support fep, the whole job is aborted.
> 
> For just running tests normally, you shouldn't need to test for fep.
> 
> Some of the functional tests which compare Xen's emulated behaviour to
> real hardware have an algorithm which runs first against real hardware,
> then emulated (usually via fep).  For these tests, the first half will
> always run (and may fail if the test is broken), and then either skip or
> run the fep part of the test.
> 
> These are still worth running even if fep is unavailable, as it still
> runs the first half against real hardware.
> 
> 
> However, it might be useful to confirm that fep doesn't accidentally
> disappear, or a load of tests which were previously passing will turn
> into skip, which I presume OSSTest isn't going to be overly concerned with.

OK. I will adjust that test to check a runvar for expected result.

For 4.8 onwards, the expected result of fep test is SUCCESS and for (to
be implemented) older versions the expected result is ERROR (or FAILURE?
Doesn't matter from OSSTest's PoV) because branches for older versions
always have FEP to be off.

Wei.

> 
> ~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Wei Liu]

On Thu, Aug 04, 2016 at 12:49:08PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ...
> > +: ${TREE_XTF:=git://xenbits.xen.org/xtf.git}
> > +: ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/osstest/xtf.git}
> > +: ${BASE_TREE_XTF:=git://xenbits.xen.org/osstest/xtf.git}
> 
> We should probably have the osstest tested version in the same repo.
> 
> Andrew, do you want to have osstest run its xtf push gate between
> `staging' and `master' branches of xtf, or do you want to just push to
> xtf master, and have a separate `osstest/tested' or some such branch
> for the osstest push gate output ?
> 

It is currently arranged in latter way.

See "Create XTF branch" patch. OSSTest will push to xen-tested-master.

Wei.

> Thanks,
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Andrew Cooper]

On 04/08/16 15:03, Wei Liu wrote:
> On Thu, Aug 04, 2016 at 02:28:01PM +0100, Andrew Cooper wrote:
>> On 04/08/16 13:03, Wei Liu wrote:
>>> On Thu, Aug 04, 2016 at 12:53:33PM +0100, Andrew Cooper wrote:
>>>> On 04/08/16 12:46, Ian Jackson wrote:
>>>>> Wei Liu writes ("[OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
>>>>>> By default FEP depends on debug flag. When we are near release the debug
>>>>>> flag will be turned off. In order to test a release build, we explicitly
>>>>>> enable FEP in build configuration.
>>>>>>
>>>>>> Since we target Xen versions that already have Kconfig support, only a
>>>>>> Kconfig option is created for now.
>>>>> What do you mean by "we target" ?  osstest needs to support all
>>>>> versions of Xen which are still in security support, which goes back
>>>>> to 4.4 right now.
>>>> Xen 4.8 will have CONFIG_HVM_FEP available, and the hvm_fep command line
>>>> option is available irrespective of whether it is a debug build of the
>>>> hypervisor.
>>>>
>>>> Xen 4.5-4.7 have hvm_fep available on the command line, strictly for
>>>> debug builds of Xen only.
>>>>
>>>> Older versions have no FEP support at all.
>>>>
>>>>
>>>> XTF tests which require FEP to function will probe and SKIP if support
>>>> is not available.  This does mean that some tests can only sensibly work
>>>> against newer hypervisors.
>>>>
>>> In that case I think I would just abolish the fep test? Currently if
>>> hypervisor doesn't support fep, the whole job is aborted.
>> For just running tests normally, you shouldn't need to test for fep.
>>
>> Some of the functional tests which compare Xen's emulated behaviour to
>> real hardware have an algorithm which runs first against real hardware,
>> then emulated (usually via fep).  For these tests, the first half will
>> always run (and may fail if the test is broken), and then either skip or
>> run the fep part of the test.
>>
>> These are still worth running even if fep is unavailable, as it still
>> runs the first half against real hardware.
>>
>>
>> However, it might be useful to confirm that fep doesn't accidentally
>> disappear, or a load of tests which were previously passing will turn
>> into skip, which I presume OSSTest isn't going to be overly concerned with.
> OK. I will adjust that test to check a runvar for expected result.
>
> For 4.8 onwards, the expected result of fep test is SUCCESS and for (to
> be implemented) older versions the expected result is ERROR (or FAILURE?
> Doesn't matter from OSSTest's PoV) because branches for older versions
> always have FEP to be off.

http://xenbits.xen.org/people/andrewcoop/xen-test-framework/fep_2main_8c_source.html

Docs say success/failure.  (after all, it was your code ;p)

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 09/14] mfi-common: create xtf build job for 4.8 onwards

[Wei Liu]

On Thu, Aug 04, 2016 at 12:59:04PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 09/14] mfi-common: create xtf build job for 4.8 onwards"):
> > Signed-off-by: Wei Liu <wei.liu2@citrix.com>
> ...
> > +xenbranch_wants_xtf_tests () {
> > +    case "$xenbranch" in
> > +    xen-3.*-testing) return 1;;
> > +    xen-4.0-testing) return 1;;
> > +    xen-4.1-testing) return 1;;
> > +    xen-4.2-testing) return 1;;
> > +    xen-4.3-testing) return 1;;
> > +    xen-4.4-testing) return 1;;
> > +    xen-4.5-testing) return 1;;
> > +    xen-4.6-testing) return 1;;
> > +    xen-4.7-testing) return 1;;
> 
> Is there some reason we can't run XTF for older Xen versions too ?
> 

No particular reason, it is just not yet coded like that.

Would be trivial to change this patch, but I want to wait until
xtf-runner is capable of running on older versions.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 11/14] Introduce ts-xtf-run

[Wei Liu]

On Thu, Aug 04, 2016 at 01:19:15PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 11/14] Introduce ts-xtf-run"):
> > This is the main script for running XTF.  It will first perform
> > selftest, and then run each XTF test case as a substep.
> > 
> > It does the following things:
> > 
> > 1. Run self tests for individual environment and record the result.
> > 2. Collect tests according to available environments.
> > 3. Run the collected tests one by one.
> > 
> > The script may exit early if it detects the test host is down or
> > xtf-runner returns non-recognisable exit code.
> ...
> > +    foreach my $e (split('\n', $output)) {
> > +        push @all_environments, $e
> > +    }
> 
> Why not
>    push @all_environments, split /\n/, $output;
> ?
> 
> (NB split takes a pattern - a regexp - not a string.  Your code
> provides a literal string which is then used as a regexp.)
> 
> (Another occurrence of this pattern, later.)
> 

Ack.

> > +# Call the runner on one test case, generate a substep for it in final test
> > +# report. Return runner exit code to caller. May exit the script early if
> > +# things go really bad.
> > +sub do_one_test ($) {
> > +    my ($name) = @_;
> > +    my $tid = "xtf/$name";
> > +    my $cmd = "xtf-runner $name";
> > +
> > +    substep_start($tid, $cmd);
> > +    my $output = target_cmd_output_root($ho, <<END, 600);
> > +        $runner $name 1>&2; echo \$\?
> > +END
>                                       ^ this second \ is superfluous
> > +    my ($ret) = $output =~ /(\d+)/;
> 
> die unless the pattern matches.

Ack.

> 
> This code seems to be lacking the error handling we discussed.  In
> particular, if target_cmd_output_root fails (because the dom0
> crashed), target_cmd_output_root will die.  The script will exit
> nonzero and the step will be left `running'.
> 
> > +    # If the host doesn't respond after a test case, always make this substep
> > +    # fail and exit the script early with 0
> > +    my $msg = target_ping_check_up($ho);
> 
> I think it would be better to use
>    target_cmd_output($ho, 'echo ok.');
> than target_ping_check_up.
> 
> There are some kinds of failure which leave the host responding to
> ping but not actually usefully alive.
> 
> > +    # If xtf result can't be recognised, always make this substep fail and exit
> > +    # the script early with status 0.
> > +    if (!xtf_result_defined($ret)) {
> > +        substep_finish($tid, "fail");
> > +        exit 0;
> 
> The lack of use of `eval' (and appropriate use of `die') has resulted
> in a lot of explicit repetition of this error path.
> 
> Please arrange that all problems which ought to cause "record step as
> `fail' and run no more tests" are handled by:
>    
>     - having the code which detects the problem calling die
>     - that die being caught by a single eval instance
>     - the code after the eval handling all exceptions that way
> 

I will see what I can do regarding this. I'm not very familiar with
perl, will need to read some docs first.

> Also there is a latent bug here.  You have xtf_result_defined accept
> exit statuses 1 and 2, but those are actually not defined exit
> statuses for the xtf runner.

FAOD, 1 and 2 are defined xtf exit statuses -- reserved for anything
python interpreter related. But I think this is going to be moot because
they are going to be mapped to fail anyway.

> 
> I think that this would be best fixed by using something like this:
> 
>    +sub xtf_result_to_osstest_result ($) {
>    +    my ($xret) = @_;
>    +
>    +    return "pass" if $xret == 0;
>    +    return "skip" if $xret == 3;
>    +    return "fail" if $xret == 4;
>    +    return "fail" if $xret == 5;
>    +    die "xtf runner gave unexpected exit status $xret";
>    +}
> 
> (And, obviously, calling it within the eval.)
> 
> Then you can abolish xtf_result_defined.
> 
> And another thing: AFAICT there is nothing that prints the XTF exit
> status.  You need at least to report the numerical exit status;
> ideally you would print some human-readable interpretation of it.
> 
> The person reading the logs may not be familiar with osstest or xtf.
> They ought to be told the xtf name for the exit status as well as the
> osstest mapping of it.
> 

The XTF exit status is already available in the log, as is the
osstest result (in substep status line).

I guess you want them on the same line? One logm would do.

> > +# Run selftest for each environment, record the ones that are
> > +# funtional to get maximum coverage.
>         ^c
> 
> > +sub get_tests_list () {
> > +    foreach my $e (sort @environments) {
> > +        my $output = target_cmd_output_root($ho, <<END);
> > +            $runner --list $e --all --host
> > +END
> > +        foreach my $t (split('\n', $output)) {
> > +            push @tests, $t
> > +        }
> 
> It might be worth recording the environment for each test, for the
> log, unless the xtf runner prints that.
> 

It is encoded in test case name.

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build"):
> On Thu, Aug 04, 2016 at 12:52:26PM +0100, Ian Jackson wrote:
> > Wei Liu writes ("[OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build"):
> > > +selectbuildhost(\@ARGV);
> > > +# remaining arguments are passed as targets to "make"
> > 
> > This doesn't seem to be actually true.
> 
> Right, I will delete this.

Maybe it should be made true rather than deleted ?

I'll reply to the tarball listing in just a moment.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]

[Ian Jackson]

Wei Liu writes ("[OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build"):
> > > +sub stash () {
> > > +    built_stash($ho, $builddir, "xtfdist", "xtfdist");
> > > +}
> > 
> > Can you post a tar zvvtf of a sample xtfdist tarball ?
> 
> drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/
> drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/osstest/
...
> -rw-r--r-- osstest/osstest     563 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-168/test-hvm64-xsa-168.cfg

This is quite odd.

AFAICT this means that the original path of that file, as generated by
the invocation of `make install DESTDIR=...', was:

 /home/osstest/build.66916/build-amd64-xtf/xtf/xtfdist/home/osstest/build.66916/build-amd64-xtf/xtf/xtfdist/tests/xsa-168/test-hvm64-xsa-168.cfg

This seems perverse.  Read on...

Wei Liu writes ("Re: [OSSTEST PATCH RFC 08/14] Introduce ts-xtf-install"):
> On Thu, Aug 04, 2016 at 12:54:41PM +0100, Ian Jackson wrote:
> > Surely the extracting job should just put it wherever it wants.
> 
> No, it can't.
> 
> It has to be in the location specified during build because there are
> hard-coded paths inside test case config files.

That's a bit sad.  But, fine.

But, in that case it is not possible to coinstall multiple different
versions of xtf.

The ability to coinstall multiple different xtfs was my reason for
suggesting that the xtfdist should be unpacked into a job-specific
subdirectory of /home/osstest.  Obviously that would have to be a
directory specific to the _test job_, not the build job.

As I say, if xtfdists aren't portable to different install locations,
then this is not possible and there is no reason not to use a fixed
install location.

That is:

  Build host:

     cd /home/osstest/build.66916.build-amd64-xtf/xtf
     make xtfdir=/home/xtf
     make xtfdir=/home/xtf DESTDIR=`pwd`/xtfdist

  Resulting in, on build host, for example:

     /home/osstest/build.66916.build-amd64-xtf/xtf/xtfdist/home/xtf/tests/xsa-168/test-hvm64-xsa-168.cfg

  Note that this does not actually create anything in /home/xtf on the
  build host.  So different builds do not interfere with each other.

  Test host, for example:

     /home/xtf/tests/xsa-168/test-hvm64-xsa-168.cfg

Then there is no need for the xtfdir runvar, because the answer is
always the same.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Ian Jackson]

Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> On 04/08/16 12:49, Ian Jackson wrote:
> > Andrew, do you want to have osstest run its xtf push gate between
> > `staging' and `master' branches of xtf, or do you want to just push to
> > xtf master, and have a separate `osstest/tested' or some such branch
> > for the osstest push gate output ?
> 
> The latter please.

NP.

> We should also clarify the force push criteria.  It is moderately likely
> that we get a fix or extension to an existing test which starts showing
> up a new bug in the code under test.

Can this not be made into a new test ?

osstest would like such an extension to an existing test to be a new
`step' (in osstest terminology).  That would mean that the fact that
it fails with old versions of Xen can be seen not to be a regression.

Ie, osstest would like to be able to distinguish:
  * this version of xtf didn't have the new part of the test
  * old part of the test failed (and presumably new part was not run)
  * old part succeeded and new part failed
  * both parts were run and succeeded

> In this case, OSSTest will identify a regression, but it isn't a
> regression isn't in XTF, nor is it reasonable to call it a regression in
> Xen.  We would absolutely want to fix it in upstream, but there are no
> guarentees that the bugfix would be suitable for backport to older trees.

Indeed.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Ian Jackson]

Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
> On Thu, Aug 04, 2016 at 02:28:01PM +0100, Andrew Cooper wrote:
> > However, it might be useful to confirm that fep doesn't accidentally
> > disappear, or a load of tests which were previously passing will turn
> > into skip, which I presume OSSTest isn't going to be overly concerned with.

Yes, indeed.  That should be detected by spotting that the fep test
fails.

> OK. I will adjust that test to check a runvar for expected result.

No, that is not the right approach.

> For 4.8 onwards, the expected result of fep test is SUCCESS and for (to
> be implemented) older versions the expected result is ERROR (or FAILURE?
> Doesn't matter from OSSTest's PoV) because branches for older versions
> always have FEP to be off.

Why not just report the fep test as having failed, in old branches,
and then carry on ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Wei Liu]

On Thu, Aug 04, 2016 at 04:22:58PM +0100, Ian Jackson wrote:
> Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
> > On Thu, Aug 04, 2016 at 02:28:01PM +0100, Andrew Cooper wrote:
> > > However, it might be useful to confirm that fep doesn't accidentally
> > > disappear, or a load of tests which were previously passing will turn
> > > into skip, which I presume OSSTest isn't going to be overly concerned with.
> 
> Yes, indeed.  That should be detected by spotting that the fep test
> fails.
> 
> > OK. I will adjust that test to check a runvar for expected result.
> 
> No, that is not the right approach.
> 
> > For 4.8 onwards, the expected result of fep test is SUCCESS and for (to
> > be implemented) older versions the expected result is ERROR (or FAILURE?
> > Doesn't matter from OSSTest's PoV) because branches for older versions
> > always have FEP to be off.
> 
> Why not just report the fep test as having failed, in old branches,
> and then carry on ?
> 

How to do that? If the test script fails, surely sg-run-job would not
run the rest of the job.

Or do you mean the script should still pass, but print the result in the
log? But then, how to express the expected result for different
branches? Runvar? Script argument?

Wei.

> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 11/14] Introduce ts-xtf-run

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH RFC 11/14] Introduce ts-xtf-run"):
> On Thu, Aug 04, 2016 at 01:19:15PM +0100, Ian Jackson wrote:
> > Please arrange that all problems which ought to cause "record step as
> > `fail' and run no more tests" are handled by:
> >    
> >     - having the code which detects the problem calling die
> >     - that die being caught by a single eval instance
> >     - the code after the eval handling all exceptions that way
> > 
> 
> I will see what I can do regarding this. I'm not very familiar with
> perl, will need to read some docs first.

Sure.  I'm happy to handwave IRL or on IRC if you prefer.

If you want some full-on examples of the use of eval and indeed die,
there are:
  ts-logs-capture      (several smallish examples)
  cs-bisection-step
  Osstest::db_retry    (subrefs, hairy)
  Osstest::Executive::alloc_resources  (one nested inside another)

> > Also there is a latent bug here.  You have xtf_result_defined accept
> > exit statuses 1 and 2, but those are actually not defined exit
> > statuses for the xtf runner.
> 
> FAOD, 1 and 2 are defined xtf exit statuses -- reserved for anything
> python interpreter related. But I think this is going to be moot because
> they are going to be mapped to fail anyway.

What I meant was that exit statuses 1 and 2 mean `it has all gone
horribly wrong'.  They are not supposed to occur.  (They may be
`defined' by the xtf runner spec, but only to reserve them.)

> The XTF exit status is already available in the log, as is the
> osstest result (in substep status line).

Oh good.  Sorry for quibbling, then.

> I guess you want them on the same line? One logm would do.

But, yes, that would be nice.

> > It might be worth recording the environment for each test, for the
> > log, unless the xtf runner prints that.
> 
> It is encoded in test case name.

Jolly good.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]

[Wei Liu]

On Thu, Aug 04, 2016 at 04:17:45PM +0100, Ian Jackson wrote:
> Wei Liu writes ("[OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build"):
> > > > +sub stash () {
> > > > +    built_stash($ho, $builddir, "xtfdist", "xtfdist");
> > > > +}
> > > 
> > > Can you post a tar zvvtf of a sample xtfdist tarball ?
> > 
> > drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/
> > drwxr-xr-x osstest/osstest   0 2016-08-04 08:30 home/osstest/
> ...
> > -rw-r--r-- osstest/osstest     563 2016-08-04 08:30 home/osstest/build.66916.build-amd64-xtf/xtf/tests/xsa-168/test-hvm64-xsa-168.cfg
> 
> This is quite odd.
> 
> AFAICT this means that the original path of that file, as generated by
> the invocation of `make install DESTDIR=...', was:
> 
>  /home/osstest/build.66916/build-amd64-xtf/xtf/xtfdist/home/osstest/build.66916/build-amd64-xtf/xtf/xtfdist/tests/xsa-168/test-hvm64-xsa-168.cfg
> 
> This seems perverse.  Read on...
> 
> Wei Liu writes ("Re: [OSSTEST PATCH RFC 08/14] Introduce ts-xtf-install"):
> > On Thu, Aug 04, 2016 at 12:54:41PM +0100, Ian Jackson wrote:
> > > Surely the extracting job should just put it wherever it wants.
> > 
> > No, it can't.
> > 
> > It has to be in the location specified during build because there are
> > hard-coded paths inside test case config files.
> 
> That's a bit sad.  But, fine.
> 
> But, in that case it is not possible to coinstall multiple different
> versions of xtf.
> 
> The ability to coinstall multiple different xtfs was my reason for
> suggesting that the xtfdist should be unpacked into a job-specific
> subdirectory of /home/osstest.  Obviously that would have to be a
> directory specific to the _test job_, not the build job.
> 

I'm confused.

Note that xtf won't generate any artefacts during runtime.

The test jobs are tied to a specific build job in one flight, so I think
a build job specific path (as coded) would be the same as test job
specific path.

If you have different test jobs that depend the same build job, using
the same build should be fine. If you have test jobs that depend on
different build job, obviously the path would be different, too.

But I guess this point is moot now because I'm fine with your suggestion
below.

> As I say, if xtfdists aren't portable to different install locations,
> then this is not possible and there is no reason not to use a fixed
> install location.
> 
> That is:
> 
>   Build host:
> 
>      cd /home/osstest/build.66916.build-amd64-xtf/xtf
>      make xtfdir=/home/xtf
>      make xtfdir=/home/xtf DESTDIR=`pwd`/xtfdist
> 
>   Resulting in, on build host, for example:
> 
>      /home/osstest/build.66916.build-amd64-xtf/xtf/xtfdist/home/xtf/tests/xsa-168/test-hvm64-xsa-168.cfg
> 
>   Note that this does not actually create anything in /home/xtf on the
>   build host.  So different builds do not interfere with each other.
> 
>   Test host, for example:
> 
>      /home/xtf/tests/xsa-168/test-hvm64-xsa-168.cfg
> 
> Then there is no need for the xtfdir runvar, because the answer is
> always the same.
> 
> Thanks,
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support

[Ian Jackson]

Wei Liu writes ("Re: [Xen-devel] [OSSTEST PATCH RFC 01/14] ts-xen-build: always compile in FEP support"):
> On Thu, Aug 04, 2016 at 04:22:58PM +0100, Ian Jackson wrote:
> > Why not just report the fep test as having failed, in old branches,
> > and then carry on ?
> 
> How to do that? If the test script fails, surely sg-run-job would not
> run the rest of the job.

sg-run-job can be told that this script failing is not fatal.  This is
done by calling spawn-ts and reap-ts explicitly.  See test-guest-migr
for an example.

> Or do you mean the script should still pass, but print the result in the
> log?

No.

>  But then, how to express the expected result for different
> branches? Runvar? Script argument?

You do not need to encode this in osstest at all.  The database will
record that FEP never worked for Xen 4.4 the same way that it records
that ARM migration never worked for 4.7.  It will show up as `fail
never pass'.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Andrew Cooper]

On 04/08/16 16:21, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
>> On 04/08/16 12:49, Ian Jackson wrote:
>>> Andrew, do you want to have osstest run its xtf push gate between
>>> `staging' and `master' branches of xtf, or do you want to just push to
>>> xtf master, and have a separate `osstest/tested' or some such branch
>>> for the osstest push gate output ?
>> The latter please.
> NP.
>
>> We should also clarify the force push criteria.  It is moderately likely
>> that we get a fix or extension to an existing test which starts showing
>> up a new bug in the code under test.
> Can this not be made into a new test ?

That very much depends, and probably needs deciding on a case by case basis.

What absolutely shouldn't happen is ending up with test-$FOO,
test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
the same logical test being split up in ad-hoc ways.  While XTF is
useful for automation, it is first and foremost a tool for humans.

~Andrew

>
> osstest would like such an extension to an existing test to be a new
> `step' (in osstest terminology).  That would mean that the fact that
> it fails with old versions of Xen can be seen not to be a regression.
>
> Ie, osstest would like to be able to distinguish:
>   * this version of xtf didn't have the new part of the test
>   * old part of the test failed (and presumably new part was not run)
>   * old part succeeded and new part failed
>   * both parts were run and succeeded
>
>> In this case, OSSTest will identify a regression, but it isn't a
>> regression isn't in XTF, nor is it reasonable to call it a regression in
>> Xen.  We would absolutely want to fix it in upstream, but there are no
>> guarentees that the bugfix would be suitable for backport to older trees.
> Indeed.
>
> Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]

[Ian Jackson]

Wei Liu writes ("Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]"):
> On Thu, Aug 04, 2016 at 04:17:45PM +0100, Ian Jackson wrote:
> > The ability to coinstall multiple different xtfs was my reason for
> > suggesting that the xtfdist should be unpacked into a job-specific
> > subdirectory of /home/osstest.  Obviously that would have to be a
> > directory specific to the _test job_, not the build job.
...
> Note that xtf won't generate any artefacts during runtime.

Yes.

> The test jobs are tied to a specific build job in one flight, so I think
> a build job specific path (as coded) would be the same as test job
> specific path.

That is how the flights are constructed right now, yes.

But it is not in general a good idea to bake too many of these
assumptions into the ts-* scripts.  It's not easy to imagine today why
a single test host might want to be subjected to multiple xtf runs,
but it seems hard to rule out.

For example, in the future, host sharing may mean that different test
jobs for the same flight are run on the same host.  Of course these
wouldn't be your existing multiple xtf jobs (for several reasons).

But if there were ever some xtf test jobs which were different in some
way, then in principle it would be possible for each xtf test job to
specify a differnet xtf build job.  This would mean installing xtf
more than once, from what may be different build jobs or what may be
the same build job.

Whether they are the same build job might not depend on things
specified directly in make-flight and mfi-*.  Automatic job
constructors can reuse jobs from other flights and/or construct new
flights using old ones as templates.  The bisector will choose build
jobs as it sees fit.

If an xtfdist would work if unpacked in a different directory, which
wasn't known at build time, it would be possible for each xtf test job
to unpack its xtfdist in the directory corresponding to the test job.

Thus multiple test jobs with potentially-different xtfs could run
sequentially on the same host.

However, that is not possible.  (Unpacking the same xtf over the top
of itself might not be safe and wouldn't be a good idea.)

Therefore, we will have to remember this limitation if we ever are
likely to come across it.  Fine.

But that does mean that my suggestion for using a job-specific
location for the xtf installed version is pointless.  The
(impossible) scheme depends on a _test_-job-specific location.  There
is no point embedding the _build_ job info in the pathnames.

And as you discover, doing so means that the pathname needs to be made
available in a runvar.  This is pointless complexity.

> If you have different test jobs that depend the same build job, using
> the same build should be fine. If you have test jobs that depend on
> different build job, obviously the path would be different, too.

This seems to contemplate unpacking the same xtf over the top of
itself.  I don't think this is a good idea.  For example, if we ever
do parallel runs, it would break.  It might also overwrite useful logs
etc.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Ian Jackson]

Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> On 04/08/16 16:21, Ian Jackson wrote:
> > Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
> >> We should also clarify the force push criteria.  It is moderately likely
> >> that we get a fix or extension to an existing test which starts showing
> >> up a new bug in the code under test.
> > Can this not be made into a new test ?
> 
> That very much depends, and probably needs deciding on a case by case basis.
> 
> What absolutely shouldn't happen is ending up with test-$FOO,
> test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
> the same logical test being split up in ad-hoc ways.  While XTF is
> useful for automation, it is first and foremost a tool for humans.

You mean that this would contort the code for the test ?

Here is another approach that could be used to satisfy osstest's
desire for stability in the meaning of test names: you could
explicitly copy the entire source of the test to a new test name, and
deprecate the old test.  Later, if the old versions of Xen are fixed,
or after they are retired you would delete the old test.

The result would be that the old, less-good, test would still exist
and still detect regressions.  The new, better test, would be seen to
have never passed on old branches.

Force pushing XTF is not an answer to this problem, for two reasons:

Firstly, if the improved test correctly fails on old versions of Xen,
this will appear as a regression in each old version of Xen.  Each old
Xen tree would need to be force pushed, every time this happened.

Secondly, in any case, force pushing XTF might not be necessary.  In
the usual case, one would expect xen-unstable (which is what would
drive the XTF push gate) to be fixed so that the improved test
passes.  So osstest wouldn't spot the issue until it ran the new XTF
with old Xen.

Another possibility would be to have the capability for an single XTF
test execution to return multiple statuses.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]

[Andrew Cooper]

On 04/08/16 17:05, Ian Jackson wrote:
> Wei Liu writes ("Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]"):
>> On Thu, Aug 04, 2016 at 04:17:45PM +0100, Ian Jackson wrote:
>>> The ability to coinstall multiple different xtfs was my reason for
>>> suggesting that the xtfdist should be unpacked into a job-specific
>>> subdirectory of /home/osstest.  Obviously that would have to be a
>>> directory specific to the _test job_, not the build job.
> ...
>> Note that xtf won't generate any artefacts during runtime.
> Yes.
>
>> The test jobs are tied to a specific build job in one flight, so I think
>> a build job specific path (as coded) would be the same as test job
>> specific path.
> That is how the flights are constructed right now, yes.
>
> But it is not in general a good idea to bake too many of these
> assumptions into the ts-* scripts.  It's not easy to imagine today why
> a single test host might want to be subjected to multiple xtf runs,
> but it seems hard to rule out.
>
> For example, in the future, host sharing may mean that different test
> jobs for the same flight are run on the same host.  Of course these
> wouldn't be your existing multiple xtf jobs (for several reasons).
>
> But if there were ever some xtf test jobs which were different in some
> way, then in principle it would be possible for each xtf test job to
> specify a differnet xtf build job.  This would mean installing xtf
> more than once, from what may be different build jobs or what may be
> the same build job.
>
> Whether they are the same build job might not depend on things
> specified directly in make-flight and mfi-*.  Automatic job
> constructors can reuse jobs from other flights and/or construct new
> flights using old ones as templates.  The bisector will choose build
> jobs as it sees fit.
>
> If an xtfdist would work if unpacked in a different directory, which
> wasn't known at build time, it would be possible for each xtf test job
> to unpack its xtfdist in the directory corresponding to the test job.
>
> Thus multiple test jobs with potentially-different xtfs could run
> sequentially on the same host.
>
> However, that is not possible.  (Unpacking the same xtf over the top
> of itself might not be safe and wouldn't be a good idea.)
>
> Therefore, we will have to remember this limitation if we ever are
> likely to come across it.  Fine.
>
> But that does mean that my suggestion for using a job-specific
> location for the xtf installed version is pointless.  The
> (impossible) scheme depends on a _test_-job-specific location.  There
> is no point embedding the _build_ job info in the pathnames.
>
> And as you discover, doing so means that the pathname needs to be made
> available in a runvar.  This is pointless complexity.
>
>> If you have different test jobs that depend the same build job, using
>> the same build should be fine. If you have test jobs that depend on
>> different build job, obviously the path would be different, too.
> This seems to contemplate unpacking the same xtf over the top of
> itself.  I don't think this is a good idea.  For example, if we ever
> do parallel runs, it would break.  It might also overwrite useful logs
> etc.

The current hard-coded paths exist only because of current limitiations
in xl/libxl (identified here:
http://xenbits.xen.org/people/andrewcoop/xen-test-framework/#errata)

With those issues fixed, XTF can be made to be entirely filesystem
location independent.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]

[Ian Jackson]

Andrew Cooper writes ("Re: [Xen-devel] [OSSTEST PATCH RFC 06/14] Introduce ts-xtf-build [and 1 more messages]"):
> The current hard-coded paths exist only because of current limitiations
> in xl/libxl (identified here:
> http://xenbits.xen.org/people/andrewcoop/xen-test-framework/#errata)
> 
> With those issues fixed, XTF can be made to be entirely filesystem
> location independent.

Great.  If that ever happens we can make osstest install its xtf(s)
whereever it likes and the build location will be irrelevant.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree

[Andrew Cooper]

On 04/08/16 17:13, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
>> On 04/08/16 16:21, Ian Jackson wrote:
>>> Andrew Cooper writes ("Re: [OSSTEST PATCH RFC 04/14] ap-common: add xtf tree"):
>>>> We should also clarify the force push criteria.  It is moderately likely
>>>> that we get a fix or extension to an existing test which starts showing
>>>> up a new bug in the code under test.
>>> Can this not be made into a new test ?
>> That very much depends, and probably needs deciding on a case by case basis.
>>
>> What absolutely shouldn't happen is ending up with test-$FOO,
>> test-$FOO-2, test-more-$FOO, test-$FOO-again because that will result in
>> the same logical test being split up in ad-hoc ways.  While XTF is
>> useful for automation, it is first and foremost a tool for humans.
> You mean that this would contort the code for the test ?
>
> Here is another approach that could be used to satisfy osstest's
> desire for stability in the meaning of test names: you could
> explicitly copy the entire source of the test to a new test name, and
> deprecate the old test.  Later, if the old versions of Xen are fixed,
> or after they are retired you would delete the old test.
>
> The result would be that the old, less-good, test would still exist
> and still detect regressions.  The new, better test, would be seen to
> have never passed on old branches.
>
> Force pushing XTF is not an answer to this problem, for two reasons:
>
> Firstly, if the improved test correctly fails on old versions of Xen,
> this will appear as a regression in each old version of Xen.  Each old
> Xen tree would need to be force pushed, every time this happened.
>
> Secondly, in any case, force pushing XTF might not be necessary.  In
> the usual case, one would expect xen-unstable (which is what would
> drive the XTF push gate) to be fixed so that the improved test
> passes.  So osstest wouldn't spot the issue until it ran the new XTF
> with old Xen.
>
> Another possibility would be to have the capability for an single XTF
> test execution to return multiple statuses.

After an IRL discussion, the following has been proposed.

Each test can have a revision number associated with it.  The revision
gets bumped every time there is a change to the test which adds new
functionality (including bugfixes which cause it to spot a preexisting
error).

By default, a human will just want the latest and greatest.  Test
systems on the other hand will want to test "the same thing as before".

The XTF build system will support a way to build older revisions of the
tests.  It is likely that this will involve recompiling the current code
with an older revision id, and leave the onus on the programmer to
ensure that the result is functionally equivalent to the older test.

This way, test automation systems can internal treat different revisions
of the test as logically separate, for the purpose of identifying
regressions.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel