[PATCH RFC 0/9] Running benchmarks via OSSTest

[PATCH RFC 0/9] Running benchmarks via OSSTest

[Dario Faggioli]

Hello Everyone,

This is something I've been working in the latest few days, and it's about
making it possible to run benchmarks via OSSTest. I've (of course) tested it in
standalone mode, but nothing forbids to integrate it in our push gate, as we
were saying at the hackathon. Of course, there are a lot of things to be
decided. Where to run benchmarks, which benchmarks, how frequently, what to do
with the results, etc, but it's a start!  :-P

So, in standalone mode, after applying the patches, you can do the following,
and you'll have a PV guest running unixbench for a while, and the results
fetched for you on the OSSTest standalone controller stash area:

# Retrieve the benchmark archive
./mg-unixbench-download
# Reset, create the benchmarking flight, install the host and build Xen and Linux
./standalone-reset -t bench
./standalone run-job -R -h ghoul3 build-amd64   # consider --lvextendmax=XXGB, if necessary
./standalone run-job -h ghoul3 build-amd64-pvops
# Or set-paths --if build happened already-- like this:
#./standalone set-paths build-amd64
#./standalone set-paths build-amd64-pvops
# Install Xen on the host
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-xen-install
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-host-reboot
# Install (and start) the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-debian-install
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-debian-fixup debian
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-guest-start debian
# Prepare the benchmarking environment in the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-prep debian
# Run the benchmark in the guest
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-run debian
# Retrieve the benchmark results:
./standalone run-test -h ghoul3 bench-unixbench-amd64-amd64-xl-credit ts-unixbench-reslts debian

A few jobs are being created. In case you want to use one, instead of invoking
all the ts-s, run the following:

# Retrieve the benchmark archive
./mg-unixbench-download    
# Reset, create the benchmarking flight, install the host and build Xen and Linux
./standalone-reset -t bench
./standalone run-job -R -h ghoul3 build-amd64   # consider --lvextendmax=XXGB, if necessary
./standalone run-job -h ghoul3 build-amd64-pvops
# Run the benchmarking job:
./standalone run-job -h ghoul3 bench-unixbench-amd64-amd64-xl-credit2

Almost all the loginc to use an HVM guest are there, a just need a few fixups
in ts-debian-hvm-install that I'll submit as a separate patch.

I tested both the ways in standalone mode, and it seems to be working here.

And that's it... So, what do you think about it? IanC? IanJ?

I now it's still a sort of a prototype, but I'd love to hear your thoughts
before proceeding any further.

One thing about the last two patches. They are there to show how the code will
look like if we take a slightly different path than what's outlined in the
first part of the series. The idea is, in fact, to have the mg-xxx-download and
the ts-xxx-{prep,run,reslts} scripts to be more generic, accept one (or more)
benchmark name(s) as parameter and prepare the target (host or guest) for them.
That would make the single scripts slightly more complex, but will avoid having
to have 4 scripts (downloading, preparing, running and fetching results) for
each new benchmark we want to introduce support for.

Personally, I think I prefer them to be generic, i.e., something similar to
what they look like after all the series is applied, as compared to how they
look only until patch 7. This is exacly an example of something I'd like to
hear your thoughts about before going forward. :-)

The series is also available here:

 git://xenbits.xen.org/people/dariof/osstest.git benchmarking-with-osstest-RFC

Regards,
Dario
---
Dario Faggioli (9):
      mg-unixbench-download: new script for downloading the unixbench archive
      ts-unixbench-prep: prep the environment for running unixbench
      ts-unixbench-run: kick off the benchmark on the target
      ts-unixbench-reslts: for retrieving the results
      sg-run-job: new recipe for a unixbench job
      make-bench-flight: to create a benchmarking flight
      make-flight: introduce a new -t option
      mg-unixbench-download: make it generic
      ts-unixbench-prep: make it generic


 ap-common             |    8 ++++
 make-bench-flight     |   92 ++++++++++++++++++++++++++++++++++++++++++++++
 mg-unixbench-download |   49 ++++++++++++++++++++++++
 sg-run-job            |   28 ++++++++++++++
 standalone-reset      |    9 +++-
 ts-unixbench-prep     |   99 +++++++++++++++++++++++++++++++++++++++++++++++++
 ts-unixbench-reslts   |   56 ++++++++++++++++++++++++++++
 ts-unixbench-run      |   45 ++++++++++++++++++++++
 8 files changed, 383 insertions(+), 3 deletions(-)
 create mode 100755 make-bench-flight
 create mode 100755 mg-unixbench-download
 create mode 100755 ts-unixbench-prep
 create mode 100755 ts-unixbench-reslts
 create mode 100755 ts-unixbench-run

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-------------------------------------------------------------------
Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[PATCH RFC 1/9] mg-unixbench-download: new script for downloading the unixbench archive

[Dario Faggioli]

The script fetches it, and saves it in c{Images}/benchs.

Default values for URL and actual filename are in ap-common, and
can be overridden as usual.

 XXX: this may or may not be something suitable to do in a
      function in cri-common, called by ap-fetch-version. It's
      not a git tree, and it's not something that changes much
      so I think it's fine like it is in this patch. If it is
      not, advise otherwise and I'll cope.

igned-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 ap-common             |    4 ++++
 mg-unixbench-download |   34 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100755 mg-unixbench-download

diff --git a/ap-common b/ap-common
index ea4b04c..000ca69 100644
--- a/ap-common
+++ b/ap-common
@@ -136,3 +136,7 @@ info_linux_tree () {
 
 	return 0
 }
+
+# UnixBench. By default we try to grab version 5.1.3
+: ${REMOTEFILE_UNIXBENCH:=http://byte-unixbench.googlecode.com/files/unixbench-5.1.3.tgz}
+: ${LOCALFILE_UNIXBENCH:=unixbench.tgz}
diff --git a/mg-unixbench-download b/mg-unixbench-download
new file mode 100755
index 0000000..891b0b1
--- /dev/null
+++ b/mg-unixbench-download
@@ -0,0 +1,34 @@
+#!/bin/bash
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -e
+
+if [ -f standalone.config ] ; then
+    . standalone.config
+fi
+
+. cri-getconfig
+. ap-common
+
+fail () { echo >&2 "$0: $1"; exit 1; }
+
+images=`getconfig Images`;
+dstdir="${images}/benchs"
+mkdir -p $dstdir
+
+wget ${REMOTEFILE_UNIXBENCH} -O ${dstdir}/${LOCALFILE_UNIXBENCH} || \
+    fail "failed downloading the benchmark"

[PATCH RFC 2/9] ts-unixbench-prep: prep the environment for running unixbench

[Dario Faggioli]

by installing some dependencies, shipping the archive, untaring
and building the sources.

This accepts two parametrs, in the form 'host=somehost someguest',
as most of the ts-guest-xxx scripts. If only the first one is
provided, it must be 'host=somehost', and the script will prep
the host.

 XXX: I see it useful that this works for the host too, as
      at some point we may want to run benchmarks on dom0
      or even bare metal.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 ts-unixbench-prep |   79 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 79 insertions(+)
 create mode 100755 ts-unixbench-prep

diff --git a/ts-unixbench-prep b/ts-unixbench-prep
new file mode 100755
index 0000000..607d1a8
--- /dev/null
+++ b/ts-unixbench-prep
@@ -0,0 +1,79 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+use File::Basename;
+
+use feature 'switch';
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host=<somehost> [<someguest>]
+our $ho= selecthost($ARGV[0]); shift @ARGV;
+our $gho= $ho;
+if ( (0+@ARGV) == 1) {
+  $gho= selectguest($ARGV[0], $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+logm("Prep the environment for running unixbench");
+
+# packages needed to build and run
+target_install_packages_norec($gho, qw(build-essential libx11-dev
+                                       libgl1-mesa-dev libxext-dev));
+
+# Ship the benchmark to the target machine. By default, we expect to find
+# UnixBench 5.1.3, stored in $c{Images}/benchs/unixbench.tgz.
+# To use something different, define r{'unixbench_file'}.
+#
+# We also assume we're dealing with a tarball, and we try to figure out
+# whether or not it is compressed, and if yes, how. Check is rather basic,
+# relying only on the file extension.
+#
+my $unixbench_file= (defined($r{'unixbench_file'})) ? $r{'unixbench_file'} :
+    "$c{Images}/benchs/unixbench.tgz";
+target_putfile_root($gho, 60, "$unixbench_file", "/root");
+
+my @exts= qw(.tar .tgz .tbz2. .bz2 .gzip);
+my $extract_cmd= 'xf';
+my ($name, $dir, $ext) = fileparse($unixbench_file, @exts);
+
+given ($ext) {
+  $extract_cmd= 'xf' when ('.tar');
+  $extract_cmd= 'zxf' when ['.tgz', '.gzip'];
+  $extract_cmd= 'jxf' when  ['.tbz2', '.bz2'];
+  default { die "unrecognised file extension $ext" }
+}
+
+# Prepare the target, by installing dependencies, and build the benchmark
+target_install_packages_norec($gho, qw(build-essential libx11-dev
+                                       libgl1-mesa-dev libxext-dev
+                                       x11-apps));
+target_cmd_root($gho, <<END, 200);
+        set -ex
+        rm -rf /root/unixbench/
+        mkdir /root/unixbench
+        tar $extract_cmd /root/$name$ext -C /root/unixbench --strip-components=1
+        cd /root/unixbench
+        sed -e "s/^# GRAPHIC_TESTS =/GRAPHIC_TESTS =/" -i Makefile
+        make
+END

[PATCH RFC 3/9] ts-unixbench-run: kick off the benchmark on the target

[Dario Faggioli]

by simply running it, as it requires.

There is a runvar called 'unixbench_params', for specifying
the arguments.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 ts-unixbench-run |   45 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 45 insertions(+)
 create mode 100755 ts-unixbench-run

diff --git a/ts-unixbench-run b/ts-unixbench-run
new file mode 100755
index 0000000..dc81a73
--- /dev/null
+++ b/ts-unixbench-run
@@ -0,0 +1,45 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host=<somehost> [<someguest>]
+our $ho= selecthost($ARGV[0]); shift @ARGV;
+our $gho= $ho;
+if ( (0+@ARGV) == 1) {
+  $gho= selectguest($ARGV[0], $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+my $args= $r{ 'unixbench_params' };
+
+logm("Run unixbench with following parameters: $args");
+
+# XXX Arbitrary timeout...
+target_cmd_root($gho, <<END, 20000);
+        set -ex
+        cd /root/unixbench/
+        rm -rf results/*
+        time ./Run $args
+END

[PATCH RFC 4/9] ts-unixbench-reslts: for retrieving the results

[Dario Faggioli]

and store them in $c{Stash}/$flight/$job/results.

XXX: - I think $c{Stash}/<flight>/<job>/results is a fine
       place (very similar to what build jobs do, IMO). If
       a different location should be used, please advise.
     - This is, for now, only fetching the results. Of
       course we'd like to do something with them
       (printing, plotting, storing for long term usage
       or comparison, etc.). Let's have a discussion
       about this and I'll implementi it too

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 ts-unixbench-reslts |   56 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)
 create mode 100755 ts-unixbench-reslts

diff --git a/ts-unixbench-reslts b/ts-unixbench-reslts
new file mode 100755
index 0000000..37bcfc6
--- /dev/null
+++ b/ts-unixbench-reslts
@@ -0,0 +1,56 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use Osstest;
+use DBI;
+use IO::File;
+use POSIX;
+use Osstest::TestSupport;
+use File::Basename;
+
+tsreadconfig();
+
+# what we expect as argument list is:
+#  host=<somehost> [<someguest>]
+our $ho= selecthost($ARGV[0]); shift @ARGV;
+our $gho= $ho;
+if ( (0+@ARGV) == 1) {
+  $gho= selectguest($ARGV[0], $ho);
+  my $err= guest_check_ip($gho);
+  die "$err $gho->{Name}" if defined $err;
+}
+
+my $lresults_dir= "$stash/results";
+my $rresults_dir= "/root/unixbench/results";
+ensuredir($lresults_dir);
+
+my $resultspat= "$rresults_dir/$gho->{Guest}*-01*";
+
+my $rresfiles_str= target_cmd_output_root($gho, <<END);
+    chmod a+r $resultspat 2>&1 ||:
+    echo $resultspat
+END
+
+my @rresfiles= split(' ', $rresfiles_str);
+
+foreach (@rresfiles) {
+  my ($name, $dir, $ext) = fileparse($_, qr/\.[^.]*/);
+  target_getfile_root($gho, 60, "$_", "$lresults_dir/$gho->{Name}-unixbench$ext");
+}
+
+logm("results captured to $lresults_dir");

[PATCH RFC 5/9] sg-run-job: new recipe for a unixbench job

[Dario Faggioli]

Installs and run unixbench inside a PV and an HVM guest.

XXX - The HVM case needs that a few fixup are performed
      after ts-debian-hvm-install. I'll do that as a
      separate and independent patch.
    - This is mostly a proof of concept. I think the
      recipes it defines are actually useful, but of
      course we can discuss of what is the best course
      of actions, depending how we want to achieve.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 sg-run-job |   28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index e903869..36b6870 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -317,6 +317,34 @@ proc run-job/test-rumpuserxen {} {
     run-ts . =   ts-guest-destroy          + host $g
 }
 
+#-------- benchmarks --------
+
+proc bench-unixbench-guest {g} {
+    run-ts . = ts-unixbench-prep   + host $g
+    run-ts . = ts-unixbench-run    + host $g
+    run-ts . = ts-unixbench-reslts + host $g
+    run-ts . = ts-guest-stop       + host $g
+}
+
+proc bench-unixbench-pv {} {
+    run-ts . = ts-debian-install + host
+    run-ts . = ts-debian-fixup   + host debian
+    run-ts . = ts-guest-start    + host debian
+    bench-unixbench-guest debian
+}
+
+proc bench-unixbench-hvm {} {
+    run-ts . = ts-debian-hvm-install
+    bench-unixbench-guest debianhvm
+}
+
+proc need-hosts/bench-unixbench {} { return host }
+proc run-job/bench-unixbench {} {
+    bench-unixbench-pv
+    #bench-unixbench-hvm TODO: a few fixup needed, will submit as
+    #                          separate patches.
+}
+
 #---------- builds ----------
 
 proc need-hosts/build {} { return BUILD }

[PATCH RFC 6/9] make-bench-flight: to create a benchmarking flight

[Dario Faggioli]

instead of polluting make-flight.

Two bench-unixbench based jobs are created, one using
credit, the other credit2.

XXX - Wheter or not to create a new script was discussed
      briefly with IanC (the last time I was in Cambridge).
    - As for the previous patch, this is mostly a proof of
      concept... We can do everything here, let's discuss
      about it!

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 make-bench-flight |   92 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 92 insertions(+)
 create mode 100755 make-bench-flight

diff --git a/make-bench-flight b/make-bench-flight
new file mode 100755
index 0000000..4dd69cc
--- /dev/null
+++ b/make-bench-flight
@@ -0,0 +1,92 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2013 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. ap-common
+. cri-common
+. mfi-common
+
+defsuite=`getconfig DebianSuite`
+defguestsuite=`getconfig GuestDebianSuite`
+
+if [ x$buildflight = x ]; then
+
+  if [ "x$BUILD_LVEXTEND_MAX" != x ]; then
+     BUILD_RUNVARS+=" build_lvextend_max=$BUILD_LVEXTEND_MAX "
+  fi
+
+  create_build_jobs
+
+else
+
+  bfi=$buildflight.
+
+fi
+
+job_create_test_filter_callback () {
+    :
+}
+
+test_matrix_branch_filter_callback () {
+    :
+}
+
+do_unixbench_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 ]; then
+    return
+  fi
+
+  for sched in credit credit2; do
+    job_create_test bench-unixbench-$xenarch$kern-$dom0arch-xl-${sched}        \
+      bench-unixbench xl $xenarch $dom0arch guests_vcpus=4                     \
+      xen_boot_append="sched=${sched}" unixbench_params='-i 5 -c 1 -c 2 -c 4'  \
+      $debian_runvars bios=seabios debianhvm_image=debian-7.2.0-amd64-CD-1.iso \
+      all_hostflags=$most_hostflags
+  done
+}
+
+test_matrix_do_one () {
+  case ${xenarch} in
+  amd64) domUarches="amd64";;
+  i386)  domUarches="";;
+  armhf) domUarches="armhf";;
+  esac
+
+  for domU in $domUarches ; do
+      do_unixbench_tests
+  done
+}
+
+test_matrix_iterate
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:

[PATCH RFC 7/9] make-flight: introduce a new -t option

[Dario Faggioli]

to make it possible to call the new make-bench-flight,
and generate the benchmarking jobs. It's best combined
with -f, so that one can create a benchmarking flight
with all the benchmarking jobs.

It is generic, so, when passing '-t sometype', a script
called make-sometype-flight is what will be invoked.

XXX Just FTR, this was also part of that conversation
    with IanC.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 standalone-reset |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/standalone-reset b/standalone-reset
index 8555039..f041e6d 100755
--- a/standalone-reset
+++ b/standalone-reset
@@ -23,14 +23,17 @@ usage(){
 usage: ./standalone-reset [<options>] [<branch> [<xenbranch> [<buildflight>]]]
  branch and xenbranch default, separately, to xen-unstable
 options:
- -f<flight>     generate flight "flight", default is "standalone"
+ -f <flight>      generate flight "flight", default is "standalone"
+ -t <flight_type> generate a different type of flight (it calls
+                  make-<flight_type>-flight)
 END
 }
 
 flight="standalone"
-while getopts "f:" opt; do
+while getopts "f:t:" opt; do
     case "$opt" in
         f) flight=${OPTARG};;
+        t) flight_type="-${OPTARG}";;
         *) usage; exit 1;;
     esac
 done
@@ -140,7 +143,7 @@ fi
 export BUILD_LVEXTEND_MAX
 
 OSSTEST_FLIGHT=$flight \
-./make-flight "$branch" "$xenbranch" play $buildflight >/dev/null
+./make${flight_type}-flight "$branch" "$xenbranch" play $buildflight >/dev/null
 
 #---------- done ----------

[PATCH RFC 8/9] mg-unixbench-download: make it generic

[Dario Faggioli]

so that it can handle the retrieval of different
benchmarks. The name of the benchmark to fetch should
be among the supported one, and is passed as an argument.

XXX This is a stab. As detailed in the cover letter, I
    just wanted to give an idea of how things would look
    like, if we take this path. I personally prefer it
    like this, but let's talk about it.

    Of course, if we decide to go this way, the name
    will be changed from mg-unixbench-download to
    mg-bench-download, and I'll structure the file
    like this from the beginning.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 ap-common             |    4 ++++
 mg-unixbench-download |   19 +++++++++++++++++--
 2 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/ap-common b/ap-common
index 000ca69..ff0f205 100644
--- a/ap-common
+++ b/ap-common
@@ -140,3 +140,7 @@ info_linux_tree () {
 # UnixBench. By default we try to grab version 5.1.3
 : ${REMOTEFILE_UNIXBENCH:=http://byte-unixbench.googlecode.com/files/unixbench-5.1.3.tgz}
 : ${LOCALFILE_UNIXBENCH:=unixbench.tgz}
+
+# hackbench. By default, directly from mingo page!
+: ${REMOTEFILE_HACKBENCH:=http://people.redhat.com/mingo/cfs-scheduler/tools/hackbench.c}
+: ${LOCALFILE_HACKBENCH:=hackbench.c}
diff --git a/mg-unixbench-download b/mg-unixbench-download
index 891b0b1..c4d5ab3 100755
--- a/mg-unixbench-download
+++ b/mg-unixbench-download
@@ -30,5 +30,20 @@ images=`getconfig Images`;
 dstdir="${images}/benchs"
 mkdir -p $dstdir
 
-wget ${REMOTEFILE_UNIXBENCH} -O ${dstdir}/${LOCALFILE_UNIXBENCH} || \
-    fail "failed downloading the benchmark"
+for bench in "$@"; do
+  case $bench in
+    unixbench)
+      bench_url=${REMOTEFILE_UNIXBENCH}
+      bench_dst=${dstdir}/${LOCALFILE_UNIXBENCH}
+      ;;
+    hackbench)
+      bench_url=${REMOTEFILE_HACKBENCH}
+      bench_dst=${dstdir}/${LOCALFILE_HACKBENCH}
+      ;;
+    *)
+      fail "unknown benchmark $bench"
+      ;;
+    esac
+
+    wget ${bench_url} -O ${bench_dst} || fail "failed downloading ${bench}"
+done

[PATCH RFC 9/9] ts-unixbench-prep: make it generic

[Dario Faggioli]

to allow dealing with preparation of multiple benchmarks.

XXX As for the previous patch, this is only to show how
    the code will look if we go this way.

Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
---
 ts-unixbench-prep |   94 ++++++++++++++++++++++++++++++++---------------------
 1 file changed, 57 insertions(+), 37 deletions(-)

diff --git a/ts-unixbench-prep b/ts-unixbench-prep
index 607d1a8..f0258ab 100755
--- a/ts-unixbench-prep
+++ b/ts-unixbench-prep
@@ -26,7 +26,10 @@ use feature 'switch';
 tsreadconfig();
 
 # what we expect as argument list is:
-#  host=<somehost> [<someguest>]
+#  host=<somehost> [<someguest>] <benchname>
+@ARGV= reverse @ARGV;
+our $benchname= $ARGV[0]; shift @ARGV;
+@ARGV= reverse @ARGV;
 our $ho= selecthost($ARGV[0]); shift @ARGV;
 our $gho= $ho;
 if ( (0+@ARGV) == 1) {
@@ -35,45 +38,62 @@ if ( (0+@ARGV) == 1) {
   die "$err $gho->{Name}" if defined $err;
 }
 
-logm("Prep the environment for running unixbench");
+# In general, we know what benchmarks comes in archives. We assume them to be
+# tarballs, and we try to figure out whether or not it is compressed, and if
+# yes, how. Check is rather basic, relying only on the file extension.
+sub archive_extract_cmd($) {
+  my $benchfile_path= @_;
+  my @exts= qw(.tar .tgz .tbz2. .bz2 .gzip);
+  my ($name, $dir, $ext) = fileparse($benchfile_path, @exts);
+  my $cmd;
 
-# packages needed to build and run
-target_install_packages_norec($gho, qw(build-essential libx11-dev
-                                       libgl1-mesa-dev libxext-dev));
+  given ($ext) {
+    $cmd= 'xf' when ('.tar');
+    $cmd= 'zxf' when ['.tgz', '.gzip'];
+    $cmd= 'jxf' when  ['.tbz2', '.bz2'];
+    default { die "unrecognised file extension $ext" }
+  }
+  return $cmd
+}
 
-# Ship the benchmark to the target machine. By default, we expect to find
-# UnixBench 5.1.3, stored in $c{Images}/benchs/unixbench.tgz.
-# To use something different, define r{'unixbench_file'}.
-#
-# We also assume we're dealing with a tarball, and we try to figure out
-# whether or not it is compressed, and if yes, how. Check is rather basic,
-# relying only on the file extension.
-#
-my $unixbench_file= (defined($r{'unixbench_file'})) ? $r{'unixbench_file'} :
-    "$c{Images}/benchs/unixbench.tgz";
-target_putfile_root($gho, 60, "$unixbench_file", "/root");
+# Ship the benchmark to the target machine and prepare the environment
+logm("Prep the environment for running $benchname");
 
-my @exts= qw(.tar .tgz .tbz2. .bz2 .gzip);
-my $extract_cmd= 'xf';
-my ($name, $dir, $ext) = fileparse($unixbench_file, @exts);
+if ($benchname eq "unixbench") {
+  # By default, we expect to find UnixBench 5.1.3, stored in
+  # $c{Images}/benchs/unixbench.tgz. To use something different,
+  # define r{'unixbench_file'}.
+  my $unixbench_file= (defined($r{'unixbench_file'})) ? $r{'unixbench_file'} :
+      "$c{Images}/benchs/unixbench.tgz";
+  target_putfile_root($gho, 60, "$unixbench_file", "/root");
 
-given ($ext) {
-  $extract_cmd= 'xf' when ('.tar');
-  $extract_cmd= 'zxf' when ['.tgz', '.gzip'];
-  $extract_cmd= 'jxf' when  ['.tbz2', '.bz2'];
-  default { die "unrecognised file extension $ext" }
-}
+  my $extract_cmd= archive_extract_cmd($unixbench_file);
+
+  # Packages needed to build and run
+  target_install_packages_norec($gho, qw(build-essential libx11-dev
+                                         libgl1-mesa-dev libxext-dev));
 
-# Prepare the target, by installing dependencies, and build the benchmark
-target_install_packages_norec($gho, qw(build-essential libx11-dev
-                                       libgl1-mesa-dev libxext-dev
-                                       x11-apps));
-target_cmd_root($gho, <<END, 200);
-        set -ex
-        rm -rf /root/unixbench/
-        mkdir /root/unixbench
-        tar $extract_cmd /root/$name$ext -C /root/unixbench --strip-components=1
-        cd /root/unixbench
-        sed -e "s/^# GRAPHIC_TESTS =/GRAPHIC_TESTS =/" -i Makefile
-        make
+  target_cmd_root($gho, <<END, 200);
+          set -ex
+          rm -rf /root/unixbench/
+          mkdir /root/unixbench
+          tar $extract_cmd /root/$name$ext -C /root/unixbench --strip-components=1
+          cd /root/unixbench
+          sed -e "s/^# GRAPHIC_TESTS =/GRAPHIC_TESTS =/" -i Makefile
+          make
 END
+} elsif ($benchname eq "hackbench") {
+  # In case of hackbench, it is just a .c file.
+  my $hackbench_file= (defined($r{'hackbench_file'})) ? $r{'hackbench_file'} :
+      "$c{Images}/benchs/hackbench.c";
+  target_putfile_root($gho, 60, "$hackbench_file", "/root");
+
+  # Packages needed to build and run
+  target_install_packages_norec($gho, qw(build-essential));
+
+  target_cmd_root($gho, <<END, 200);
+          set -ex
+          rm -f /root/hackbench
+          gcc -g -Wall -O2 -o /root/hackbench /root/hackbench.c -lpthread
+END
+}

Re: [PATCH RFC 9/9] ts-unixbench-prep: make it generic

[Dario Faggioli]

[-- Attachment #1.1.1: Type: text/plain, Size: 1122 bytes --]

On gio, 2014-06-26 at 15:03 +0200, Dario Faggioli wrote:
> to allow dealing with preparation of multiple benchmarks.
> 
> XXX As for the previous patch, this is only to show how
>     the code will look if we go this way.
> 
Not only it's a stub, it also has a bug in it! :-P

Attached a correct version (and I also updated the branch so that, if
you clone it, everything should work fine).

 git://xenbits.xen.org/people/dariof/osstest.git benchmarking-with-osstest-RFC

Hopefully, this does not affect too much the review of the series, given
the nature of this very patch.

In summary, if you want to try and see how the thing works, either apply
only patches 1-7, or, if you want to apply 9 as well, use the version
attached to this email. Or just clone the git repo (it has the fixed
version patch 9/9).

Sorry and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.1.2: turn-ts-unixbench-into-generic.patch --]
[-- Type: text/x-patch, Size: 5106 bytes --]

commit d61f362e934421a74b2b7238035af3fce6ac307b
Author: Dario Faggioli <dario.faggioli@citrix.com>
Date:   Wed Jun 25 16:55:56 2014 +0200

    ts-unixbench-prep: make it generic
    
    to allow dealing with preparation of multiple benchmarks.
    
    XXX As for the previous patch, this is only to show how
        the code will look if we go this way.
    
    Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>

diff --git a/ts-unixbench-prep b/ts-unixbench-prep
index 607d1a8..55816b7 100755
--- a/ts-unixbench-prep
+++ b/ts-unixbench-prep
@@ -26,7 +26,10 @@ use feature 'switch';
 tsreadconfig();
 
 # what we expect as argument list is:
-#  host=<somehost> [<someguest>]
+#  host=<somehost> [<someguest>] <benchname>
+@ARGV= reverse @ARGV;
+our $benchname= $ARGV[0]; shift @ARGV;
+@ARGV= reverse @ARGV;
 our $ho= selecthost($ARGV[0]); shift @ARGV;
 our $gho= $ho;
 if ( (0+@ARGV) == 1) {
@@ -35,45 +38,62 @@ if ( (0+@ARGV) == 1) {
   die "$err $gho->{Name}" if defined $err;
 }
 
-logm("Prep the environment for running unixbench");
+# In general, we know what benchmarks comes in archives. We assume them to be
+# tarballs, and we try to figure out whether or not it is compressed, and if
+# yes, how. Check is rather basic, relying only on the file extension.
+sub archive_extract_cmd($) {
+  my $benchfile_path= "@_";
+  my @exts= qw(.tar .tgz .tbz2. .bz2 .gzip);
+  my ($name, $dir, $ext) = fileparse($benchfile_path, @exts);
+  my $cmd;
 
-# packages needed to build and run
-target_install_packages_norec($gho, qw(build-essential libx11-dev
-                                       libgl1-mesa-dev libxext-dev));
+  given ($ext) {
+    $cmd= 'xf' when ('.tar');
+    $cmd= 'zxf' when ['.tgz', '.gzip'];
+    $cmd= 'jxf' when  ['.tbz2', '.bz2'];
+    default { die "unrecognised file extension $ext" }
+  }
+  return ($cmd, $name, $ext);
+}
 
-# Ship the benchmark to the target machine. By default, we expect to find
-# UnixBench 5.1.3, stored in $c{Images}/benchs/unixbench.tgz.
-# To use something different, define r{'unixbench_file'}.
-#
-# We also assume we're dealing with a tarball, and we try to figure out
-# whether or not it is compressed, and if yes, how. Check is rather basic,
-# relying only on the file extension.
-#
-my $unixbench_file= (defined($r{'unixbench_file'})) ? $r{'unixbench_file'} :
-    "$c{Images}/benchs/unixbench.tgz";
-target_putfile_root($gho, 60, "$unixbench_file", "/root");
+# Ship the benchmark to the target machine and prepare the environment
+logm("Prep the environment for running $benchname");
 
-my @exts= qw(.tar .tgz .tbz2. .bz2 .gzip);
-my $extract_cmd= 'xf';
-my ($name, $dir, $ext) = fileparse($unixbench_file, @exts);
+if ($benchname eq "unixbench") {
+  # By default, we expect to find UnixBench 5.1.3, stored in
+  # $c{Images}/benchs/unixbench.tgz. To use something different,
+  # define r{'unixbench_file'}.
+  my $unixbench_file= (defined($r{'unixbench_file'})) ? $r{'unixbench_file'} :
+      "$c{Images}/benchs/unixbench.tgz";
+  target_putfile_root($gho, 60, "$unixbench_file", "/root");
 
-given ($ext) {
-  $extract_cmd= 'xf' when ('.tar');
-  $extract_cmd= 'zxf' when ['.tgz', '.gzip'];
-  $extract_cmd= 'jxf' when  ['.tbz2', '.bz2'];
-  default { die "unrecognised file extension $ext" }
-}
+  my ($extract_cmd, $name, $ext)= archive_extract_cmd($unixbench_file);
+
+  # Packages needed to build and run
+  target_install_packages_norec($gho, qw(build-essential libx11-dev
+                                         libgl1-mesa-dev libxext-dev));
 
-# Prepare the target, by installing dependencies, and build the benchmark
-target_install_packages_norec($gho, qw(build-essential libx11-dev
-                                       libgl1-mesa-dev libxext-dev
-                                       x11-apps));
-target_cmd_root($gho, <<END, 200);
-        set -ex
-        rm -rf /root/unixbench/
-        mkdir /root/unixbench
-        tar $extract_cmd /root/$name$ext -C /root/unixbench --strip-components=1
-        cd /root/unixbench
-        sed -e "s/^# GRAPHIC_TESTS =/GRAPHIC_TESTS =/" -i Makefile
-        make
+  target_cmd_root($gho, <<END, 200);
+          set -ex
+          rm -rf /root/unixbench/
+          mkdir /root/unixbench
+          tar $extract_cmd /root/$name$ext -C /root/unixbench --strip-components=1
+          cd /root/unixbench
+          sed -e "s/^# GRAPHIC_TESTS =/GRAPHIC_TESTS =/" -i Makefile
+          make
 END
+} elsif ($benchname eq "hackbench") {
+  # In case of hackbench, it is just a .c file.
+  my $hackbench_file= (defined($r{'hackbench_file'})) ? $r{'hackbench_file'} :
+      "$c{Images}/benchs/hackbench.c";
+  target_putfile_root($gho, 60, "$hackbench_file", "/root");
+
+  # Packages needed to build and run
+  target_install_packages_norec($gho, qw(build-essential));
+
+  target_cmd_root($gho, <<END, 200);
+          set -ex
+          rm -f /root/hackbench
+          gcc -g -Wall -O2 -o /root/hackbench /root/hackbench.c -lpthread
+END
+}

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH RFC 1/9] mg-unixbench-download: new script for downloading the unixbench archive

[Ian Campbell]

On Thu, 2014-06-26 at 15:02 +0200, Dario Faggioli wrote:
> The script fetches it, and saves it in c{Images}/benchs.
> 
> Default values for URL and actual filename are in ap-common, and
> can be overridden as usual.

ap-* are to do with the pushing machinery, so I'm not sure this is the
right place for it.

I think it would be OK to hardcode the URL into the script and put the
version number in the config file (e.g. production-config). Perhaps with
a default (in the script).

>  XXX: this may or may not be something suitable to do in a
>       function in cri-common, called by ap-fetch-version. It's
>       not a git tree, and it's not something that changes much
>       so I think it's fine like it is in this patch. If it is
>       not, advise otherwise and I'll cope.

The usual approach would be to have a setup script (as you have done
here) to populate an output directory rather than to run the dl on every
run (as you propose in this comment, I think).

All ones for Ian J though really.

Ian

Re: [PATCH RFC 2/9] ts-unixbench-prep: prep the environment for running unixbench

[Ian Campbell]

On Thu, 2014-06-26 at 15:02 +0200, Dario Faggioli wrote:
> by installing some dependencies,

this isn't currently possible due to the (possibly futuristic) potential
to share build hosts. ts-xen-build-prep is special in this regard
(AIUI). I don't see you installing anything which would be so
intolerable to install on every machine.

I have an old patch "apt: lock osstest's usages of apt-get against each
other" which is supposed to help with this, but I need to revisit and
fix some comments from Ian I think.

>  shipping the archive, untaring
> and building the sources.

ts-unixbench-build would be the more conventional name I think.

> This accepts two parametrs, in the form 'host=somehost someguest',
> as most of the ts-guest-xxx scripts. If only the first one is
> provided, it must be 'host=somehost', and the script will prep
> the host.
> 
>  XXX: I see it useful that this works for the host too, as
>       at some point we may want to run benchmarks on dom0
>       or even bare metal.

Absolutely. I realise now that the comments about apt install above is
only really relevant to the host case.

> Signed-off-by: Dario Faggioli <dario.faggioli@citrix.com>
> ---
>  ts-unixbench-prep |   79 +++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 79 insertions(+)
>  create mode 100755 ts-unixbench-prep
> 
> diff --git a/ts-unixbench-prep b/ts-unixbench-prep
> new file mode 100755
> index 0000000..607d1a8
> --- /dev/null
> +++ b/ts-unixbench-prep
> @@ -0,0 +1,79 @@
> +#!/usr/bin/perl -w
> +# This is part of "osstest", an automated testing framework for Xen.
> +# Copyright (C) 2009-2013 Citrix Inc.

Out of date.

> +tsreadconfig();
> +
> +# what we expect as argument list is:
> +#  host=<somehost> [<someguest>]
> +our $ho= selecthost($ARGV[0]); shift @ARGV;

IIRC Ian just made select* consume ARGV, so the shift wouldn't be needed
any more.

> +our $gho= $ho;
> +if ( (0+@ARGV) == 1) {

Is the 0+ needed, I thought arrays in scalar context evaluated to their
length already without forcing it like that.

> +given ($ext) {
> +  $extract_cmd= 'xf' when ('.tar');
> +  $extract_cmd= 'zxf' when ['.tgz', '.gzip'];
> +  $extract_cmd= 'jxf' when  ['.tbz2', '.bz2'];
> +  default { die "unrecognised file extension $ext" }

I think "xaf" will automatically handle different compression algs. I'm
just not quite sure what it does if the tarball is uncompressed.

> +}
> +
> +# Prepare the target, by installing dependencies, and build the benchmark
> +target_install_packages_norec($gho, qw(build-essential libx11-dev
> +                                       libgl1-mesa-dev libxext-dev
> +                                       x11-apps));
> +target_cmd_root($gho, <<END, 200);
> +        set -ex
> +        rm -rf /root/unixbench/

Ideally you would use the osstest user. I'm not sure if that is set up
for guests though, might be host only.

> +        mkdir /root/unixbench
> +        tar $extract_cmd /root/$name$ext -C /root/unixbench --strip-components=1
> +        cd /root/unixbench
> +        sed -e "s/^# GRAPHIC_TESTS =/GRAPHIC_TESTS =/" -i Makefile
> +        make
> +END

Ian.

Re: [PATCH RFC 4/9] ts-unixbench-reslts: for retrieving the results

[Ian Campbell]

On Thu, 2014-06-26 at 15:02 +0200, Dario Faggioli wrote:

> +foreach (@rresfiles) {
> +  my ($name, $dir, $ext) = fileparse($_, qr/\.[^.]*/);
> +  target_getfile_root($gho, 60, "$_", "$lresults_dir/$gho->{Name}-unixbench$ext");

There's a general need for a target_getfile_(root_)stash I think, as an
analogue to e.g. target_putfilecontents_stash.

> +}
> +
> +logm("results captured to $lresults_dir");
> 

Re: [PATCH RFC 2/9] ts-unixbench-prep: prep the environment for running unixbench

[Ian Jackson]

Ian Campbell writes ("Re: [PATCH RFC 2/9] ts-unixbench-prep: prep the environment for running unixbench"):
> On Thu, 2014-06-26 at 15:02 +0200, Dario Faggioli wrote:
> > by installing some dependencies,
> 
> this isn't currently possible due to the (possibly futuristic) potential
> to share build hosts. ts-xen-build-prep is special in this regard
> (AIUI). I don't see you installing anything which would be so
> intolerable to install on every machine.

We do actually share build hosts right now.  I think you meant share
test hosts.

> I have an old patch "apt: lock osstest's usages of apt-get against each
> other" which is supposed to help with this, but I need to revisit and
> fix some comments from Ian I think.

That's what I remember.


I'll try to look at the rest of Darios series RSN.

Ian.

Re: [PATCH RFC 7/9] make-flight: introduce a new -t option

[Ian Campbell]

On Thu, 2014-06-26 at 15:03 +0200, Dario Faggioli wrote:
> to make it possible to call the new make-bench-flight,
> and generate the benchmarking jobs. It's best combined
> with -f, so that one can create a benchmarking flight
> with all the benchmarking jobs.
> 
> It is generic, so, when passing '-t sometype', a script
> called make-sometype-flight is what will be invoked.
> 
> XXX Just FTR, this was also part of that conversation
>     with IanC.

WFM. Perhaps you'd like to plumb this into "standalone make-flight" too?

Ian.

Re: [PATCH RFC 7/9] make-flight: introduce a new -t option

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 927 bytes --]

On mer, 2014-07-16 at 16:13 +0100, Ian Campbell wrote:
> On Thu, 2014-06-26 at 15:03 +0200, Dario Faggioli wrote:
> > to make it possible to call the new make-bench-flight,
> > and generate the benchmarking jobs. It's best combined
> > with -f, so that one can create a benchmarking flight
> > with all the benchmarking jobs.
> > 
> > It is generic, so, when passing '-t sometype', a script
> > called make-sometype-flight is what will be invoked.
> > 
> > XXX Just FTR, this was also part of that conversation
> >     with IanC.
> 
> WFM. Perhaps you'd like to plumb this into "standalone make-flight" too?
> 
Sure!

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH RFC 2/9] ts-unixbench-prep: prep the environment for running unixbench

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2566 bytes --]

On mer, 2014-07-16 at 16:09 +0100, Ian Campbell wrote:
> On Thu, 2014-06-26 at 15:02 +0200, Dario Faggioli wrote:
> > by installing some dependencies,
> 
> this isn't currently possible due to the (possibly futuristic) potential
> to share build hosts. ts-xen-build-prep is special in this regard
> (AIUI). I don't see you installing anything which would be so
> intolerable to install on every machine.
> 
> I have an old patch "apt: lock osstest's usages of apt-get against each
> other" which is supposed to help with this, but I need to revisit and
> fix some comments from Ian I think.
> 
Mm... I'd be fine with installing everything on every test host.
However, as we grow the number of supported benchmark, dependencies will
also grow. E.g., if we want to run SpecJBB, we'll need JVM stuff. If we
want to run even a subset of Phoronix test suite, we'll need a lot of
stuff, I think... Is this still acceptable?

> >  shipping the archive, untaring
> > and building the sources.
> 
> ts-unixbench-build would be the more conventional name I think.
> 
Indeed. Fact is, there are benchmarks that require building (e.g., this
one), and benchmark that does not (e.g., kernbench).

I thought it would be good to have all the prepping script to have a
similar name (if not to include all the prep work in just one
ts-bench-prep that takes the benchmark name as a parameter! Look at the
very last patch of the series).

That's the reason for the 'prep' instead than 'build'. What do you
think?

> > This accepts two parametrs, in the form 'host=somehost someguest',
> > as most of the ts-guest-xxx scripts. If only the first one is
> > provided, it must be 'host=somehost', and the script will prep
> > the host.
> > 
> >  XXX: I see it useful that this works for the host too, as
> >       at some point we may want to run benchmarks on dom0
> >       or even bare metal.
> 
> Absolutely. I realise now that the comments about apt install above is
> only really relevant to the host case.
> 
Which would make the 'path' for running benches on host and guest a bit
different, but that's certainly fine. If we're installing *everything*
on hosts, I've got no problem making this prepping scripts available for
guests only.

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH RFC 4/9] ts-unixbench-reslts: for retrieving the results

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 776 bytes --]

On mer, 2014-07-16 at 16:11 +0100, Ian Campbell wrote:
> On Thu, 2014-06-26 at 15:02 +0200, Dario Faggioli wrote:
> 
> > +foreach (@rresfiles) {
> > +  my ($name, $dir, $ext) = fileparse($_, qr/\.[^.]*/);
> > +  target_getfile_root($gho, 60, "$_", "$lresults_dir/$gho->{Name}-unixbench$ext");
> 
> There's a general need for a target_getfile_(root_)stash I think, as an
> analogue to e.g. target_putfilecontents_stash.
> 
I certainly can look into implementing that one.

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH RFC 2/9] ts-unixbench-prep: prep the environment for running unixbench

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3068 bytes --]

On mer, 2014-07-16 at 16:09 +0100, Ian Campbell wrote:
> > ---
> >  ts-unixbench-prep |   79 +++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 79 insertions(+)
> >  create mode 100755 ts-unixbench-prep
> > 
> > diff --git a/ts-unixbench-prep b/ts-unixbench-prep
> > new file mode 100755
> > index 0000000..607d1a8
> > --- /dev/null
> > +++ b/ts-unixbench-prep
> > @@ -0,0 +1,79 @@
> > +#!/usr/bin/perl -w
> > +# This is part of "osstest", an automated testing framework for Xen.
> > +# Copyright (C) 2009-2013 Citrix Inc.
> 
> Out of date.
> 
Oops. :-)

> > +tsreadconfig();
> > +
> > +# what we expect as argument list is:
> > +#  host=<somehost> [<someguest>]
> > +our $ho= selecthost($ARGV[0]); shift @ARGV;
> 
> IIRC Ian just made select* consume ARGV, so the shift wouldn't be needed
> any more.
> 
Yeah, I saw it. I'll build on top of that as far as it will hit the
repo.

> > +our $gho= $ho;
> > +if ( (0+@ARGV) == 1) {
> 
> Is the 0+ needed, I thought arrays in scalar context evaluated to their
> length already without forcing it like that.
> 
Ehm, my Perl-fu is still very poor. ISTR trying and having some issues,
but yeah, I'll give it another shoot at doing things properly. Sorry :-)

> > +given ($ext) {
> > +  $extract_cmd= 'xf' when ('.tar');
> > +  $extract_cmd= 'zxf' when ['.tgz', '.gzip'];
> > +  $extract_cmd= 'jxf' when  ['.tbz2', '.bz2'];
> > +  default { die "unrecognised file extension $ext" }
> 
> I think "xaf" will automatically handle different compression algs. I'm
> just not quite sure what it does if the tarball is uncompressed.
> 
Oh, I did not know that, I'll give it a try. About the uncompressed
tarball case, I don't think we should worry much about being too
flexible here. Perhaps what I was trying to do above is already too
much. I mean, we expect the file to have a specific name, and to be in a
specific place already, we can well ask for it to be compressed, and
bail if it is not. What do you think?

> > +}
> > +
> > +# Prepare the target, by installing dependencies, and build the benchmark
> > +target_install_packages_norec($gho, qw(build-essential libx11-dev
> > +                                       libgl1-mesa-dev libxext-dev
> > +                                       x11-apps));
> > +target_cmd_root($gho, <<END, 200);
> > +        set -ex
> > +        rm -rf /root/unixbench/
> 
> Ideally you would use the osstest user. I'm not sure if that is set up
> for guests though, might be host only.
> 
I don't recall that either, and I'll check. There are benchmarks,
though, that wants to run as root... I guess that is not a big deal, at
least for guests, is it? For hosts, I see it could be a bigger issue...

Thanks again and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH RFC 1/9] mg-unixbench-download: new script for downloading the unixbench archive

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1489 bytes --]

On mer, 2014-07-16 at 16:02 +0100, Ian Campbell wrote:
> On Thu, 2014-06-26 at 15:02 +0200, Dario Faggioli wrote:
> > The script fetches it, and saves it in c{Images}/benchs.
> > 
> > Default values for URL and actual filename are in ap-common, and
> > can be overridden as usual.
> 
> ap-* are to do with the pushing machinery, so I'm not sure this is the
> right place for it.
> 
Right. Sorry then, I must have missed this.

> I think it would be OK to hardcode the URL into the script and put the
> version number in the config file (e.g. production-config). Perhaps with
> a default (in the script).
> 
I'll do it this way.

> >  XXX: this may or may not be something suitable to do in a
> >       function in cri-common, called by ap-fetch-version. It's
> >       not a git tree, and it's not something that changes much
> >       so I think it's fine like it is in this patch. If it is
> >       not, advise otherwise and I'll cope.
> 
> The usual approach would be to have a setup script (as you have done
> here) to populate an output directory rather than to run the dl on every
> run (as you propose in this comment, I think).
> 
Great, less (re)work! :-)

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel