Re: [PATCH v3] OSSTEST: introduce a raisin build test

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-05-08 at 13:36 +0100, Stefano Stabellini wrote:
> Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> 
> ---
> Changes in v3:
> - use $raisindir throughout ts-raisin-build
> - do not specify ENABLED_COMPONENTS so that empty REVISION variables can
> be used to disable building a raisin component

Is this existing raisin behaviour, or is there a patch pending for that?

> 
> Changes in v2:
> - set revision_* variables in mfi-common;
> - in ts-raisin-build set the *_REVISION config options based on the
>   revision_* variables;
> - in ts-raisin-build, call store_revision appropriately;
> - divide the output in an hypervisor and a tools tarball.
> ---
>  ap-common       |    5 ++
>  mfi-common      |   25 +++++++++
>  sg-run-job      |    5 ++
>  ts-raisin-build |  161 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 196 insertions(+)
>  create mode 100755 ts-raisin-build
> 
> diff --git a/ap-common b/ap-common
> index 64749e3..985eeec 100644
> --- a/ap-common
> +++ b/ap-common
> @@ -47,13 +47,18 @@
>  # rumpsrc-related runvars needed only for old rumpuser-xen
>  # (ie ones which need $bodges=1 in ts-rumpuserxen-build)
>  
> +: ${TREE_RAISIN:=git://xenbits.xen.org/people/sstabellini/raisin.git}
> +: ${DEFAULT_REVISION_RAISIN:=master}
> +
>  : ${TREE_SEABIOS_UPSTREAM:=git://git.seabios.org/seabios.git}
>  : ${PUSH_TREE_SEABIOS:=$XENBITS:/home/xen/git/osstest/seabios.git}
>  : ${BASE_TREE_SEABIOS:=git://xenbits.xen.org/osstest/seabios.git}
> +: ${TREE_SEABIOS:=$TREE_SEABIOS_UPSTREAM}
>  
>  : ${TREE_OVMF_UPSTREAM:=https://github.com/tianocore/edk2.git}
>  : ${PUSH_TREE_OVMF:=$XENBITS:/home/xen/git/osstest/ovmf.git}
>  : ${BASE_TREE_OVMF:=git://xenbits.xen.org/osstest/ovmf.git}
> +: ${TREE_OVMF:=$TREE_OVMF_UPSTREAM}

What are these two doing? If it's something legitimate then it really
needs to be mentioned in the commit log, but using
git://git.seabios.org/seabios.git here seems wrong, and I think we
change things for older branches and the existing ts-xen-build job.

I have a feeling that these relate to trees which we previously allowed
xen.git:Config.mk to choose, which means we need to have a little think
about the behaviour of cr-daily-branch vs this script here.

Perhaps you should do for TREE_{SEABIOS,OVMF} as you have done for the
revision, i.e. use ${TREE_SEABIOS:-<SOMEDEFAULT>}

Where <SOMEDEFAULT> is whatever is in xen.git Config.mk today.

Or maybe raisin has some way to say "whatever you think is best"?

The question here is, I suppose, where should the replacement for the
Config.mk supplied defaults live now, in osstest.git or raisin.git?

>  : ${TREE_LINUXFIRMWARE:=git://xenbits.xen.org/osstest/linux-firmware.git}
>  : ${PUSH_TREE_LINUXFIRMWARE:=$XENBITS:/home/osstest/ext/linux-firmware.git}
> diff --git a/mfi-common b/mfi-common
> index 16fc8c5..57bbf1c 100644
> --- a/mfi-common
> +++ b/mfi-common
> @@ -215,6 +215,31 @@ create_build_jobs () {
>  
>      fi
>  
> +    if [ "x$REVISION_RAISIN" != xdisable ]; then
> +
> +    ./cs-job-create $flight build-$arch-raisin build-raisin                  \
> +                arch=$arch                                                   \
> +        tree_xen=$TREE_XEN                                                   \
> +                $RUNVARS $BUILD_RUNVARS $BUILD_RAISIN_RUNVARS $arch_runvars \
> +                $suite_runvars                                               \
> +                host_hostflags=$build_hostflags                              \
> +                buildjob=${bfi}build-$arch                                   \
> +                tree_raisin=$TREE_RAISIN                                     \
> +                tree_qemuu=$TREE_QEMU_UPSTREAM                               \
> +                tree_qemu=$TREE_QEMU                                         \
> +                tree_seabios=$TREE_SEABIOS                                   \
> +                tree_libvirt=$TREE_LIBVIRT                                   \
> +                tree_ovmf=$TREE_OVMF                                         \
> +                revision_xen=${REVISION_XEN:-master}                         \
> +                revision_qemu=${REVISION_QEMU:-master}                       \
> +                revision_qemuu=${REVISION_QEMU_UPSTREAM:-master}             \
> +                revision_seabios=${REVISION_SEABIOS:-master}                 \
> +                revision_ovmf=${REVISION_OVMF:-master}                       \
> +                revision_libvirt=${REVISION_LIBVIRT:-master}                 \

I'm not sure about the fallback to master here.

In the current xen.git build stuff revision_foo='' means "use
Config.mk".

> +                revision_raisin=${REVISION_RAISIN:-${DEFAULT_REVISION_RAISIN}}\
> +
> +    fi
> +
>      if branch_wants_rumpkernel_tests; then
>  
>      case $arch in
> diff --git a/sg-run-job b/sg-run-job
> index eae159d..449118d 100755
> --- a/sg-run-job
> +++ b/sg-run-job
> @@ -346,6 +346,7 @@ proc need-hosts/build {} { return BUILD }
>  proc need-hosts/build-kern {} { return BUILD }
>  proc need-hosts/build-libvirt {} { return BUILD }
>  proc need-hosts/build-rumpuserxen {} { return BUILD }
> +proc need-hosts/build-raisin {} { return BUILD }
>  
>  proc run-job/build {} {
>      run-ts . = ts-xen-build
> @@ -364,6 +365,10 @@ proc run-job/build-rumpuserxen {} {
>      run-ts . = ts-xen-build + host tools
>  }
>  
> +proc run-job/build-raisin {} {
> +    run-ts . = ts-raisin-build
> +}
> +
>  proc prepare-build-host {} {
>      global jobinfo
>      run-ts broken = ts-hosts-allocate + host
> diff --git a/ts-raisin-build b/ts-raisin-build
> new file mode 100755
> index 0000000..94c6447
> --- /dev/null
> +++ b/ts-raisin-build
> @@ -0,0 +1,161 @@
> +#!/usr/bin/perl -w
> +# This is part of "osstest", an automated testing framework for Xen.
> +# Copyright (C) 2009-2013 Citrix Inc.
> +# 
> +# This program is free software: you can redistribute it and/or modify
> +# it under the terms of the GNU Affero General Public License as published by
> +# the Free Software Foundation, either version 3 of the License, or
> +# (at your option) any later version.
> +# 
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU Affero General Public License for more details.
> +# 
> +# You should have received a copy of the GNU Affero General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +
> +use strict qw(vars);
> +use DBI;
> +use Osstest;
> +use File::Path;
> +use POSIX;
> +use Osstest::TestSupport;
> +use Osstest::BuildSupport;
> +
> +tsreadconfig();
> +selectbuildhost(\@ARGV);
> +# remaining arguments are passed as targets to "make"
> +builddirsprops();
> +
> +my $raisindir="$builddir/raisin";
> +
> +sub checkout () {
> +    prepbuilddirs();
> +    build_clone($ho, 'raisin', $builddir, 'raisin');
> +
> +    target_cmd_build($ho, 100, $builddir, <<END);
> +        cd $raisindir
> +	>config
> +
> +    echo >>config MAKE=\\"make $makeflags\\"
> +    echo >>config PREFIX=\\"/usr\\"
> +    echo >>config DESTDIR=dist
> +
> +    echo >>config XEN_URL=\\"$r{tree_xen}\\"
> +    echo >>config QEMU_URL=\\"$r{tree_qemuu}\\"
> +    echo >>config QEMU_TRADITIONAL_URL=\\"$r{tree_qemu}\\"
> +    echo >>config SEABIOS_URL=\\"$r{tree_seabios}\\"
> +    echo >>config LIBVIRT_URL=\\"$r{tree_libvirt}\\"
> +    echo >>config OVMF_URL=\\"$r{tree_ovmf}\\"
> +
> +    echo >>config XEN_REVISION=\\"$r{revision_xen}\\"
> +    echo >>config QEMU_REVISION=\\"$r{revision_qemuu}\\"
> +    echo >>config QEMU_TRADITIONAL_REVISION=\\"$r{revision_qemu}\\"
> +    echo >>config SEABIOS_REVISION=\\"$r{revision_seabios}\\"
> +    echo >>config LIBVIRT_REVISION=\\"$r{revision_libvirt}\\"
> +    echo >>config OVMF_REVISION=\\"$r{revision_ovmf}\\"
> +END
> +#/;
> +}
> +
> +sub build () {
> +    target_cmd_root($ho, <<END);
> +        cd $raisindir
> +        ./raise -y install-builddep
> +END
> +#/;
> +    target_cmd_build($ho, 9000, $builddir, <<END);
> +        cd $raisindir
> +        ./raise -n build
> +END
> +#/;
> +}
> +
> +sub checkoutput () {
> +    target_cmd_build($ho, 100, $builddir, <<END);
> +            cd $raisindir/dist
> +            ls boot/xen.gz
> +            ls usr/sbin/xenstored
> +            ls usr/sbin/xenconsoled
> +            ls usr/lib/libxenlight.so
> +            ls usr/sbin/xl
> +            ls usr/lib/xen/boot/hvmloader
> +            ls usr/lib/xen/bin/qemu-dm
> +            ls usr/lib/xen/bin/qemu-system-i386
> +            ls usr/sbin/libvirtd
> +END
> +#/;
> +}
> +
> +sub collectversions () {
> +    store_revision($ho, 'xen', "$raisindir/xen-dir", 1);
> +    store_revision($ho, 'qemu', "$raisindir/qemu-traditional-dir", 1);
> +    store_revision($ho, 'qemuu', "$raisindir/qemu-dir", 1);
> +    store_revision($ho, 'seabios', "$raisindir/seabios-dir", 1);
> +    store_revision($ho, 'ovmf', "$raisindir/ovmf-dir", 1);
> +    store_revision($ho, 'libvirt', "$raisindir/libvirt-dir", 1);
> +}
> +
> +sub divide () {
> +    # Only move hv to xeninstall, so that we can have
> +    # xenpolicy in tools tarball.
> +    #
> +    # The files inside boot/ after `make dist' are
> +    # xen-$XEN_VERSION: Xen binary
> +    # xen.gz/xen: symlink to xen-$XEN_VERSION
> +    # xen-$MAJOR: symlink to xen-$XEN_VERSION
> +    # xen-$MAJOR.$MINOR: symlink to xen-$XEN_VERSION
> +    # xen-sym-$XEN_VERSION: Xen symbol
> +    # xenpolicy-$XEN_VERSION: flask policy binary if xsm is enabled
> +    #
> +    # So the following snippet will leave xenpolicy* in
> +    # install/boot and get packaged to tools tarball.
> +    target_cmd_build($ho, 100, $builddir, <<END);
> +        cd raisin
> +        mkdir xendist
> +        for f in *dist; do
> +            mkdir -p \$f/lib
> +        done
> +        if test -d dist/boot; then
> +            if test -f dist/boot/xen.gz || test -f dist/boot/xen; then
> +                mkdir xendist/boot
> +                mvfiles=`find dist/boot -name 'xen[a-z]*' -prune -o -name 'xen*' -print`
> +                mv \$mvfiles xendist/boot/.
> +            fi
> +        fi
> +END
> +}
> +
> +sub stash () {
> +    foreach my $part ('', 'xen') {
> +        built_stash($ho, $builddir,
> +                    "raisin/${part}dist",
> +                    "${part}dist");
> +    }
> +    built_stash_file($ho, $builddir, "xen-syms", "raisin/xen-dir/xen/xen-syms", 1);
> +    built_stash_file($ho, $builddir, "raisin-config", "raisin/config", 1);
> +    built_stash_file($ho, $builddir, "seabios-config",
> +             "raisin/seabios-dir/.config", 1);
> +    built_compress_stashed("xen-syms");
> +}
> +
> +
> +our @probs;
> +
> +sub trapping ($) {
> +    my ($sub) = @_;
> +    my $tok= eval { $sub->(); 1; };
> +    if (!$tok) { push @probs, $@; print STDERR "failure (trapped): $@\n"; }
> +}
> +
> +checkout();
> +trapping(\&build);
> +trapping(\&checkoutput);
> +trapping(\&collectversions);
> +divide();
> +stash();
> +
> +die "*** something failed:\n\n".(join "\n\n",@probs)."\n** something failed"
> +    if @probs;
> +