[PATCH v3 0/8] osstest: initial FreeBSD support

[PATCH v3 0/8] osstest: initial FreeBSD support

[Roger Pau Monne]

Hello,

This series introduces initial FreeBSD host support to osstest. The
current series allow installing a bare-metal host with FreeBSD and
building FreeBSD on it in order to generate new install media that can
be fed into the installer script.

This is still very limited, since no Xen testing is done on those
hosts, however it sets the base to add a Xen build test for FreeBSD.

Note that it should be quite easy to add more steps to the build
script so that FreeBSD VM images are also generated, that could be
used by osstest.

I've tried to add a detailed commit log at each relevant patch, so not
much more to add here in the cover letter.

The series can also be found on my git repo:

git://xenbits.xen.org/people/royger/osstest.git freebsd_v3

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 1/8] osstest: make built_stash_file store a path_ runvar for each file

[Roger Pau Monne]

And introduce built_stash_debugfile in order the keep the previous
behavior of built_stash_file.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/TestSupport.pm | 14 ++++++++++++--
 ts-kernel-build        |  4 ++--
 ts-xen-build           |  8 ++++----
 3 files changed, 18 insertions(+), 8 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index c23ac135..5f13eb0b 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -83,7 +83,7 @@ BEGIN {
                       get_stashed open_unique_stashfile compress_stashed
                       dir_identify_vcs
                       build_url_vcs build_clone
-                      built_stash built_stash_file
+                      built_stash built_stash_file built_stash_debugfile
                       built_compress_stashed
                       hg_dir_revision git_dir_revision vcs_dir_revision
                       store_revision store_vcs_revision
@@ -1445,7 +1445,7 @@ END
     store_runvar("path_$item", $stashleaf);
 }
 
-sub built_stash_file ($$$$;$) {
+sub built_stash_debugfile ($$$$;$) {
     my ($ho, $builddir, $item, $fname, $optional) = @_;
     my $build= "build";
     my $stashleaf= "$build/$item";
@@ -1458,6 +1458,16 @@ sub built_stash_file ($$$$;$) {
                    "$stash/$stashleaf");
 }
 
+sub built_stash_file ($$$$;$) {
+    my ($ho, $builddir, $item, $fname, $optional) = @_;
+    my $build= "build";
+    my $stashleaf= "$build/$item";
+
+    built_stash_debugfile($ho, $builddir, $item, $fname, $optional);
+    store_runvar("path_$item", $stashleaf);
+}
+
+
 sub built_compress_stashed($) {
     my ($path) = @_;
     compress_stashed("build/$path");
diff --git a/ts-kernel-build b/ts-kernel-build
index 94e67a47..5b87f5a7 100755
--- a/ts-kernel-build
+++ b/ts-kernel-build
@@ -438,9 +438,9 @@ if ($r{tree_linuxfirmware}) {
     fwinstall();
 }
 built_stash($ho, $builddir, 'dist', 'kerndist');
-built_stash_file($ho, $builddir, 'vmlinux', 'linux/vmlinux');
+built_stash_debugfile($ho, $builddir, 'vmlinux', 'linux/vmlinux');
 built_compress_stashed('vmlinux');
-built_stash_file($ho, $builddir, 'config', 'linux/.config');
+built_stash_debugfile($ho, $builddir, 'config', 'linux/.config');
 
 sub enable_xen_config () {
     return <<'END';
diff --git a/ts-xen-build b/ts-xen-build
index 31acb9dd..0152ea05 100755
--- a/ts-xen-build
+++ b/ts-xen-build
@@ -209,10 +209,10 @@ sub stash () {
                     "xen/dist/${part}install",
                     "${part}dist");
     }
-    built_stash_file($ho, $builddir, "xen-syms", "xen/xen/xen-syms", 1);
-    built_stash_file($ho, $builddir, "xen-config", "xen/.config", 1);
-    built_stash_file($ho, $builddir, "xen-hv-config", "xen/xen/.config", 1);
-    built_stash_file($ho, $builddir, "seabios-config",
+    built_stash_debugfile($ho, $builddir, "xen-syms", "xen/xen/xen-syms", 1);
+    built_stash_debugfile($ho, $builddir, "xen-config", "xen/.config", 1);
+    built_stash_debugfile($ho, $builddir, "xen-hv-config", "xen/xen/.config", 1);
+    built_stash_debugfile($ho, $builddir, "seabios-config",
 		     "xen/tools/firmware/seabios-dir-remote/.config", 1);
     built_compress_stashed("xen-syms");
 }
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 2/8] osstest: move known_hosts generation to TestSupport

[Roger Pau Monne]

This is equivalent to the already existing authorized_keys function,
and generates the contents of the known_hosts file that should be
installed on targets.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 Osstest/Debian.pm      | 36 +-----------------------------------
 Osstest/TestSupport.pm | 41 ++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 41 insertions(+), 36 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 8ba48bfa..87539822 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -709,41 +709,7 @@ sub preseed_ssh ($$) {
     my ($ho,$sfx) = @_;
 
     my $authkeys_url= create_webfile($ho, "authkeys$sfx", authorized_keys());
-
-    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
-    my $hostkey= get_filecontents($hostkeyfile);
-    chomp($hostkey); $hostkey.="\n";
-    my $knownhosts= '';
-
-    my $hostsq= $dbh_tests->prepare(<<END);
-        SELECT val FROM runvars
-         WHERE flight=? AND name LIKE '%host'
-         GROUP BY val
-END
-    $hostsq->execute($flight);
-    while (my ($node) = $hostsq->fetchrow_array()) {
-        my $defaultfqdn = $node;
-        $defaultfqdn .= ".$c{TestHostDomain}" unless $defaultfqdn =~ m/\./;
-
-        my %props;
-        $mhostdb->get_properties($node, \%props);
-
-        my $longname= $props{Fqdn} // $defaultfqdn;
-        my (@hostent)= gethostbyname($longname);
-        if (!@hostent) {
-            logm("skipping host key for nonexistent host $longname");
-            next;
-        }
-        my $specs= join ',', $longname, $node, map {
-            join '.', unpack 'W4', $_;
-        } @hostent[4..$#hostent];
-        logm("adding host key for $specs");
-        $knownhosts.= "$specs ".$hostkey;
-    }
-    $hostsq->finish();
-
-    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
-    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", $knownhosts);
+    my $knownhosts_url= create_webfile($ho, "known_hosts$sfx", known_hosts());
 
     preseed_hook_command($ho, 'late_command', $sfx, <<END);
 #!/bin/sh
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 5f13eb0b..8c7078c5 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -89,7 +89,7 @@ BEGIN {
                       store_revision store_vcs_revision
                       git_massage_url
 
-                      sshopts authorized_keys
+                      sshopts authorized_keys known_hosts
                       cfg_tftp_di_version
                       remote_perl_script_open remote_perl_script_done
                       host_reboot target_reboot target_reboot_hard            
@@ -2356,6 +2356,45 @@ sub authorized_keys () {
     return $authkeys;
 }
 
+sub known_hosts () {
+    my $hostkeyfile= "$c{OverlayLocal}/etc/ssh/ssh_host_rsa_key.pub";
+    my $hostkey= get_filecontents($hostkeyfile);
+    chomp($hostkey); $hostkey.="\n";
+
+    my $knownhosts= '';
+
+    my $hostsq= $dbh_tests->prepare(<<END);
+        SELECT val FROM runvars
+         WHERE flight=? AND name LIKE '%host'
+         GROUP BY val
+END
+    $hostsq->execute($flight);
+    while (my ($node) = $hostsq->fetchrow_array()) {
+        my $defaultfqdn = $node;
+        $defaultfqdn .= ".$c{TestHostDomain}" unless $defaultfqdn =~ m/\./;
+
+        my %props;
+        $mhostdb->get_properties($node, \%props);
+
+        my $longname= $props{Fqdn} // $defaultfqdn;
+        my (@hostent)= gethostbyname($longname);
+        if (!@hostent) {
+            logm("skipping host key for nonexistent host $longname");
+            next;
+        }
+        my $specs= join ',', $longname, $node, map {
+            join '.', unpack 'W4', $_;
+        } @hostent[4..$#hostent];
+        logm("adding host key for $specs");
+        $knownhosts.= "$specs ".$hostkey;
+    }
+    $hostsq->finish();
+
+    $knownhosts.= "localhost,127.0.0.1 ".$hostkey;
+
+    return $knownhosts;
+}
+
 sub cfg_tftp_di_version ($) {
     my ($suite) = @_;
     $suite //= 'x def suite'; # will not find $c{...}
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 3/8] osstest: introduce helper to get per-host tftp prefix

[Roger Pau Monne]

This is used in order to get the per-host tftp prefix, used to store
the host initrd file.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v1:
 - New in this version.
---
 Osstest/TestSupport.pm | 8 +++++++-
 ts-host-install        | 2 +-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 8c7078c5..8b41a4bc 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -67,7 +67,7 @@ BEGIN {
                       target_run_apt
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
-                      target_extract_jobdistpath
+                      target_extract_jobdistpath target_tftp_prefix
                       lv_create lv_dev_mapper
 
                       poll_loop tcpconnect await_tcp
@@ -2278,6 +2278,12 @@ sub target_extract_jobdistpath ($$$$$) {
     target_cmd_root($ho, "cd / && tar -hzxf $distcopy", 300);
 }
 
+sub target_tftp_prefix ($) {
+    my ($ho) = @_;
+
+    return "$ho->{Tftp}{TmpDir}/" . hostnamepath($ho);
+}
+
 sub guest_find_domid ($$) {
     my ($ho,$gho) = @_;
     return if defined $gho->{Domid};
diff --git a/ts-host-install b/ts-host-install
index 299eae48..9f9c84fd 100755
--- a/ts-host-install
+++ b/ts-host-install
@@ -253,7 +253,7 @@ END
     push @initrds, "$initrd_overlay.cpio.gz";
 
     logm("using initrds: @initrds");
-    my $initrd= "$ho->{Tftp}{TmpDir}".hostnamepath($ho)."--initrd.gz";
+    my $initrd= target_tftp_prefix($ho) . "--initrd.gz";
     system_checked("cat -- @initrds >$ho->{Tftp}{Path}$initrd");
     
     push @dicmdline, "domain=$c{TestHostDomain}";
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 4/8] osstest: add a FreeBSD host install recipe

[Roger Pau Monne]

The installation is performed using the bsdinstall tool, which is part
of the FreeBSD base system. The installer image is setup with the
osstest ssh keys and sshd enabled by default, which allows the test
harness to just ssh into the box, create the install config file and
launch the scripted install.

Currently the installation is done with ZFS only, in stripe mode, and
a single disk.

In order to support the FreeBSD installer a new method is added, that
allows setting the pxe boot of a host using a memdisk.

The install script either picks the binary images from the output of a
previous FreeBSD buildjob (yet to be introduced), or from the folder
pointed by freebsd_distpath. This folder should contain the installer
image (install.img) and the compressed install sets (kernel.txz,
base.txz) together with the MANIFEST file that contains the checksums.

Note that the FreeBSD installer image uses "memdisk" from pxelinux, so
the tftp root needs to provide a copy of memdisk.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Use sysrc instead of echo to modify rc.conf.

Changes since v1:
 - Use pkg bootstrap instead of pkg update.
 - Use freebsdbuildjob instead of freebsd_buildjob.
 - add "append raw" to the memdisk command line.
 - Use with-lock-ex instead of flock.
 - Use system_checked and an array to hold the locked command to
   execute.
 - Use an array to store the list of possible disk devices.
 - Use target_putfilecontents_root_stash in order to upload the
   install script to the target.
 - Remove the usage of TftpFreeBSDBase.
 - Use target_tftp_prefix in order to get the prefix for the tftp
   target image path.
 - Add a missing linewrap.
 - Replace the split used to get the runvar paths with a regexp.
 - Place the install sets at /root/osstest_sets instead of
   /tmp/osstest_sets.
 - Pick the sets/installer image from the freebsdbuildjob
   path_freebsdist or from the runvar variable freebsd_distpath.
 - Add a comment about which FreeBSD specific inputs the script
   consumes.
---
 Osstest/TestSupport.pm  |  22 +++-
 ts-freebsd-host-install | 269 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 290 insertions(+), 1 deletion(-)
 create mode 100755 ts-freebsd-host-install

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 8b41a4bc..ba1d6cc9 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -118,7 +118,7 @@ BEGIN {
                       await_webspace_fetch_byleaf create_webfile
                       file_link_contents get_timeout
                       setup_netboot_di setup_netboot_local host_netboot_file
-		      subst_netboot_template
+                      subst_netboot_template setup_netboot_memdisk
 
                       ether_prefix
 
@@ -2559,6 +2559,26 @@ default local
 END
 }
 
+sub setup_netboot_memdisk ($$) {
+    my ($ho, $img) = @_;
+    setup_netboot_bootcfg($ho, <<END);
+serial 0 $c{Baud}
+timeout 5
+label overwrite
+        menu label ^Overwrite
+        menu default
+        kernel memdisk
+        initrd $img
+        # NB: according to the memdisk syslinux wikipage [0]
+        # adding "append raw" is required in order to boot on
+        # some boxes, and in fact some hardware will not boot
+        # without it.
+        # [0] http://www.syslinux.org/wiki/index.php?title=MEMDISK#Memory_access_method
+        append raw
+default overwrite
+END
+}
+
 # uboot emulates pxelinux, so reuse BIOS stuff
 sub setup_netboot_di_uboot ($$$$$;%) { return &setup_netboot_di_bios; }
 sub setup_netboot_local_uboot ($) { return &setup_netboot_local_bios; }
diff --git a/ts-freebsd-host-install b/ts-freebsd-host-install
new file mode 100755
index 00000000..19254c8c
--- /dev/null
+++ b/ts-freebsd-host-install
@@ -0,0 +1,269 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# The FreeBSD installer script consumes either one of the following
+# runvars:
+#
+# freebsd_distpath: points to a folder accessible by the controller that
+# should contain the installer image and the compressed install sets,
+# together with the MANIFEST file.
+#
+# freebsdbuildjob: points to a previous FreeBSD build job, that has
+# produced an installer image and compressed sets. The only runvar used
+# form the job pointed to by freebsdbuildjob is the path_freebsddist one,
+# that points to the folder containing the files described above.
+#
+# The installer image must be named 'install.img', and the sets
+# 'kernel.txz', 'base.txz' and finally the 'MANIFEST' file that contains
+# the checksums.
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our %xopts;
+
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+exit 0 if $ho->{Flags}{'no-reinstall'};
+exit 0 if $ho->{SharedReady};
+
+our $timeout = 1000;
+
+our @sets = qw(base.txz kernel.txz);
+
+our $path_prefix = $r{"freebsd_distpath"} ||
+                   get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
+
+sub get_sets_path () {
+    my @paths;
+
+    foreach my $set (@sets, "MANIFEST") {
+        push @paths, { name => "$set", path => "$path_prefix/$set" };
+    }
+
+    return @paths;
+}
+
+sub create_ssh_overlay () {
+    my $url = create_webfile($ho, "ssh.tar", sub {
+        my ($fh) = @_;
+        contents_make_cpio($fh, 'ustar',  "$c{OverlayLocal}/etc/ssh/");
+    });
+
+    return $url;
+}
+
+sub setup_netboot_installer () {
+    my $image = "$path_prefix/install.img";
+    my $pxeimg = target_tftp_prefix($ho) . "--freebsd.img";
+    my $hash = `sha256sum $image | head -c 16` or die $!;
+    my $tftp_freebsd = "$ho->{Tftp}{Path}/$ho->{Tftp}{TmpDir}/freebsd-images/";
+    my $script = <<'END';
+basedir=$0
+imagepath=$1
+sharedpath=$2
+targetpath=$3
+cd $basedir
+if [ ! -f $sharedpath ]; then
+    mkdir -p `dirname $sharedpath`
+    cp $imagepath $sharedpath
+fi
+rm -f $targetpath
+ln $sharedpath $targetpath
+# Dir format from basedir is $arch/$hash/install.img
+for hashdir in `find -mindepth 2 -maxdepth 2 -type d`; do
+    count=`stat -c %h $hashdir/install.img`
+    if [ $count -eq 1 ]; then
+        rm -rf $hashdir
+    fi
+done
+END
+    my @cmd = ( "with-lock-ex", "-w", "$tftp_freebsd/lock",
+                "bash", "-exc", "$script",
+                "$tftp_freebsd", "$image", "$r{arch}/$hash/install.img",
+                "$ho->{Tftp}{Path}/$pxeimg" );
+
+    ensuredir("$tftp_freebsd");
+    system_checked(@cmd);
+
+    # Setup the pxelinux config file
+    logm("Booting from installer image at $pxeimg");
+    setup_netboot_memdisk($ho, $pxeimg);
+}
+
+sub install () {
+    my $authkeys = authorized_keys();
+    my $knownhosts = known_hosts();
+    my $sshd_keys_url = create_ssh_overlay();
+    my @disk_names = qw(ada0 da0 ad0);
+    my $target_sets = "/root/osstest_sets";
+    my $disk;
+    my $nic;
+
+    target_cmd_root($ho, 'chsh -s /bin/sh', 10);
+
+    logm("Trying to find a disk to install to");
+    $disk = target_cmd_output_root($ho, <<END, 30);
+for disk in @disk_names; do
+    if [ -c "/dev/\$disk" ]; then
+        echo \$disk
+        exit 0
+    fi
+done
+exit 1
+END
+    defined($disk) or die "Unable to find a valid disk";
+    logm("Using $disk as destination disk device");
+
+    logm("Trying to figure out primary nic device name");
+    $nic = target_cmd_output_root($ho, <<END, 30);
+nics=`ifconfig -l`
+for nic in \$nics; do
+    addr=`ifconfig \$nic inet|grep inet|awk {'print \$2'}`
+    if [ "\$addr" = "$ho->{Ip}" ]; then
+        echo \$nic
+        exit 0
+    fi
+done
+exit 1
+END
+    defined($nic) or die "Unable to find primary network interface";
+    logm("Using $nic as primary network interface");
+
+    logm("Uploading the install sets to the system");
+    target_cmd_root($ho, <<END, 30);
+mkdir -p $target_sets
+mount -o size=1G -t tmpfs tmpfs $target_sets
+END
+
+    foreach (get_sets_path()) {
+        target_putfile_root($ho, 600, $_->{path},
+                            "$target_sets/$_->{name}");
+    }
+
+    logm("Creating the installer script");
+    target_putfilecontents_root_stash($ho, 10, <<END, '~/installscript');
+set -a
+BSDINSTALL_DISTDIR="$target_sets"
+ZFSBOOT_DISKS="$disk"
+DISTRIBUTIONS="@sets"
+nonInteractive=1
+
+#!/bin/sh
+set -ex
+
+# Setup nic and sshd
+sysrc ifconfig_$nic=DHCP
+sysrc sshd_enable=YES
+
+# Use syncronious DHCP in case the server is slow
+sysrc synchronous_dhclient=YES
+
+# Disable sendmail
+sysrc sendmail_enable=NO
+sysrc sendmail_submit_enable=NO
+sysrc sendmail_outbound_enable=NO
+sysrc sendmail_msp_queue_enable=NO
+
+# Set proxy for the pkg manager
+mkdir -p /usr/local/etc/
+cat << ENDPKG >> /usr/local/etc/pkg.conf
+pkg_env: { http_proxy = $c{HttpProxy} }
+default_always_yes: true
+assume_always_yes: true
+ENDPKG
+
+# Bootstap the package manager
+export HTTP_PROXY=$c{HttpProxy}
+export ASSUME_ALWAYS_YES=yes
+pkg bootstrap
+
+# Allow root user login and setup ssh keys
+chsh -s /bin/sh root
+echo 'PermitRootLogin yes' >> /etc/ssh/sshd_config
+mkdir -p /root/.ssh
+cat << ENDKEYS > /root/.ssh/authorized_keys
+$authkeys
+ENDKEYS
+cat << ENDHOSTS > /root/.ssh/known_hosts
+$knownhosts
+ENDHOSTS
+
+# Fetch host keys
+fetch $sshd_keys_url -o - | tar -xf - -C /etc/ssh/
+# Set correct permissions
+chown root:wheel /etc/ssh/ssh_host_*_key*
+chmod 0600 /etc/ssh/ssh_host_*_key
+chmod 0644 /etc/ssh/ssh_host_*_key.pub
+
+# Add a osstest user
+pw useradd osstest -m
+chsh -s /bin/sh osstest
+mkdir -p /home/osstest/.ssh
+cat << ENDKEYS > /home/osstest/.ssh/authorized_keys
+$authkeys
+ENDKEYS
+cat << ENDHOSTS > /home/osstest/.ssh/known_hosts
+$knownhosts
+ENDHOSTS
+
+# Setup serial console
+printf "%s" "-h -S$c{Baud}" >> /boot.config
+cat << ENDBOOT >> /boot/loader.conf
+boot_serial="YES"
+comconsole_speed="$c{Baud}"
+console="comconsole"
+boot_verbose="YES"
+beastie_disable="YES"
+ENDBOOT
+END
+
+    logm("Launch the installer");
+    target_cmd_root($ho, 'bsdinstall script installscript', 1200);
+
+    target_reboot($ho);
+
+    logm("Waiting for the host to boot");
+    await_tcp(get_timeout($ho,'reboot',$timeout), 5, $ho);
+
+    logm("FreeBSD installed succesfully");
+}
+
+# Switch off, setup PXE and switch on to the installer
+power_state($ho, 0);
+setup_netboot_installer();
+power_cycle_sleep($ho);
+power_state($ho, 1);
+
+# Wait for the host to finish booting
+logm("Waiting for the installer to boot");
+await_tcp(get_timeout($ho,'reboot',$timeout), 5, $ho);
+
+# Next boot will be from local disk
+setup_netboot_local($ho);
+
+# Proceed with the install
+install();
+
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 5/8] osstest: introduce a FreeBSD build script

[Roger Pau Monne]

In order to generate the FreeBSD installer image and the install
media.

The install sets are the vanilla ones generated by the 'ftp' release
target.  The installer image is handcrafted based on the filesystem
created by the 'bootonly' target, which is then populated with the ssh
host keys, and setup in order to use the serial console. The other
difference from upstream FreeBSD installer images is that the one
built by osstest uses a ramdisk instead of relying on the installer
media to be somehow attached, either on a CD or USB drive. This is
required in order to boot the image from pxelinux (where no CD or USB
is actually attached to the host, and everything is fetched from
tftp).

Due to the nature of the FreeBSD build, the outputs are different from
what osstest expects from a buildjob, more specifically
path_freebsddist points to a folder that contains the several outputs
form this buildjob.

The following new helpers are also introduced, that are only used by
the FreeBSD build script: buildcmd_stamped_logged_root and
target_cmd_build_root, they behave exactly the same as the non-root
counterparts.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Increase build target timeouts.
 - Use sysrc instead echo to set rc.conf options.

Changes since v1:
 - Remove the ts-build-check FreeBSD hack.
 - Use pkg-static instead of pkg.
 - Introduce buildcmd_stamped_logged_root and target_cmd_build_root.
 - Use target_cmd_build_root and buildcmd_stamped_logged_root in the
   ts-freebsd-build script.
 - Fix the script snippets to use <<END.<<'END' in order to avoid
   escaping the shell variables.
 - Set path_freebsddist runvar to point to the folder where the build
   files are stashed.
 - Add a comment at the top of the file describing what runvars are
   consumed/produced by the build script.
---
 Osstest/BuildSupport.pm |  26 +++++-
 Osstest/TestSupport.pm  |  28 +++++-
 ts-freebsd-build        | 240 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 286 insertions(+), 8 deletions(-)
 create mode 100755 ts-freebsd-build

diff --git a/Osstest/BuildSupport.pm b/Osstest/BuildSupport.pm
index 4c2b658b..b96ce766 100644
--- a/Osstest/BuildSupport.pm
+++ b/Osstest/BuildSupport.pm
@@ -36,7 +36,7 @@ BEGIN {
                       $whhost $ho
 
                       builddirsprops
-                      buildcmd_stamped_logged
+                      buildcmd_stamped_logged buildcmd_stamped_logged_root
                       $builddir $makeflags
 		      $buildcmd_global_prefix
 
@@ -60,9 +60,9 @@ our ($xendist);
 
 our $buildcmd_global_prefix //= '';
 
-sub buildcmd_stamped_logged ($$$$$$) {
-    my ($timeout, $component, $stampname, $prefix, $cmd, $suffix) = @_;
-    target_cmd_build($ho, $timeout, $builddir, <<END);
+sub buildcmd_stamped_logged_script ($$$$$) {
+    my ($component, $stampname, $prefix, $cmd, $suffix) = @_;
+    my $script = <<END;
         cd $component
         $buildcmd_global_prefix
         $prefix
@@ -73,6 +73,24 @@ sub buildcmd_stamped_logged ($$$$$$) {
         echo ok.
 END
 #/;
+
+    return $script;
+}
+
+sub buildcmd_stamped_logged_root ($$$$$$) {
+    my ($timeout, $component, $stampname, $prefix, $cmd, $suffix) = @_;
+    my $script = buildcmd_stamped_logged_script($component, $stampname,
+                                                $prefix, $cmd, $suffix);
+
+    target_cmd_build_root($ho, $timeout, $builddir, $script);
+}
+
+sub buildcmd_stamped_logged ($$$$$$) {
+    my ($timeout, $component, $stampname, $prefix, $cmd, $suffix) = @_;
+    my $script = buildcmd_stamped_logged_script($component, $stampname,
+                                                $prefix, $cmd, $suffix);
+
+    target_cmd_build($ho, $timeout, $builddir, $script);
 }
 
 sub selectbuildhost {
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index ba1d6cc9..3f136777 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -53,6 +53,7 @@ BEGIN {
                       stashfilecontents
 
                       target_cmd_root target_cmd target_cmd_build
+                      target_cmd_build_root
                       target_cmd_output_root target_cmd_output
                       target_cmd_inputfh_root sshuho
                       target_getfile target_getfile_root
@@ -635,8 +636,8 @@ sub target_editfile      ($$$;$$) { teditfileex('osstest',@_); }
     # my ($ho,$rfile, $lleaf,$rdest) = @_;
     #                 ^^^^^^^^^^^^^ optional
 
-sub target_cmd_build ($$$$) {
-    my ($ho,$timeout,$builddir,$script) = @_;
+sub target_cmd_build_script ($$$) {
+    my ($ho, $builddir, $script) = @_;
 
     my $distcc_hosts = get_host_property($ho,'DistccHosts',undef);
     my $distcc = defined($distcc_hosts) ? <<END : "";
@@ -648,8 +649,8 @@ END
 
     my $httpproxy = http_proxy_envsettings($ho);
 
-    target_cmd($ho, <<END.$distcc.<<END.$script, $timeout);
-	set -xe
+    my $cmd =  <<END.$distcc.<<END.$script;
+        set -xe
         LC_ALL=C; export LC_ALL
         PATH=/usr/lib/ccache:\$PATH:/usr/lib/git-core
         $httpproxy
@@ -658,8 +659,27 @@ END
         cd $builddir
         rm -f build-ok-stamp
 END
+
+    return $cmd;
+}
+
+sub target_cmd_build ($$$$) {
+    my ($ho,$timeout,$builddir,$script) = @_;
+
+    my $cmd = target_cmd_build_script($ho, $builddir, $script);
+
+    target_cmd($ho, $cmd, $timeout);
+}
+
+sub target_cmd_build_root ($$$$) {
+    my ($ho,$timeout,$builddir,$script) = @_;
+
+    my $cmd = target_cmd_build_script($ho, $builddir, $script);
+
+    target_cmd_root($ho, $cmd, $timeout);
 }
 
+
 sub target_ping_check_core {
     my ($ho, $exp) = @_;
     my $out= `ping -c 5 $ho->{Ip} 2>&1`;
diff --git a/ts-freebsd-build b/ts-freebsd-build
new file mode 100755
index 00000000..6c10eece
--- /dev/null
+++ b/ts-freebsd-build
@@ -0,0 +1,240 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Consumes the following input runvars:
+#
+# revision_freebsd: FreeBSD commit ID to generate the install media from.
+# tree_freebsd: FreeBSD git tree to fetch the source code from.
+#
+# Produces the following output:
+#
+# Compressed install sets: kernel.txz, base.txz
+# Compressed debug symbols for the kernel: kernel-dbg.txz
+# Manifest file (checkums): MANIFEST
+# Installer image: install.img
+#
+# Sets the following runvar:
+#
+# path_freebsddist: points to the folder where the above files are stored.
+# freebsd_buildversion: version of FreeBSD built.
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+
+selectbuildhost(\@ARGV);
+builddirsprops();
+
+sub install_deps () {
+    target_cmd_root($ho, 'pkg-static install git', 300);
+}
+
+sub checkout () {
+    prepbuilddirs();
+
+    # Remove the directory as root, there might be files owned by root
+    target_cmd_build_root($ho, 300, $builddir, <<'END');
+# chflags will return error if the directory doesn't exist
+chflags -fR noschg freebsd || true
+rm -rf freebsd
+END
+
+    logm("Checkout the FreeBSD source tree");
+    build_clone($ho, 'freebsd', $builddir, 'freebsd', );
+}
+
+sub build () {
+    my $authkeys = authorized_keys();
+    my $target = "bootonly";
+    my $prefix = <<END;
+export MAKEOBJDIRPREFIX=$builddir/obj
+export TARGET=$r{arch}
+END
+
+    # Build process as documented in the handbook:
+    # https://www.freebsd.org/doc/handbook/updating-src.html
+
+    logm("Cleaning up previous builds");
+    buildcmd_stamped_logged(300, 'freebsd', 'cleanworld',
+                            $prefix, 'make cleanworld', '');
+
+    logm("Building world");
+    buildcmd_stamped_logged(25200, 'freebsd', 'buildworld',
+                            $prefix, <<END, '');
+make $makeflags buildworld
+END
+
+    logm("Building kernel");
+    buildcmd_stamped_logged(3600, 'freebsd', 'buildkernel',
+                            $prefix, <<END, '');
+make $makeflags buildkernel
+END
+
+    logm("Creating the install sets");
+    # NB: the steps below need to be done as root or the permissions
+    # of the files won't be properly set (and the target will fail).
+    buildcmd_stamped_logged_root(3600, 'freebsd', 'release-ftp',
+                                 $prefix, <<END, '');
+make -C release ftp
+END
+
+    logm("Populating the installer image");
+    buildcmd_stamped_logged_root(3600, 'freebsd', "release-$target",
+                                 $prefix, <<END, '');
+make -C release $target
+END
+
+    logm("Placing ssh host keys");
+    foreach my $file (<$c{OverlayLocal}/etc/ssh/ssh_host_*_key*>) {
+        target_putfile_root($ho, 30, $file,
+                            "$builddir/freebsd/release/$target/etc/ssh/");
+    }
+
+    logm("Configuring the installer image");
+    target_cmd_build_root($ho, 30, $builddir, <<END.<<'END');
+authkeys="$authkeys"
+bauds="$c{Baud}"
+cd freebsd/release/$target
+END
+# Enable sshd by default
+sysrc -f etc/rc.conf sshd_enable=YES
+
+# Allow root login and copy the keys
+echo 'PermitRootLogin yes' >> etc/ssh/sshd_config
+mkdir -p root/.ssh
+cat << ENDKEYS > root/.ssh/authorized_keys
+$authkeys
+ENDKEYS
+
+# Set host keys permissions
+chown root:wheel etc/ssh/ssh_host_*_key*
+chmod 0600 etc/ssh/ssh_host_*_key
+chmod 0644 etc/ssh/ssh_host_*_key.pub
+
+# Setup serial console output for stage1
+printf "%s" "-h -S$bauds" >> boot.config
+cat << ENDBOOT >> boot/loader.conf
+# Serial console configuration
+boot_serial="YES"
+comconsole_speed="$bauds"
+console="comconsole"
+boot_verbose="YES"
+beastie_disable="YES"
+
+# mfs boot parameters
+mfs_load="YES"
+mfs_type="mfs_root"
+mfs_name="/mfsroot"
+vfs.root.mountfrom="ufs:/dev/ufs/FreeBSD_Install"
+ENDBOOT
+
+# Enable DHCP on all network interfaces
+sysrc -f etc/rc.conf ifconfig_DEFAULT=DHCP
+
+# Remove the local script that launches the installer by default
+rm -rf etc/rc.local
+
+# Create a temporary fstab with the root dir
+echo '/dev/ufs/FreeBSD_Install / ufs rw 1 1' > etc/fstab
+
+# Remove the linked resolv.conf
+rm -rf etc/resolv.conf
+END
+
+    logm("Create the installer");
+    target_cmd_build_root($ho, 900, $builddir, <<END.<<'END');
+target="freebsd/release/$target"
+output="install.img"
+END
+mkdir -p $output.tmp
+
+# Do some pruning
+rm -rf $target/usr/share/man
+rm -rf $target/usr/share/examples
+rm -rf $target/usr/share/doc
+rm -rf $target/usr/share/dtrace
+
+# Create a mfs root image
+makefs -b 10% -B little -o label=FreeBSD_Install $output.tmp/mfsroot $target
+# Compress image
+gzip $output.tmp/mfsroot
+
+# Copy boot to the staging dir
+cp -r $target/boot $output.tmp/
+cp $target/boot.config $output.tmp/
+
+# The loader doesn't need any modules in order to boot into the mfsroot.
+# The rest of the modules can be loaded from the mfs root itself.
+rm -f $output.tmp/boot/kernel/*.ko
+
+# Compress the kernel
+gzip $output.tmp/boot/kernel/kernel
+
+makefs -B little $output.part $output.tmp
+
+# Make the image bootable
+mkimg -s gpt -b $target/boot/pmbr -p efi:=$target/boot/boot1.efifat \
+    -p freebsd-boot:=$target/boot/gptboot -p freebsd-ufs:=$output.part \
+    -p freebsd-swap::1M -o $output
+
+rm $output.part
+rm -rf $output.tmp
+END
+}
+
+sub stash () {
+    my @sets = qw(MANIFEST base.txz kernel.txz kernel-dbg.txz);
+    my @symbols = qw(kernel-dbg.txz);
+
+
+    logm("Stashing FreeBSD build output");
+    foreach my $set (@sets) {
+        built_stash_file($ho, $builddir, $set,
+                         "freebsd/release/ftp/$set", 0);
+    }
+    foreach my $symbol (@symbols) {
+        built_stash_debugfile($ho, $builddir, $symbol,
+                              "freebsd/release/ftp/$symbol", 0);
+    }
+    built_stash_file($ho, $builddir, "install.img", "install.img", 0);
+
+
+    my $srcversion = target_cmd_output_root($ho, <<END, 30);
+awk '/^\\\#define[[:space:]]*__FreeBSD_version/ { print \$3 }' \\
+    $builddir/freebsd/sys/sys/param.h | cut -c1-2
+END
+    store_runvar("freebsd_buildversion", "$srcversion");
+
+    # Set path_freebsddist to point to the build output folder
+    # in order to make ts-build-check happy.
+    store_runvar("path_freebsddist", "build/");
+}
+
+install_deps();
+checkout();
+build();
+stash();
+
+logm("FreeBSD build successful");
+
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs

[Roger Pau Monne]

Due to the nature of the FreeBSD install media, which is
self-generated from the ts-freebsd-build script, the hostflags set to
FreeBSD jobs are related to the current version under test.

The following hostflags might need to be fetched from the runvars of a
previous build-$arch-freebsd job:

 - share-build-freebsd-$arch-$hash: the $hash used here is calculated
   from the checksum of the installer image used by this specific job.
   This allows osstest to share FreeBSD build hosts, and be sure the
   exact desired FreeBSD version is used.
 - freebsd-$version: $version contains the major FreeBSD version under
   test. This version is obtained from the FreeBSD source code used to
   create the install media.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - New in this version.
---
 ts-freebsd-set-hostflags | 64 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)
 create mode 100755 ts-freebsd-set-hostflags

diff --git a/ts-freebsd-set-hostflags b/ts-freebsd-set-hostflags
new file mode 100755
index 00000000..e78e95d4
--- /dev/null
+++ b/ts-freebsd-set-hostflags
@@ -0,0 +1,64 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# This script sets the host_hostflags for a FreeBSD job based on the runvars
+# provided and the contents of the extra_hostflags runvar.
+#
+# If the freebsd_distpath runvar is set the installer image will be retrieved
+# from "freebsd_distpath"/install.img, and the FreeBSD version from the
+# freebsd_version runvar. Note that both those runvars should be set on the
+# current job.
+#
+# If freebsd_distpath is not set, it is assumed that freebsdbuildjob runvar is
+# set and the installer image will be retrieved from the path pointed to by
+# "path_freebsdddist"/install.img, and the FreeBSD version will be obtained
+# from the "freebsd_buildversion" runvar. Both of those runvars belong to the
+# flight and job pointed to by freebsdbuildjob.
+#
+# As output upon successful completion this script will set the host_hostflags
+# runvar for the current job. Note that this _must_ be done before running
+# ts-host-allocate.
+#
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+sub get_freebsd_image_hash() {
+    my $distpath =  $r{"freebsd_distpath"} ||
+                    get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
+
+    return `sha256sum $distpath/install.img|head -c 16`;
+}
+
+sub get_freebsd_version() {
+    return $r{"freebsd_version"} ||
+           get_runvar("freebsd_buildversion", $r{"freebsdbuildjob"});
+}
+
+my $hash = get_freebsd_image_hash();
+my $version = get_freebsd_version();
+my $arch = $r{"arch"};
+
+store_runvar("host_hostflags", $r{"extra_hostflags"} .
+             ",share-build-freebsd-$arch-$hash,freebsd-$version");
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 7/8] osstest: introduce a script to create a FreeBSD flight

[Roger Pau Monne]

This requires changes in several places in order to accommodate the
FreeBSD build, which although it's a build job, it doesn't have the
same set of dependencies as the current builds.

First, a new build recipe is added to sg-run-job, and accordingly
sg-run-job is also made aware about the differences between a Linux
and a FreeBSD build job. A Linux build job requires ts-host-allocate +
ts-host-install-twice + ts-xen-build-prep, while a FreeBSD build job
requires ts-freebsd-set-hostflags + ts-host-allocate +
ts-freebsd-host-install.

All the current build jobs are kept to use the Linux build recipe,
while the newly added build-freebsd is using the new FreeBSD recipe.

Finally, the logic to create a FreeBSD build job is added to
make-freebsd-flight. This includes creating a FreeBSD build job, and
also testing the output of that build job (by creating another build
job that depends on the output of the first).

Note that the FreeBSD build job needs some input in order to setup a
FreeBSD host, and that can be fetched from different places:

1. Env variable FREEBSD_BUILDJOB: use the output from a previous
build-<arch>-freebsd.

2. Env variables FREEBSD_DIST and FREEBSD_VERSION: set before calling
into make-flight, provide the path to the installer image and sets
folder and the version being installed.

3. Config file FreeBSDDist and FreeBSDVersion: same as 2. except that
they are set on the config file.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v1:
 - Replace freebsd_buildjob with freebsdbuildjob.
 - Replace FREEBSD_SETS/IMAGE with a single FREEBSD_DIST that points
   to a folder that should contain both things.
 - Document the FreeBSDDist and FreeBSDVersion config file options in
   the README file.
---
 README              | 11 +++++++
 ap-common           |  4 +++
 make-freebsd-flight | 92 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 sg-run-job          | 43 ++++++++++++++++++-------
 4 files changed, 138 insertions(+), 12 deletions(-)
 create mode 100755 make-freebsd-flight

diff --git a/README b/README
index b45058da..51243974 100644
--- a/README
+++ b/README
@@ -423,6 +423,17 @@ HostGroupFlags_<group>
 DebianPreseed
    Text to add to the debian-installer preseed file.  Optional.
 
+FreeBSDDist
+   Path to the folder that contains the FreeBSD install image and
+   the FreeBSD compressed install sets, together with the MANIFEST
+   file that holds the checksums. This is required in order to run
+   a FreeBSD host install if no previous FreeBSD buildjob is
+   available (ie: for example when running in standalone mode).
+
+FreeBSDVersion
+   Numeric value holding the major FreeBSD version of the media
+   provided in FreeBSDDist (ie: 12).
+
 ========================================
 
 Config settings relevant only to standalone mode
diff --git a/ap-common b/ap-common
index cbb815ce..d4fa7aef 100644
--- a/ap-common
+++ b/ap-common
@@ -37,6 +37,10 @@
 : ${PUSH_TREE_XTF:=$XENBITS:/home/xen/git/xtf.git}
 : ${BASE_TREE_XTF:=git://xenbits.xen.org/xtf.git}
 
+: ${TREE_FREEBSD:=git://github.com/freebsd/freebsd.git}
+: ${PUSH_TREE_FREEBSD:=$XENBITS:/home/xen/git/freebsd.git}
+: ${BASE_TREE_FREEBSD:=git://xenbits.xen.org/freebsd.git}
+
 : ${TREE_LIBVIRT:=git://libvirt.org/libvirt.git}
 : ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git}
 : ${BASE_TREE_LIBVIRT:=git://xenbits.xen.org/libvirt.git}
diff --git a/make-freebsd-flight b/make-freebsd-flight
new file mode 100755
index 00000000..93bb2cf3
--- /dev/null
+++ b/make-freebsd-flight
@@ -0,0 +1,92 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e -o posix
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. ./cri-common
+. ./ap-common
+. ./mfi-common
+
+arch=amd64
+
+job_create_build_filter_callback () {
+    :
+}
+
+get_freebsdjob_flags () {
+    arch=$1
+
+    # Figure out where are the installer binaries. The order is the following:
+    #
+    # 1. Env variable FREEBSD_BUILDJOB: use the output from a previous
+    # build-<arch>-freebsd.
+    #
+    # 2. Env variables FREEBSD_DIST, FREEBSD_VERSION: set before calling into
+    # make-flight, provide the path to the installer image, the sets to install
+    # and the version being installed.
+    #
+    # 3. Config file FreeBSDDist, FreeBSDVersion: same as 2. except that they
+    # are set on the config file.
+    #
+    if [ -n "$FREEBSD_BUILDJOB" ]; then
+        flags="freebsdbuildjob=$FREEBSD_BUILDJOB"
+    elif [ -n "$FREEBSD_DIST" ] && [ -n "$FREEBSD_VERSION" ]; then
+        flags="freebsd_distpath=$FREEBSD_DIST freebsd_version=$FREEBSD_VERSION"
+    else
+        distpath=`getconfig "FreeBSDDist"`
+        version=`getconfig "FreeBSDVersion"`
+        flags="freebsd_distpath=$distpath freebsd_version=$version"
+    fi
+
+    echo $flags
+}
+
+flags=`get_freebsdjob_flags $arch`
+job_create_build build-$arch-freebsd build-freebsd                       \
+            arch=$arch                                                   \
+            $RUNVARS $BUILD_RUNVARS $BUILD_FREEBSD_RUNVARS $arch_runvars \
+            tree_freebsd=$TREE_FREEBSD                                   \
+            revision_freebsd=$REVISION_FREEBSD                           \
+            extra_hostflags=arch-$arch,purpose-build                     \
+            $flags
+
+# Create an identical job that's going to use the build output from
+# the previous one.
+job_create_build build-$arch-freebsd-again build-freebsd                 \
+            arch=$arch                                                   \
+            $RUNVARS $BUILD_RUNVARS $BUILD_FREEBSD_RUNVARS $arch_runvars \
+            extra_hostflags=arch-$arch,purpose-build                     \
+            tree_freebsd=$TREE_FREEBSD                                   \
+            revision_freebsd=$REVISION_FREEBSD                           \
+            freebsdbuildjob=build-$arch-freebsd
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/sg-run-job b/sg-run-job
index ceb79800..239446b8 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -53,12 +53,15 @@ proc run-job {job} {
     set skip_globs     [jobdb::read-runvar $flight $job skip_testids]
 
     set nh [need-hosts/$jobinfo(recipe)]
-    if {![string compare $nh BUILD]} {
+    if {![string compare $nh BUILD_LINUX]} {
         set need_xen_hosts {}
-        set need_build_host 1
+        set need_build_host LINUX
+    } elseif {![string compare $nh BUILD_FREEBSD]} {
+        set need_xen_hosts {}
+        set need_build_host FREEBSD
     } else {
         set need_xen_hosts $nh
-        set need_build_host 0
+        set need_build_host ""
     }
     set nested_layers_hosts {}
 
@@ -69,7 +72,11 @@ proc run-job {job} {
         eval run-ts broken  =             ts-hosts-allocate + $need_xen_hosts
     }
 
-    if {$need_build_host} { catching-otherwise broken prepare-build-host }
+    if {![string compare $need_build_host LINUX]} {
+        catching-otherwise broken prepare-build-host-linux
+    } elseif {![string compare $need_build_host FREEBSD]}  {
+        catching-otherwise broken prepare-build-host-freebsd
+    }
 
     if {$ok} { setstatus running                                          }
 
@@ -89,7 +96,7 @@ proc run-job {job} {
         set need_xen_hosts [lunappend nested_layers_hosts]
     }
 
-    if {$need_build_host && $anyfailed} {
+    if {[string length $need_build_host] && $anyfailed} {
 	run-ts  !broken capture-logs      ts-logs-capture + host
     }
 
@@ -106,7 +113,7 @@ proc run-job {job} {
 
     if {$ok} { setstatus pass                                             }
 
-    if {$need_build_host && $ok} { jobdb::preserve-task 90 }
+    if {[string length $need_build_host] && $ok} { jobdb::preserve-task 90 }
 
     if {$anyfailed} {
         jobdb::logputs stdout "at least one test failed"
@@ -535,11 +542,12 @@ proc need-hosts/host-examine-linux {} {
 
 #---------- builds ----------
 
-proc need-hosts/build {} { return BUILD }
-proc need-hosts/build-kern {} { return BUILD }
-proc need-hosts/build-libvirt {} { return BUILD }
-proc need-hosts/build-rumprun {} { return BUILD }
-proc need-hosts/build-xtf {} { return BUILD }
+proc need-hosts/build {} { return BUILD_LINUX }
+proc need-hosts/build-kern {} { return BUILD_LINUX }
+proc need-hosts/build-libvirt {} { return BUILD_LINUX }
+proc need-hosts/build-rumprun {} { return BUILD_LINUX }
+proc need-hosts/build-xtf {} { return BUILD_LINUX }
+proc need-hosts/build-freebsd {} { return BUILD_FREEBSD }
 
 proc run-job/build {} {
     run-ts . = ts-xen-build
@@ -566,13 +574,24 @@ proc run-job/build-xtf {} {
     run-ts . = ts-xtf-build
 }
 
-proc prepare-build-host {} {
+proc run-job/build-freebsd {} {
+    run-ts . = ts-freebsd-build
+}
+
+proc prepare-build-host-linux {} {
     global jobinfo
     run-ts broken = ts-hosts-allocate + host
     run-ts broken host-install(*) ts-host-install-twice
     run-ts . host-build-prep ts-xen-build-prep
 }
 
+proc prepare-build-host-freebsd {} {
+    global jobinfo
+    run-ts broken = ts-freebsd-set-hostflags
+    run-ts broken = ts-hosts-allocate + host
+    run-ts broken host-install(*) ts-freebsd-host-install
+}
+
 proc need-hosts/coverity {} { return BUILD }
 proc run-job/coverity {} {
     run-ts . = ts-coverity-build + host
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3 8/8] osstest: hook FreeBSD flight into cr-daily-branch

[Roger Pau Monne]

Add the missing pieces so that cr-daily-branch knows how to create a
FreeBSD flight. This has been tested by calling `./cr-daily-branch
freebsd`.

Note that there are two missing pieces. The first one is correctly
setting the FREEBSD_BUILDJOB environment variable when calling
make-freebsd-flight, without it the installer media will always be
picked from the configuration file. This requires keeping track of
previous FreeBSD successful build jobs, marking them as permanent and
fetching them from the database in order to pass them on the env
variable.

The second missing bit is setting daily-cron-email-freebsd to send
emails to the xen-devel mailing list, ATM it's only sending emails to
my personal @FreeBSD.org address (this has been done for testing
purposes).

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
 ap-common                | 1 +
 ap-fetch-version         | 4 ++++
 ap-fetch-version-old     | 5 +++++
 ap-print-url             | 3 +++
 ap-push                  | 5 +++++
 cr-daily-branch          | 9 +++++++++
 cr-for-branches          | 2 +-
 cri-common               | 1 +
 daily-cron-email-freebsd | 1 +
 9 files changed, 30 insertions(+), 1 deletion(-)
 create mode 100644 daily-cron-email-freebsd

diff --git a/ap-common b/ap-common
index d4fa7aef..0d8a667a 100644
--- a/ap-common
+++ b/ap-common
@@ -86,6 +86,7 @@ fi
 : ${LOCALREV_SEABIOS:=daily-cron.$branch}
 : ${LOCALREV_OVMF:=daily-cron.$branch}
 : ${LOCALREV_XTF:=daily-cron.$branch}
+: ${LOCALREV_FREEBSD:=daily-cron.$branch}
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
diff --git a/ap-fetch-version b/ap-fetch-version
index a107c93d..365ae018 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -106,6 +106,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
 	;;
+freebsd)
+	repo_tree_rev_fetch_git freebsd \
+		$TREE_FREEBSD master $LOCALREV_FREEBSD
+	;;
 osstest)
         if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
 	    git update-ref -m "Arranging to test HEAD" \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 3cbc1768..9b086401 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -34,6 +34,7 @@ check_ap_fetch_placeholders
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_XTF:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
+: ${BASE_LOCALREV_FREEBSD:=daily-cron.$branch.old}
 : ${BASE_TAG_LIBVIRT:=xen-tested-master}
 
 if info_linux_tree "$branch"; then
@@ -114,6 +115,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$BASE_TREE_OVMF xen-tested-master $BASE_LOCALREV_OVMF
 	;;
+freebsd)
+	repo_tree_rev_fetch_git freebsd \
+		$BASE_TREE_FREEBSD xen-tested-master $BASE_LOCALREV_FREEBSD
+	;;
 osstest)
 	if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
 	    git fetch -f $HOME/testing.git production:ap-fetch
diff --git a/ap-print-url b/ap-print-url
index 93c14b35..57126292 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -64,6 +64,9 @@ xtf)
 ovmf)
 	echo $TREE_OVMF_UPSTREAM
 	;;
+freebsd)
+	echo $TREE_FREEBSD
+	;;
 osstest)
 	echo none:;
 	;;
diff --git a/ap-push b/ap-push
index a27ccc22..5b6ce7fc 100755
--- a/ap-push
+++ b/ap-push
@@ -41,6 +41,7 @@ TREE_RUMPRUN=$PUSH_TREE_RUMPRUN
 TREE_SEABIOS=$PUSH_TREE_SEABIOS
 TREE_OVMF=$PUSH_TREE_OVMF
 TREE_XTF=$PUSH_TREE_XTF
+TREE_FREEBSD=$PUSH_TREE_FREEBSD
 
 if info_linux_tree "$branch"; then
 	cd $repos/linux
@@ -125,6 +126,10 @@ xtf)
 	cd $repos/xtf
 	git push $TREE_XTF $revision:refs/heads/xen-tested-master
 	;;
+freebsd)
+	cd $repos/freebsd
+	git push $TREE_FREEBSD $revision:refs/heads/xen-tested-master
+	;;
 ovmf)
 	cd $repos/ovmf
 	git push $TREE_OVMF $revision:refs/heads/xen-tested-master
diff --git a/cr-daily-branch b/cr-daily-branch
index 8a314ce9..996be350 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -207,6 +207,10 @@ if [ "x$REVISION_LINUXFIRMWARE" = x ]; then
 	determine_version REVISION_LINUXFIRMWARE linuxfirmware LINUXFIRMWARE
         export REVISION_LINUXFIRMWARE
 fi
+if [ "x$REVISION_FREEBSD" = x ]; then
+	determine_version REVISION_FREEBSD freebsd FREEBSD
+	export REVISION_FREEBSD
+fi
 
 if [ "x$branch" != "xxen-unstable" ]; then
         export REVISION_LINUX_OLD=disable
@@ -260,6 +264,10 @@ ovmf)
 	realtree=ovmf
 	NEW_REVISION=$REVISION_OVMF
 	;;
+freebsd)
+	realtree=freebsd
+	NEW_REVISION=$REVISION_FREEBSD
+	;;
 *)
 	NEW_REVISION=''
 	wantpush=false
@@ -279,6 +287,7 @@ examine)	makeflight=make_hosts_flight
 			./make-hosts-flight $1 $2 blessed-$3 $3 $4
 		}
 		;;
+freebsd)	makeflight=./make-freebsd-flight ;;
 *)         	makeflight=./make-flight ;;
 esac
 
diff --git a/cr-for-branches b/cr-for-branches
index a6912735..23e4ca84 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -31,7 +31,7 @@ scriptoptions="$1"; shift
 LOGFILE=tmp/cr-for-branches.log
 export LOGFILE
 
-: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-4.8-testing xen-4.9-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing qemu-upstream-4.8-testing qemu-upstream-4.9-testing linux-linus linux-4.9 linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf ${EXTRA_BRANCHES}}
+: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-4.8-testing xen-4.9-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing qemu-upstream-4.8-testing qemu-upstream-4.9-testing linux-linus linux-4.9 linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf freebsd ${EXTRA_BRANCHES}}
 export BRANCHES
 
 fetchwlem=$wlem
diff --git a/cri-common b/cri-common
index 903fb4e2..a3e679cf 100644
--- a/cri-common
+++ b/cri-common
@@ -80,6 +80,7 @@ select_xenbranch () {
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
 	xtf)			tree=xtf;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
+	freebsd)		tree=freebsd;	xenbranch=xen-unstable ;;
 	distros-*|examine)	tree=none;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
 	esac
diff --git a/daily-cron-email-freebsd b/daily-cron-email-freebsd
new file mode 100644
index 00000000..b64821af
--- /dev/null
+++ b/daily-cron-email-freebsd
@@ -0,0 +1 @@
+To: royger@FreeBSD.org
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v3.1 8/8] osstest: hook FreeBSD flight into cr-daily-branch

[Roger Pau Monne]

Add the missing pieces so that cr-daily-branch knows how to create a
FreeBSD flight. This has been tested by calling `./cr-daily-branch
freebsd`.

Note that there are two missing pieces. The first one is correctly
setting the FREEBSD_BUILDJOB environment variable when calling
make-freebsd-flight, without it the installer media will always be
picked from the configuration file. This requires keeping track of
previous FreeBSD successful build jobs, marking them as permanent and
fetching them from the database in order to pass them on the env
variable.

The second missing bit is setting daily-cron-email-freebsd to send
emails to the xen-devel mailing list, ATM it's only sending emails to
my personal @FreeBSD.org address (this has been done for testing
purposes).

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v3:
 - Fix the daily-cron-email-* settings for FreeBSD branch flights.
---
 ap-common                       | 1 +
 ap-fetch-version                | 4 ++++
 ap-fetch-version-old            | 5 +++++
 ap-print-url                    | 3 +++
 ap-push                         | 5 +++++
 cr-daily-branch                 | 9 +++++++++
 cr-for-branches                 | 2 +-
 cri-common                      | 1 +
 daily-cron-email-adhoc--freebsd | 1 +
 daily-cron-email-play--freebsd  | 1 +
 daily-cron-email-real--freebsd  | 4 ++++
 11 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100644 daily-cron-email-adhoc--freebsd
 create mode 100644 daily-cron-email-play--freebsd
 create mode 100644 daily-cron-email-real--freebsd

diff --git a/ap-common b/ap-common
index d4fa7aef..0d8a667a 100644
--- a/ap-common
+++ b/ap-common
@@ -86,6 +86,7 @@ fi
 : ${LOCALREV_SEABIOS:=daily-cron.$branch}
 : ${LOCALREV_OVMF:=daily-cron.$branch}
 : ${LOCALREV_XTF:=daily-cron.$branch}
+: ${LOCALREV_FREEBSD:=daily-cron.$branch}
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
diff --git a/ap-fetch-version b/ap-fetch-version
index a107c93d..365ae018 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -106,6 +106,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
 	;;
+freebsd)
+	repo_tree_rev_fetch_git freebsd \
+		$TREE_FREEBSD master $LOCALREV_FREEBSD
+	;;
 osstest)
         if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
 	    git update-ref -m "Arranging to test HEAD" \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 3cbc1768..9b086401 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -34,6 +34,7 @@ check_ap_fetch_placeholders
 : ${BASE_LOCALREV_SEABIOS:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_XTF:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
+: ${BASE_LOCALREV_FREEBSD:=daily-cron.$branch.old}
 : ${BASE_TAG_LIBVIRT:=xen-tested-master}
 
 if info_linux_tree "$branch"; then
@@ -114,6 +115,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$BASE_TREE_OVMF xen-tested-master $BASE_LOCALREV_OVMF
 	;;
+freebsd)
+	repo_tree_rev_fetch_git freebsd \
+		$BASE_TREE_FREEBSD xen-tested-master $BASE_LOCALREV_FREEBSD
+	;;
 osstest)
 	if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
 	    git fetch -f $HOME/testing.git production:ap-fetch
diff --git a/ap-print-url b/ap-print-url
index 93c14b35..57126292 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -64,6 +64,9 @@ xtf)
 ovmf)
 	echo $TREE_OVMF_UPSTREAM
 	;;
+freebsd)
+	echo $TREE_FREEBSD
+	;;
 osstest)
 	echo none:;
 	;;
diff --git a/ap-push b/ap-push
index a27ccc22..5b6ce7fc 100755
--- a/ap-push
+++ b/ap-push
@@ -41,6 +41,7 @@ TREE_RUMPRUN=$PUSH_TREE_RUMPRUN
 TREE_SEABIOS=$PUSH_TREE_SEABIOS
 TREE_OVMF=$PUSH_TREE_OVMF
 TREE_XTF=$PUSH_TREE_XTF
+TREE_FREEBSD=$PUSH_TREE_FREEBSD
 
 if info_linux_tree "$branch"; then
 	cd $repos/linux
@@ -125,6 +126,10 @@ xtf)
 	cd $repos/xtf
 	git push $TREE_XTF $revision:refs/heads/xen-tested-master
 	;;
+freebsd)
+	cd $repos/freebsd
+	git push $TREE_FREEBSD $revision:refs/heads/xen-tested-master
+	;;
 ovmf)
 	cd $repos/ovmf
 	git push $TREE_OVMF $revision:refs/heads/xen-tested-master
diff --git a/cr-daily-branch b/cr-daily-branch
index 8a314ce9..996be350 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -207,6 +207,10 @@ if [ "x$REVISION_LINUXFIRMWARE" = x ]; then
 	determine_version REVISION_LINUXFIRMWARE linuxfirmware LINUXFIRMWARE
         export REVISION_LINUXFIRMWARE
 fi
+if [ "x$REVISION_FREEBSD" = x ]; then
+	determine_version REVISION_FREEBSD freebsd FREEBSD
+	export REVISION_FREEBSD
+fi
 
 if [ "x$branch" != "xxen-unstable" ]; then
         export REVISION_LINUX_OLD=disable
@@ -260,6 +264,10 @@ ovmf)
 	realtree=ovmf
 	NEW_REVISION=$REVISION_OVMF
 	;;
+freebsd)
+	realtree=freebsd
+	NEW_REVISION=$REVISION_FREEBSD
+	;;
 *)
 	NEW_REVISION=''
 	wantpush=false
@@ -279,6 +287,7 @@ examine)	makeflight=make_hosts_flight
 			./make-hosts-flight $1 $2 blessed-$3 $3 $4
 		}
 		;;
+freebsd)	makeflight=./make-freebsd-flight ;;
 *)         	makeflight=./make-flight ;;
 esac
 
diff --git a/cr-for-branches b/cr-for-branches
index a6912735..23e4ca84 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -31,7 +31,7 @@ scriptoptions="$1"; shift
 LOGFILE=tmp/cr-for-branches.log
 export LOGFILE
 
-: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-4.8-testing xen-4.9-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing qemu-upstream-4.8-testing qemu-upstream-4.9-testing linux-linus linux-4.9 linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf ${EXTRA_BRANCHES}}
+: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-4.8-testing xen-4.9-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing qemu-upstream-4.8-testing qemu-upstream-4.9-testing linux-linus linux-4.9 linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf freebsd ${EXTRA_BRANCHES}}
 export BRANCHES
 
 fetchwlem=$wlem
diff --git a/cri-common b/cri-common
index 903fb4e2..a3e679cf 100644
--- a/cri-common
+++ b/cri-common
@@ -80,6 +80,7 @@ select_xenbranch () {
 	seabios)		tree=seabios;	xenbranch=xen-unstable ;;
 	xtf)			tree=xtf;	xenbranch=xen-unstable ;;
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
+	freebsd)		tree=freebsd;	xenbranch=xen-unstable ;;
 	distros-*|examine)	tree=none;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
 	esac
diff --git a/daily-cron-email-adhoc--freebsd b/daily-cron-email-adhoc--freebsd
new file mode 100644
index 00000000..b64821af
--- /dev/null
+++ b/daily-cron-email-adhoc--freebsd
@@ -0,0 +1 @@
+To: royger@FreeBSD.org
diff --git a/daily-cron-email-play--freebsd b/daily-cron-email-play--freebsd
new file mode 100644
index 00000000..b64821af
--- /dev/null
+++ b/daily-cron-email-play--freebsd
@@ -0,0 +1 @@
+To: royger@FreeBSD.org
diff --git a/daily-cron-email-real--freebsd b/daily-cron-email-real--freebsd
new file mode 100644
index 00000000..6ee2fa4d
--- /dev/null
+++ b/daily-cron-email-real--freebsd
@@ -0,0 +1,4 @@
+To: xen-devel@lists.xenproject.org,
+    osstest-admin@xenproject.org,
+    royger@FreeBSD.org
+Bcc: osstest-output@lists.xenproject.org
-- 
2.11.0 (Apple Git-81)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 3/8] osstest: introduce helper to get per-host tftp prefix

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 3/8] osstest: introduce helper to get per-host tftp prefix"):
> This is used in order to get the per-host tftp prefix, used to store
> the host initrd file.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 4/8] osstest: add a FreeBSD host install recipe

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 4/8] osstest: add a FreeBSD host install recipe"):
> The installation is performed using the bsdinstall tool, which is part
> of the FreeBSD base system. The installer image is setup with the
> osstest ssh keys and sshd enabled by default, which allows the test
> harness to just ssh into the box, create the install config file and
> launch the scripted install.

thanks.

> +# The installer image must be named 'install.img', and the sets
> +# 'kernel.txz', 'base.txz' and finally the 'MANIFEST' file that contains
> +# the checksums.

Thanks for the good doc comment.

> +sub setup_netboot_installer () {
> +    my $image = "$path_prefix/install.img";
> +    my $pxeimg = target_tftp_prefix($ho) . "--freebsd.img";
> +    my $hash = `sha256sum $image | head -c 16` or die $!;

You should do this `  -' stripping in Perl.  That way, also, you won't
lose the exit status from sha256sum as you do here.

> +    my $tftp_freebsd = "$ho->{Tftp}{Path}/$ho->{Tftp}{TmpDir}/freebsd-images/";
> +    my $script = <<'END';
> +basedir=$0
> +imagepath=$1
> +sharedpath=$2
> +targetpath=$3

Please pass a dummy argument to the script (I usually use `x')
and shift all these arguments along.  Passing a real argument as $0 is
IMO strange.

> +cd $basedir
> +if [ ! -f $sharedpath ]; then
> +    mkdir -p `dirname $sharedpath`
> +    cp $imagepath $sharedpath

Please use the copy-to-tempfile-and-rename pattern.

AFAICT your filename pattern for the per-hash filename is
   $tftp_freebsd/$r{arch}/$hash/install.img

Is there some reason why this needs
 - to be qualified with $r{arch}
 - to have a bunch of per-hash directories containing one file each
?

Why not
   $tftp_freebsd/by-hash/$hash.img
?

> +# Dir format from basedir is $arch/$hash/install.img
> +for hashdir in `find -mindepth 2 -maxdepth 2 -type d`; do

1. use xargs or -exec -rm
2. use find -links
3. add -ctime +7 or something, so we don't delete things which
   have just been added (or used)

> +    my @cmd = ( "with-lock-ex", "-w", "$tftp_freebsd/lock",
> +                "bash", "-exc", "$script",
> +                "$tftp_freebsd", "$image", "$r{arch}/$hash/install.img",
> +                "$ho->{Tftp}{Path}/$pxeimg" );

Use of qw() would make this a lot less visually noisy.

> +    ensuredir("$tftp_freebsd");

" " are redundant in Perl.  It's not sh :-).

> +    logm("Trying to find a disk to install to");
> +    $disk = target_cmd_output_root($ho, <<END, 30);
> +for disk in @disk_names; do

Did you want to add set -e ?

> +    if [ -c "/dev/\$disk" ]; then
> +        echo \$disk
> +        exit 0
> +    fi
> +done
> +exit 1
> +END
> +    defined($disk) or die "Unable to find a valid disk";

target_cmd_output_root will never return undef.  I think this
error check is redundant, therefore.

> +    logm("Trying to figure out primary nic device name");
> +    $nic = target_cmd_output_root($ho, <<END, 30);
> +nics=`ifconfig -l`

I think this definitely needs set -e.

> +for nic in \$nics; do
> +    addr=`ifconfig \$nic inet|grep inet|awk {'print \$2'}`
> +    if [ "\$addr" = "$ho->{Ip}" ]; then
> +        echo \$nic
> +        exit 0
> +    fi
> +done
> +exit 1
> +END

Is it likely that the disk or network device name, for a particular
device, will change, for example across versions of FreeBSD ?

> +    logm("Uploading the install sets to the system");
> +    target_cmd_root($ho, <<END, 30);
> +mkdir -p $target_sets

Missing set -e.

> +    logm("Creating the installer script");
> +    target_putfilecontents_root_stash($ho, 10, <<END, '~/installscript');
> +set -a
> +BSDINSTALL_DISTDIR="$target_sets"
> +ZFSBOOT_DISKS="$disk"
> +DISTRIBUTIONS="@sets"
> +nonInteractive=1
> +
> +#!/bin/sh
> +set -ex

There's a #! halfway through this script.

> +# Setup serial console
> +printf "%s" "-h -S$c{Baud}" >> /boot.config

Are you deliberately leaving /boot.config with no final newline ?

> +cat << ENDBOOT >> /boot/loader.conf
> +boot_serial="YES"
> +comconsole_speed="$c{Baud}"
> +console="comconsole"
> +boot_verbose="YES"
> +beastie_disable="YES"

:-) re beastie.


Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 5/8] osstest: introduce a FreeBSD build script

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 5/8] osstest: introduce a FreeBSD build script"):
> The following new helpers are also introduced, that are only used by
> the FreeBSD build script: buildcmd_stamped_logged_root and
> target_cmd_build_root, they behave exactly the same as the non-root
> counterparts.

Please split these out into a separate patch.

> diff --git a/ts-freebsd-build b/ts-freebsd-build
> new file mode 100755
> index 00000000..6c10eece
> --- /dev/null
> +++ b/ts-freebsd-build
...
> +sub install_deps () {
> +    target_cmd_root($ho, 'pkg-static install git', 300);
> +}

This needs to be concurrency-safe, I think, since it might run
simultaneously in differnet jobs.  Is it ?

I wonder if the right answer is for you to call
   target_install_packages
and teach target_install_packages a mapping from the Debian package
names to FreeBSD ones, and how to do package installation on FreeBSD.

> +    logm("Cleaning up previous builds");
> +    buildcmd_stamped_logged(300, 'freebsd', 'cleanworld',
> +                            $prefix, 'make cleanworld', '');
> +
> +    logm("Building world");
> +    buildcmd_stamped_logged(25200, 'freebsd', 'buildworld',
> +                            $prefix, <<END, '');
> +make $makeflags buildworld
> +END
> +
> +    logm("Building kernel");
> +    buildcmd_stamped_logged(3600, 'freebsd', 'buildkernel',
> +                            $prefix, <<END, '');
> +make $makeflags buildkernel
> +END

These are quite formulaic, aren't they ?  Maybe you want to make a
sub for them (either global in this file, or an anon subref).

> +    # Set path_freebsddist to point to the build output folder
> +    # in order to make ts-build-check happy.
> +    store_runvar("path_freebsddist", "build/");

Heh.  Fine by me.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs"):
> Due to the nature of the FreeBSD install media, which is
> self-generated from the ts-freebsd-build script, the hostflags set to
> FreeBSD jobs are related to the current version under test.
> 
> The following hostflags might need to be fetched from the runvars of a
> previous build-$arch-freebsd job:

I think it is wrong to have a script set the normally baked-in runvar
host_hostflags.

In fact, I don't know how this script can have worked for you.
Currently many build jobs have the runvar "host_hostflags" including
many flags including "arch-i386" say, which I assume your FreeBSD
build jobs will have from make-flight too.  (It is forbidden, and
prevented, for a ts-* script to use store_runvar to modify a runvar
provided as part of the job definition.)

I think you should probably invent something like
  runtime_IDENT_hostflags
and teach ts-hosts-allocate-Executive about it.

> +sub get_freebsd_image_hash() {
> +    my $distpath =  $r{"freebsd_distpath"} ||
> +                    get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
> +
> +    return `sha256sum $distpath/install.img|head -c 16`;

This pattern again.  I commented on it before, but now that you are
repeating it, it should become a helper function.

Now that I think about this some more, why not use Digest::SHA and
$sha->addfile ?

> +store_runvar("host_hostflags", $r{"extra_hostflags"} .
> +             ",share-build-freebsd-$arch-$hash,freebsd-$version");

"extra_hostflags" would be the host flags for the host ident extra.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 7/8] osstest: introduce a script to create a FreeBSD flight

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 7/8] osstest: introduce a script to create a FreeBSD flight"):
> +FreeBSDDist
> +   Path to the folder that contains the FreeBSD install image and
> +   the FreeBSD compressed install sets, together with the MANIFEST
> +   file that holds the checksums. This is required in order to run
> +   a FreeBSD host install if no previous FreeBSD buildjob is
> +   available (ie: for example when running in standalone mode).
> +
> +FreeBSDVersion
> +   Numeric value holding the major FreeBSD version of the media
> +   provided in FreeBSDDist (ie: 12).

This means that a user who is setting these manually needs to supply
both.  Is there a way to avoid that ?

> +get_freebsdjob_flags () {
> +    arch=$1

You repeated use the word "flags" here for things which are runvars.
flags are, in osstest, strictly boolean.

> +flags=`get_freebsdjob_flags $arch`
> +job_create_build build-$arch-freebsd build-freebsd                       \
> +            arch=$arch                                                   \
> +            $RUNVARS $BUILD_RUNVARS $BUILD_FREEBSD_RUNVARS $arch_runvars \
> +            tree_freebsd=$TREE_FREEBSD                                   \
> +            revision_freebsd=$REVISION_FREEBSD                           \
> +            extra_hostflags=arch-$arch,purpose-build                     \
> +            $flags

See my comments about extra_hostflags in the previous patch.  Use
host_hostflags here.

> diff --git a/sg-run-job b/sg-run-job
> index ceb79800..239446b8 100755
> --- a/sg-run-job
> +++ b/sg-run-job

IWBN to split this out.  Firstly, please split the sg-run-job changes
from the make-flight changes.

Secondly, can you split the sg-run-job patch into two ?  1. change the
meaning of need_build_host but not introduce the FreeBSD versions;
2. add the FreeBSD cases.  sg-run-job is a bit fragile to do this kind
of work in.

Also I'm afraid you're going to find some conflicts with my recent
syslog work.  You will probably want to rebase onto pretest, which I
think is very likely to pass and be pushed soon.

> @@ -53,12 +53,15 @@ proc run-job {job} {
>      set skip_globs     [jobdb::read-runvar $flight $job skip_testids]
>  
>      set nh [need-hosts/$jobinfo(recipe)]
> -    if {![string compare $nh BUILD]} {
> +    if {![string compare $nh BUILD_LINUX]} {

How about
       if {[string match $nh BUILD_*]} {
and then set need_build_host to the RHS ?

> -    if {$need_build_host} { catching-otherwise broken prepare-build-host }
> +    if {![string compare $need_build_host LINUX]} {
> +        catching-otherwise broken prepare-build-host-linux
> +    } elseif {![string compare $need_build_host FREEBSD]}  {
> +        catching-otherwise broken prepare-build-host-freebsd
> +    }

Please use switch(3tcl).

> -proc prepare-build-host {} {
> +proc prepare-build-host-linux {} {

This rename should be in the first patch.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 8/8] osstest: hook FreeBSD flight into cr-daily-branch

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3 8/8] osstest: hook FreeBSD flight into cr-daily-branch"):
> +++ b/daily-cron-email-freebsd
> @@ -0,0 +1 @@
> +To: royger@FreeBSD.org

Please at least Bcc osstest-output.  See daily-cron-email-osstest for
an example.

Please also provide information about the changes to flights etc.,
for the whole series, by diffing the output of
  OSSTEST_CONFIG=standalone-config-example eatmydata standalone-generate-dump-flight-runvars
or some such.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3.1 8/8] osstest: hook FreeBSD flight into cr-daily-branch

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v3.1 8/8] osstest: hook FreeBSD flight into cr-daily-branch"):
> +++ b/daily-cron-email-real--freebsd
> @@ -0,0 +1,4 @@
> +To: xen-devel@lists.xenproject.org,
> +    osstest-admin@xenproject.org,
> +    royger@FreeBSD.org
> +Bcc: osstest-output@lists.xenproject.org

FTR this is good.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 4/8] osstest: add a FreeBSD host install recipe

[Roger Pau Monne]

On Fri, Jun 23, 2017 at 03:45:45PM +0100, Ian Jackson wrote:
> Roger Pau Monne writes ("[PATCH v3 4/8] osstest: add a FreeBSD host install recipe"):
> > +sub setup_netboot_installer () {
> > +    my $image = "$path_prefix/install.img";
> > +    my $pxeimg = target_tftp_prefix($ho) . "--freebsd.img";
> > +    my $hash = `sha256sum $image | head -c 16` or die $!;
> 
> You should do this `  -' stripping in Perl.  That way, also, you won't
> lose the exit status from sha256sum as you do here.

I've added a pre-patch that introduces a sha256file helper in order to
calculate the hash of a file.

> > +    my $tftp_freebsd = "$ho->{Tftp}{Path}/$ho->{Tftp}{TmpDir}/freebsd-images/";
> > +    my $script = <<'END';
> > +basedir=$0
> > +imagepath=$1
> > +sharedpath=$2
> > +targetpath=$3
> 
> Please pass a dummy argument to the script (I usually use `x')
> and shift all these arguments along.  Passing a real argument as $0 is
> IMO strange.

I agree, I was surprised to find out I could do it like that.

> > +cd $basedir
> > +if [ ! -f $sharedpath ]; then
> > +    mkdir -p `dirname $sharedpath`
> > +    cp $imagepath $sharedpath
> 
> Please use the copy-to-tempfile-and-rename pattern.
> 
> AFAICT your filename pattern for the per-hash filename is
>    $tftp_freebsd/$r{arch}/$hash/install.img
> 
> Is there some reason why this needs
>  - to be qualified with $r{arch}
>  - to have a bunch of per-hash directories containing one file each
> ?
> 
> Why not
>    $tftp_freebsd/by-hash/$hash.img
> ?

Yes, I've changed the path now to the suggested one.

> > +# Dir format from basedir is $arch/$hash/install.img
> > +for hashdir in `find -mindepth 2 -maxdepth 2 -type d`; do
> 
> 1. use xargs or -exec -rm
> 2. use find -links
> 3. add -ctime +7 or something, so we don't delete things which
>    have just been added (or used)

Done:

find `dirname $sharedpath` -links 1 -ctime +7 -delete

> > +for nic in \$nics; do
> > +    addr=`ifconfig \$nic inet|grep inet|awk {'print \$2'}`
> > +    if [ "\$addr" = "$ho->{Ip}" ]; then
> > +        echo \$nic
> > +        exit 0
> > +    fi
> > +done
> > +exit 1
> > +END
> 
> Is it likely that the disk or network device name, for a particular
> device, will change, for example across versions of FreeBSD ?

I don't think it happens usually, but I guess the driver name could
change, and in turn make the device name change. As I said I think
this is mostly stable, so that users don't get nasty surprises.

> > +    logm("Uploading the install sets to the system");
> > +    target_cmd_root($ho, <<END, 30);
> > +mkdir -p $target_sets
> 
> Missing set -e.
> 
> > +    logm("Creating the installer script");
> > +    target_putfilecontents_root_stash($ho, 10, <<END, '~/installscript');
> > +set -a
> > +BSDINSTALL_DISTDIR="$target_sets"
> > +ZFSBOOT_DISKS="$disk"
> > +DISTRIBUTIONS="@sets"
> > +nonInteractive=1
> > +
> > +#!/bin/sh
> > +set -ex
> 
> There's a #! halfway through this script.

Yes, that's intended, it's the format of the install script. This
first part contains variables (ie: input data) used by the installer.
The second part is automatically executed chrooted into the root dir
of the newly installed system once done.

https://www.freebsd.org/cgi/man.cgi?query=bsdinstall

See the SCRIPTING section.

> > +# Setup serial console
> > +printf "%s" "-h -S$c{Baud}" >> /boot.config
> 
> Are you deliberately leaving /boot.config with no final newline ?

That's what I've usually done, but it doesn't matter.

> > +cat << ENDBOOT >> /boot/loader.conf
> > +boot_serial="YES"
> > +comconsole_speed="$c{Baud}"
> > +console="comconsole"
> > +boot_verbose="YES"
> > +beastie_disable="YES"
> 
> :-) re beastie.

It just consumes serial log space and has a nasty countdown.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 5/8] osstest: introduce a FreeBSD build script

[Roger Pau Monne]

On Fri, Jun 23, 2017 at 03:56:20PM +0100, Ian Jackson wrote:
> Roger Pau Monne writes ("[PATCH v3 5/8] osstest: introduce a FreeBSD build script"):
> > The following new helpers are also introduced, that are only used by
> > the FreeBSD build script: buildcmd_stamped_logged_root and
> > target_cmd_build_root, they behave exactly the same as the non-root
> > counterparts.
> 
> Please split these out into a separate patch.

Done.

> > diff --git a/ts-freebsd-build b/ts-freebsd-build
> > new file mode 100755
> > index 00000000..6c10eece
> > --- /dev/null
> > +++ b/ts-freebsd-build
> ...
> > +sub install_deps () {
> > +    target_cmd_root($ho, 'pkg-static install git', 300);
> > +}
> 
> This needs to be concurrency-safe, I think, since it might run
> simultaneously in differnet jobs.  Is it ?

Yes, it needs to be concurrency safe and it's not. I've added a
pre-patch that adds FreeBSD support to target_install_packages,
although there's no translation layer yet.

I've used lockf [0] instead of with-lock-ex, which is the default lock
file utility in FreeBSD.

> I wonder if the right answer is for you to call
>    target_install_packages
> and teach target_install_packages a mapping from the Debian package
> names to FreeBSD ones, and how to do package installation on FreeBSD.
> 
> > +    logm("Cleaning up previous builds");
> > +    buildcmd_stamped_logged(300, 'freebsd', 'cleanworld',
> > +                            $prefix, 'make cleanworld', '');
> > +
> > +    logm("Building world");
> > +    buildcmd_stamped_logged(25200, 'freebsd', 'buildworld',
> > +                            $prefix, <<END, '');
> > +make $makeflags buildworld
> > +END
> > +
> > +    logm("Building kernel");
> > +    buildcmd_stamped_logged(3600, 'freebsd', 'buildkernel',
> > +                            $prefix, <<END, '');
> > +make $makeflags buildkernel
> > +END
> 
> These are quite formulaic, aren't they ?  Maybe you want to make a
> sub for them (either global in this file, or an anon subref).

Yes, since the code is already quite small I didn't made a stub, but I
certainly don't mind adding one.

Thanks, Roger.

[0] https://www.freebsd.org/cgi/man.cgi?query=lockf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs

[Roger Pau Monne]

On Fri, Jun 23, 2017 at 04:10:47PM +0100, Ian Jackson wrote:
> Roger Pau Monne writes ("[PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs"):
> > Due to the nature of the FreeBSD install media, which is
> > self-generated from the ts-freebsd-build script, the hostflags set to
> > FreeBSD jobs are related to the current version under test.
> > 
> > The following hostflags might need to be fetched from the runvars of a
> > previous build-$arch-freebsd job:
> 
> I think it is wrong to have a script set the normally baked-in runvar
> host_hostflags.

Note that host_hostflags is not set before this script runs.

> In fact, I don't know how this script can have worked for you.
> Currently many build jobs have the runvar "host_hostflags" including
> many flags including "arch-i386" say, which I assume your FreeBSD
> build jobs will have from make-flight too.  (It is forbidden, and
> prevented, for a ts-* script to use store_runvar to modify a runvar
> provided as part of the job definition.)

In this case host_hostflags is not part of the job definition.

> I think you should probably invent something like
>   runtime_IDENT_hostflags
> and teach ts-hosts-allocate-Executive about it.

What should I store in this runvar?

The problem here is that I need to set two host_hostflags properties:

freebsd-<version>
share-build-freebsd-<arch>-<hash>

arch is not a problem because it's available at job creation, but both
<version> and <hash> are more difficult to get, because they might
come from a previous flight, and get_runvar must be executed from a
job context, or else it fails. That's the reason I needed to set
host_hostflags from a ts script, so that I could use get_runvar.

> > +sub get_freebsd_image_hash() {
> > +    my $distpath =  $r{"freebsd_distpath"} ||
> > +                    get_stashed("path_freebsddist", $r{"freebsdbuildjob"});
> > +
> > +    return `sha256sum $distpath/install.img|head -c 16`;
> 
> This pattern again.  I commented on it before, but now that you are
> repeating it, it should become a helper function.

It's done :).

> > +store_runvar("host_hostflags", $r{"extra_hostflags"} .
> > +             ",share-build-freebsd-$arch-$hash,freebsd-$version");
> 
> "extra_hostflags" would be the host flags for the host ident extra.

Just so that this is clear to me. When creating the job the
host_hostflags variable should not be set (when calling
job_create_build), and ts-hosts-allocate-Executive should set the
host_hostflags itself?

I don't really see how that's much different from what I was doing
here, the only difference being that I was introducing a new script
instead of modifying ts-hosts-allocate-Executive.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs

[Ian Jackson]

Roger Pau Monne writes ("Re: [PATCH v3 6/8] osstest: introduce a script to set the hostflags for FreeBSD jobs"):
> On Fri, Jun 23, 2017 at 04:10:47PM +0100, Ian Jackson wrote:
> > In fact, I don't know how this script can have worked for you.
> > Currently many build jobs have the runvar "host_hostflags" including
> > many flags including "arch-i386" say, which I assume your FreeBSD
> > build jobs will have from make-flight too.  (It is forbidden, and
> > prevented, for a ts-* script to use store_runvar to modify a runvar
> > provided as part of the job definition.)
> 
> In this case host_hostflags is not part of the job definition.

That is precisely what I am complaining about.

The very same runvar cannot be set both at job creation and later by a
ts-* script.

I know you are not doing this with the very same runvar (because you
are setting host_hostflags at runtime and all_hostflags at job
creation time), but IMO similar runvars should not normally be set at
job creation in some cases and by ts-* scripts in others.  This is
certainly true of "portmanteau" runvars like the hostflags ones, whose
contents need to be assembled out of various pieces.

If you do things the way you do right now, we would be unable to use
the host_hostflags runvar for its current purpose, if we decided we
wanted to in the future, because you'd used it as a runtime-set
variable.

Currently, FOO_hostflags variables are all set during test definition
and I would like that to remain true.

You legitimately need to set some hostflags, to affect host
allocation, in a ts-* script.  So your new runtime-defined runvars
should have new names.  This is why I suggested this:

> > I think you should probably invent something like
> >   runtime_IDENT_hostflags
> > and teach ts-hosts-allocate-Executive about it.
> 
> What should I store in this runvar?

The same thing as you are currently storing in host_hostflags.  A
comma-separated list of the flags.

> arch is not a problem because it's available at job creation, but both
> <version> and <hash> are more difficult to get, because they might
> come from a previous flight, and get_runvar must be executed from a
> job context, or else it fails. That's the reason I needed to set
> host_hostflags from a ts script, so that I could use get_runvar.

I'm saying you should introduce a new runvar, which is exactly like
host_hostflags except that we expect to define it with a ts-* script
rather than in the job definition.

In fact, a new _family_ of runvars
  runtime_IDENT_hostflags
  runtime_all_hostflags
mirroring the existing
  IDENT_hostflags
  all_hostflags

> > > +store_runvar("host_hostflags", $r{"extra_hostflags"} .
> > > +             ",share-build-freebsd-$arch-$hash,freebsd-$version");
> > 
> > "extra_hostflags" would be the host flags for the host ident extra.
> 
> Just so that this is clear to me. When creating the job the
> host_hostflags variable should not be set (when calling
> job_create_build), and ts-hosts-allocate-Executive should set the
> host_hostflags itself?

No.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v3 7/8] osstest: introduce a script to create a FreeBSD flight

[Roger Pau Monne]

On Fri, Jun 23, 2017 at 04:18:28PM +0100, Ian Jackson wrote:
> Roger Pau Monne writes ("[PATCH v3 7/8] osstest: introduce a script to create a FreeBSD flight"):
> > +FreeBSDDist
> > +   Path to the folder that contains the FreeBSD install image and
> > +   the FreeBSD compressed install sets, together with the MANIFEST
> > +   file that holds the checksums. This is required in order to run
> > +   a FreeBSD host install if no previous FreeBSD buildjob is
> > +   available (ie: for example when running in standalone mode).
> > +
> > +FreeBSDVersion
> > +   Numeric value holding the major FreeBSD version of the media
> > +   provided in FreeBSDDist (ie: 12).
>
> This means that a user who is setting these manually needs to supply
> both.  Is there a way to avoid that ?

Er, not really. Sadly the MANIFEST file doesn't contain the FreeBSD
version number. I know it's inconvenient.

> > +get_freebsdjob_flags () {
> > +    arch=$1
> 
> You repeated use the word "flags" here for things which are runvars.
> flags are, in osstest, strictly boolean.
> 
> > +flags=`get_freebsdjob_flags $arch`
> > +job_create_build build-$arch-freebsd build-freebsd                       \
> > +            arch=$arch                                                   \
> > +            $RUNVARS $BUILD_RUNVARS $BUILD_FREEBSD_RUNVARS $arch_runvars \
> > +            tree_freebsd=$TREE_FREEBSD                                   \
> > +            revision_freebsd=$REVISION_FREEBSD                           \
> > +            extra_hostflags=arch-$arch,purpose-build                     \
> > +            $flags
> 
> See my comments about extra_hostflags in the previous patch.  Use
> host_hostflags here.
> 
> > diff --git a/sg-run-job b/sg-run-job
> > index ceb79800..239446b8 100755
> > --- a/sg-run-job
> > +++ b/sg-run-job
> 
> IWBN to split this out.  Firstly, please split the sg-run-job changes
> from the make-flight changes.

Sure, np.

> Secondly, can you split the sg-run-job patch into two ?  1. change the
> meaning of need_build_host but not introduce the FreeBSD versions;
> 2. add the FreeBSD cases.  sg-run-job is a bit fragile to do this kind
> of work in.
> 
> Also I'm afraid you're going to find some conflicts with my recent
> syslog work.  You will probably want to rebase onto pretest, which I
> think is very likely to pass and be pushed soon.

I think I've already sorted those out.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel