[PATCH v12 00/33] osstest: FreeBSD host support

[PATCH v12 00/33] osstest: FreeBSD host support

[Roger Pau Monne]

Hello,

These are the remainig (not acked) patches of the FreeBSD host install
series.

The full branch can be found at:

git://xenbits.xen.org/people/royger/osstest.git freebsd_v12

Note that before attempting to run a FreeBSD flight an examine flight
with blessing real must be executed, so that the right memdisk
properties are set in the database. At the moment both the Massachusetts
and the internal Citrix osstest instance already have an anointed
FreeBSD image, so the examine flight can be run without issues.

Ian Jackson (2):
  anointments: Introduce anointment scheme, and mg-anoint script
  anointments: Provide ./mg-anoint list-prepared

Roger Pau Monne (31):
  osstest: make built_stash_file store a path_ runvar for each file
  osstest: move known_hosts generation to TestSupport
  osstest: introduce helper to get per-host tftp prefix
  osstest: introduce a helper to calculate the sha256 of a given file
  osstest: introduce a helper to setup a host to boot using memdisk
  osstest: add executive prefix to resource_shared_mark_ready
  osstest: introduce resource_shared_mark_ready
  osstest: add a FreeBSD host install script
  osstest: introduce build helpers for FreeBSD
  osstest: add prototypes to target_install_packages{_norec}
  osstest: introduce an OS $ho field
  osstest: add support for the FreeBSD package manager
  osstest: introduce a FreeBSD build script
  osstest: add support for runtime_IDENT_hostflags
  osstest: introduce a script to set the runtime hostflags runvar for
    FreeBSD jobs
  osstest: change the meaning of need_build_host
  osstest: add support for FreeBSD buildjobs to sg-run-job
  osstest: introduce a script to create a FreeBSD flight
  osstest: hook FreeBSD flight into cr-daily-branch
  osstest: save/retrieve the last successfully tested FreeBSD build
  netboot_memdisk: allow each host to have different append values
  ts-freebsd-host-install: fix image permissions
  sg-run-job: fix typo in the examine jobs
  HostDB: introduce set_property
  mfi-common: move set_freebsd_runvars to mfi-common
  TestSupport: introduce hostprop_putative_record
  ts-freebsd-host-install: add arguments to test memdisk append options
  ts-memdisk-try-append: introduce a script to test memdisk options
  ts-examine-hostprops-save: introduce a script to save properties
  make-hosts-flight: set runvars for FreeBSD test
  sg-run-job: hook the memdisk test into examine

 Osstest/BuildSupport.pm                |  26 ++-
 Osstest/Debian.pm                      |  36 +---
 Osstest/Executive.pm                   |   5 +-
 Osstest/HostDB/Executive.pm            |  23 ++
 Osstest/HostDB/Static.pm               |   7 +
 Osstest/JobDB/Executive.pm             |   2 +-
 Osstest/TestSupport.pm                 | 193 +++++++++++++++--
 README                                 |  16 ++
 README.planner                         |   4 +
 ap-common                              |   5 +
 ap-fetch-version                       |   8 +
 ap-fetch-version-old                   |   9 +
 ap-print-url                           |   3 +
 ap-push                                |   9 +
 cr-daily-branch                        |  84 ++++++++
 cr-for-branches                        |   2 +-
 cri-common                             |   1 +
 daily-cron-email-adhoc--freebsd-master |   1 +
 daily-cron-email-play--freebsd-master  |   1 +
 daily-cron-email-real--freebsd-master  |   4 +
 make-freebsd-flight                    |  64 ++++++
 make-hosts-flight                      |   3 +
 mfi-common                             |  31 +++
 mg-anoint                              | 376 +++++++++++++++++++++++++++++++++
 sg-run-job                             |  53 +++--
 ts-examine-hostprops-save              |  46 ++++
 ts-freebsd-build                       | 239 +++++++++++++++++++++
 ts-freebsd-host-install                | 280 ++++++++++++++++++++++++
 ts-freebsd-set-hostflags               |  78 +++++++
 ts-host-install                        |   2 +-
 ts-kernel-build                        |   4 +-
 ts-memdisk-try-append                  |  45 ++++
 ts-xen-build                           |   8 +-
 ts-xen-build-prep                      |   4 +-
 34 files changed, 1582 insertions(+), 90 deletions(-)
 create mode 100644 daily-cron-email-adhoc--freebsd-master
 create mode 100644 daily-cron-email-play--freebsd-master
 create mode 100644 daily-cron-email-real--freebsd-master
 create mode 100755 make-freebsd-flight
 create mode 100755 mg-anoint
 create mode 100755 ts-examine-hostprops-save
 create mode 100755 ts-freebsd-build
 create mode 100755 ts-freebsd-host-install
 create mode 100755 ts-freebsd-set-hostflags
 create mode 100755 ts-memdisk-try-append

-- 
2.13.5 (Apple Git-94)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v12 26/33] HostDB: introduce set_property

[Roger Pau Monne]

And provide a helper in TestSupport to use it. This allows osstest to
set host properties from test script themselves (instead of using
the mg-hosts clu).

Note that the setting of host properties is limited to flights with
intended blessing real, and it will fail for any other blessing.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Die if attempting to modify a host prop with intended blessing !=
   real.
---
 Osstest/HostDB/Executive.pm | 23 +++++++++++++++++++++++
 Osstest/HostDB/Static.pm    |  7 +++++++
 Osstest/TestSupport.pm      |  8 +++++++-
 3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/Osstest/HostDB/Executive.pm b/Osstest/HostDB/Executive.pm
index 300178bb..2a961b6d 100644
--- a/Osstest/HostDB/Executive.pm
+++ b/Osstest/HostDB/Executive.pm
@@ -51,6 +51,29 @@ END
     }
 }
 
+sub set_property($$$$) {
+    my ($hd, $ho, $prop, $val) = @_;
+    my $rmq = $dbh_tests->prepare(<<END);
+        DELETE FROM resource_properties
+               WHERE restype='host' and resname=? AND name=?
+END
+    my $addq = $dbh_tests->prepare(<<END);
+        INSERT INTO resource_properties (restype,resname,name,val)
+               VALUES ('host', ?,?,?)
+END
+    my $blessing = intended_blessing();
+
+    die "Attempting to modify host props with blessing $blessing != real"
+        if $blessing ne "real";
+
+    db_retry($dbh_tests, [qw(resources)], sub {
+        $rmq->execute($ho->{Name}, $prop);
+        if (length $val) {
+            $addq->execute($ho->{Name}, $prop, $val);
+       }
+    });
+}
+
 sub get_flags ($$) {
     my ($hd, $ho) = @_;
 
diff --git a/Osstest/HostDB/Static.pm b/Osstest/HostDB/Static.pm
index 60f5d3c2..3191c565 100644
--- a/Osstest/HostDB/Static.pm
+++ b/Osstest/HostDB/Static.pm
@@ -40,6 +40,13 @@ sub get_properties ($$$) { #method
     my ($hd, $name, $hp) = @_;
 }
 
+sub set_property($$$$) {
+    my ($hd, $ho, $prop, $val) = @_;
+
+    die
+    "Cannot set property in standalone mode for $ho->{Name} $prop = $val\n";
+}
+
 sub get_flags ($$) { #method
     my ($hd, $ho) = @_;
 
diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 585e2286..ec834da7 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -80,7 +80,7 @@ BEGIN {
                       get_target_property get_host_native_linux_console
                       hostnamepath hostnamepath_list set_runtime_hostflag
                       power_state power_cycle power_cycle_sleep
-                      serial_fetch_logs
+                      serial_fetch_logs set_host_property
                       propname_massage propname_check
          
                       get_stashed open_unique_stashfile compress_stashed
@@ -1183,6 +1183,12 @@ sub get_host_property ($$;$) {
     return defined($val) ? $val : $defval;
 }
 
+sub set_host_property ($$$) {
+    my ($ho,$prop,$val) = @_;
+
+    $mhostdb->set_property($ho, $prop, $val);
+}
+
 sub get_target_property ($$;$);
 sub get_target_property ($$;$) {
     my ($ho, $prop, $defval) = @_;
-- 
2.13.5 (Apple Git-94)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v12 29/33] ts-freebsd-host-install: add arguments to test memdisk append options

[Roger Pau Monne]

This is needed in order to figure out which memdisk options should be
used to boot the images on each specific box.

Note that when passed the --recordappend argument upon success the
script stores the tentative host property in the runvars.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Fix commit message.

Changes since v1:
 - Provide a --recordappend argument to force the recording the
   memdisk parameters.
 - Exit gracefully if a bootonly test is attempted against a
   non-supported architecture.
 - Use NONE instead of an empty string when calling
   setup_netboot_memdisk if nothing should be appended.
 - Do not perform any arch test in ts-freebsd-host-install.
---
 ts-freebsd-host-install | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/ts-freebsd-host-install b/ts-freebsd-host-install
index 483b9aec..50af5dd3 100755
--- a/ts-freebsd-host-install
+++ b/ts-freebsd-host-install
@@ -41,6 +41,23 @@ use Osstest::TestSupport;
 
 tsreadconfig();
 
+our $bootonly;
+our $memdisk_append;
+our $record_append;
+while (@ARGV && $ARGV[0] =~ m/^-/g) {
+    if ($ARGV[0] =~ m/^--memdiskappend=(.*)/) {
+        $memdisk_append = $1;
+    } elsif ($ARGV[0] eq "--testboot") {
+        $memdisk_append //= "NONE";
+        $bootonly = 1;
+    } elsif ($ARGV[0] eq "--recordappend") {
+        $record_append = 1;
+    } else {
+        die "Unknown argument $ARGV[0]";
+    }
+    shift @ARGV;
+}
+
 our ($whhost) = @ARGV;
 $whhost ||= 'host';
 our $ho= selecthost($whhost);
@@ -95,7 +112,7 @@ END
 
     # Setup the pxelinux config file
     logm("Booting from installer image at $pxeimg");
-    setup_netboot_memdisk($ho, $pxeimg);
+    setup_netboot_memdisk($ho, $pxeimg, $memdisk_append);
 }
 
 sub install () {
@@ -247,6 +264,12 @@ power_state($ho, 1);
 logm("Waiting for the installer to boot");
 await_tcp(get_timeout($ho,'reboot',$timeout), 5, $ho);
 
+if ($bootonly) {
+    hostprop_putative_record($ho, "MemdiskAppend", $memdisk_append)
+        if $record_append;
+    exit 0;
+}
+
 # Next boot will be from local disk
 setup_netboot_local($ho);
 
-- 
2.13.5 (Apple Git-94)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v12 31/33] ts-examine-hostprops-save: introduce a script to save properties

[Roger Pau Monne]

This script turns the properties stored in the runvars using the
format hostprop/$ident/$prop=$val into host properties stored in the
database.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Call selecthost based on the idents passed in the putative
   hostprops runvar.
 - Fix commit message.
 - Use '/' instead of '_' in the runvars.
 - Do a dry run if flight blessing != real.
 - Fix parentheses indentation.

Changes since v1:
 - Select a host for setting the properties.
 - Print a message before exiting if blessing != real.
 - Skip properties that don't contain the selected host.
---
 ts-examine-hostprops-save | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 ts-examine-hostprops-save

diff --git a/ts-examine-hostprops-save b/ts-examine-hostprops-save
new file mode 100755
index 00000000..99a4627b
--- /dev/null
+++ b/ts-examine-hostprops-save
@@ -0,0 +1,46 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2017 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+unshift @INC, qw(.);
+use Osstest;
+use Osstest::TestSupport;
+
+tsreadconfig();
+
+our $blessing = intended_blessing();
+
+logm("setting host properties");
+
+foreach my $k (sort keys %r) {
+    next unless $k =~ m/^hostprop\/([^\/]*)\/([^\/]*)$/;
+    my $ho = selecthost($1);
+    my $prop = $2;
+
+    logm("recording for $ho->{Name} $prop=$r{$k}");
+
+    # NB: in order to aid debug only attempt to save the host props
+    # on flights with real blessing, for the rest just do a dry run.
+    if ($blessing eq "real") {
+        set_host_property($ho, $prop, $r{$k});
+    } else {
+        logm("not saving host prop with blessing $blessing != real");
+    }
+}
-- 
2.13.5 (Apple Git-94)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH v12 33/33] sg-run-job: hook the memdisk test into examine

[Roger Pau Monne]

Hook the memdisk parameter detection and the saving of the host
properties into the examine jobs.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Changes since v2:
 - Do not pass a host ident to ts-examine-hostprops-save.
 - Use .- for ts-memdisk-try-append so that the rest of the job will
   run even if this step fails.

Changes since v1:
 - Run the memdisk test first (so that we don't leave the host in a
   weird state).
 - Pass a host to the examine-hostprops-save.
---
 sg-run-job | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index ed1ed3c8..de6e3f76 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -655,6 +655,7 @@ proc examine-host-install-xen {} {
 proc examine-host-examine {install} {
     global ok
     catching-otherwise fail {
+	run-ts -.  =            ts-memdisk-try-append + host
 	examine-host-install-$install
 	run-ts .   =            ts-examine-serial-pre + host
 	run-ts .   reboot       ts-host-reboot        + host
@@ -663,6 +664,7 @@ proc examine-host-examine {install} {
     if {$ok} {
 	run-ts -.  =           ts-examine-serial-post + host
 	run-ts .   =           ts-examine-logs-save   + host
+	run-ts .   =           ts-examine-hostprops-save
     }
 }
 
-- 
2.13.5 (Apple Git-94)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v12 26/33] HostDB: introduce set_property

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v12 26/33] HostDB: introduce set_property"):
> And provide a helper in TestSupport to use it. This allows osstest to
> set host properties from test script themselves (instead of using
> the mg-hosts clu).
> 
> Note that the setting of host properties is limited to flights with
> intended blessing real, and it will fail for any other blessing.
...
> +    die "Attempting to modify host props with blessing $blessing != real"
> +        if $blessing ne "real";

Thanks.  You should write "intended blessing" in the commit message
and error message, since there is also a current blessing etc.

With that changed,

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>


Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v12 29/33] ts-freebsd-host-install: add arguments to test memdisk append options

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v12 29/33] ts-freebsd-host-install: add arguments to test memdisk append options"):
> This is needed in order to figure out which memdisk options should be
> used to boot the images on each specific box.
> 
> Note that when passed the --recordappend argument upon success the
> script stores the tentative host property in the runvars.

Last time, I said:

  Roger Pau Monne writes ("[PATCH OSSTEST v2 07/11] ts-freebsd-host-install: add arguments to test memdisk append options"):
  > This is needed in order to figure out which memdisk options should be
  > used to boot the images on each specific box.
  > 
  > Note that upon success the script stores the tentative host property
  > in the runvars.
  ...
  > +    } elsif ($ARGV[0] eq "--recordappend") {
  > +        $record_append = 1;
  ...
  > +if ($bootonly) {
  > +    hostprop_putative_record($ho, "MemdiskAppend", $memdisk_append)
  > +        if $record_append;
  > +    exit 0;

  This is surely wrong.

The same code seems to be here, unchanged:

> +if ($bootonly) {
> +    hostprop_putative_record($ho, "MemdiskAppend", $memdisk_append)
> +        if $record_append;
> +    exit 0;
> +}

What I mean is that you only do the work for $record_append if
$bootonly is also set.  If --record-append is meaningful only with
--test-boot then you should die if it's specified without --test-boot.

Also, I have just noticed that the option names ought to be
   --record-append --test-boot --memdisk-append
(ie with the hyphens that are conventional in multi-word long option
names).

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v12 33/33] sg-run-job: hook the memdisk test into examine

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v12 33/33] sg-run-job: hook the memdisk test into examine"):
> Hook the memdisk parameter detection and the saving of the host
> properties into the examine jobs.
> 
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Changes since v2:
>  - Do not pass a host ident to ts-examine-hostprops-save.
>  - Use .- for ts-memdisk-try-append so that the rest of the job will
>    run even if this step fails.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v12 31/33] ts-examine-hostprops-save: introduce a script to save properties

[Ian Jackson]

Roger Pau Monne writes ("[PATCH v12 31/33] ts-examine-hostprops-save: introduce a script to save properties"):
> This script turns the properties stored in the runvars using the
> format hostprop/$ident/$prop=$val into host properties stored in the
> database.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Although,

> +foreach my $k (sort keys %r) {
> +    next unless $k =~ m/^hostprop\/([^\/]*)\/([^\/]*)$/;
> +    my $ho = selecthost($1);
> +    my $prop = $2;
> +
> +    logm("recording for $ho->{Name} $prop=$r{$k}");
> +
> +    # NB: in order to aid debug only attempt to save the host props
> +    # on flights with real blessing, for the rest just do a dry run.
> +    if ($blessing eq "real") {
> +        set_host_property($ho, $prop, $r{$k});
> +    } else {
> +        logm("not saving host prop with blessing $blessing != real");

This will be annoyingly noisy in non-real flights with multiple
hostprops.

Better would be to move the $blessing eq "real" test to the outside of
the loop and have a variable to day whether to actually set things.
Then the log message about "not saving" could be printed once.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v12 00/33] osstest: FreeBSD host support

[Ian Jackson]

We have decided:

 We will push the anoint and examine parts of this series to osstest
 pretest.  (You're going to give me a suitable branch on Monday.)
 This should work because we have anointed FreeBSD builds already.

 If this works (passes pretest) we will then run a special invocation
 of the real examination flight (which normally only runs once a
 month) to collect the new host properties.

 All being well we will then run mg-branch-setup and try to run the
 full series (including the new freebsd tests) with --real, and then,
 if that is good, push it to pretest (where the self-test should then
 be a formality).

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v12 00/33] osstest: FreeBSD host support

[Roger Pau Monné]

On Fri, Oct 20, 2017 at 04:32:44PM +0100, Ian Jackson wrote:
> We have decided:
> 
>  We will push the anoint and examine parts of this series to osstest
>  pretest.  (You're going to give me a suitable branch on Monday.)
>  This should work because we have anointed FreeBSD builds already.

Sorry for the delay, had to cherry-pick some commits from the FreeBSD
host install series in order for the examine one to work. I've pushed
this to the following branch:

git://xenbits.xen.org/people/royger/osstest.git examine

Here is the output of a sample examine flight with the contents of the
branch:

http://osstest.xs.citrite.net/~osstest/testlogs/logs/72345/

Note that patch "ts-freebsd-host-install: add arguments to test
memdisk append options" is missing an Ack (you requested changes to
it).

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH v12 00/33] osstest: FreeBSD host support

[Ian Jackson]

Roger Pau Monné writes ("Re: [PATCH v12 00/33] osstest: FreeBSD host support"):
> Sorry for the delay, had to cherry-pick some commits from the FreeBSD
> host install series in order for the examine one to work. I've pushed
> this to the following branch:
> 
> git://xenbits.xen.org/people/royger/osstest.git examine

I have now rebased this onto my small queue (of patches to admin tools
which should not cause trouble) and pushed it to pretest.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel