[OSSTEST PATCH v6 0/3] Have OpenStack tested on top of xen's master and libvirt's master.

[OSSTEST PATCH v6 0/3] Have OpenStack tested on top of xen's master and libvirt's master.

[Anthony PERARD]

Hi,

I have looked into getting OpenStack been tested on the latest Xen via
osstest.

The ts-openstack-deploy script does prepare a bit more the host, clone
devstack and other OpenStack trees, then run ./stack.sh, which is a bit
like raisin and deploy OpenStack on the host. Once the machine is ready,
the integration test suite from OpenStack, Tempest, is started by
ts-openstack-tempest.

For the last patch that create a flight plan, I've tested only with
`./standalone make-flight openstack` and looked into the database to check
that only the necessary build job and the only test job are there.
(build-amd64{,-pvops,-libvirt} test-amd64-amd64-devstack)

Thanks.

Changes in V6:
  - rebased
  - fix ts-openstack-deploy script to work with newer devstach and new debian.

Change in V5:
  - on small change in patch 1, but I keeped the acked-by
  - few changes in the way the flight is created (in the last patch)

Change in V4:
  few changes listed in second and third patch, mostly cleanup.

Change in V3:
  - Track Nova tree instead of devstack.
      Nova is the service we care about from a Xen point of view.
      Also it is updated much more often than devstack.
  - Cleanups, see change log in patches.

Changes in V2:
  - no more Osstest::Toolstack::OpenStack.
  - osstest now clone every single tree that devstack is going to need.
    And ./stack.sh should fail if one tree is missing.
  - avoid build-*-xsm for an openstack flight
  - rename ts-devstack to ts-openstack-devstack
  - New test script ts-openstack-tempest
  - Add CONFIG_CRYPTO_XTS=m to the kernel build.
  - new possible runvar $dom0_mem to control dom0 memory
  - several fix to have volume tests working.
  - have OpenStack deploy from it's builddir instead of /opt/stack
  - have 4GB for dom0 instead of relying on balloning.

Anthony PERARD (3):
  ts-openstack-deploy: Deploy OpenStack on a host with devstack
  ts-openstack-tempest: Run Tempest to check OpenStack
  Create a flight to test OpenStack with xen-unstable and libvirt

 ap-common            |  12 ++
 ap-fetch-version     |   4 +
 ap-fetch-version-old |   5 +
 ap-print-url         |   3 +
 ap-push              |   5 +
 cr-daily-branch      |   8 ++
 cr-for-branches      |   2 +-
 cri-common           |   1 +
 make-flight          |  52 ++++++++
 sg-run-job           |   6 +
 ts-openstack-deploy  | 338 +++++++++++++++++++++++++++++++++++++++++++++++++++
 ts-openstack-tempest |  65 ++++++++++
 12 files changed, 500 insertions(+), 1 deletion(-)
 create mode 100755 ts-openstack-deploy
 create mode 100755 ts-openstack-tempest

-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v6 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack

[Anthony PERARD]

This script installs any necessary packages and clones all of the OpenStack
trees which are used by devstack to deploy OpenStack.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>

---
Changes in V6:
- rebased
- fix issues due to new debian and newer devstack:
  - add missing libvirt group
  - switch back to old nova-network instead of neutron
  - have devstack use 'service' instead of 'systemctl' to restart
    services

Only change in V5:
- edit stackrc from devstack file to change the hardcoded path DEST

No change in V4:
- acked

Change in V3:
- Use host as argument to run the job.
- Use selectjob() and get rid of the unused $gho.
- Use target_jobdir() instead of builddirsprops().
- Remove GIT_BASE from devstack config.
- Rename the script to ts-openstack-deploy (from ts-openstack-devstack).
---
 sg-run-job          |   5 +
 ts-openstack-deploy | 338 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 343 insertions(+)
 create mode 100755 ts-openstack-deploy

diff --git a/sg-run-job b/sg-run-job
index 9f8d003..5146dd1 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -474,6 +474,11 @@ proc run-job/test-rumprun {} {
                  ts-guest-destroy-hard        host   $g   +
 }
 
+proc need-hosts/test-devstack {} { return host }
+proc run-job/test-devstack {} {
+  run-ts . = ts-openstack-deploy host
+}
+
 if {[file exists sg-run-job-adhoc]} {
     source sg-run-job-adhoc
 }
diff --git a/ts-openstack-deploy b/ts-openstack-deploy
new file mode 100755
index 0000000..5758f82
--- /dev/null
+++ b/ts-openstack-deploy
@@ -0,0 +1,338 @@
+#!/usr/bin/perl
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho = selecthost($whhost);
+our $builddir = target_jobdir($ho);
+
+sub tgt_init ();
+
+sub packages () {
+  # Install open-iscsi ahead of devstack ...
+  target_install_packages($ho, qw(git sudo open-iscsi));
+  # ... and start open-iscsi to have /etc/iscsi/initiatorname.iscsi
+  # generated. This is done on install on Ubuntu.
+  target_cmd_root($ho, 'service open-iscsi start');
+}
+
+sub checkout () {
+  prepbuilddirs();
+  build_clone($ho, 'cinder', $builddir, 'cinder');
+  build_clone($ho, 'devstack', $builddir, 'devstack');
+  build_clone($ho, 'glance', $builddir, 'glance');
+  build_clone($ho, 'keystone', $builddir, 'keystone');
+  build_clone($ho, 'nova', $builddir, 'nova');
+  build_clone($ho, 'requirements', $builddir, 'requirements');
+  build_clone($ho, 'tempest', $builddir, 'tempest');
+
+  my $vg = target_choose_vg($ho, 10*1024); # 10GB
+  target_putfilecontents_stash($ho, 60, <<END, $builddir.'/devstack/local.conf');
+[[local|localrc]]
+# Everything should be cloned by osstest, so devstack don't have to do it
+ERROR_ON_CLONE=True
+USE_SCREEN=False
+ADMIN_PASSWORD=secretadmin
+DATABASE_PASSWORD=secretdatabase
+RABBIT_PASSWORD=secretrabbit
+SERVICE_PASSWORD=secretservice
+SERVICE_TOKEN=atokenserviced
+# make it small because there is no way to not have this lvm volume created
+VOLUME_BACKING_FILE_SIZE=500M
+DEST=$builddir
+LOGFILE=\$DEST/logs/stack.sh.log
+# stackrc set this but don't take \$DEST into account
+SERVICE_DIR=\${DEST}/status
+LOG_COLOR=False
+LIBVIRT_TYPE=xen
+disable_service horizon
+disable_service n-novnc
+disable_service dstat
+enable_service n-obj
+# Disable neutron and switch back to nova-network
+disable_service q-svc
+disable_service q-dhcp
+disable_service q-meta
+disable_service q-agt
+disable_service q-l3
+enable_service n-net
+[[post-config|\$CINDER_CONF]]
+[lvmdriver-1]
+volume_group = $vg
+END
+
+  # stackrc does not take $DEST from local.conf into account, so fix it here
+  target_editfile($ho, "$builddir/devstack/stackrc", sub {
+      while (<EI>) {
+        if (m/^DEST=\/opt\/stack$/) {
+          s/DEST=\/opt\/stack/DEST=$builddir/;
+        }
+        print EO or die $!;
+      }
+  });
+
+  # libvirt is already installed, but not as a package, so avoid installation of
+  # the libvirt package with devstack
+  target_editfile($ho, "$builddir/devstack/files/debs/nova", sub {
+      while (<EI>) {
+        next if m/.*libvirt.*/;
+        print EO or die $!;
+      }
+  });
+  target_editfile($ho, "$builddir/devstack/lib/nova_plugins/functions-libvirt", sub {
+      while (<EI>) {
+        next if m/install_package.*libvirt.*/;
+        print EO or die $!;
+      }
+  });
+
+  # devstack blindly assume that systemd is used if systemctl is present
+  target_editfile($ho, "$builddir/devstack/functions-common", sub {
+      while (<EI>) {
+        if (m%\[ -x /bin/systemctl%) {
+          s%\[ -x /bin/systemctl \]%false%
+        }
+        print EO or die $!;
+      }
+  });
+
+  # OpenStack needs access to libvirt from a user.
+  target_cmd_root($ho, <<END);
+    if ! getent group libvirt >/dev/null; then
+      groupadd libvirt
+    fi
+    cat >> /etc/libvirt/libvirtd.conf <<EOF
+unix_sock_group = "libvirt"
+unix_sock_ro_perms = "0777"
+unix_sock_rw_perms = "0770"
+EOF
+    # For unknown reason, restart fail when done by devstack
+    # so stop the service here, and devstack will start it.
+    service libvirtd stop
+END
+
+  # devstack is going to setup the host, install some dependency.
+  target_putfilecontents_root_stash($ho, 100, <<END,"/etc/sudoers.d/devstack");
+osstest ALL=(ALL) NOPASSWD:ALL
+END
+
+  target_putfilecontents_root_stash($ho, 60, tgt_init(), "/etc/init.d/tgt");
+  target_cmd_root($ho, <<END);
+    chmod +x /etc/init.d/tgt
+END
+}
+
+sub deploy() {
+  target_cmd($ho, <<END, 1800);
+    cd $builddir/devstack
+    ./stack.sh
+END
+}
+
+packages();
+checkout();
+deploy();
+
+# This is missing from Debian but required by devstack
+# Got it from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=577925
+sub tgt_init () {
+  return <<'END';
+#!/bin/bash
+### BEGIN INIT INFO
+# Provides:          tgtd
+# Required-Start:    $remote_fs $syslog
+# Required-Stop:     $remote_fs $syslog
+# Should-Start:      zfs
+# Should-Stop:       zfs
+# Default-Start:     2 3 4 5
+# Default-Stop:      0 1 6
+# Short-Description: iscsi target daemon
+# Description:       iscsi target daemon
+### END INIT INFO
+
+DESC="target framework daemon"
+NAME=tgtd
+DAEMON=/usr/sbin/${NAME}
+
+TGTD_CONFIG=/etc/tgt/targets.conf
+
+TASK=$1
+
+. /lib/lsb/init-functions
+
+[ -x $DAEMON ] || exit 0
+
+start()
+{
+        log_daemon_msg "Starting $DESC" "$NAME"
+        # Start tgtd first.
+        tgtd &>/dev/null
+        RETVAL=$?
+        if [ "$RETVAL" -ne 0 ] ; then
+                log_end_msg 1
+                exit 1
+        else
+                log_end_msg 0
+        fi
+        # Put tgtd into "offline" state until all the targets are configured.
+        # We don't want initiators to (re)connect and fail the connection
+        # if it's not ready.
+        tgtadm --op update --mode sys --name State -v offline
+        # Configure the targets.
+        tgt-admin -e -c $TGTD_CONFIG
+        # Put tgtd into "ready" state.
+        tgtadm --op update --mode sys --name State -v ready
+}
+
+stop()
+{
+        if [ "$RUNLEVEL" == 0 -o "$RUNLEVEL" == 6 ] ; then
+            forcedstop
+        fi
+        log_daemon_msg "Stopping $DESC" "$NAME"
+        # Remove all targets. It only removes targets which are not in use.
+        tgt-admin --update ALL -c /dev/null &>/dev/null
+        # tgtd will exit if all targets were removed
+        tgtadm --op delete --mode system &>/dev/null
+        RETVAL=$?
+        if [ "$RETVAL" -eq 107 ] ; then
+                if [ "$TASK" != "restart" ] ; then
+                        log_end_msg 1
+                        exit 1
+                else
+                        log_end_msg 0
+                fi
+        elif [ "$RETVAL" -ne 0 ] ; then
+                log_end_msg 1
+                echo "Some initiators are still connected - could not stop tgtd"
+                exit 2
+        else
+                log_end_msg 0
+        fi
+        echo -n
+}
+
+forcedstop()
+{
+        # NOTE: Forced shutdown of the iscsi target may cause data corruption
+        # for initiators that are connected.
+        echo "Force-stopping target framework daemon"
+        # Offline everything first. May be needed if we're rebooting, but
+        # expect the initiators to reconnect cleanly when we boot again
+        # (i.e. we don't want them to reconnect to a tgtd which is still
+        # working, but the target is gone).
+        tgtadm --op update --mode sys --name State -v offline &>/dev/null
+        RETVAL=$?
+        if [ "$RETVAL" -eq 107 ] ; then
+            echo "tgtd is not running"
+            [ "$TASK" != "restart" ] && exit 1
+        else
+            tgt-admin --offline ALL
+            # Remove all targets, even if they are still in use.
+            tgt-admin --update ALL -c /dev/null -f
+            # It will shut down tgtd only after all targets were removed.
+            tgtadm --op delete --mode system
+            RETVAL=$?
+            if [ "$RETVAL" -ne 0 ] ; then
+                echo "Failed to shutdown tgtd"
+                exit 1
+            fi
+        fi
+        echo -n
+}
+
+reload()
+{
+        log_daemon_msg "Reloading configuration of $DESC" "$NAME"
+        # Update configuration for targets. Only targets which
+        # are not in use will be updated.
+        tgt-admin --update ALL -c $TGTD_CONFIG &>/dev/null
+        RETVAL=$?
+        if [ "$RETVAL" -eq 107 ] ; then
+                log_end_msg 1
+                echo "tgtd is not running"
+                exit 1
+        fi
+        log_end_msg 0
+}
+
+forcedreload()
+{
+        log_daemon_msg "Forced-reload configuration of $DESC" "$NAME"
+        # Update configuration for targets, even those in use.
+        tgt-admin --update ALL -f -c $TGTD_CONFIG &>/dev/null
+        RETVAL=$?
+        if [ "$RETVAL" -eq 107 ] ; then
+                log_end_msg 1
+                echo "tgtd is not running"
+                exit 1
+        else
+                log_end_msg 0
+        fi
+}
+
+status()
+{
+        # Don't name this script "tgtd"...
+        TGTD_PROC=$(ps -C tgtd | grep -c tgtd)
+        if [ "$TGTD_PROC" -eq 2 ] ; then
+            echo "tgtd is running. Run 'tgt-admin -s' to see detailed target info."
+        else
+            echo "tgtd is NOT running."
+        fi
+}
+
+case $1 in
+        start)
+                start
+                ;;
+        stop)
+                stop
+                ;;
+        forcedstop)
+                forcedstop
+                ;;
+        restart)
+                TASK=restart
+                stop && start
+                ;;
+        forcedrestart)
+                TASK=restart
+                forcedstop && start
+                ;;
+        reload)
+                reload
+                ;;
+        force-reload)
+                forcedreload
+                ;;
+        status)
+                status
+                ;;
+        *)
+                echo "Usage: $0 {start|stop|forcedstop|restart|forcedrestart|reload|force-reload|status}"
+                exit 2
+                ;;
+esac
+END
+}
-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v6 2/3] ts-openstack-tempest: Run Tempest to check OpenStack

[Anthony PERARD]

This script runs the OpenStack integration test suite, Tempest.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>

---
No change in V5

Change in V4:
- use \Q\E for tests names
- write the full name of the tests to skip
- use push @ignored_test then join()
- use variables to store common prefix
- rewrite comments

Change in V3:
- Use host as argument of the script.
- Use selecthost() and get rid of $gho.
- Use target_jobdir() instead of builddirsprops().
- Put the ignored Tempest tests into a var and try to explain why there are
  skip.
---
 sg-run-job           |  1 +
 ts-openstack-tempest | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 66 insertions(+)
 create mode 100755 ts-openstack-tempest

diff --git a/sg-run-job b/sg-run-job
index 5146dd1..932d7bf 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -477,6 +477,7 @@ proc run-job/test-rumprun {} {
 proc need-hosts/test-devstack {} { return host }
 proc run-job/test-devstack {} {
   run-ts . = ts-openstack-deploy host
+  run-ts . = ts-openstack-tempest host
 }
 
 if {[file exists sg-run-job-adhoc]} {
diff --git a/ts-openstack-tempest b/ts-openstack-tempest
new file mode 100755
index 0000000..91c5f3a
--- /dev/null
+++ b/ts-openstack-tempest
@@ -0,0 +1,65 @@
+#!/usr/bin/perl
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+our ($whhost) = @ARGV;
+$whhost ||= 'host';
+our $ho = selecthost($whhost);
+our $builddir = target_jobdir($ho);
+
+sub tempest() {
+  my @ignored_tests;
+  my $scenario = 'tempest.scenario';
+  my $volume_boot_pattern =
+      "$scenario.test_volume_boot_pattern.TestVolumeBootPattern";
+  my $shelve_instance = "$scenario.test_shelve_instance.TestShelveInstance";
+
+  # Ignore tests which try to boot a guest with /dev/vda as boot device name.
+  push @ignored_tests,
+      "^\Q$volume_boot_pattern.test_volume_boot_pattern\E";
+  push @ignored_tests,
+      "^\Q$volume_boot_pattern.test_create_ebs_image_and_check_boot\E";
+  push @ignored_tests,
+      "^\Q$shelve_instance.test_shelve_volume_backed_instance\E";
+
+  # Those tests access a volume through iSCSI. This does not work when both the
+  # server and client of iSCSI are on the same Xen host, Linux 4.0 is the first
+  # Linux to have a fix.
+  push @ignored_tests,
+      "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E";
+  push @ignored_tests,
+      "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E";
+
+  # This regex below select the tests to run and exclude the ones marked as
+  # slow as well as the explicit tests listed above.
+  # It is based on the one that can be found in tempest.git/tox.ini
+  # in the section [testenv:full].
+  my $ignored_tests = join("|", @ignored_tests);
+  my $regex =
+      "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))";
+
+  target_cmd($ho, <<END, 7200);
+    $builddir/tempest/run_tempest.sh --virtual-env -- --concurrency=2 '$regex'
+END
+}
+
+tempest();
-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[OSSTEST PATCH v6 3/3] Create a flight to test OpenStack with xen-unstable and libvirt

[Anthony PERARD]

This patch should create a flight "openstack-nova", with those jobs:
  build-amd64
  build-amd64-xsm
  build-amd64-pvops
  build-amd64-libvirt
  test-amd64-amd64-devstack
  test-amd64-amd64-devstack-xsm

About the runvars revision_* of test-*-*-devstack:
  only REVISION_OPENSTACK_NOVA is set, the others are unset.
  Empty revision_* runvar would clone the default branch, which should
  be master for every openstack repos.

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>

---
Change in V6:
- rebased

Change in V5:
- rename flight openstack to openstack-nova
- add -xsm variant of the test
- run test-devstack only on openstack-nova flight

Change in V4:
- also skip build-*-oldkern in make flight
- fix select_xenbranch
- set revision_*=$REVISION_OPENSTACK_* in make-flight
  (was revision_*=master before)
  only REVISION_OPENSTACK_NOVA is set, the others are unset.
  empty revision_* runvar would clone the default branch, which should
  be master for every openstack repos

Change in V3:
- Switch to "track" Nova tree instead of devstack.
    Nova is the service we care about from a Xen point of view.
    Also it is updated much more often than devstack.
- Use TREE_OPENSTACK_ as prefix for all trees variables.
- Change the filter, keep only *-devstack jobs.
- Add stuff into ./ap-push
- Add stuff into ./cr-daily-branch.
- Add 'openstack' into ./cr-for-branches.
---
 ap-common            | 12 ++++++++++++
 ap-fetch-version     |  4 ++++
 ap-fetch-version-old |  5 +++++
 ap-print-url         |  3 +++
 ap-push              |  5 +++++
 cr-daily-branch      |  8 ++++++++
 cr-for-branches      |  2 +-
 cri-common           |  1 +
 make-flight          | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 9 files changed, 91 insertions(+), 1 deletion(-)

diff --git a/ap-common b/ap-common
index cbb815c..b148170 100644
--- a/ap-common
+++ b/ap-common
@@ -54,6 +54,17 @@
 : ${PUSH_TREE_OVMF:=$XENBITS:/home/xen/git/osstest/ovmf.git}
 : ${BASE_TREE_OVMF:=git://xenbits.xen.org/osstest/ovmf.git}
 
+: ${GIT_OPENSTACK_ORG:=git://git.openstack.org}
+: ${TREE_OPENSTACK_CINDER:=$GIT_OPENSTACK_ORG/openstack/cinder.git}
+: ${TREE_OPENSTACK_DEVSTACK:=$GIT_OPENSTACK_ORG/openstack-dev/devstack.git}
+: ${TREE_OPENSTACK_GLANCE:=$GIT_OPENSTACK_ORG/openstack/glance.git}
+: ${TREE_OPENSTACK_KEYSTONE:=$GIT_OPENSTACK_ORG/openstack/keystone.git}
+: ${TREE_OPENSTACK_NOVA:=$GIT_OPENSTACK_ORG/openstack/nova.git}
+: ${TREE_OPENSTACK_REQUIREMENTS:=$GIT_OPENSTACK_ORG/openstack/requirements.git}
+: ${TREE_OPENSTACK_TEMPEST:=$GIT_OPENSTACK_ORG/openstack/tempest.git}
+: ${PUSH_TREE_OPENSTACK_NOVA:=$XENBITS:/home/xen/git/osstest/openstack-nova.git}
+: ${BASE_TREE_OPENSTACK_NOVA:=git://xenbits.xen.org/osstest/openstack-nova.git}
+
 : ${TREE_LINUXFIRMWARE:=git://xenbits.xen.org/osstest/linux-firmware.git}
 : ${PUSH_TREE_LINUXFIRMWARE:=$XENBITS:/home/osstest/ext/linux-firmware.git}
 : ${UPSTREAM_TREE_LINUXFIRMWARE:=$GIT_KERNEL_ORG/pub/scm/linux/kernel/git/firmware/linux-firmware.git}
@@ -82,6 +93,7 @@ fi
 : ${LOCALREV_SEABIOS:=daily-cron.$branch}
 : ${LOCALREV_OVMF:=daily-cron.$branch}
 : ${LOCALREV_XTF:=daily-cron.$branch}
+: ${LOCALREV_OPENSTACK_NOVA:=daily-cron.$branch}
 
 : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27}
 
diff --git a/ap-fetch-version b/ap-fetch-version
index a107c93..a714ee2 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -106,6 +106,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
 	;;
+openstack-nova)
+	repo_tree_rev_fetch_git openstack-nova \
+		$TREE_OPENSTACK_NOVA master $LOCALREV_OPENSTACK_NOVA
+	;;
 osstest)
         if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
 	    git update-ref -m "Arranging to test HEAD" \
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index 3cbc176..6dddbb7 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -35,6 +35,7 @@ check_ap_fetch_placeholders
 : ${BASE_LOCALREV_XTF:=daily-cron.$branch.old}
 : ${BASE_LOCALREV_OVMF:=daily-cron.$branch.old}
 : ${BASE_TAG_LIBVIRT:=xen-tested-master}
+: ${BASE_LOCALREV_OPENSTACK_NOVA:=daily-cron.$branch.old}
 
 if info_linux_tree "$branch"; then
 	repo_tree_rev_fetch_git linux \
@@ -114,6 +115,10 @@ ovmf)
 	repo_tree_rev_fetch_git ovmf \
 		$BASE_TREE_OVMF xen-tested-master $BASE_LOCALREV_OVMF
 	;;
+openstack-nova)
+	repo_tree_rev_fetch_git openstack-nova \
+		$BASE_TREE_OPENSTACK_NOVA xen-tested-master $BASE_LOCALREV_OPENSTACK_NOVA
+	;;
 osstest)
 	if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
 	    git fetch -f $HOME/testing.git production:ap-fetch
diff --git a/ap-print-url b/ap-print-url
index 93c14b3..6f4e6b1 100755
--- a/ap-print-url
+++ b/ap-print-url
@@ -67,6 +67,9 @@ ovmf)
 osstest)
 	echo none:;
 	;;
+openstack-nova)
+	echo $TREE_OPENSTACK_NOVA
+	;;
 *)
 	echo >&2 "branch $branch ?"
 	exit 1
diff --git a/ap-push b/ap-push
index a27ccc2..136d1b6 100755
--- a/ap-push
+++ b/ap-push
@@ -41,6 +41,7 @@ TREE_RUMPRUN=$PUSH_TREE_RUMPRUN
 TREE_SEABIOS=$PUSH_TREE_SEABIOS
 TREE_OVMF=$PUSH_TREE_OVMF
 TREE_XTF=$PUSH_TREE_XTF
+TREE_OPENSTACK_NOVA=$PUSH_TREE_OPENSTACK_NOVA
 
 if info_linux_tree "$branch"; then
 	cd $repos/linux
@@ -129,6 +130,10 @@ ovmf)
 	cd $repos/ovmf
 	git push $TREE_OVMF $revision:refs/heads/xen-tested-master
 	;;
+openstack-nova)
+	cd $repos/openstack-nova
+	git push $TREE_OPENSTACK_NOVA $revision:refs/heads/xen-tested-master
+	;;
 osstest)
 	git push $HOME/testing.git $revision:production
 	if [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then
diff --git a/cr-daily-branch b/cr-daily-branch
index 158554d..45a2f69 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -207,6 +207,10 @@ if [ "x$REVISION_LINUXFIRMWARE" = x ]; then
 	determine_version REVISION_LINUXFIRMWARE linuxfirmware LINUXFIRMWARE
         export REVISION_LINUXFIRMWARE
 fi
+if [ "x$REVISION_OPENSTACK_NOVA" = x ]; then
+        determine_version REVISION_OPENSTACK_NOVA openstack-nova OPENSTACK_NOVA
+        export REVISION_OPENSTACK_NOVA
+fi
 
 if [ "x$branch" != "xxen-unstable" ]; then
         export REVISION_LINUX_OLD=disable
@@ -260,6 +264,10 @@ ovmf)
 	realtree=ovmf
 	NEW_REVISION=$REVISION_OVMF
 	;;
+openstack-nova)
+	realtree=openstack-nova
+	NEW_REVISION=$REVISION_OPENSTACK_NOVA
+	;;
 *)
 	NEW_REVISION=''
 	wantpush=false
diff --git a/cr-for-branches b/cr-for-branches
index 5e8b1a4..ebe8986 100755
--- a/cr-for-branches
+++ b/cr-for-branches
@@ -31,7 +31,7 @@ scriptoptions="$1"; shift
 LOGFILE=tmp/cr-for-branches.log
 export LOGFILE
 
-: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf ${EXTRA_BRANCHES}}
+: ${BRANCHES:=osstest xen-4.0-testing xen-4.1-testing xen-4.2-testing xen-4.3-testing xen-4.4-testing xen-4.5-testing xen-4.6-testing xen-4.7-testing xen-unstable qemu-mainline qemu-upstream-unstable qemu-upstream-4.2-testing qemu-upstream-4.3-testing qemu-upstream-4.4-testing qemu-upstream-4.5-testing qemu-upstream-4.6-testing qemu-upstream-4.7-testing linux-4.1 linux-3.18 linux-3.16 linux-3.14 linux-3.10 linux-3.4 linux-arm-xen seabios ovmf xtf openstack-nova ${EXTRA_BRANCHES}}
 export BRANCHES
 
 fetchwlem=$wlem
diff --git a/cri-common b/cri-common
index ac360de..f48dfa5 100644
--- a/cri-common
+++ b/cri-common
@@ -82,6 +82,7 @@ select_xenbranch () {
 	ovmf)			tree=ovmf;	xenbranch=xen-unstable ;;
 	distros-*)		tree=none;	xenbranch=xen-unstable ;;
 	osstest)		tree=osstest;	xenbranch=xen-unstable ;;
+	openstack-nova)	   tree=openstack-nova;	xenbranch=xen-unstable ;;
 	esac
 	if [ "x$tree" = xlinux ]; then
 		linuxbranch=$branch
diff --git a/make-flight b/make-flight
index 7bb536f..ad2608d 100755
--- a/make-flight
+++ b/make-flight
@@ -167,6 +167,17 @@ job_create_test_filter_callback () {
         *) return 1;;
       esac
       ;;
+    openstack-nova)
+      case "$job" in
+        *-devstack) ;;
+        *-devstack-xsm) ;;
+        *) return 1;;
+      esac
+      case $dom0arch in
+          amd64) ;;
+          *) return 1;;
+      esac
+      ;;
     *)
       case "$job" in
         *-qemuu-*)
@@ -198,6 +209,12 @@ arch_branch_filter_callback () {
         rumprun) return 1;;
         seabios) return 1;;
         ovmf) return 1;;
+        openstack-nova) return 1;;
+        esac
+        ;;
+  i386)
+        case "$branch" in
+        openstack-nova) return 1;;
         esac
         ;;
   i386|amd64)
@@ -606,6 +623,39 @@ do_pv_debian_tests () {
   $endfmt$dom0arch
 }
 
+do_openstack_tests () {
+  local xsms=$(xenbranch_xsm_variants)
+
+  if [ $dom0arch != amd64 ]; then
+    return
+  fi
+  if [ $branch != openstack-nova ]; then
+    return
+  fi
+
+  for xsm in $xsms ; do
+    job_create_test test-$xenarch$kern-$dom0arch-devstack \
+      test-devstack libvirt $xenarch $dom0arch \
+      tree_cinder=$TREE_OPENSTACK_CINDER \
+      tree_devstack=$TREE_OPENSTACK_DEVSTACK \
+      tree_glance=$TREE_OPENSTACK_GLANCE \
+      tree_keystone=$TREE_OPENSTACK_KEYSTONE \
+      tree_nova=$TREE_OPENSTACK_NOVA \
+      tree_requirements=$TREE_OPENSTACK_REQUIREMENTS \
+      tree_tempest=$TREE_OPENSTACK_TEMPEST \
+      revision_nova=$REVISION_OPENSTACK_NOVA \
+      revision_cinder=$REVISION_OPENSTACK_CINDER \
+      revision_devstack=$REVISION_OPENSTACK_DEVSTACK \
+      revision_glance=$REVISION_OPENSTACK_GLANCE \
+      revision_keystone=$REVISION_OPENSTACK_KEYSTONE \
+      revision_requirements=$REVISION_OPENSTACK_REQUIREMENTS \
+      revision_tempest=$REVISION_OPENSTACK_TEMPEST \
+      dom0_mem=4000 \
+      enable_xsm=$xsm \
+      all_hostflags=$most_hostflags
+  done
+}
+
 test_matrix_do_one () {
 
   do_pv_debian_tests
@@ -753,6 +803,8 @@ test_matrix_do_one () {
   do_pvgrub_tests
 
   do_xtf_tests
+
+  do_openstack_tests
 }
 
 if [ x$buildflight = x ]; then
-- 
Anthony PERARD


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v6 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v6 1/3] ts-openstack-deploy: Deploy OpenStack on a host with devstack"):
> This script installs any necessary packages and clones all of the OpenStack
> trees which are used by devstack to deploy OpenStack.

Thanks for this contribution.  I have no knowledge of OpenStack so I
can't really know whether what you are doing here is right, but it
looks basically OK to me.


I have some minor comments.  The most annoying one will be that I
would like you to file bugs in appropriate upstream bug trackers
(OpenStack or Debian), for each of the the workarounds.  Sorry...



> +sub packages () {
> +  # Install open-iscsi ahead of devstack ...
> +  target_install_packages($ho, qw(git sudo open-iscsi));

The indentation is a bit odd.  I won't insist you fix it, but usual
indent in osstest is 4 spaces.

> +  # ... and start open-iscsi to have /etc/iscsi/initiatorname.iscsi
> +  # generated. This is done on install on Ubuntu.
> +  target_cmd_root($ho, 'service open-iscsi start');

This looks like it is a workaround.  Can you please
 1. file a bug against Debian
 2. quote the bug number in your code
 3. make the workaround conditional on the suite

> +sub checkout () {
> +  prepbuilddirs();
> +  build_clone($ho, 'cinder', $builddir, 'cinder');
> +  build_clone($ho, 'devstack', $builddir, 'devstack');
> +  build_clone($ho, 'glance', $builddir, 'glance');
> +  build_clone($ho, 'keystone', $builddir, 'keystone');
> +  build_clone($ho, 'nova', $builddir, 'nova');
> +  build_clone($ho, 'requirements', $builddir, 'requirements');
> +  build_clone($ho, 'tempest', $builddir, 'tempest');

This could profitably be reformatted with extra whitespace so that the
columns line up.  (Again I won't insist on this.)

> +LOGFILE=\$DEST/logs/stack.sh.log
> +# stackrc set this but don't take \$DEST into account

ITYM "sets this but doesn't take".

> +  # stackrc does not take $DEST from local.conf into account, so fix it here
> +  target_editfile($ho, "$builddir/devstack/stackrc", sub {

This is a workround for an upstream bug ?  Is there a corresponding
upstream bug report ?  Should there be ?

In osstest I like to reduce, where possible, unconditional workarounds
for bugs in the software under test.  In general there should be an
upstream bug report, referred to in the osstest code.  Ideally there
would be a way to automatically disable the workaround eventually,
although that's probably too difficult here.

> +      while (<EI>) {
> +        if (m/^DEST=\/opt\/stack$/) {
> +          s/DEST=\/opt\/stack/DEST=$builddir/;

Using a regexp delimiter other than // would make this much clearer.

  +        if (m{^DEST=/opt/stack$}) {
  +        if (m#^DEST=/opt/stack$}) {
  +          s{DEST=/opt/stack}{DEST=$builddir};
  +          s#DEST=/opt/stack#DEST=$builddir#;

For example.  Take your pick.  I see later you used % % which is also
OK, although it is not the most idiomatic Perl (since the human reader
will tend to see % as a reference to a hash).

> +  # libvirt is already installed, but not as a package, so avoid installation of
> +  # the libvirt package with devstack
> +  target_editfile($ho, "$builddir/devstack/files/debs/nova", sub {
> +      while (<EI>) {
> +        next if m/.*libvirt.*/;
> +        print EO or die $!;
> +      }
> +  });
> +  target_editfile($ho, "$builddir/devstack/lib/nova_plugins/functions-libvirt", sub {
> +      while (<EI>) {
> +        next if m/install_package.*libvirt.*/;
> +        print EO or die $!;
> +      }
> +  });

These look like they ought to come with an upstream wishlist bug
asking for a front-door way to achieve this.

> +  # devstack blindly assume that systemd is used if systemctl is present
> +  target_editfile($ho, "$builddir/devstack/functions-common", sub {
> +      while (<EI>) {
> +        if (m%\[ -x /bin/systemctl%) {
> +          s%\[ -x /bin/systemctl \]%false%
> +        }
> +        print EO or die $!;
> +      }
> +  });

Again, an upstream bug.

> +  # OpenStack needs access to libvirt from a user.
> +  target_cmd_root($ho, <<END);
> +    if ! getent group libvirt >/dev/null; then
> +      groupadd libvirt

You should probably use "addgroup --system".  That is idempotent, so
you do not need the test.

> +    # For unknown reason, restart fail when done by devstack
> +    # so stop the service here, and devstack will start it.

Again, upstream bug.

> +  # devstack is going to setup the host, install some dependency.
> +  target_putfilecontents_root_stash($ho, 100, <<END,"/etc/sudoers.d/devstack");

How ... exciting.  This is rather mad but I don't think it is harmful
in the context of osstest.  So, fine.

> +  target_putfilecontents_root_stash($ho, 60, tgt_init(), "/etc/init.d/tgt");
> +  target_cmd_root($ho, <<END);
> +    chmod +x /etc/init.d/tgt
> +END
> +}

You may find it better to combine some of these target_cmd_root
stanzas.  That would reduce the overall amount of clutter in the perl
code.

> +# This is missing from Debian but required by devstack
> +# Got it from https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=577925
> +sub tgt_init () {

This should be conditional on the suite.  The bug seems to be fixed in
2014 and is allegedly fixed in stable (jessie).  So the workaround is
no longer necessary ?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v6 2/3] ts-openstack-tempest: Run Tempest to check OpenStack

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v6 2/3] ts-openstack-tempest: Run Tempest to check OpenStack"):
> This script runs the OpenStack integration test suite, Tempest.
...
> +  push @ignored_tests,
> +      "^\Q$volume_boot_pattern.test_volume_boot_pattern\E";
> +  push @ignored_tests,
> +      "^\Q$volume_boot_pattern.test_create_ebs_image_and_check_boot\E";
> +  push @ignored_tests,
> +      "^\Q$shelve_instance.test_shelve_volume_backed_instance\E";
> +
> +  # Those tests access a volume through iSCSI. This does not work when both the
> +  # server and client of iSCSI are on the same Xen host, Linux 4.0 is the first
> +  # Linux to have a fix.
> +  push @ignored_tests,
> +      "^\Q${volume_boot_pattern}V2.test_volume_boot_pattern\E";
> +  push @ignored_tests,
> +      "^\Q${volume_boot_pattern}V2.test_create_ebs_image_and_check_boot\E";
> +
> +  # This regex below select the tests to run and exclude the ones marked as
> +  # slow as well as the explicit tests listed above.
> +  # It is based on the one that can be found in tempest.git/tox.ini
> +  # in the section [testenv:full].
> +  my $ignored_tests = join("|", @ignored_tests);
> +  my $regex =
> +      "(?!.*\\[.*\\bslow\\b.*\\]|$ignored_tests)(^tempest\\.(api|scenario|thirdparty))";

This is all quite nightmarish.  I don't have a better suggestion,
though.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v6 3/3] Create a flight to test OpenStack with xen-unstable and libvirt

[Ian Jackson]

Anthony PERARD writes ("[OSSTEST PATCH v6 3/3] Create a flight to test OpenStack with xen-unstable and libvirt"):
> This patch should create a flight "openstack-nova", with those jobs:
>   build-amd64
>   build-amd64-xsm
>   build-amd64-pvops
>   build-amd64-libvirt
>   test-amd64-amd64-devstack
>   test-amd64-amd64-devstack-xsm

Do we intend to try to gate xen-unstable on this eventually ?

AFAICT you pull in new versions of all the subtrees willy-nilly.
Isn't that likely to cause regressions ?

> +: ${TREE_OPENSTACK_CINDER:=$GIT_OPENSTACK_ORG/openstack/cinder.git}
> +: ${TREE_OPENSTACK_DEVSTACK:=$GIT_OPENSTACK_ORG/openstack-dev/devstack.git}
> +: ${TREE_OPENSTACK_GLANCE:=$GIT_OPENSTACK_ORG/openstack/glance.git}
> +: ${TREE_OPENSTACK_KEYSTONE:=$GIT_OPENSTACK_ORG/openstack/keystone.git}
> +: ${TREE_OPENSTACK_NOVA:=$GIT_OPENSTACK_ORG/openstack/nova.git}

Regardless of your answer to the above, this and the part in
make-flight could really do with some metaprogramming.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [OSSTEST PATCH v6 3/3] Create a flight to test OpenStack with xen-unstable and libvirt

[Ian Jackson]

We just discussed this IRL, and here are our conclusions:

Ian Jackson writes ("Re: [OSSTEST PATCH v6 3/3] Create a flight to test OpenStack with xen-unstable and libvirt"):
> Do we intend to try to gate xen-unstable on this eventually ?

Maybe eventually, but not right now.

> AFAICT you pull in new versions of all the subtrees willy-nilly.
> Isn't that likely to cause regressions ?

The openstack components other than nova are already tested by the
openstack upstream CI loop, whose output we are going to be
consuming, so regressions there will be less frequent.  They might
block our openstack-nova push gate occasionally, but it would unblock
when upstream fixed it.

Your proposed arrangements would allow the bisector to work, across
all the relevant trees.

The obvious alternative would be to set up a push gate for each
openstack subtree, which obviously not something we want to do.

There is a less-obvious alternative, to have cr-daily-branch fetch all
the openstack trees' versions together, and then push them together to
a multitude of push-gate-output branches.  This would be fairly fiddly
work in cr-daily-branch, and introduce (yet another) a new special
case there.  It's not clear that we want to do this at all.
Definitely neither of us want to do that *now*.

So overall the conclusion is that your approach is OK.

> > +: ${TREE_OPENSTACK_CINDER:=$GIT_OPENSTACK_ORG/openstack/cinder.git}
> > +: ${TREE_OPENSTACK_DEVSTACK:=$GIT_OPENSTACK_ORG/openstack-dev/devstack.git}
> > +: ${TREE_OPENSTACK_GLANCE:=$GIT_OPENSTACK_ORG/openstack/glance.git}
> > +: ${TREE_OPENSTACK_KEYSTONE:=$GIT_OPENSTACK_ORG/openstack/keystone.git}
> > +: ${TREE_OPENSTACK_NOVA:=$GIT_OPENSTACK_ORG/openstack/nova.git}
> 
> Regardless of your answer to the above, this and the part in
> make-flight could really do with some metaprogramming.

Anthony is going to go and write some shell functions.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel