[raisin][PATCH 00/11] pygrub fix + cirros tests

[raisin][PATCH 00/11] pygrub fix + cirros tests

[Géza Gémes]

First patch adds a symlink from python2.7/site-packages/*
to python2.7/dist-packages/ at debian package build, this fixes
pygrub imports.

The rest of the patches are the cirros tests rebased to current master
and the python fix, plus a trivial fix for the pygrub test. in this
new patchset pygrub test is also added to the ones in the default set.

Géza Gémes (11):
  Fix installation of python libs
  Fix lopartsetup parsing of fdisk output
  Introduce cirros tests
  Introduce cirros-separate-kernel-pv test
  Introduce cirros-pygrub-pv test
  Introduce cirros-pvgrub2-pv test
  Introduce cirros-qemu-hvm test
  Introduce cirros-minios-stubdom-hvm test
  Introduce cirros-qemu-pvhvm test
  Introduce cirros-minios-stubdom-pvhvm test
  Enable cirros tests in the default config

 configs/config-cirros             |  44 ++++++++++++++++
 defconfig                         |   4 +-
 lib/common-tests.sh               | 102 ++++++++++++++++++++++++++++++++++++++
 scripts/lopartsetup               |   6 ++-
 scripts/mkdeb                     |   9 ++++
 tests/cirros-minios-stubdom-hvm   |  29 +++++++++++
 tests/cirros-minios-stubdom-pvhvm |  30 +++++++++++
 tests/cirros-pvgrub2-pv           |  27 ++++++++++
 tests/cirros-pygrub-pv            |  27 ++++++++++
 tests/cirros-qemu-hvm             |  28 +++++++++++
 tests/cirros-qemu-pvhvm           |  29 +++++++++++
 tests/cirros-separate-kernel-pv   |  28 +++++++++++
 tests/series                      |   7 +++
 13 files changed, 368 insertions(+), 2 deletions(-)
 create mode 100644 configs/config-cirros
 create mode 100644 tests/cirros-minios-stubdom-hvm
 create mode 100644 tests/cirros-minios-stubdom-pvhvm
 create mode 100644 tests/cirros-pvgrub2-pv
 create mode 100644 tests/cirros-pygrub-pv
 create mode 100644 tests/cirros-qemu-hvm
 create mode 100644 tests/cirros-qemu-pvhvm
 create mode 100644 tests/cirros-separate-kernel-pv

-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 01/11] Fix installation of python libs

[Géza Gémes]

Change deb package build in order to symlink the files installed
to site-packages to dist-packages to have them inluded in the
default PYTHONPATH

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 scripts/mkdeb | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/scripts/mkdeb b/scripts/mkdeb
index 3796300..23bcc2a 100755
--- a/scripts/mkdeb
+++ b/scripts/mkdeb
@@ -46,6 +46,15 @@ then
     rm -rf deb/usr/lib64
 fi
 
+# Make sure that python libs will be in the path
+mkdir -p deb/usr/lib/python2.7/dist-packages/
+cd deb/usr/lib/python2.7/dist-packages/
+for file in `ls ../site-packages`
+do
+    ln -s ../site-packages/$file .
+done
+cd -
+
 # Fill in the debian boilerplate
 mkdir -p deb/DEBIAN
 cat >deb/DEBIAN/control <<EOF
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 02/11] Fix lopartsetup parsing of fdisk output

[Géza Gémes]

Change lopartsetup in order to handle partitions, which have the
boot flag enabled.

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 scripts/lopartsetup | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/scripts/lopartsetup b/scripts/lopartsetup
index bf33a28..04ce3cc 100755
--- a/scripts/lopartsetup
+++ b/scripts/lopartsetup
@@ -58,7 +58,11 @@ index=0
 for i in "`fdisk -lu $filename 2>/dev/null | grep -e "^$filename"`"
 do
     index=$((index+1))
-    offset=`echo $i | tr -s " " | cut -d " " -f 2`
+    offset=`echo "$i" | tr -s " " | cut -d " " -f 2`
+    if [[ "$offset" == "*" ]]
+    then
+        offset=`echo "$i" | tr -s " " | cut -d " " -f 3`
+    fi
     offset=$((unit*offset))
 
     _create_loop_device "$filename" "$index" "$offset"
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 03/11] Introduce cirros tests

[Géza Gémes]

Add support for using cirros images in raisin tests

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 configs/config-cirros |  44 ++++++++++++++++++++++
 defconfig             |   2 +
 lib/common-tests.sh   | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 148 insertions(+)
 create mode 100644 configs/config-cirros

diff --git a/configs/config-cirros b/configs/config-cirros
new file mode 100644
index 0000000..fa2823e
--- /dev/null
+++ b/configs/config-cirros
@@ -0,0 +1,44 @@
+CIRROS_BASE_URL="https://download.cirros-cloud.net/"
+CIRROS_VERSION="0.3.5"
+
+source `pwd`/lib/common-functions.sh
+get_arch
+case $RAISIN_ARCH in
+    x86_64)
+        CIRROS_ARCH=x86_64
+        ;;
+    x86_32)
+        CIRROS_ARCH=i386
+        ;;
+    *)
+        echo $PREPEND cirros tests only valid on x86, 32 or 64 bit
+        exit 1
+esac
+
+CIRROS_KERNEL_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-kernel
+CIRROS_INITRD_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-initramfs
+CIRROS_ROOTFS_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img
+CIRROS_DISK_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img
+CIRROS_KERNEL_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_KERNEL_FILE}
+CIRROS_INITRD_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_INITRD_FILE}
+CIRROS_ROOTFS_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_ROOTFS_FILE}.gz
+CIRROS_DISK_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_DISK_FILE}
+
+CIRROS_GRUB_CFG="(xen/xvda,msdos1)/boot/grub/grub.cfg"
+
+set +e
+QEMU_IMG=`which qemu-img`
+set -e
+if [[ -z "$QEMU_IMG" ]]
+then
+    QEMU_IMG="/usr/lib/xen/bin/qemu-img"
+fi
+
+set +e
+PVGRUB=`which grub-${CIRROS_ARCH}-xen`
+set -e
+if [[ -z "$PVGRUB" ]]
+then
+    PVGRUB="/usr/lib/xen/boot/grub-${CIRROS_ARCH}-xen"
+fi
+
diff --git a/defconfig b/defconfig
index f8ef398..111554e 100644
--- a/defconfig
+++ b/defconfig
@@ -32,3 +32,5 @@ GIT_TRANSPORT="git"
 ## All tests: busybox-pv busybox-hvm
 ## ENABLED_TESTS is the list of test run by raise test
 ENABLED_TESTS="busybox-pv busybox-hvm"
+
+. configs/config-cirros
diff --git a/lib/common-tests.sh b/lib/common-tests.sh
index d346af4..79815ce 100644
--- a/lib/common-tests.sh
+++ b/lib/common-tests.sh
@@ -178,3 +178,105 @@ function get_host_initrd() {
         exit 1
     fi
 }
+
+function cirros_network_init() {
+    rootdir=$1
+    # Configure static ip
+    $SUDO sed -i -e 's/iface eth0 inet dhcp/iface eth0 inet static/' ${rootdir}/etc/network/interfaces
+    $SUDO sed -i -e '/iface eth0 inet static/a\    address 169.254.0.2' ${rootdir}/etc/network/interfaces
+    $SUDO sed -i -e '/address/a\    network 169.254.0.0' ${rootdir}/etc/network/interfaces
+    $SUDO sed -i -e '/network/a\    broadcast 169.254.0.255' ${rootdir}/etc/network/interfaces
+    $SUDO sed -i -e '/broadcast/a\    netmask 255.255.255.0' ${rootdir}/etc/network/interfaces
+    # Disable cloud-init
+    $SUDO rm -f ${rootdir}/etc/rc3.d/S*cirros*ds*
+    $SUDO rm -f ${rootdir}/etc/rc3.d/S*-cirros-userdata
+}
+
+function get_cirros_kernel() {
+    bootdir=$1
+    basename `find $bootdir -name vmlinuz* 2>/dev/null | head -1`
+}
+
+function get_cirros_initrd() {
+    bootdir=$1
+    basename `find $bootdir -name initrd* 2>/dev/null | head -1`
+}
+
+function cirros_grub_cfg() {
+    rootdir=$1
+    grubcfg="`echo $CIRROS_GRUB_CFG | cut -d ')' -f 2`"
+    grubdir=`dirname $grubcfg`
+    bootdir=`dirname $grubdir`
+    tmpgrubcfg=`mktemp`
+    cat > $tmpgrubcfg <<EOF
+root='(xen/xvda,msdos1)'
+insmod xzio
+insmod gzio
+insmod btrfs
+insmod ext2
+set timeout=1
+set default=0
+menuentry Cirros {
+    linux `echo $bootdir`/`get_cirros_kernel ${rootdir}/${bootdir}` root=/dev/xvda1 ro
+    initrd `echo $bootdir`/`get_cirros_initrd ${rootdir}/${bootdir}`
+}
+EOF
+    $SUDO mv $tmpgrubcfg ${rootdir}/${grubcfg}
+}
+
+function set_up_cirros_tests() {
+    verbose_echo "Setting up environment for cirros tests"
+    tmpdir=`mktemp -d`
+    wget -q $CIRROS_KERNEL_URL -P $tmpdir
+    wget -q $CIRROS_INITRD_URL -P $tmpdir
+    wget -q $CIRROS_ROOTFS_URL -P $tmpdir
+    wget -q $CIRROS_DISK_URL -P $tmpdir
+    gunzip ${tmpdir}/${CIRROS_ROOTFS_FILE}.gz
+    mv ${tmpdir}/${CIRROS_DISK_FILE} ${tmpdir}/${CIRROS_DISK_FILE}.qcow2
+    $QEMU_IMG convert -f qcow2 -O raw ${tmpdir}/${CIRROS_DISK_FILE}.qcow2 ${tmpdir}/${CIRROS_DISK_FILE}
+    CIRROS_ROOTFS_LOOP=`create_loop ${tmpdir}/${CIRROS_ROOTFS_FILE}`
+    CIRROS_DISK_LOOP_P0=`$SUDO $BASEDIR/scripts/lopartsetup ${tmpdir}/${CIRROS_DISK_FILE} | head -1 | cut -d ":" -f 1`
+    CIRROS_ROOTFS_MNTPT=`mktemp -d`
+    CIRROS_DISK_MNTPT=`mktemp -d`
+    $SUDO mount $CIRROS_ROOTFS_LOOP $CIRROS_ROOTFS_MNTPT
+    $SUDO mount $CIRROS_DISK_LOOP_P0 $CIRROS_DISK_MNTPT
+    cirros_network_init $CIRROS_ROOTFS_MNTPT
+    cirros_network_init $CIRROS_DISK_MNTPT
+    cirros_grub_cfg $CIRROS_DISK_MNTPT
+    $SUDO umount $CIRROS_ROOTFS_MNTPT
+    $SUDO umount $CIRROS_DISK_MNTPT
+    $SUDO rmdir $CIRROS_ROOTFS_MNTPT
+    $SUDO rmdir $CIRROS_DISK_MNTPT
+    $SUDO losetup -d $CIRROS_ROOTFS_LOOP
+    $SUDO losetup -d $CIRROS_DISK_LOOP_P0
+}
+
+function tear_down_cirros_tests() {
+    tmpdir=$1
+    if [[ `$SUDO xl vm-list | grep -e "raisin-test" | wc -l` -gt 0 ]]
+    then
+        $SUDO xl destroy "raisin-test"
+    fi
+    number_of_cirros_tests=0
+    for test in $TESTS
+    do
+        if [[ "`echo $test | cut -d '-' -f 1`" == "cirros" ]]
+        then
+            number_of_cirros_tests=$((number_of_cirros_tests+1))
+        fi
+    done
+    number_of_run_cirros_tests=0
+    for test in $TESTS
+    do
+        if [[ -f ${tmpdir}/${test}.cfg ]]
+        then
+            number_of_run_cirros_tests=$((number_of_run_cirros_tests+1))
+        fi
+    done
+    if [[ $number_of_cirros_tests == $number_of_run_cirros_tests ]]
+    then
+        verbose_echo "Deleting environment of cirros tests"
+        $SUDO rm -rf $tmpdir
+    fi
+}
+
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 04/11] Introduce cirros-separate-kernel-pv test

[Géza Gémes]

This test is the cirros equivalent of the bussybox-pv test

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 tests/cirros-separate-kernel-pv | 28 ++++++++++++++++++++++++++++
 tests/series                    |  1 +
 2 files changed, 29 insertions(+)
 create mode 100644 tests/cirros-separate-kernel-pv

diff --git a/tests/cirros-separate-kernel-pv b/tests/cirros-separate-kernel-pv
new file mode 100644
index 0000000..bbbcf7d
--- /dev/null
+++ b/tests/cirros-separate-kernel-pv
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+set -e
+
+function cirros-separate-kernel-pv-cleanup() {
+    tear_down_cirros_tests $tmpdir
+}
+
+function cirros-separate-kernel-pv-test() {
+    if [[ ! -d $tmpdir ]]
+    then
+        set_up_cirros_tests
+    fi
+    cd $tmpdir
+    cat >cirros-separate-kernel-pv.cfg <<EOF
+name = "raisin-test"
+memory = 128
+vcpus = 2
+kernel = "${tmpdir}/${CIRROS_KERNEL_FILE}"
+ramdisk = "${tmpdir}/${CIRROS_INITRD_FILE}"
+disk = [ '${tmpdir}/${CIRROS_ROOTFS_FILE},raw,xvda,rw' ]
+extra = "root=/dev/xvda"
+vif = [ 'bridge=xenbr1' ]
+EOF
+
+    $SUDO xl create cirros-separate-kernel-pv.cfg
+    check_guest_alive
+}
diff --git a/tests/series b/tests/series
index a2f4b34..38ee35a 100644
--- a/tests/series
+++ b/tests/series
@@ -1,3 +1,4 @@
 busybox-pv
 busybox-hvm
 busybox-hvm-migrate
+cirros-separate-kernel-pv
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 05/11] Introduce cirros-pygrub-pv test

[Géza Gémes]

This test verifies booting the cirros image using pygrub

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 tests/cirros-pygrub-pv | 27 +++++++++++++++++++++++++++
 tests/series           |  1 +
 2 files changed, 28 insertions(+)
 create mode 100644 tests/cirros-pygrub-pv

diff --git a/tests/cirros-pygrub-pv b/tests/cirros-pygrub-pv
new file mode 100644
index 0000000..6b86573
--- /dev/null
+++ b/tests/cirros-pygrub-pv
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+
+set -e
+
+function cirros-pygrub-pv-cleanup() {
+    tear_down_cirros_tests $tmpdir
+}
+
+function cirros-pygrub-pv-test() {
+    if [[ ! -d $tmpdir ]]
+    then
+        set_up_cirros_tests
+    fi
+    cd $tmpdir
+    cat >cirros-pygrub-pv.cfg <<EOF
+name = "raisin-test"
+memory = 128
+vcpus = 2
+bootloader = "pygrub"
+disk = [ '${tmpdir}/${CIRROS_ROOTFS_FILE},raw,xvda,rw' ]
+extra = "root=/dev/xvda1"
+vif = [ 'bridge=xenbr1' ]
+EOF
+
+    $SUDO xl create cirros-pygrub-pv.cfg
+    check_guest_alive
+}
diff --git a/tests/series b/tests/series
index 38ee35a..372fafb 100644
--- a/tests/series
+++ b/tests/series
@@ -2,3 +2,4 @@ busybox-pv
 busybox-hvm
 busybox-hvm-migrate
 cirros-separate-kernel-pv
+cirros-pygrub-pv
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 06/11] Introduce cirros-pvgrub2-pv test

[Géza Gémes]

This test verifies booting the cirros image using pvgrub2

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 tests/cirros-pvgrub2-pv | 27 +++++++++++++++++++++++++++
 tests/series            |  1 +
 2 files changed, 28 insertions(+)
 create mode 100644 tests/cirros-pvgrub2-pv

diff --git a/tests/cirros-pvgrub2-pv b/tests/cirros-pvgrub2-pv
new file mode 100644
index 0000000..5c746fe
--- /dev/null
+++ b/tests/cirros-pvgrub2-pv
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+
+set -e
+
+function cirros-pvgrub2-pv-cleanup() {
+    tear_down_cirros_tests $tmpdir
+}
+
+function cirros-pvgrub2-pv-test() {
+    if [[ ! -d $tmpdir ]]
+    then
+        set_up_cirros_tests
+    fi
+    cd $tmpdir
+    cat >cirros-pvgrub2-pv.cfg <<EOF
+name = "raisin-test"
+memory = 128
+vcpus = 2
+kernel = "$PVGRUB"
+disk = [ '${tmpdir}/${CIRROS_DISK_FILE},raw,xvda,rw' ]
+extra = "$CIRROS_GRUB_CFG"
+vif = [ 'bridge=xenbr1' ]
+EOF
+
+    $SUDO xl create cirros-pvgrub2-pv.cfg
+    check_guest_alive
+}
diff --git a/tests/series b/tests/series
index 372fafb..1fec245 100644
--- a/tests/series
+++ b/tests/series
@@ -3,3 +3,4 @@ busybox-hvm
 busybox-hvm-migrate
 cirros-separate-kernel-pv
 cirros-pygrub-pv
+cirros-pvgrub2-pv
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 07/11] Introduce cirros-qemu-hvm test

[Géza Gémes]

This test is the cirros equivalent of the bussybox-hvm test

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 tests/cirros-qemu-hvm | 28 ++++++++++++++++++++++++++++
 tests/series          |  1 +
 2 files changed, 29 insertions(+)
 create mode 100644 tests/cirros-qemu-hvm

diff --git a/tests/cirros-qemu-hvm b/tests/cirros-qemu-hvm
new file mode 100644
index 0000000..963d380
--- /dev/null
+++ b/tests/cirros-qemu-hvm
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+
+set -e
+
+function cirros-qemu-hvm-cleanup() {
+    tear_down_cirros_tests $tmpdir
+}
+
+function cirros-qemu-hvm-test() {
+    if [[ ! -d $tmpdir ]]
+    then
+        set_up_cirros_tests
+    fi
+    cd $tmpdir
+    cat >cirros-qemu-hvm.cfg <<EOF
+name = "raisin-test"
+builder = "hvm"
+memory = 128
+vcpus = 2
+disk = [ '${tmpdir}/${CIRROS_DISK_FILE},raw,hda,rw' ]
+vif = [ 'bridge=xenbr1' ]
+serial="pty"
+boot="c"
+EOF
+
+    $SUDO xl create cirros-qemu-hvm.cfg
+    check_guest_alive
+}
diff --git a/tests/series b/tests/series
index 1fec245..49a9257 100644
--- a/tests/series
+++ b/tests/series
@@ -4,3 +4,4 @@ busybox-hvm-migrate
 cirros-separate-kernel-pv
 cirros-pygrub-pv
 cirros-pvgrub2-pv
+cirros-qemu-hvm
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 08/11] Introduce cirros-minios-stubdom-hvm test

[Géza Gémes]

This test verifies a cirros based hvm domain using a minios based
stubdom as its device model

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 tests/cirros-minios-stubdom-hvm | 29 +++++++++++++++++++++++++++++
 tests/series                    |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 tests/cirros-minios-stubdom-hvm

diff --git a/tests/cirros-minios-stubdom-hvm b/tests/cirros-minios-stubdom-hvm
new file mode 100644
index 0000000..86ced49
--- /dev/null
+++ b/tests/cirros-minios-stubdom-hvm
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+set -e
+
+function cirros-minios-stubdom-hvm-cleanup() {
+    tear_down_cirros_tests $tmpdir
+}
+
+function cirros-minios-stubdom-hvm-test() {
+    if [[ ! -d $tmpdir ]]
+    then
+        set_up_cirros_tests
+    fi
+    cd $tmpdir
+    cat >cirros-minios-stubdom-hvm.cfg <<EOF
+name = "raisin-test"
+builder = "hvm"
+device_model_stubdomain_override = 1
+memory = 128
+vcpus = 2
+disk = [ '${tmpdir}/${CIRROS_DISK_FILE},raw,hda,rw' ]
+vif = [ 'bridge=xenbr1' ]
+serial="pty"
+boot="c"
+EOF
+
+    $SUDO xl create cirros-minios-stubdom-hvm.cfg
+    check_guest_alive
+}
diff --git a/tests/series b/tests/series
index 49a9257..7b1ad8e 100644
--- a/tests/series
+++ b/tests/series
@@ -5,3 +5,4 @@ cirros-separate-kernel-pv
 cirros-pygrub-pv
 cirros-pvgrub2-pv
 cirros-qemu-hvm
+cirros-minios-stubdom-hvm
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 09/11] Introduce cirros-qemu-pvhvm test

[Géza Gémes]

This test is similar to cirros-qemu-hvm test, differing by
explicitly exposing the xen netfront and blockfront drivers

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 tests/cirros-qemu-pvhvm | 29 +++++++++++++++++++++++++++++
 tests/series            |  1 +
 2 files changed, 30 insertions(+)
 create mode 100644 tests/cirros-qemu-pvhvm

diff --git a/tests/cirros-qemu-pvhvm b/tests/cirros-qemu-pvhvm
new file mode 100644
index 0000000..28c4f9d
--- /dev/null
+++ b/tests/cirros-qemu-pvhvm
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+set -e
+
+function cirros-qemu-pvhvm-cleanup() {
+    tear_down_cirros_tests $tmpdir
+}
+
+function cirros-qemu-pvhvm-test() {
+    if [[ ! -d $tmpdir ]]
+    then
+        set_up_cirros_tests
+    fi
+    cd $tmpdir
+    cat >cirros-qemu-pvhvm.cfg <<EOF
+name = "raisin-test"
+builder = "hvm"
+xen_platform_pci = 1
+memory = 128
+vcpus = 2
+disk = [ '${tmpdir}/${CIRROS_DISK_FILE},raw,xvda,rw' ]
+vif = [ 'bridge=xenbr1' ]
+serial="pty"
+boot="c"
+EOF
+
+    $SUDO xl create cirros-qemu-pvhvm.cfg
+    check_guest_alive
+}
diff --git a/tests/series b/tests/series
index 7b1ad8e..51dbe17 100644
--- a/tests/series
+++ b/tests/series
@@ -6,3 +6,4 @@ cirros-pygrub-pv
 cirros-pvgrub2-pv
 cirros-qemu-hvm
 cirros-minios-stubdom-hvm
+cirros-qemu-pvhvm
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 10/11] Introduce cirros-minios-stubdom-pvhvm test

[Géza Gémes]

This test is similar to cirros-minios-stubdom-hvm test, differing by
explicitly exposing the xen netfront and blockfront drivers

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 tests/cirros-minios-stubdom-pvhvm | 30 ++++++++++++++++++++++++++++++
 tests/series                      |  1 +
 2 files changed, 31 insertions(+)
 create mode 100644 tests/cirros-minios-stubdom-pvhvm

diff --git a/tests/cirros-minios-stubdom-pvhvm b/tests/cirros-minios-stubdom-pvhvm
new file mode 100644
index 0000000..73b6834
--- /dev/null
+++ b/tests/cirros-minios-stubdom-pvhvm
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+
+set -e
+
+function cirros-minios-stubdom-pvhvm-cleanup() {
+    tear_down_cirros_tests $tmpdir
+}
+
+function cirros-minios-stubdom-pvhvm-test() {
+    if [[ ! -d $tmpdir ]]
+    then
+        set_up_cirros_tests
+    fi
+    cd $tmpdir
+    cat >cirros-minios-stubdom-pvhvm.cfg <<EOF
+name = "raisin-test"
+builder = "hvm"
+device_model_stubdomain_override = 1
+xen_platform_pci = 1
+memory = 128
+vcpus = 2
+disk = [ '${tmpdir}/${CIRROS_DISK_FILE},raw,xvda,rw' ]
+vif = [ 'bridge=xenbr1' ]
+serial="pty"
+boot="c"
+EOF
+
+    $SUDO xl create cirros-minios-stubdom-pvhvm.cfg
+    check_guest_alive
+}
diff --git a/tests/series b/tests/series
index 51dbe17..ffa8e69 100644
--- a/tests/series
+++ b/tests/series
@@ -7,3 +7,4 @@ cirros-pvgrub2-pv
 cirros-qemu-hvm
 cirros-minios-stubdom-hvm
 cirros-qemu-pvhvm
+cirros-minios-stubdom-pvhvm
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[PATCH 11/11] Enable cirros tests in the default config

[Géza Gémes]

The existing cirros tests are enabled, with the following
exceptions:

cirros-minios-stubdom-hvm and cirros-minios-stubdom-pvhvm are
skipped as raisin does not install the stubdom

Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
---
 defconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/defconfig b/defconfig
index 111554e..f3b1759 100644
--- a/defconfig
+++ b/defconfig
@@ -31,6 +31,6 @@ GIT_TRANSPORT="git"
 # Tests
 ## All tests: busybox-pv busybox-hvm
 ## ENABLED_TESTS is the list of test run by raise test
-ENABLED_TESTS="busybox-pv busybox-hvm"
+ENABLED_TESTS="busybox-pv busybox-hvm cirros-separate-kernel-pv cirros-pygrub-pv cirros-pvgrub2-pv cirros-qemu-hvm cirros-qemu-pvhvm"
 
 . configs/config-cirros
-- 
2.7.4


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/11] Fix installation of python libs

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 970 bytes --]

On Sun, 19 Mar 2017, Géza Gémes wrote:
> Change deb package build in order to symlink the files installed
> to site-packages to dist-packages to have them inluded in the
> default PYTHONPATH
> 
> Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
> ---
>  scripts/mkdeb | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/scripts/mkdeb b/scripts/mkdeb
> index 3796300..23bcc2a 100755
> --- a/scripts/mkdeb
> +++ b/scripts/mkdeb
> @@ -46,6 +46,15 @@ then
>      rm -rf deb/usr/lib64
>  fi
>  
> +# Make sure that python libs will be in the path
> +mkdir -p deb/usr/lib/python2.7/dist-packages/
> +cd deb/usr/lib/python2.7/dist-packages/
> +for file in `ls ../site-packages`
> +do
> +    ln -s ../site-packages/$file .
> +done
> +cd -

Actually for Debian I think it makes sense to move (not link) the files
to dist-packages, what do you think?


>  # Fill in the debian boilerplate
>  mkdir -p deb/DEBIAN
>  cat >deb/DEBIAN/control <<EOF
> -- 
> 2.7.4
> 

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 02/11] Fix lopartsetup parsing of fdisk output

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 949 bytes --]

On Sun, 19 Mar 2017, Géza Gémes wrote:
> Change lopartsetup in order to handle partitions, which have the
> boot flag enabled.
> 
> Signed-off-by: Géza Gémes <geza.gemes@gmail.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  scripts/lopartsetup | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/lopartsetup b/scripts/lopartsetup
> index bf33a28..04ce3cc 100755
> --- a/scripts/lopartsetup
> +++ b/scripts/lopartsetup
> @@ -58,7 +58,11 @@ index=0
>  for i in "`fdisk -lu $filename 2>/dev/null | grep -e "^$filename"`"
>  do
>      index=$((index+1))
> -    offset=`echo $i | tr -s " " | cut -d " " -f 2`
> +    offset=`echo "$i" | tr -s " " | cut -d " " -f 2`
> +    if [[ "$offset" == "*" ]]
> +    then
> +        offset=`echo "$i" | tr -s " " | cut -d " " -f 3`
> +    fi
>      offset=$((unit*offset))
>  
>      _create_loop_device "$filename" "$index" "$offset"
> -- 
> 2.7.4
> 

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/11] Introduce cirros tests

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 8047 bytes --]

On Sun, 19 Mar 2017, Géza Gémes wrote:
> Add support for using cirros images in raisin tests
> 
> Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
> ---
>  configs/config-cirros |  44 ++++++++++++++++++++++
>  defconfig             |   2 +
>  lib/common-tests.sh   | 102 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 148 insertions(+)
>  create mode 100644 configs/config-cirros
> 
> diff --git a/configs/config-cirros b/configs/config-cirros
> new file mode 100644
> index 0000000..fa2823e
> --- /dev/null
> +++ b/configs/config-cirros

The files under configs are meant to contain the git urls and software
versions of each component to build. It is not the right place for all
these cirros related variables.

Instead, I would move config-cirros to a new top level directory, maybe
I would call it "tests-configs".



> @@ -0,0 +1,44 @@
> +CIRROS_BASE_URL="https://download.cirros-cloud.net/"
> +CIRROS_VERSION="0.3.5"
> +
> +source `pwd`/lib/common-functions.sh
> +get_arch
> +case $RAISIN_ARCH in
> +    x86_64)
> +        CIRROS_ARCH=x86_64
> +        ;;
> +    x86_32)
> +        CIRROS_ARCH=i386
> +        ;;
> +    *)
> +        echo $PREPEND cirros tests only valid on x86, 32 or 64 bit
> +        exit 1
> +esac
> +
> +CIRROS_KERNEL_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-kernel
> +CIRROS_INITRD_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-initramfs
> +CIRROS_ROOTFS_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img
> +CIRROS_DISK_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img
> +CIRROS_KERNEL_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_KERNEL_FILE}
> +CIRROS_INITRD_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_INITRD_FILE}
> +CIRROS_ROOTFS_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_ROOTFS_FILE}.gz
> +CIRROS_DISK_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_DISK_FILE}
> +
> +CIRROS_GRUB_CFG="(xen/xvda,msdos1)/boot/grub/grub.cfg"
> +

I like to keep the config files as simple as possible, with just
parameters and variable assignments. Ideally, we would have one
config-cirros file with only varibles, like configs/config-master for
example. Everything else should be elsewhere. I would even go as far as
removing the get_arch call above, writing two fully static cirros config
files, one for x86_64 and one for x86_32.


> +set +e
> +QEMU_IMG=`which qemu-img`
> +set -e
> +if [[ -z "$QEMU_IMG" ]]
> +then
> +    QEMU_IMG="/usr/lib/xen/bin/qemu-img"
> +fi
> +
> +set +e
> +PVGRUB=`which grub-${CIRROS_ARCH}-xen`
> +set -e
> +if [[ -z "$PVGRUB" ]]
> +then
> +    PVGRUB="/usr/lib/xen/boot/grub-${CIRROS_ARCH}-xen"
> +fi

I would move the detection of QEMU_IMG and PVGRUB to functions in
lib/common-functions.sh.


> diff --git a/defconfig b/defconfig
> index f8ef398..111554e 100644
> --- a/defconfig
> +++ b/defconfig
> @@ -32,3 +32,5 @@ GIT_TRANSPORT="git"
>  ## All tests: busybox-pv busybox-hvm
>  ## ENABLED_TESTS is the list of test run by raise test
>  ENABLED_TESTS="busybox-pv busybox-hvm"
> +
> +. configs/config-cirros
> diff --git a/lib/common-tests.sh b/lib/common-tests.sh
> index d346af4..79815ce 100644
> --- a/lib/common-tests.sh
> +++ b/lib/common-tests.sh
> @@ -178,3 +178,105 @@ function get_host_initrd() {
>          exit 1
>      fi
>  }
> +
> +function cirros_network_init() {
> +    rootdir=$1
> +    # Configure static ip
> +    $SUDO sed -i -e 's/iface eth0 inet dhcp/iface eth0 inet static/' ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/iface eth0 inet static/a\    address 169.254.0.2' ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/address/a\    network 169.254.0.0' ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/network/a\    broadcast 169.254.0.255' ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/broadcast/a\    netmask 255.255.255.0' ${rootdir}/etc/network/interfaces

I think that it would be more future-proof to just generate and
overwrite ${rootdir}/etc/network/interfaces with our own. Also in
general we don't use ${} for variables.


> +    # Disable cloud-init
> +    $SUDO rm -f ${rootdir}/etc/rc3.d/S*cirros*ds*
> +    $SUDO rm -f ${rootdir}/etc/rc3.d/S*-cirros-userdata
> +}
> +
> +function get_cirros_kernel() {
> +    bootdir=$1
> +    basename `find $bootdir -name vmlinuz* 2>/dev/null | head -1`
> +}
> +
> +function get_cirros_initrd() {
> +    bootdir=$1
> +    basename `find $bootdir -name initrd* 2>/dev/null | head -1`
> +}
> +
> +function cirros_grub_cfg() {
> +    rootdir=$1
> +    grubcfg="`echo $CIRROS_GRUB_CFG | cut -d ')' -f 2`"
> +    grubdir=`dirname $grubcfg`
> +    bootdir=`dirname $grubdir`
> +    tmpgrubcfg=`mktemp`
> +    cat > $tmpgrubcfg <<EOF
> +root='(xen/xvda,msdos1)'
> +insmod xzio
> +insmod gzio
> +insmod btrfs
> +insmod ext2
> +set timeout=1
> +set default=0
> +menuentry Cirros {
> +    linux `echo $bootdir`/`get_cirros_kernel ${rootdir}/${bootdir}` root=/dev/xvda1 ro
> +    initrd `echo $bootdir`/`get_cirros_initrd ${rootdir}/${bootdir}`
> +}
> +EOF
> +    $SUDO mv $tmpgrubcfg ${rootdir}/${grubcfg}
> +}
> +
> +function set_up_cirros_tests() {
> +    verbose_echo "Setting up environment for cirros tests"
> +    tmpdir=`mktemp -d`

Reading later patches, I noticed that you rely on the variable being
called "tmpdir". If you need such as a variable please give a more
meaningful name.


> +    wget -q $CIRROS_KERNEL_URL -P $tmpdir
> +    wget -q $CIRROS_INITRD_URL -P $tmpdir
> +    wget -q $CIRROS_ROOTFS_URL -P $tmpdir
> +    wget -q $CIRROS_DISK_URL -P $tmpdir

Shouldn't we check whether they have been already downloaded? So that,
if you run 'raise tests' twice, the second time you don't need to wait?
In fact, I would download them to a "downloads" directory, rather than a
tmpdir.

Today, each busybox test setups its own disk and loop device. If
possible I would do the same for the cirros test. We can reuse already
downloaded images to save time and bandwidth. Of course, we can share
the same common init functions in all cirros tests. For example, in this
function we setup the disk for both pv and hvm tests. I would create two
separate functions, one for each.


> +    gunzip ${tmpdir}/${CIRROS_ROOTFS_FILE}.gz
> +    mv ${tmpdir}/${CIRROS_DISK_FILE} ${tmpdir}/${CIRROS_DISK_FILE}.qcow2
> +    $QEMU_IMG convert -f qcow2 -O raw ${tmpdir}/${CIRROS_DISK_FILE}.qcow2 ${tmpdir}/${CIRROS_DISK_FILE}
> +    CIRROS_ROOTFS_LOOP=`create_loop ${tmpdir}/${CIRROS_ROOTFS_FILE}`
> +    CIRROS_DISK_LOOP_P0=`$SUDO $BASEDIR/scripts/lopartsetup ${tmpdir}/${CIRROS_DISK_FILE} | head -1 | cut -d ":" -f 1`
> +    CIRROS_ROOTFS_MNTPT=`mktemp -d`
> +    CIRROS_DISK_MNTPT=`mktemp -d`

local variables should be lower case and declared "local"


> +    $SUDO mount $CIRROS_ROOTFS_LOOP $CIRROS_ROOTFS_MNTPT
> +    $SUDO mount $CIRROS_DISK_LOOP_P0 $CIRROS_DISK_MNTPT
> +    cirros_network_init $CIRROS_ROOTFS_MNTPT
> +    cirros_network_init $CIRROS_DISK_MNTPT
> +    cirros_grub_cfg $CIRROS_DISK_MNTPT
> +    $SUDO umount $CIRROS_ROOTFS_MNTPT
> +    $SUDO umount $CIRROS_DISK_MNTPT
> +    $SUDO rmdir $CIRROS_ROOTFS_MNTPT
> +    $SUDO rmdir $CIRROS_DISK_MNTPT
> +    $SUDO losetup -d $CIRROS_ROOTFS_LOOP
> +    $SUDO losetup -d $CIRROS_DISK_LOOP_P0
> +}
> +
> +function tear_down_cirros_tests() {
> +    tmpdir=$1
> +    if [[ `$SUDO xl vm-list | grep -e "raisin-test" | wc -l` -gt 0 ]]
> +    then
> +        $SUDO xl destroy "raisin-test"
> +    fi
> +    number_of_cirros_tests=0
> +    for test in $TESTS
> +    do
> +        if [[ "`echo $test | cut -d '-' -f 1`" == "cirros" ]]
> +        then
> +            number_of_cirros_tests=$((number_of_cirros_tests+1))
> +        fi
> +    done
> +    number_of_run_cirros_tests=0
> +    for test in $TESTS
> +    do
> +        if [[ -f ${tmpdir}/${test}.cfg ]]
> +        then
> +            number_of_run_cirros_tests=$((number_of_run_cirros_tests+1))
> +        fi
> +    done
> +    if [[ $number_of_cirros_tests == $number_of_run_cirros_tests ]]
> +    then
> +        verbose_echo "Deleting environment of cirros tests"
> +        $SUDO rm -rf $tmpdir
> +    fi
> +}

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 04/11] Introduce cirros-separate-kernel-pv test

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1725 bytes --]

On Sun, 19 Mar 2017, Géza Gémes wrote:
> This test is the cirros equivalent of the bussybox-pv test
> 
> Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
> ---
>  tests/cirros-separate-kernel-pv | 28 ++++++++++++++++++++++++++++
>  tests/series                    |  1 +
>  2 files changed, 29 insertions(+)
>  create mode 100644 tests/cirros-separate-kernel-pv
> 
> diff --git a/tests/cirros-separate-kernel-pv b/tests/cirros-separate-kernel-pv
> new file mode 100644
> index 0000000..bbbcf7d
> --- /dev/null
> +++ b/tests/cirros-separate-kernel-pv
> @@ -0,0 +1,28 @@
> +#!/usr/bin/env bash
> +
> +set -e
> +
> +function cirros-separate-kernel-pv-cleanup() {
> +    tear_down_cirros_tests $tmpdir
> +}
> +
> +function cirros-separate-kernel-pv-test() {
> +    if [[ ! -d $tmpdir ]]
> +    then
> +        set_up_cirros_tests

As I wrote in reply to the other patch, I would remove tmpdir and create
one from scratch for each test (but download the cirros images only
once).

Also, we need a pv specific init function here.

The rest looks good, thanks!


> +    fi
> +    cd $tmpdir
> +    cat >cirros-separate-kernel-pv.cfg <<EOF
> +name = "raisin-test"
> +memory = 128
> +vcpus = 2
> +kernel = "${tmpdir}/${CIRROS_KERNEL_FILE}"
> +ramdisk = "${tmpdir}/${CIRROS_INITRD_FILE}"
> +disk = [ '${tmpdir}/${CIRROS_ROOTFS_FILE},raw,xvda,rw' ]
> +extra = "root=/dev/xvda"
> +vif = [ 'bridge=xenbr1' ]
> +EOF
> +
> +    $SUDO xl create cirros-separate-kernel-pv.cfg
> +    check_guest_alive
> +}
> diff --git a/tests/series b/tests/series
> index a2f4b34..38ee35a 100644
> --- a/tests/series
> +++ b/tests/series
> @@ -1,3 +1,4 @@
>  busybox-pv
>  busybox-hvm
>  busybox-hvm-migrate
> +cirros-separate-kernel-pv
> -- 
> 2.7.4
> 

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [raisin][PATCH 00/11] pygrub fix + cirros tests

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2323 bytes --]

Thanks for the patches! :-)

I committed patch #2. Please resend the rest of the series addressing
the comments. I didn't reply to each patch, but the cirros tests look
good except for the few comments regarding the way they interact with
the common functions.

Cheers,

Stefano


On Sun, 19 Mar 2017, Géza Gémes wrote:
> First patch adds a symlink from python2.7/site-packages/*
> to python2.7/dist-packages/ at debian package build, this fixes
> pygrub imports.
> 
> The rest of the patches are the cirros tests rebased to current master
> and the python fix, plus a trivial fix for the pygrub test. in this
> new patchset pygrub test is also added to the ones in the default set.
> 
> Géza Gémes (11):
>   Fix installation of python libs
>   Fix lopartsetup parsing of fdisk output
>   Introduce cirros tests
>   Introduce cirros-separate-kernel-pv test
>   Introduce cirros-pygrub-pv test
>   Introduce cirros-pvgrub2-pv test
>   Introduce cirros-qemu-hvm test
>   Introduce cirros-minios-stubdom-hvm test
>   Introduce cirros-qemu-pvhvm test
>   Introduce cirros-minios-stubdom-pvhvm test
>   Enable cirros tests in the default config
> 
>  configs/config-cirros             |  44 ++++++++++++++++
>  defconfig                         |   4 +-
>  lib/common-tests.sh               | 102 ++++++++++++++++++++++++++++++++++++++
>  scripts/lopartsetup               |   6 ++-
>  scripts/mkdeb                     |   9 ++++
>  tests/cirros-minios-stubdom-hvm   |  29 +++++++++++
>  tests/cirros-minios-stubdom-pvhvm |  30 +++++++++++
>  tests/cirros-pvgrub2-pv           |  27 ++++++++++
>  tests/cirros-pygrub-pv            |  27 ++++++++++
>  tests/cirros-qemu-hvm             |  28 +++++++++++
>  tests/cirros-qemu-pvhvm           |  29 +++++++++++
>  tests/cirros-separate-kernel-pv   |  28 +++++++++++
>  tests/series                      |   7 +++
>  13 files changed, 368 insertions(+), 2 deletions(-)
>  create mode 100644 configs/config-cirros
>  create mode 100644 tests/cirros-minios-stubdom-hvm
>  create mode 100644 tests/cirros-minios-stubdom-pvhvm
>  create mode 100644 tests/cirros-pvgrub2-pv
>  create mode 100644 tests/cirros-pygrub-pv
>  create mode 100644 tests/cirros-qemu-hvm
>  create mode 100644 tests/cirros-qemu-pvhvm
>  create mode 100644 tests/cirros-separate-kernel-pv
> 
> -- 
> 2.7.4
> 

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 11/11] Enable cirros tests in the default config

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 904 bytes --]

On Sun, 19 Mar 2017, Géza Gémes wrote:
> The existing cirros tests are enabled, with the following
> exceptions:
> 
> cirros-minios-stubdom-hvm and cirros-minios-stubdom-pvhvm are
> skipped as raisin does not install the stubdom
> 
> Signed-off-by: Géza Gémes <geza.gemes@gmail.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

> ---
>  defconfig | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/defconfig b/defconfig
> index 111554e..f3b1759 100644
> --- a/defconfig
> +++ b/defconfig
> @@ -31,6 +31,6 @@ GIT_TRANSPORT="git"
>  # Tests
>  ## All tests: busybox-pv busybox-hvm
>  ## ENABLED_TESTS is the list of test run by raise test
> -ENABLED_TESTS="busybox-pv busybox-hvm"
> +ENABLED_TESTS="busybox-pv busybox-hvm cirros-separate-kernel-pv cirros-pygrub-pv cirros-pvgrub2-pv cirros-qemu-hvm cirros-qemu-pvhvm"
>  
>  . configs/config-cirros
> -- 
> 2.7.4
> 

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/11] Fix installation of python libs

[Géza Gémes]

[-- Attachment #1.1: Type: text/plain, Size: 1316 bytes --]

2017. márc. 21. 0:54 ezt írta ("Stefano Stabellini" <sstabellini@kernel.org
>):

On Sun, 19 Mar 2017, Géza Gémes wrote:
> Change deb package build in order to symlink the files installed
> to site-packages to dist-packages to have them inluded in the
> default PYTHONPATH
>
> Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
> ---
>  scripts/mkdeb | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/scripts/mkdeb b/scripts/mkdeb
> index 3796300..23bcc2a 100755
> --- a/scripts/mkdeb
> +++ b/scripts/mkdeb
> @@ -46,6 +46,15 @@ then
>      rm -rf deb/usr/lib64
>  fi
>
> +# Make sure that python libs will be in the path
> +mkdir -p deb/usr/lib/python2.7/dist-packages/
> +cd deb/usr/lib/python2.7/dist-packages/
> +for file in `ls ../site-packages`
> +do
> +    ln -s ../site-packages/$file .
> +done
> +cd -

Actually for Debian I think it makes sense to move (not link) the files
to dist-packages, what do you think?


As I haven't tried on anything else than ubuntu 14.04 I was afraid of
breaking it elsewhere. But if you are aware that Debian and forks are using
dist-packages, I will change the symlink to a move.



>  # Fill in the debian boilerplate
>  mkdir -p deb/DEBIAN
>  cat >deb/DEBIAN/control <<EOF
> --
> 2.7.4
>


Cheers

Géza

[-- Attachment #1.2: Type: text/html, Size: 2377 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 03/11] Introduce cirros tests

[Géza Gémes]

[-- Attachment #1.1: Type: text/plain, Size: 8494 bytes --]

2017. márc. 21. 1:55 ezt írta ("Stefano Stabellini" <sstabellini@kernel.org
>):

On Sun, 19 Mar 2017, Géza Gémes wrote:
> Add support for using cirros images in raisin tests
>
> Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
> ---
>  configs/config-cirros |  44 ++++++++++++++++++++++
>  defconfig             |   2 +
>  lib/common-tests.sh   | 102 ++++++++++++++++++++++++++++++
++++++++++++++++++++
>  3 files changed, 148 insertions(+)
>  create mode 100644 configs/config-cirros
>
> diff --git a/configs/config-cirros b/configs/config-cirros
> new file mode 100644
> index 0000000..fa2823e
> --- /dev/null
> +++ b/configs/config-cirros

The files under configs are meant to contain the git urls and software
versions of each component to build. It is not the right place for all
these cirros related variables.

Instead, I would move config-cirros to a new top level directory, maybe
I would call it "tests-configs".



> @@ -0,0 +1,44 @@
> +CIRROS_BASE_URL="https://download.cirros-cloud.net/"
> +CIRROS_VERSION="0.3.5"
> +
> +source `pwd`/lib/common-functions.sh
> +get_arch
> +case $RAISIN_ARCH in
> +    x86_64)
> +        CIRROS_ARCH=x86_64
> +        ;;
> +    x86_32)
> +        CIRROS_ARCH=i386
> +        ;;
> +    *)
> +        echo $PREPEND cirros tests only valid on x86, 32 or 64 bit
> +        exit 1
> +esac
> +
> +CIRROS_KERNEL_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-kernel
> +CIRROS_INITRD_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-initramfs
> +CIRROS_ROOTFS_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img
> +CIRROS_DISK_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img
> +CIRROS_KERNEL_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${
CIRROS_KERNEL_FILE}
> +CIRROS_INITRD_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${
CIRROS_INITRD_FILE}
> +CIRROS_ROOTFS_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${
CIRROS_ROOTFS_FILE}.gz
> +CIRROS_DISK_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_DISK_FILE}
> +
> +CIRROS_GRUB_CFG="(xen/xvda,msdos1)/boot/grub/grub.cfg"
> +

I like to keep the config files as simple as possible, with just
parameters and variable assignments. Ideally, we would have one
config-cirros file with only varibles, like configs/config-master for
example. Everything else should be elsewhere. I would even go as far as
removing the get_arch call above, writing two fully static cirros config
files, one for x86_64 and one for x86_32.


> +set +e
> +QEMU_IMG=`which qemu-img`
> +set -e
> +if [[ -z "$QEMU_IMG" ]]
> +then
> +    QEMU_IMG="/usr/lib/xen/bin/qemu-img"
> +fi
> +
> +set +e
> +PVGRUB=`which grub-${CIRROS_ARCH}-xen`
> +set -e
> +if [[ -z "$PVGRUB" ]]
> +then
> +    PVGRUB="/usr/lib/xen/boot/grub-${CIRROS_ARCH}-xen"
> +fi

I would move the detection of QEMU_IMG and PVGRUB to functions in
lib/common-functions.sh.


> diff --git a/defconfig b/defconfig
> index f8ef398..111554e 100644
> --- a/defconfig
> +++ b/defconfig
> @@ -32,3 +32,5 @@ GIT_TRANSPORT="git"
>  ## All tests: busybox-pv busybox-hvm
>  ## ENABLED_TESTS is the list of test run by raise test
>  ENABLED_TESTS="busybox-pv busybox-hvm"
> +
> +. configs/config-cirros
> diff --git a/lib/common-tests.sh b/lib/common-tests.sh
> index d346af4..79815ce 100644
> --- a/lib/common-tests.sh
> +++ b/lib/common-tests.sh
> @@ -178,3 +178,105 @@ function get_host_initrd() {
>          exit 1
>      fi
>  }
> +
> +function cirros_network_init() {
> +    rootdir=$1
> +    # Configure static ip
> +    $SUDO sed -i -e 's/iface eth0 inet dhcp/iface eth0 inet static/'
${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/iface eth0 inet static/a\    address 169.254.0.2'
${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/address/a\    network 169.254.0.0'
${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/network/a\    broadcast 169.254.0.255'
${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/broadcast/a\    netmask 255.255.255.0'
${rootdir}/etc/network/interfaces

I think that it would be more future-proof to just generate and
overwrite ${rootdir}/etc/network/interfaces with our own. Also in
general we don't use ${} for variables.


> +    # Disable cloud-init
> +    $SUDO rm -f ${rootdir}/etc/rc3.d/S*cirros*ds*
> +    $SUDO rm -f ${rootdir}/etc/rc3.d/S*-cirros-userdata
> +}
> +
> +function get_cirros_kernel() {
> +    bootdir=$1
> +    basename `find $bootdir -name vmlinuz* 2>/dev/null | head -1`
> +}
> +
> +function get_cirros_initrd() {
> +    bootdir=$1
> +    basename `find $bootdir -name initrd* 2>/dev/null | head -1`
> +}
> +
> +function cirros_grub_cfg() {
> +    rootdir=$1
> +    grubcfg="`echo $CIRROS_GRUB_CFG | cut -d ')' -f 2`"
> +    grubdir=`dirname $grubcfg`
> +    bootdir=`dirname $grubdir`
> +    tmpgrubcfg=`mktemp`
> +    cat > $tmpgrubcfg <<EOF
> +root='(xen/xvda,msdos1)'
> +insmod xzio
> +insmod gzio
> +insmod btrfs
> +insmod ext2
> +set timeout=1
> +set default=0
> +menuentry Cirros {
> +    linux `echo $bootdir`/`get_cirros_kernel ${rootdir}/${bootdir}`
root=/dev/xvda1 ro
> +    initrd `echo $bootdir`/`get_cirros_initrd ${rootdir}/${bootdir}`
> +}
> +EOF
> +    $SUDO mv $tmpgrubcfg ${rootdir}/${grubcfg}
> +}
> +
> +function set_up_cirros_tests() {
> +    verbose_echo "Setting up environment for cirros tests"
> +    tmpdir=`mktemp -d`

Reading later patches, I noticed that you rely on the variable being
called "tmpdir". If you need such as a variable please give a more
meaningful name.


> +    wget -q $CIRROS_KERNEL_URL -P $tmpdir
> +    wget -q $CIRROS_INITRD_URL -P $tmpdir
> +    wget -q $CIRROS_ROOTFS_URL -P $tmpdir
> +    wget -q $CIRROS_DISK_URL -P $tmpdir

Shouldn't we check whether they have been already downloaded? So that,
if you run 'raise tests' twice, the second time you don't need to wait?
In fact, I would download them to a "downloads" directory, rather than a
tmpdir.

Today, each busybox test setups its own disk and loop device. If
possible I would do the same for the cirros test. We can reuse already
downloaded images to save time and bandwidth. Of course, we can share
the same common init functions in all cirros tests. For example, in this
function we setup the disk for both pv and hvm tests. I would create two
separate functions, one for each.


> +    gunzip ${tmpdir}/${CIRROS_ROOTFS_FILE}.gz
> +    mv ${tmpdir}/${CIRROS_DISK_FILE} ${tmpdir}/${CIRROS_DISK_FILE}.qcow2
> +    $QEMU_IMG convert -f qcow2 -O raw ${tmpdir}/${CIRROS_DISK_FILE}.qcow2
${tmpdir}/${CIRROS_DISK_FILE}
> +    CIRROS_ROOTFS_LOOP=`create_loop ${tmpdir}/${CIRROS_ROOTFS_FILE}`
> +    CIRROS_DISK_LOOP_P0=`$SUDO $BASEDIR/scripts/lopartsetup
${tmpdir}/${CIRROS_DISK_FILE} | head -1 | cut -d ":" -f 1`
> +    CIRROS_ROOTFS_MNTPT=`mktemp -d`
> +    CIRROS_DISK_MNTPT=`mktemp -d`

local variables should be lower case and declared "local"


> +    $SUDO mount $CIRROS_ROOTFS_LOOP $CIRROS_ROOTFS_MNTPT
> +    $SUDO mount $CIRROS_DISK_LOOP_P0 $CIRROS_DISK_MNTPT
> +    cirros_network_init $CIRROS_ROOTFS_MNTPT
> +    cirros_network_init $CIRROS_DISK_MNTPT
> +    cirros_grub_cfg $CIRROS_DISK_MNTPT
> +    $SUDO umount $CIRROS_ROOTFS_MNTPT
> +    $SUDO umount $CIRROS_DISK_MNTPT
> +    $SUDO rmdir $CIRROS_ROOTFS_MNTPT
> +    $SUDO rmdir $CIRROS_DISK_MNTPT
> +    $SUDO losetup -d $CIRROS_ROOTFS_LOOP
> +    $SUDO losetup -d $CIRROS_DISK_LOOP_P0
> +}
> +
> +function tear_down_cirros_tests() {
> +    tmpdir=$1
> +    if [[ `$SUDO xl vm-list | grep -e "raisin-test" | wc -l` -gt 0 ]]
> +    then
> +        $SUDO xl destroy "raisin-test"
> +    fi
> +    number_of_cirros_tests=0
> +    for test in $TESTS
> +    do
> +        if [[ "`echo $test | cut -d '-' -f 1`" == "cirros" ]]
> +        then
> +            number_of_cirros_tests=$((number_of_cirros_tests+1))
> +        fi
> +    done
> +    number_of_run_cirros_tests=0
> +    for test in $TESTS
> +    do
> +        if [[ -f ${tmpdir}/${test}.cfg ]]
> +        then
> +            number_of_run_cirros_tests=$((number_of_run_cirros_tests+1))
> +        fi
> +    done
> +    if [[ $number_of_cirros_tests == $number_of_run_cirros_tests ]]
> +    then
> +        verbose_echo "Deleting environment of cirros tests"
> +        $SUDO rm -rf $tmpdir
> +    fi
> +}

Thank you for reviewing these. I agree with your comments and will send the
changes soon.

Cheers

Géza

[-- Attachment #1.2: Type: text/html, Size: 11506 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [PATCH 01/11] Fix installation of python libs

[Stefano Stabellini]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1603 bytes --]

On Tue, 21 Mar 2017, Géza Gémes wrote:
> 2017. márc. 21. 0:54 ezt írta ("Stefano Stabellini" <sstabellini@kernel.org>):
>       On Sun, 19 Mar 2017, Géza Gémes wrote:
>       > Change deb package build in order to symlink the files installed
>       > to site-packages to dist-packages to have them inluded in the
>       > default PYTHONPATH
>       >
>       > Signed-off-by: Géza Gémes <geza.gemes@gmail.com>
>       > ---
>       >  scripts/mkdeb | 9 +++++++++
>       >  1 file changed, 9 insertions(+)
>       >
>       > diff --git a/scripts/mkdeb b/scripts/mkdeb
>       > index 3796300..23bcc2a 100755
>       > --- a/scripts/mkdeb
>       > +++ b/scripts/mkdeb
>       > @@ -46,6 +46,15 @@ then
>       >      rm -rf deb/usr/lib64
>       >  fi
>       >
>       > +# Make sure that python libs will be in the path
>       > +mkdir -p deb/usr/lib/python2.7/dist-packages/
>       > +cd deb/usr/lib/python2.7/dist-packages/
>       > +for file in `ls ../site-packages`
>       > +do
>       > +    ln -s ../site-packages/$file .
>       > +done
>       > +cd -
> 
> Actually for Debian I think it makes sense to move (not link) the files
> to dist-packages, what do you think?
> 
> 
> As I haven't tried on anything else than ubuntu 14.04 I was afraid of breaking it elsewhere. But if you are aware that Debian and forks are using dist-packages, I will change
> the symlink to a move.

Yes, I think that's the case.




>       >  # Fill in the debian boilerplate
>       >  mkdir -p deb/DEBIAN
>       >  cat >deb/DEBIAN/control <<EOF
>       > --
>       > 2.7.4
>       >

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel