[OSSTEST Nested PATCH v9 0/9] Introduction of netsted HVM test job

[OSSTEST Nested PATCH v9 0/9] Introduction of netsted HVM test job

[longtao.pang]

This patch set adds nested HVM test case for osstest.
In this test case, a Xen hypervisor (L1) runs on top of another Xen 
hypervisor (L0). 
Upon L1 hypervisor, we will then create a nested guest (L2), and test if the 
Linux guest can then be installed and run well.
About nested Xen virtualization, 
refer to http://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen.

Test steps
        0. To run osstest in standalone mode, write a config file in
           '~/.xen-osstest/config', and then create a standalone.config file 
           to define 'TREE_LINUX', 'REVISION_LINUX' which will be used for
           nested test. The directory path of 'Debian Images' could be defined 
           in '~/.xen-osstest/config'. 
        1. run './standalone-reset' to generate standalone.db firstly then run 
           'build-amd64' job and then 'build-amd64-pvops', to prepare xen
           installation tarball and hvm guest kernel.
        2. run 'test-amd64-amd6-qemuu-nested' job, it does following:
                a. invoke test step of 'ts-debain-hvm-install' to install 
                   a normal HVM guest
                b. invoke test step of 'ts-nested-setup' to make some
                   appropriate runvars which selecthost() would recognise and
                   prepare the configurations for installing L2 guest VM. 
                c. invoke test step of 'ts-xen-install' to install xen on 
                   the normal guest, alter it into a L1 hypervisor
                d. invoke test step of 'ts-debain-hvm-install' again, but 
                   take the L1 hypervisor as host, install the L2 guest on it
                e. invoke test step of 'ts-guest-stop', stop L2 guest.
                f. invoke test step of 'ts-guest-destroy' to destroy L1 guest.

This patch set reuse 'ts-debian-hvm-install' for both L1 installation
and L2 installation, use 'nestedl1' as L1's guestname and identity 
and use 'nestedl2 as L2's guestname.
It also reuses 'ts-xen-install' with L1's identity 'nestedl1' input parameter 
to differentiate from L0 Xen installation.
This patch series has been tested on test machines of amd64 arch,
Debian-7.2.0-amd64 as guests OS, with hvm domain0 of Linux kernel 3.18.5, 
in standalone mode.
Also, we use linux-stable tree as domain0 kernel source.
----------------------------------------------------------------
longtao.pang (9):
      parsing grub which has 'submenu' primitive
      Changes to support '/boot' leading paths of kernel, xen, in grub
      Refactor installation of overlays for guest used
      Edit some APIs in TestSupport.pm for nested test
      Move the codes about memory size setting into prep()
      Changes on test step of Debian hvm guest install
      Add new script to customize nested test configuration
      Compose the main recipe of nested test job
      Add test job for nest test case

 Osstest/Debian.pm      |   22 ++++++++++++----
 Osstest/TestSupport.pm |   27 ++++++++++++--------
 make-flight            |   32 +++++++++++++++++++++++
 sg-run-job             |   11 ++++++++
 ts-debian-hvm-install  |   37 +++++++++++++++++----------
 ts-nested-setup        |   66 ++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 167 insertions(+), 28 deletions(-)
 create mode 100755 ts-nested-setup

[OSSTEST Nested PATCH v9 1/9] parsing grub which has 'submenu' primitive

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

>From a hvm kernel build from Linux stable Kernel tree,
the auto generated grub2 menu will have 'submenu' primitive, upon the
'menuentry' items. Xen boot entries will be grouped into a submenu. This
patch adds capability to support such grub formats.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
 Osstest/Debian.pm |   13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index bd506a6..1e7e8e3 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -407,10 +407,18 @@ sub setupboot_grub2 ($$$$) {
     
         my $count= 0;
         my $entry;
+        my $submenu;
         while (<$f>) {
             next if m/^\s*\#/ || !m/\S/;
             if (m/^\s*\}\s*$/) {
-                die unless $entry;
+                die unless $entry || $submenu;
+                if(!defined $entry && defined $submenu){
+                    logm("Met end of a submenu starting from ".
+                        "$submenu->{StartLine}. ".
+                        "Our want kern is $want_kernver");
+                    $submenu=undef;
+                    next;
+                }
                 my (@missing) =
                     grep { !defined $entry->{$_} } 
 		        (defined $xenhopt
@@ -441,6 +449,9 @@ sub setupboot_grub2 ($$$$) {
                 $entry= { Title => $1, StartLine => $., Number => $count };
                 $count++;
             }
+            if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
+                $submenu={ StartLine =>$.};
+            }
             if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
                 die unless $entry;
                 $entry->{Hv}= $1;
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 1/9] parsing grub which has 'submenu' primitive

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> From: "longtao.pang" <longtaox.pang@intel.com>
> 
> From a hvm kernel build from Linux stable Kernel tree,
> the auto generated grub2 menu will have 'submenu' primitive, upon the
> 'menuentry' items. Xen boot entries will be grouped into a submenu. This
> patch adds capability to support such grub formats.
> 
> Signed-off-by: longtao.pang <longtaox.pang@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Can I ask that when a patch is Acked (as I did this one in v8) that you
add that Acked-by to the patch (immediately after your Signed-off-by)
for the next posting, unless there have been major modifications which
might invalidate the review (use your judgement here).

That essentially serves as a reminder to me and others that there is no
need to look at the patch again which speeds up the review process for
subsequent iterations.

Ian.

[OSSTEST Nested PATCH v9 2/9] Changes to support '/boot' leading paths of kernel, xen, in grub

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

Support situations of grub that have vmlinuz and other things starting
with path of '/boot' rather than '/'.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
 Osstest/Debian.pm |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 1e7e8e3..6691ff6 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -452,21 +452,21 @@ sub setupboot_grub2 ($$$$) {
             if (m/^submenu\s+[\'\"](.*)[\'\"].*\{\s*$/) {
                 $submenu={ StartLine =>$.};
             }
-            if (m/^\s*multiboot\s*\/(xen\-[0-9][-+.0-9a-z]*\S+)/) {
+            if (m/^\s*multiboot\s*(?:\/boot)?\/(xen\S+)/) {
                 die unless $entry;
                 $entry->{Hv}= $1;
             }
-            if (m/^\s*multiboot\s*\/(vmlinu[xz]-(\S+))/) {
+            if (m/^\s*multiboot\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
                 die unless $entry;
                 $entry->{KernOnly}= $1;
                 $entry->{KernVer}= $2;
             }
-            if (m/^\s*module\s*\/(vmlinu[xz]-(\S+))/) {
+            if (m/^\s*module\s*(?:\/boot)?\/(vmlinu[xz]-(\S+))/) {
                 die unless $entry;
                 $entry->{KernDom0}= $1;
                 $entry->{KernVer}= $2;
             }
-            if (m/^\s*module\s*\/(initrd\S+)/) {
+            if (m/^\s*module\s*(?:\/boot)?\/(initrd\S+)/) {
                 $entry->{Initrd}= $1;
             }
 	    if (m/^\s*module\s*\/(xenpolicy\S+)/) {
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 2/9] Changes to support '/boot' leading paths of kernel, xen, in grub

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> From: "longtao.pang" <longtaox.pang@intel.com>
> 
> Support situations of grub that have vmlinuz and other things starting
> with path of '/boot' rather than '/'.
> 
> Signed-off-by: longtao.pang <longtaox.pang@intel.com>

Per v8: Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST Nested PATCH v9 3/9] Refactor installation of overlays for guest used

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

Based on Ian Campbell's v5_patch [04,05,06], I create this patch
to refactor installation of overlays for guest used.

Link of Ian Campbell's patch:
http://lists.xenproject.org/archives/html/xen-devel/2015-04/msg01452.html
http://lists.xenproject.org/archives/html/xen-devel/2015-04/msg01447.html

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
 Osstest/Debian.pm |    1 +
 1 file changed, 1 insertion(+)

diff --git a/Osstest/Debian.pm b/Osstest/Debian.pm
index 6691ff6..4af6957 100644
--- a/Osstest/Debian.pm
+++ b/Osstest/Debian.pm
@@ -624,6 +624,7 @@ sub preseed_base ($$$$;@) {
 
     preseed_ssh($ho, $sfx);
     preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
+    preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');
 
     my $preseed = <<"END";
 d-i mirror/suite string $suite
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 3/9] Refactor installation of overlays for guest used

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> From: "longtao.pang" <longtaox.pang@intel.com>
> 
> Based on Ian Campbell's v5_patch [04,05,06], I create this patch
> to refactor installation of overlays for guest used.
> 
> Link of Ian Campbell's patch:
> http://lists.xenproject.org/archives/html/xen-devel/2015-04/msg01452.html
> http://lists.xenproject.org/archives/html/xen-devel/2015-04/msg01447.html

This information is useful in the context of posting, but it should be
after the main body of the commit log and a --- marker so it isn't
included when committing.

The actual body of the commit message should offer some explanation of
how/why it is safe to apply the overlay to guests as well as hosts as
well as why you want it for guests.

Something like:

        Apply overlay to guests as well as hosts by moving to
        preseed_base.
        
        We do this because we want to be able to:
        
              * Use /etc/init.d/osstest-confirm-booted for guests as
                well as hosts (in particular when testing nestedhvm we
                sometimes want to treat a guest as a host)
              * Use grub's 20_linux_xen which is harmless unless Xen is
                installed in the guest which is the case only for
                nestedhvm testing when we want these changes.
        
        The other things in the overlay are some initscripts (xenbridge
        and xenlightdaemons) which are not enabled for guests and hence
        are therefore harmless.
        
        Signed-off-by: longtao.pang <longtaox.pang@intel.com>
        ---
        Base on Ian Campbell's v5_patch [04,05,06], I create this
        patch... etc

> @@ -624,6 +624,7 @@ sub preseed_base ($$$$;@) {
>  
>      preseed_ssh($ho, $sfx);
>      preseed_hook_overlay($ho, $sfx, $c{OverlayLocal}, 'overlay-local.tar');
> +    preseed_hook_overlay($ho, $sfx, 'overlay', 'overlay.tar');

I think you should also remove the existing one from preseed_create,
else it will happen twice for L0 host install.

Ian.

[OSSTEST Nested PATCH v9 4/9] Edit some APIs in TestSupport.pm for nested test

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

1. If vif model is defined for guest, use it in guest's configuration
2. In L2 installation context, its host (L1) shall not have a static IP
but stored in runvar in L1 installation before; use this stored runvar
IP.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
Changes in v9:
1. Refactor 'selecthost' to set L1 guest's IpAddr for nested job.
2. Change $vifmodel to a reasonable format.
3. Refactor 'prepareguest' to use a specific disksize for nested L1
guest.
---
 Osstest/TestSupport.pm |   27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 1bde67d..2d7ecc0 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -865,13 +865,17 @@ sub selecthost ($) {
     serial_host_setup($ho);
 
     $ho->{IpStatic} = get_host_property($ho,'ip-addr');
-    if (!defined $ho->{IpStatic}) {
-	my $ip_packed= gethostbyname($ho->{Fqdn});
-	die "$ho->{Fqdn} ?" unless $ip_packed;
-	$ho->{IpStatic}= inet_ntoa($ip_packed);
-	die "$ho->{Fqdn} ?" unless defined $ho->{IpStatic};
+    if (defined $r{"${ident}_ip"}) {
+        $ho->{Ip} = $r{"${ident}_ip"};
+    } else {
+        if (!defined $ho->{IpStatic}) {
+	    my $ip_packed= gethostbyname($ho->{Fqdn});
+	    die "$ho->{Fqdn} ?" unless $ip_packed;
+	    $ho->{IpStatic}= inet_ntoa($ip_packed);
+	    die "$ho->{Fqdn} ?" unless defined $ho->{IpStatic};
+        }
+        $ho->{Ip}= $ho->{IpStatic};
     }
-    $ho->{Ip}= $ho->{IpStatic};
 
     #----- tftp -----
 
@@ -1513,14 +1517,15 @@ sub prepareguest ($$$$$$) {
     store_runvar("${gn}_hostname", $hostname);
     store_runvar("${gn}_tcpcheckport", $tcpcheckport);
     store_runvar("${gn}_boot_timeout", $boot_timeout);
+    my $gho= selectguest($gn, $ho);
+    store_runvar("${gn}_domname", $gho->{Name});
+    #if we have defined guest specific disksize, override it
+    $mb = guest_var($gho,'disksize',$mb);
 
     if (defined $mb) {
 	store_runvar("${gn}_disk_lv", $r{"${gn}_hostname"}.'-disk');
     }
 
-    my $gho= selectguest($gn, $ho);
-    store_runvar("${gn}_domname", $gho->{Name});
-
     if (defined $mb) {
 	store_runvar("${gn}_vg", '');
 	if (!length $r{"${gn}_vg"}) {
@@ -1548,11 +1553,13 @@ sub prepareguest_part_xencfg ($$$$$) {
     my $oncrash= $xopts->{OnCrash} || 'preserve';
     my $vcpus= guest_var($gho, 'vcpus', $xopts->{DefVcpus} || 2);
     my $xoptcfg= $xopts->{ExtraConfig};
+    my $vif= guest_var($gho, 'vifmodel','');
+    my $vifmodel= $vif ? ",model=$vif" : '';
     $xoptcfg='' unless defined $xoptcfg;
     my $xencfg= <<END;
 name        = '$gho->{Name}'
 memory = ${ram_mb}
-vif         = [ 'type=ioemu,mac=$gho->{Ether}' ]
+vif         = [ 'type=ioemu,mac=$gho->{Ether}${vifmodel}' ]
 #
 on_poweroff = '$onpoweroff'
 on_reboot   = '$onreboot'
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 4/9] Edit some APIs in TestSupport.pm for nested test

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> From: "longtao.pang" <longtaox.pang@intel.com>
> 
> 1. If vif model is defined for guest, use it in guest's configuration
> 2. In L2 installation context, its host (L1) shall not have a static IP
> but stored in runvar in L1 installation before; use this stored runvar
> IP.
> 
> Signed-off-by: longtao.pang <longtaox.pang@intel.com>

The code looks good, thanks. I have some minor formatting/wording nits.

> @@ -1513,14 +1517,15 @@ sub prepareguest ($$$$$$) {
>      store_runvar("${gn}_hostname", $hostname);
>      store_runvar("${gn}_tcpcheckport", $tcpcheckport);
>      store_runvar("${gn}_boot_timeout", $boot_timeout);

Blank line here please.

> +    my $gho= selectguest($gn, $ho);
> +    store_runvar("${gn}_domname", $gho->{Name});

A blank line here please.

> +    #if we have defined guest specific disksize, override it

Space after "#" please and ideally "If" not "if".

I would say "use it" rather than "override it".

> +    $mb = guest_var($gho,'disksize',$mb);
>  

And no blank line here. (i.e. assignment of $mb nestles against the
following if and not the preceding store_runvar)

With that stuff fixed: Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST Nested PATCH v9 5/9] Move the codes about memory size setting into prep()

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

Move the codes about memory size setting from outside into prep(),
where these $ram_mb operations are only used; and in next patch,
for L2 guest installation, $ram_mb will be override by a runvar value,
which needs $gho in scope.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
 ts-debian-hvm-install |   24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index 0085d82..fd28abc 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -149,6 +149,18 @@ sub prep () {
     target_putfilecontents_root_stash($ho, 10, preseed(),
                                       $preseed_file_path);
 
+    # If host has >8G free memory, create a guest with 4G memory to catch
+    # any error that triggers cross 4G boundary
+    my $host_freemem_mb = host_get_free_memory($ho);
+    my $ram_minslop = 100;
+    my $ram_lots = 5000;
+    if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
+        $ram_mb = $ram_lots;
+    } else {
+        $ram_mb = 768;
+    }
+    logm("Host has $host_freemem_mb MB free memory, setting guest memory size to $ram_mb MB");
+
     more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
                           OnReboot => 'preserve',
                           Bios => $r{bios},
@@ -166,18 +178,6 @@ sub prep () {
     });
 }
 
-# If host has >8G free memory, create a guest with 4G memory to catch
-# any error that triggers cross 4G boundary
-my $host_freemem_mb = host_get_free_memory($ho);
-my $ram_minslop = 100;
-my $ram_lots = 5000;
-if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
-    $ram_mb = $ram_lots;
-} else {
-    $ram_mb = 768;
-}
-logm("Host has $host_freemem_mb MB free memory, setting guest memory size to $ram_mb MB");
-
 if (!$stage) {
     prep();
     guest_create($gho);
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 5/9] Move the codes about memory size setting into prep()

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> From: "longtao.pang" <longtaox.pang@intel.com>
> 
> Move the codes about memory size setting from outside into prep(),

"code for setting memory size"

> where these $ram_mb operations are only used; and in next patch,
> for L2 guest installation, $ram_mb will be override by a runvar value,

"overridden"

> which needs $gho in scope.

It is useful to say explicitly:

    This is pure code motion. No functional change.


> Signed-off-by: longtao.pang <longtaox.pang@intel.com>

With those minor changes:
        Acked-by: Ian Campbell <ian.campbell@citrix.com>

[OSSTEST Nested PATCH v9 6/9] Changes on test step of Debian hvm guest install

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

1. Increase disk size to accommodate to nested test requirement.
2. Since 'Debain-xxx-.iso' image will be stored in rootfs of L1 guest,
therefore needs more disk capacity, increase root partition size to
10000.
3. In L1 installation context, assign more memory (defined in runvar) to
it; Since it acts as a nested hypervisor anyway.
4. Comment out CDROM entry in sources.list to make HTTP URL entry
available for L1 hvm guest.
5. Enable nestedhvm feature in ExtraConfig for nested job.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
Changes in v9:
1. Using 'guest_var' to get nested L1 guest's specific disk size and
specific memory size.
2. Using 'guest_var' to enable or disable nestedhvm feature.
---
 ts-debian-hvm-install |   17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
index fd28abc..d823233 100755
--- a/ts-debian-hvm-install
+++ b/ts-debian-hvm-install
@@ -59,7 +59,7 @@ d-i partman-auto/expert_recipe string \\
                         use_filesystem{ } filesystem{ vfat } \\
                         mountpoint{ /boot/efi } \\
                 . \\
-                5000 50 5000 ext4 \\
+                10000 50 10000 ext4 \\
                         method{ format } format{ } \\
                         use_filesystem{ } filesystem{ ext4 } \\
                         mountpoint{ / } \\
@@ -73,6 +73,7 @@ d-i apt-setup/cdrom/set-first boolean false
 d-i preseed/late_command string \\
         in-target mkdir -p /boot/efi/EFI/boot; \\
         in-target cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/boot/bootx64.efi ;\\
+        in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list;
 END
 
     $preseed_file .= preseed_hook_cmds();
@@ -149,6 +150,11 @@ sub prep () {
     target_putfilecontents_root_stash($ho, 10, preseed(),
                                       $preseed_file_path);
 
+    my $extra_config='';
+    $extra_config .="nestedhvm=1\n"
+        # Use guest_var to get boolean value to enable nestedhvm
+        # feature or not
+        if guest_var($gho,"enable_nestedhvm",'false') eq 'true';
     # If host has >8G free memory, create a guest with 4G memory to catch
     # any error that triggers cross 4G boundary
     my $host_freemem_mb = host_get_free_memory($ho);
@@ -157,13 +163,18 @@ sub prep () {
     if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
         $ram_mb = $ram_lots;
     } else {
-        $ram_mb = 768;
+        # Use guest_var to get specific memsize, or will use default '768'
+        $ram_mb= guest_var($gho,'memsize',768);
     }
-    logm("Host has $host_freemem_mb MB free memory, setting guest memory size to $ram_mb MB");
+    logm("Host has $host_freemem_mb MB free memory,
+         setting guest memory size to $ram_mb MB");
+    # Use guest_var to get specific disk size, or will use default $disk_mb
+    $disk_mb= guest_var($gho,'disksize',$disk_mb);
 
     more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
                           OnReboot => 'preserve',
                           Bios => $r{bios},
+                          ExtraConfig => $extra_config,
                           PostImageHook => sub {
         my $cmds = iso_copy_content_from_image($gho, $newiso);
         $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 6/9] Changes on test step of Debian hvm guest install

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> From: "longtao.pang" <longtaox.pang@intel.com>
> 
> 1. Increase disk size to accommodate to nested test requirement.
                       ^ in preseed recipe (I think?)

> 2. Since 'Debain-xxx-.iso' image will be stored in rootfs of L1 guest,
> therefore needs more disk capacity, increase root partition size to
> 10000.
> 3. In L1 installation context, assign more memory (defined in runvar) to
> it; Since it acts as a nested hypervisor anyway.
> 4. Comment out CDROM entry in sources.list to make HTTP URL entry
> available for L1 hvm guest.
> 5. Enable nestedhvm feature in ExtraConfig for nested job.
> 
> Signed-off-by: longtao.pang <longtaox.pang@intel.com>
> ---
> Changes in v9:
> 1. Using 'guest_var' to get nested L1 guest's specific disk size and
> specific memory size.
> 2. Using 'guest_var' to enable or disable nestedhvm feature.
> ---
>  ts-debian-hvm-install |   17 ++++++++++++++---
>  1 file changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/ts-debian-hvm-install b/ts-debian-hvm-install
> index fd28abc..d823233 100755
> --- a/ts-debian-hvm-install
> +++ b/ts-debian-hvm-install
> @@ -73,6 +73,7 @@ d-i apt-setup/cdrom/set-first boolean false
>  d-i preseed/late_command string \\
>          in-target mkdir -p /boot/efi/EFI/boot; \\
>          in-target cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/boot/bootx64.efi ;\\
> +        in-target sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list;

When this is rebased onto v6 of my distro testing series this will need
to become/use preseed_hook_command and you won't want to mix it with the
EFI stuff since that has/will become conditional.

That change is probably enough to invalidate any Ack I might give now,
or I would do so.

>  END
>  
>      $preseed_file .= preseed_hook_cmds();
> @@ -149,6 +150,11 @@ sub prep () {
>      target_putfilecontents_root_stash($ho, 10, preseed(),
>                                        $preseed_file_path);
>  
> +    my $extra_config='';
> +    $extra_config .="nestedhvm=1\n"
> +        # Use guest_var to get boolean value to enable nestedhvm
> +        # feature or not
> +        if guest_var($gho,"enable_nestedhvm",'false') eq 'true';

IMHO a better formatting would be:

+    # Use guest_var to get boolean value to enable nestedhvm
+    # feature or not
+    $extra_config .="nestedhvm=1\n"
+        if guest_var($gho,"enable_nestedhvm",'false') eq 'true';

Although I think you could just omit the comment, it's not adding much.

> 
>      # If host has >8G free memory, create a guest with 4G memory to catch
>      # any error that triggers cross 4G boundary
>      my $host_freemem_mb = host_get_free_memory($ho);
> @@ -157,13 +163,18 @@ sub prep () {
>      if ($host_freemem_mb > $ram_lots * 2 + $ram_minslop) {
>          $ram_mb = $ram_lots;
>      } else {
> -        $ram_mb = 768;
> +        # Use guest_var to get specific memsize, or will use default '768'
> +        $ram_mb= guest_var($gho,'memsize',768);
>      }
> -    logm("Host has $host_freemem_mb MB free memory, setting guest memory size to $ram_mb MB");
> +    logm("Host has $host_freemem_mb MB free memory,
> +         setting guest memory size to $ram_mb MB");

Blank line here please.

> +    # Use guest_var to get specific disk size, or will use default $disk_mb
> +    $disk_mb= guest_var($gho,'disksize',$disk_mb);
>  
>      more_prepareguest_hvm($ho,$gho, $ram_mb, $disk_mb,
>                            OnReboot => 'preserve',
>                            Bios => $r{bios},
> +                          ExtraConfig => $extra_config,
>                            PostImageHook => sub {
>          my $cmds = iso_copy_content_from_image($gho, $newiso);
>          $cmds .= prepare_initrd($initrddir,$newiso,$preseed_file_path);

[OSSTEST Nested PATCH v9 7/9] Add new script to customize nested test configuration

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

1. In this script, make some appropriate runvars which selecthost would
recognise.
2. Prepare the configurations for installing L2 guest VM.
3. Create a lv disk in L0 and hot-attach it to L1, need to restart L1 to
make the block disk to be recognized by L1 system, then using this disk
to create a VG that used for installing L2.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
Changes in v9:
1. For L1, identical its ident and guestname as 'nestedl1'.
2. Reuse Ian Campbell's v5_patch[04,05,06] to avoid duplicated codes
about 'ssh host keys' and 'overlays' when installing nested guest VM.
3. Refactor LV disk creating for L2 guest.
---
 ts-nested-setup |   66 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100755 ts-nested-setup

diff --git a/ts-nested-setup b/ts-nested-setup
new file mode 100755
index 0000000..566186c
--- /dev/null
+++ b/ts-nested-setup
@@ -0,0 +1,66 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Intel Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use Osstest::Debian;
+use Osstest::TestSupport;
+
+tsreadconfig();
+# Pass L0's ident and L1's guestname
+our ($l0_ident,$l1_gn) = @ARGV;
+our ($l0,$l1) = ts_get_host_guest($l0_ident,$l1_gn);
+
+guest_check_ip($l1);
+# L1 guest's ident is same as guestname
+our $l1_ident = $l1->{Guest};
+
+store_runvar($l1_ident,$l1->{Guest});
+store_runvar("${l1_ident}_ip",$l1->{Ip});
+
+# Since Ian Campbell's v5_patch[04,05,06] has a bug that the
+# 'preseed/late_command' are unavailable, so add below line here to
+# make HTTP URL entry available in sources.list for L1 hvm guest.
+# Onece the bug is fiexed, this line code could be removed.
+target_cmd_root($l1, "sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list");
+target_cmd_root($l1, "update-rc.d osstest-confirm-booted start 99 2 .");
+
+# Inside L0, create storage lv, attach this lv to L1, and then inside L1 using the storage for installing L2
+my $vgname = $l1->{Vg};
+my $guest_storage_lv_name = "${l1_ident}_guest_storage";
+my $guest_storage_lv_size = guest_var($l1,'guest_storage_size',undef);
+my $guest_storage_lvdev = "/dev/${vgname}/${guest_storage_lv_name}";
+
+target_cmd_root($l0, <<END);
+    lvremove -f $guest_storage_lvdev ||:
+    lvcreate -L ${guest_storage_lv_size}M -n $guest_storage_lv_name $vgname
+    dd if=/dev/zero of=$guest_storage_lvdev count=10
+    xl block-attach $l1->{Name} ${guest_storage_lvdev},raw,sdb,rw
+END
+# This is probably a bug here, if we use 'xvdx' as block 'dev'
+# after installing xen and boot into xen kernel, the disk will not be
+# recognized by OS. Also, if we use 'sde' instead of 'sdb' as block 'dev',
+# after installing xen and boot into xen kernel, the disk will be recognized
+# as 'sdb' not 'sde', so we will have to use 'sdb' here.
+
+target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
+# After block-attach lvdev to L1, need to reboot L1 to make the attach
+# to take effect. I think this is a bug for kernel.
+target_reboot($l1);
+# Create a vg in L1 guest and vg name is ${l1_gn}-disk
+target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l1_gn}-disk /dev/xvdb");
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 7/9] Add new script to customize nested test configuration

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> +# Since Ian Campbell's v5_patch[04,05,06] has a bug that the
> +# 'preseed/late_command' are unavailable, so add below line here to
> +# make HTTP URL entry available in sources.list for L1 hvm guest.
> +# Onece the bug is fiexed, this line code could be removed.
> +target_cmd_root($l1, "sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list");

This should be fixed in my v6 posting of that series.

> +target_cmd_root($l1, "update-rc.d osstest-confirm-booted start 99 2 .");
> +
> +# Inside L0, create storage lv, attach this lv to L1, and then inside L1 using the storage for installing L2
> +my $vgname = $l1->{Vg};
> +my $guest_storage_lv_name = "${l1_ident}_guest_storage";
> +my $guest_storage_lv_size = guest_var($l1,'guest_storage_size',undef);
> +my $guest_storage_lvdev = "/dev/${vgname}/${guest_storage_lv_name}";

Perhaps:
        die "guest_storage_lv_size" unless $guest_storage_lv_size;
?

> +
> +target_cmd_root($l0, <<END);
> +    lvremove -f $guest_storage_lvdev ||:
> +    lvcreate -L ${guest_storage_lv_size}M -n $guest_storage_lv_name $vgname
> +    dd if=/dev/zero of=$guest_storage_lvdev count=10

I have some ideas/plans on how to refactor things so we can use
prepareguest_part_lvmdisk for secondary disks too, but this is OK as it
is for now IMHO.

> +    xl block-attach $l1->{Name} ${guest_storage_lvdev},raw,sdb,rw

This relies on/assumes the toolstack is xl (sorry for not spotting this
before).

Really it should use a new block_attach method on toolstack($ho) (i.e.
Osstest/Toolstack/*.pm) so it can work with libvirt etc.

However I think this hardcoding is tolerable for now (you've encoded
that it is xl only in make-flight already), but please could you add
near the top of this script:

        die "toolstack $r{toolstack}" unless $r{toolstack} eq "xl";
        
so it fails in an immediately obvious way if someone tries this with
another toolstack.

Ian: Is that OK with you?

> +END
> +# This is probably a bug here, if we use 'xvdx' as block 'dev'
> +# after installing xen and boot into xen kernel, the disk will not be
> +# recognized by OS. Also, if we use 'sde' instead of 'sdb' as block 'dev',
> +# after installing xen and boot into xen kernel, the disk will be recognized
> +# as 'sdb' not 'sde', so we will have to use 'sdb' here.
> +
> +target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));

Please can you put this above, just after the osstest-confirm-started
stuff so that it is not mixed in with the stuff dealing with the
additional disk.

> +# After block-attach lvdev to L1, need to reboot L1 to make the attach
> +# to take effect. I think this is a bug for kernel.

I've just tried this and it worked for me, i.e. /dev/xvdb appears
immediately after xl block-attach with no reboot required. Perhaps this
was an artefact of a bug in some previous iteration of the series?

I would replace this comment, and the "This is probably a bug here" and
"Inside L0, create storage lv" ones above with a single (long) comment
before all of this disk stuff which explains:

        We need to attach an extra disk to the L1 guest to be used as L2
        guest storage.
        
        When running in a nested HVM environment the L1 domain is acting
        as both a guest to L0 and a host to L2 guests and therefore
        potentially sees connections to two independent xenstore
        instances, one provided by the L0 host and one which is provided
        by the L1 instance of xenstore.
        
        Unfortunately the kernel is not capable of dealing with this and
        is only able to cope with a single xenstore connection. Since
        the L1 toolstack and L2 guests absolutely require xenstore to
        function we therefore cannot use the L0 xenstore and therefore
        cannot use PV devices (xvdX etc) in the L1 guest and must use
        emulated devices (sdX etc).
        
        However at the moment we have not yet rebooted L1 into Xen and
        so it does have PV devices available and sdb actually appears as
        xvdb. We could disable the Xen platform device and use emulated
        devices for the install phase too but that would be needlessly
        slow.

It's a bit long (to say the least) but I think it is worth explaining
since limitation of nestedhvm isn't obvious without thinking about it a
bit...

> +target_reboot($l1);
> +# Create a vg in L1 guest and vg name is ${l1_gn}-disk
> +target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l1_gn}-disk /dev/xvdb");

Re: [OSSTEST Nested PATCH v9 7/9] Add new script to customize nested test configuration

[Pang, LongtaoX]

> -----Original Message-----
> From: Ian Campbell [mailto:ian.campbell@citrix.com]
> Sent: Wednesday, May 06, 2015 7:37 PM
> To: Pang, LongtaoX
> Cc: xen-devel@lists.xen.org; Ian.Jackson@eu.citrix.com; wei.liu2@citrix.com; Hu,
> Robert
> Subject: Re: [OSSTEST Nested PATCH v9 7/9] Add new script to customize nested
> test configuration
> 
> On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> > +# Since Ian Campbell's v5_patch[04,05,06] has a bug that the
> > +# 'preseed/late_command' are unavailable, so add below line here to
> > +# make HTTP URL entry available in sources.list for L1 hvm guest.
> > +# Onece the bug is fiexed, this line code could be removed.
> > +target_cmd_root($l1, "sed -i 's/^deb *cdrom/#&/g' /etc/apt/sources.list");
> 
> This should be fixed in my v6 posting of that series.
> 
> > +target_cmd_root($l1, "update-rc.d osstest-confirm-booted start 99 2 .");
> > +
> > +# Inside L0, create storage lv, attach this lv to L1, and then inside L1 using the
> storage for installing L2
> > +my $vgname = $l1->{Vg};
> > +my $guest_storage_lv_name = "${l1_ident}_guest_storage";
> > +my $guest_storage_lv_size = guest_var($l1,'guest_storage_size',undef);
> > +my $guest_storage_lvdev = "/dev/${vgname}/${guest_storage_lv_name}";
> 
> Perhaps:
>         die "guest_storage_lv_size" unless $guest_storage_lv_size;
> ?
> 
> > +
> > +target_cmd_root($l0, <<END);
> > +    lvremove -f $guest_storage_lvdev ||:
> > +    lvcreate -L ${guest_storage_lv_size}M -n $guest_storage_lv_name
> $vgname
> > +    dd if=/dev/zero of=$guest_storage_lvdev count=10
> 
> I have some ideas/plans on how to refactor things so we can use
> prepareguest_part_lvmdisk for secondary disks too, but this is OK as it
> is for now IMHO.
> 
> > +    xl block-attach $l1->{Name} ${guest_storage_lvdev},raw,sdb,rw
> 
> This relies on/assumes the toolstack is xl (sorry for not spotting this
> before).
> 
> Really it should use a new block_attach method on toolstack($ho) (i.e.
> Osstest/Toolstack/*.pm) so it can work with libvirt etc.
> 
> However I think this hardcoding is tolerable for now (you've encoded
> that it is xl only in make-flight already), but please could you add
> near the top of this script:
> 
>         die "toolstack $r{toolstack}" unless $r{toolstack} eq "xl";
> 
> so it fails in an immediately obvious way if someone tries this with
> another toolstack.
> 
> Ian: Is that OK with you?
> 
> > +END
> > +# This is probably a bug here, if we use 'xvdx' as block 'dev'
> > +# after installing xen and boot into xen kernel, the disk will not be
> > +# recognized by OS. Also, if we use 'sde' instead of 'sdb' as block 'dev',
> > +# after installing xen and boot into xen kernel, the disk will be recognized
> > +# as 'sdb' not 'sde', so we will have to use 'sdb' here.
> > +
> > +target_install_packages_norec($l1, qw(lvm2 rsync genisoimage));
> 
> Please can you put this above, just after the osstest-confirm-started
> stuff so that it is not mixed in with the stuff dealing with the
> additional disk.
> 
> > +# After block-attach lvdev to L1, need to reboot L1 to make the attach
> > +# to take effect. I think this is a bug for kernel.
> 
> I've just tried this and it worked for me, i.e. /dev/xvdb appears
> immediately after xl block-attach with no reboot required. Perhaps this
> was an artefact of a bug in some previous iteration of the series?
> 
Thanks for trying, I tried it in my environment, a reboot is not required now.
I think this bug happened just because I have used ' Debian-7.6.0-amd64-DVD-1.iso ' as installing L1's guest OS before, 
it did required a reboot and I assumed it's as a kernel bug. 
According to you former comment, I change to use ' Debian-7.2.0-amd64-CD-1.iso' as installing L1's guest OS. 
I assumed that the bug still exist and keep to reboot L1 after block-attach 'lvdev'. 
In fact, I should take a try for that, but I missed. Sorry for that.
So, the reboot action is no need now.

> I would replace this comment, and the "This is probably a bug here" and
> "Inside L0, create storage lv" ones above with a single (long) comment
> before all of this disk stuff which explains:
> 
>         We need to attach an extra disk to the L1 guest to be used as L2
>         guest storage.
> 
>         When running in a nested HVM environment the L1 domain is acting
>         as both a guest to L0 and a host to L2 guests and therefore
>         potentially sees connections to two independent xenstore
>         instances, one provided by the L0 host and one which is provided
>         by the L1 instance of xenstore.
> 
>         Unfortunately the kernel is not capable of dealing with this and
>         is only able to cope with a single xenstore connection. Since
>         the L1 toolstack and L2 guests absolutely require xenstore to
>         function we therefore cannot use the L0 xenstore and therefore
>         cannot use PV devices (xvdX etc) in the L1 guest and must use
>         emulated devices (sdX etc).
> 
>         However at the moment we have not yet rebooted L1 into Xen and
>         so it does have PV devices available and sdb actually appears as
>         xvdb. We could disable the Xen platform device and use emulated
>         devices for the install phase too but that would be needlessly
>         slow.
> 
> It's a bit long (to say the least) but I think it is worth explaining
> since limitation of nestedhvm isn't obvious without thinking about it a
> bit...
> 
> > +target_reboot($l1);
> > +# Create a vg in L1 guest and vg name is ${l1_gn}-disk
> > +target_cmd_root($l1, "pvcreate /dev/xvdb && vgcreate ${l1_gn}-disk
> /dev/xvdb");
> 

Re: [OSSTEST Nested PATCH v9 7/9] Add new script to customize nested test configuration

[Ian Campbell]

On Thu, 2015-05-07 at 07:42 +0000, Pang, LongtaoX wrote:
[...]
> Thanks for trying, I tried it in my environment, a reboot is not required now.
> I think this bug happened just because I have used ' Debian-7.6.0-amd64-DVD-1.iso ' as installing L1's guest OS before, 
> it did required a reboot and I assumed it's as a kernel bug. 
> According to you former comment, I change to use ' Debian-7.2.0-amd64-CD-1.iso' as installing L1's guest OS. 
> I assumed that the bug still exist and keep to reboot L1 after block-attach 'lvdev'. 
> In fact, I should take a try for that, but I missed. Sorry for that.

No problem.

In fact it seems like you may have highlighted a regression in Debian
between the kernels used in 7.2.0 and 7.6.0.

> So, the reboot action is no need now.

Great!

Ian.

[OSSTEST Nested PATCH v9 8/9] Compose the main recipe of nested test job

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

The ident and guestname are same of 'nestedl1' for L1 guest VM.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
Changes in v9:
1. For L1, identical its ident and guestname as 'nestedl1';
And same for L2, 'nestedl2' is L2's name as well as its ident.
2. Place '+' to include parameters in the test id for ts-guest-destory
step.
---
 sg-run-job |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/sg-run-job b/sg-run-job
index eae159d..b19f9f9 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -299,6 +299,17 @@ proc run-job/test-pair {} {
 #    run-ts . remus-failover ts-remus-check         src_host dst_host + debian
 }
 
+proc need-hosts/test-nested {} {return host}
+proc run-job/test-nested {} {
+    run-ts . = ts-debian-hvm-install + host + nestedl1
+    run-ts . = ts-nested-setup + host + nestedl1
+    run-ts . = ts-xen-install nestedl1
+    run-ts . = ts-host-reboot nestedl1
+    run-ts . = ts-debian-hvm-install nestedl1 nestedl2
+    run-ts . = ts-guest-stop nestedl1 nestedl2
+    run-ts . = ts-guest-destroy + host + nestedl1
+}
+
 proc test-guest-migr {g} {
     if {[catch { run-ts . = ts-migrate-support-check + host $g }]} return
 
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 8/9] Compose the main recipe of nested test job

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> From: "longtao.pang" <longtaox.pang@intel.com>
> 
> The ident and guestname are same of 'nestedl1' for L1 guest VM.
> 
> Signed-off-by: longtao.pang <longtaox.pang@intel.com>

Acked-by: Ian Campbell <ian.campbell@citrix.com>

Ian.

[OSSTEST Nested PATCH v9 9/9] Add test job for nest test case

[longtao.pang]

From: "longtao.pang" <longtaox.pang@intel.com>

1. This patch adds creation of the nested test job, when job creation
procedure is invoked.
2. Set nested L1's vif model, nestedhvm feature, set specific disk
size and memory size for nested test by make-flight.

Signed-off-by: longtao.pang <longtaox.pang@intel.com>
---
Changes in v9:
1. Group 'nestedl1*' and 'nestedl2*' together in the list.
2. Remove duplicated runvars for nested job.
3. Avoid testing Xen branches before 4.4; Avoid flights of qemu
traditional (i.e. qemu upstream supported only).
---
 make-flight |   32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/make-flight b/make-flight
index cc8ecdb..b5c7c27 100755
--- a/make-flight
+++ b/make-flight
@@ -204,6 +204,37 @@ do_hvm_win7_x64_tests () {
             all_hostflags=$most_hostflags,hvm
 }
 
+do_hvm_debian_nested_tests () {
+  if [ $xenarch != amd64 -o $dom0arch != amd64 \
+      -o "x$qemuu_suffix" != "x-qemuu" ]; then
+    return
+  fi
+
+  case $xenbranch in
+    xen-3.*-testing)      return;;
+    xen-4.0-testing)      return;;
+    xen-4.1-testing)      return;;
+    xen-4.2-testing)      return;;
+    xen-4.3-testing)      return;;
+  esac
+  # Define bios type, in 'ts-debian-hvm-install' script will use it,
+  # or will die if "$r{bios}" undefined.
+  bios=$1
+
+  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
+            test-nested xl $xenarch $dom0arch $qemuu_runvar \
+            nestedl1_image=debian-7.2.0-amd64-CD-1.iso \
+            nestedl1_vifmodel='e1000' \
+            nestedl1_disksize='15000' \
+            nestedl1_memsize='3072' \
+            nestedl1_enable_nestedhvm='true' \
+            nestedl1_guest_storage_size='20000' \
+            nestedl2_image=debian-7.2.0-amd64-CD-1.iso \
+            nestedl2_disksize='15000' \
+            bios=$bios
+            all_hostflags=$most_hostflags,hvm
+}
+
 do_hvm_debian_test_one () {
   testname=$1
   bios=$2
@@ -408,6 +439,7 @@ test_matrix_do_one () {
     do_hvm_rhel6_tests
 
     do_hvm_debian_tests
+    do_hvm_debian_nested_tests seabios
 
   done # qemuu_suffix
 
-- 
1.7.10.4

Re: [OSSTEST Nested PATCH v9 9/9] Add test job for nest test case

[Ian Campbell]

On Sat, 2015-05-02 at 14:28 +0800, longtao.pang wrote:
> diff --git a/make-flight b/make-flight
> index cc8ecdb..b5c7c27 100755
> --- a/make-flight
> +++ b/make-flight
> @@ -204,6 +204,37 @@ do_hvm_win7_x64_tests () {
>              all_hostflags=$most_hostflags,hvm
>  }
>  
> +do_hvm_debian_nested_tests () {
> +  if [ $xenarch != amd64 -o $dom0arch != amd64 \
> +      -o "x$qemuu_suffix" != "x-qemuu" ]; then
> +    return
> +  fi
> +
> +  case $xenbranch in
> +    xen-3.*-testing)      return;;
> +    xen-4.0-testing)      return;;
> +    xen-4.1-testing)      return;;
> +    xen-4.2-testing)      return;;
> +    xen-4.3-testing)      return;;
> +  esac
> +  # Define bios type, in 'ts-debian-hvm-install' script will use it,
> +  # or will die if "$r{bios}" undefined.
> +  bios=$1

Please can you just put
        bios=$1
as the first line in the function, followed by a blank line, followed by
the $xenarch/$domarch/$qemuu_suffix test. No need for the comment.

> +
> +  job_create_test test-$xenarch$kern-$dom0arch$qemuu_suffix-nested \
> +            test-nested xl $xenarch $dom0arch $qemuu_runvar \
> +            nestedl1_image=debian-7.2.0-amd64-CD-1.iso \
> +            nestedl1_vifmodel='e1000' \
> +            nestedl1_disksize='15000' \
> +            nestedl1_memsize='3072' \
> +            nestedl1_enable_nestedhvm='true' \
> +            nestedl1_guest_storage_size='20000' \
> +            nestedl2_image=debian-7.2.0-amd64-CD-1.iso \
> +            nestedl2_disksize='15000' \
> +            bios=$bios
> +            all_hostflags=$most_hostflags,hvm

Looks good, thanks.

With the bios= moved:
        Acked-by: Ian Campbell <ian.campbell@citrix.com>

Ian.