[OSSTEST PATCH 0/6] Use mkfs rather than resize2fs for build fs

[OSSTEST PATCH 0/6] Use mkfs rather than resize2fs for build fs

[Ian Jackson]

At the moment, when we set up a machine for builds, we:
 - install Debian, leaving plenty of space in the root LVM VG
 - install the build tools
 - resize the root fs to make room for the builds

This fs resize runs (mostly) in the background, alongside the first
build(s).  This is, it turns out, not a good idea: fs resize is
massively slower than mkfs (so takes a long time) and very demanding
of IO bandwidth (so it makes builds run more slowly).

We have worked around this slowness to an extent, for example in
fbd803f7 "ts-rumpuserxen-build: Double the build timeout" and
ee31a32a "ts-xen-build-prep: Run the backgrounded resize2fs with ionice".

This series:
 - reverts our previous attempt to deal with this problem with ionice
 - reverts the (ancient) arrangements to resize in the background
 - instead, arranges to lvcreate and mkfs a new fs for /home/osstest
   (and copy the existing contents into it).

[OSSTEST PATCH 1/6] Revert "ts-xen-build-prep: Run the backgrounded resize2fs with ionice"

[Ian Jackson]

This did not really help.  We are going to do something different.

This reverts commit ee31a32af1fc9a38ad08b4a36ee69e1c58ea0828.
---
 ts-xen-build-prep |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 3684c0a..23c901b 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -159,10 +159,7 @@ sub lvextend1 ($$$) {
     if (!$background) {
 	target_cmd_root($ho, $cmd, $timeout);
     } else {
-	target_cmd_root($ho,
-			"ionice -cBest-Effort -n7 ".
-			"$cmd >>resizefs.log 2>&1 </dev/null &",
-			30);
+	target_cmd_root($ho, "$cmd >>resizefs.log 2>&1 </dev/null &", 30);
     }
 }
 
-- 
1.7.10.4

[OSSTEST PATCH 2/6] ts-xen-build-prep: Abolish "background" parameter to lvextend

[Ian Jackson]

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
---
 ts-xen-build-prep |   12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 23c901b..a9c9e4a 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -65,11 +65,11 @@ sub lvextend_stage1 () {
         fi
 END
 
-    lvextend1('disk extend initial',       30, 0);
+    lvextend1('disk extend initial',       300);
 }
 
 sub lvextend_stage2 () {
-    lvextend1('disk extend background', undef, 1);
+    lvextend1('disk extend subsequent',  undef);
 }
 
 sub vginfo () {
@@ -80,7 +80,7 @@ sub vginfo () {
 }
 
 sub lvextend1 ($$$) {
-    my ($what, $max_more_gb, $background)  = @_;
+    my ($what, $max_more_gb)  = @_;
 
     my $lv= "/dev/$vg/root";
 
@@ -156,11 +156,7 @@ sub lvextend1 ($$$) {
     my $timeout = 2000 + int($pe_size * 0.000003 * $more_pe);
     logm("$what: ${pe_size}k x $more_pe (timeout=$timeout)");
     my $cmd = "resize2fs $lv";
-    if (!$background) {
-	target_cmd_root($ho, $cmd, $timeout);
-    } else {
-	target_cmd_root($ho, "$cmd >>resizefs.log 2>&1 </dev/null &", 30);
-    }
+    target_cmd_root($ho, $cmd, $timeout);
 }
 
 sub prep () {
-- 
1.7.10.4

[OSSTEST PATCH 3/6] TestSupport: break out lv_dev_mapper

[Ian Jackson]

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>
---
 Osstest/TestSupport.pm |   17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm
index 3ff70a5..bfccaf1 100644
--- a/Osstest/TestSupport.pm
+++ b/Osstest/TestSupport.pm
@@ -61,7 +61,8 @@ BEGIN {
                       target_run_apt
                       target_install_packages target_install_packages_norec
                       target_jobdir target_extract_jobdistpath_subdir
-                      target_extract_jobdistpath target_guest_lv_name
+                      target_extract_jobdistpath
+                      lv_dev_mapper target_guest_lv_name
 
                       poll_loop tcpconnect await_tcp
                       contents_make_cpio file_simple_write_contents
@@ -694,13 +695,17 @@ sub poll_loop ($$$&) {
     logm("$what: ok. (${waited}s)");
 }
 
-sub target_guest_lv_name($$) {
-    my ($ho, $lv) = @_;
-
-    my $vg = "$ho->{Name}";
-    # Dashes are escaped in the VG name
+sub lv_dev_mapper ($$) {
+    my ($vg,$lv) = @_;
     $vg =~ s/-/--/g;
+    $lv =~ s/-/--/g;
+    # Dashes are doubled in the VG and LV names
     return "/dev/mapper/$vg-$lv";
+}    
+
+sub target_guest_lv_name($$) {
+    my ($ho, $lv) = @_;
+    return lv_dev_mapper("$ho->{Name}", $lv);
 }
 
 #---------- dhcp watching ----------
-- 
1.7.10.4

[OSSTEST PATCH 4/6] ts-xen-build-prep: Break out determine_vg_lv

[Ian Jackson]

And make $lv a global.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-xen-build-prep |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index a9c9e4a..856d97b 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -28,8 +28,9 @@ $whhost ||= 'host';
 our $ho= selecthost($whhost);
 exit 0 if $ho->{SharedReady};
 
-our ($vg);
+our ($vg,$lv);
 
+our $lvleaf = 'root';
 our $pe_size;
 our $extended_pes = 0;
 
@@ -50,12 +51,15 @@ sub overall_limit_pe ($) {
     $extended_pes += $$pevar;
 }
 
-sub lvextend_stage1 () {
+sub determine_vg_lv () {
     $vg=
         $ho->{Suite} =~ m/lenny/
         ? "$ho->{Name}.$c{TestHostDomain}"
         : $ho->{Name};
+    $lv = "/dev/$vg/$lvleaf";
+}
 
+sub lvextend_stage1 () {
     target_cmd_root($ho, <<END);
         set -ex; if ! test -f /root/swap_osstest_enabled; then
             lvcreate -L 10G -n swap_osstest_build $vg ||:
@@ -82,8 +86,6 @@ sub vginfo () {
 sub lvextend1 ($$$) {
     my ($what, $max_more_gb)  = @_;
 
-    my $lv= "/dev/$vg/root";
-
     my @vginfo = vginfo();
     $pe_size= $vginfo[12];
 
@@ -199,6 +201,7 @@ sub ccache_setup () {
 }
 
 if (!$ho->{Flags}{'no-reinstall'}) {
+    determine_vg_lv();
     lvextend_stage1();
     prep();
     ccache_setup();
-- 
1.7.10.4

[OSSTEST PATCH 5/6] ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs

[Ian Jackson]

Online resize is 40x slower than mkfs.  It appears that the
backgrounded resize2fs can starve build tasks of IO bandwidth.

So instead, use mkfs to make a new filesystem for /home/osstest.
We use rsync to copy in the old contents.

For convenience of (a) review (b) possible reversion, we keep (for
now) the lvextend machinery.  So we create a new 1-extent LV for the
lvextend machinery to work on.

But we don't call resize2fs when we extend it, because now it doesn't
have a fs on it yet.  We make the filesystem later.

We move the ccache_setup until after this is done because it's a bit
pointless to put things in the to-be-removed /home/osstest when they
could be put in the new one after it had been set up.

We take slight care to make the rune slightly idempotent: if it
completed successfully we detect this and do not run it again.  But if
it didn't, things may be messed up and running it again is unlikely to
help and may make things worse.

I have tested this on rice-weevil and the whole new target command
(including rsync, mkfs, mount etc.) takes 126s.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>
---
 ts-xen-build-prep |   46 ++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 42 insertions(+), 4 deletions(-)

diff --git a/ts-xen-build-prep b/ts-xen-build-prep
index 856d97b..4dece65 100755
--- a/ts-xen-build-prep
+++ b/ts-xen-build-prep
@@ -30,7 +30,7 @@ exit 0 if $ho->{SharedReady};
 
 our ($vg,$lv);
 
-our $lvleaf = 'root';
+our $lvleaf = 'osstest_home';
 our $pe_size;
 our $extended_pes = 0;
 
@@ -83,6 +83,11 @@ sub vginfo () {
     return @vginfo;
 }
 
+sub lvcreate () {
+    target_cmd_output_root($ho,
+			   "lvdisplay $lv || lvcreate -l 1 -n $lvleaf $vg");
+}
+
 sub lvextend1 ($$$) {
     my ($what, $max_more_gb)  = @_;
 
@@ -157,8 +162,39 @@ sub lvextend1 ($$$) {
 
     my $timeout = 2000 + int($pe_size * 0.000003 * $more_pe);
     logm("$what: ${pe_size}k x $more_pe (timeout=$timeout)");
-    my $cmd = "resize2fs $lv";
-    target_cmd_root($ho, $cmd, $timeout);
+}
+
+sub replace_home () {
+    my $dir = '/home/osstest';
+    my $mapper = lv_dev_mapper($vg,$lvleaf);
+    my ($fstype,@opts) = qw(ext3 -m 0 -O sparse_super);
+    target_cmd_root($ho, <<END, 1000);
+        set -ex
+	if mount | sed -e 's/^[^ ].* on //; s/ .*//' | grep -F '$dir'; then
+	    exit 0
+	fi
+	mkfs -t $fstype @opts $lv
+        mount $lv /mnt
+	rsync -aHx --numeric-ids $dir/. /mnt/.
+	rm -rf $dir
+	mkdir -m 2700 $dir
+	echo '$mapper $dir $fstype defaults 0 0' >>/etc/fstab
+	umount /mnt
+	mount $dir
+END
+
+        # for convenience, here is a small scriptlet to undo this:
+        <<'END';
+#!/bin/sh
+set -ex
+cd /home
+rm -rf osstest.new
+rsync -aH --numeric-ids osstest/. osstest.new
+umount /home/osstest
+rmdir osstest
+mv osstest.new osstest
+lvremove -f /dev/`uname -n`/osstest_home
+END
 }
 
 sub prep () {
@@ -202,10 +238,12 @@ sub ccache_setup () {
 
 if (!$ho->{Flags}{'no-reinstall'}) {
     determine_vg_lv();
+    lvcreate();
     lvextend_stage1();
     prep();
-    ccache_setup();
     lvextend_stage2();
+    replace_home();
+    ccache_setup();
 }
 $mjobdb->jobdb_resource_shared_mark_ready
    ($ho->{Ident}, $ho->{Name}, "build-".$ho->{Suite}."-".$r{arch});
-- 
1.7.10.4

[OSSTEST PATCH 6/6] Revert "ts-logs-capture: collect resize2fs.log"

[Ian Jackson]

This is no longer generated.

This reverts commit 9f299b936d1812bfd958d2771676a8de3cf9c060.
---
 ts-logs-capture |    2 --
 1 file changed, 2 deletions(-)

diff --git a/ts-logs-capture b/ts-logs-capture
index 4ad55b9..f6eca78 100755
--- a/ts-logs-capture
+++ b/ts-logs-capture
@@ -128,8 +128,6 @@ sub fetch_logs_host_guests () {
 
                   /var/log/xen-tools/*
 
-                  /root/resizefs.log
-
                   /boot/config*
 
                   /etc/xen/*
-- 
1.7.10.4

Re: [OSSTEST PATCH 5/6] ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs

[Ian Campbell]

On Wed, 2015-05-13 at 12:15 +0100, Ian Jackson wrote:
> +sub replace_home () {
> +    my $dir = '/home/osstest';
> +    my $mapper = lv_dev_mapper($vg,$lvleaf);
> +    my ($fstype,@opts) = qw(ext3 -m 0 -O sparse_super);
> +    target_cmd_root($ho, <<END, 1000);
> +        set -ex
> +	if mount | sed -e 's/^[^ ].* on //; s/ .*//' | grep -F '$dir'; then
> +	    exit 0
> +	fi
> +	mkfs -t $fstype @opts $lv
> +        mount $lv /mnt
> +	rsync -aHx --numeric-ids $dir/. /mnt/.
> +	rm -rf $dir
> +	mkdir -m 2700 $dir
> +	echo '$mapper $dir $fstype defaults 0 0' >>/etc/fstab
> +	umount /mnt
> +	mount $dir

This doesn't update /etc/fstab so in standalone mode (when a host may
share build and test duties) the reboots will cause the
old /home/osstest to reappear and then I'm not sure if this all does
what we want or not with the existing lv.

Re: [OSSTEST PATCH 5/6] ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs

[Ian Jackson]

Ian Campbell writes ("Re: [OSSTEST PATCH 5/6] ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs"):
> On Wed, 2015-05-13 at 12:15 +0100, Ian Jackson wrote:
> > +	echo '$mapper $dir $fstype defaults 0 0' >>/etc/fstab
> 
> This doesn't update /etc/fstab so in standalone mode (when a host may
> share build and test duties) the reboots will cause the
> old /home/osstest to reappear and then I'm not sure if this all does
> what we want or not with the existing lv.

That line there is the update to /etc/fstab.

The result is that if the host reboots the new fs should be mounted,
and then when this runs again this

> > +	if mount | sed -e 's/^[^ ].* on //; s/ .*//' | grep -F '$dir'; then
> > +	    exit 0
> > +	fi

is supposed to arrange that the whole thing is a no-op.

Also note that the script makes the mount point with mode 2700 so that
normal build scripts will crash if the fs is not mounted for some
reason:

> +	mkdir -m 2700 $dir

Ian.

Re: [OSSTEST PATCH 5/6] ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs

[Ian Campbell]

On Thu, 2015-05-14 at 18:48 +0100, Ian Jackson wrote:
> Ian Campbell writes ("Re: [OSSTEST PATCH 5/6] ts-xen-build-prep: mkfs a new /home/osstest, don't resize2fs"):
> > On Wed, 2015-05-13 at 12:15 +0100, Ian Jackson wrote:
> > > +	echo '$mapper $dir $fstype defaults 0 0' >>/etc/fstab
> > 
> > This doesn't update /etc/fstab so in standalone mode (when a host may
> > share build and test duties) the reboots will cause the
> > old /home/osstest to reappear and then I'm not sure if this all does
> > what we want or not with the existing lv.
> 
> That line there is the update to /etc/fstab.

Sorry, I triple checked for it and I've no idea how I missed it every
time.

> The result is that if the host reboots the new fs should be mounted,
> and then when this runs again this
> 
> > > +	if mount | sed -e 's/^[^ ].* on //; s/ .*//' | grep -F '$dir'; then
> > > +	    exit 0
> > > +	fi
> 
> is supposed to arrange that the whole thing is a no-op.
> 
> Also note that the script makes the mount point with mode 2700 so that
> normal build scripts will crash if the fs is not mounted for some
> reason:
> 
> > +	mkdir -m 2700 $dir

Right, all sounds exactly how I thought it should work:

Acked-by: Ian Campbell <ian.campbell@citrix.com>

There is an osstest in progress, the first bite of the cherry failed
with one of the intermittent failures. So IMHO you may as well append to
pretest when ever you are ready, if the current attempt happens to fail
again then there's no harm adding this lot into the mix for next time.

Ian.