[OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Campbell]

Currently rerunning mg-debian-install-update when the external files
have changed still produces differences in the local files produced
during post-processing.

Avoid these differences by:

  - Using gzip -n, which avoids storing a timestamp in the gzip
    header (as well as the name, which we don't need).
  - Using pax -M norm, which normalises all timestamps (among other
    things, such as the owner, which we don't care about)
  - Using tar --mtime, with a reference within the dpkg-deb created
    hierarchy (which has timestamps from the package and is therefore
    dependent only on the downloaded package revision)

With this the results of two invocations of
mg-debian-installer-update(-all) are identical (assuming no changes to
the downloaded files) as demonstrated by runnign this quick hack:

	#!/bin/bash

	set -ex

	TMP=$HOME/tmp/mg-di
	rm -rf $TMP
	mkdir -p $TMP

	cat >$TMP/config <<EOF
	TftpPath $TMP
	TftpDiBase debian-installer
	EOF
	cat $TMP/config

	mkdir -p $TMP/debian-installer

	export OSSTEST_CONFIG=production-config:$TMP/config

	# ./mg-debian-installer-update wheezy armhf firmware-bnx2
	./mg-debian-installer-update-all

	find $TMP/debian-installer -type f -print0 | xargs -0x md5sum > $TMP/SUMS.BASE

	#rm -rf $TMP/debian-installer/*
	mv $TMP/debian-installer $TMP/debian-installer.org
	mkdir -p $TMP/debian-installer

	# ./mg-debian-installer-update wheezy armhf firmware-bnx2
	./mg-debian-installer-update-all
	md5sum --quiet -c $TMP/SUMS.BASE

	exit 0

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debian-installer-update | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index 3ae50fb..5d12cf0 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -78,8 +78,8 @@ for p in $packages; do
         curl -s "$site/$pkgfile" >$p.deb
         rm -rf x
         dpkg-deb -x $p.deb x
-        cd x; pax -x sv4cpio -s '%lib%/lib%' -w lib >../cpio; cd ..
-        gzip -9f cpio
+        cd x; pax -x sv4cpio -s '%lib%/lib%' -w -M norm lib >../cpio; cd ..
+        gzip -9nf cpio
         mv cpio.gz $p.cpio.gz
         rm -rf x
 done
@@ -135,13 +135,14 @@ if [ $arch = armhf ]; then
                          -path \*/kernel/drivers/scsi/\* -o \
                          -path \*/kernel/drivers/usb/dwc3/\* -o \
                          -path \*/kernel/drivers/usb/host/\* \) \
-          |pax -x sv4cpio -s '%lib%/lib%' -d -w >../cpio; cd ..
-    gzip -9f cpio
+          |pax -x sv4cpio -s '%lib%/lib%' -d -w -M norm >../cpio; cd ..
+    gzip -9nf cpio
     mv cpio.gz armmp.cpio.gz
     rm -rf dtbs/
     mkdir dtbs/
     mv x/usr/lib/linux-image-*-armmp/*.dtb dtbs/
-    tar -caf dtbs.tar.gz dtbs
+    tar --mtime=./x/usr/lib -cf dtbs.tar dtbs
+    gzip -9nf dtbs.tar
     rm -rf x
 fi
 
-- 
2.1.1

Re: [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Campbell]

Last week we agreed that I should push a TftpDiVersion update
corresponding to the Debian 7.8 point release which is happening today.
I ran that with this patch so I have pushed both this patch and the one
below (mg-debian-installer-update has no automatic uses, it's all manual
intervention). I hope that is OK.

As noted on IRC there is some sort of proxy between the osstest control
VM and the world which is returning stale data, so I have run
mg-debian-installer-update-all on my workstation, and the files do
appear to have been updated as I would expect.

Sine that's so odd I've just done a normal push to pretest for now
instead of the force push which we discussed. I'll keep an eye on the
test results and if/when we start seeing failures due to the d-i/kernel
mismatch I'll do a force push then (since it is certain we have nothing
to lose at the point).

Ian.

commit 43716b98a912a4c1d445cb5fcca00231f7dbf542
Author: Ian Campbell <ian.campbell@citrix.com>
Date:   Sat Jan 10 16:01:00 2015 +0000

    TftpDiVersion 2015-01-14
    
    Picks up updated kernel + installer from Debian 7.8 point release.
    
    Signed-off-by: Ian Campbell <ian.campbell@citrix.com>

diff --git a/production-config b/production-config
index 3aee7db..515bd98 100644
--- a/production-config
+++ b/production-config
@@ -68,7 +68,7 @@ TftpPxeDir /
 TftpPxeTemplates %ipaddrhex%/pxelinux.cfg
 
 TftpPxeGroup osstest
-TftpDiVersion 2014-12-14
+TftpDiVersion 2015-01-10
 
 XenUsePath /usr/groups/xencore/systems/bin/xenuse
 XenUseUser osstest

Re: [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Campbell]

On Sat, 2015-01-10 at 16:10 +0000, Ian Campbell wrote:
> As noted on IRC there is some sort of proxy between the osstest control
> VM and the world which is returning stale data, so I have run
> mg-debian-installer-update-all on my workstation, and the files do
> appear to have been updated as I would expect.

We might like to consider something along these lines for the future:

------8<---------

>From ff6747252def2c69ee2bac684861f4c16bc9b8a5 Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@citrix.com>
Date: Sat, 10 Jan 2015 16:14:15 +0000
Subject: [PATCH] mg-debian-installer-update: workaround caching proxies

Which otherwise interfere with updating d-i shortly after the upstream
update.

Signed-off-by: Ian Campbell <ian.campbell@citrix.com>
---
 mg-debian-installer-update | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/mg-debian-installer-update b/mg-debian-installer-update
index 5d12cf0..9fb2968 100755
--- a/mg-debian-installer-update
+++ b/mg-debian-installer-update
@@ -32,6 +32,11 @@ fail () { echo >&2 "$0: $1"; exit 1; }
 site=http://ftp.debian.org/debian/
 sbase=$site/dists/$suite
 
+fetch () {
+	# no-cache due to transparent proxies messing things up.
+	curl -s -H 'Pragma: no-cache' $1
+}
+
 case ${suite}_${arch} in
     *_i386|*_amd64)
         src=$sbase/main/installer-$arch/current/images/netboot/
@@ -63,10 +68,10 @@ for f in $files; do
         echo >&2 "fetching $f"
         s=${f/:*} ; d=${f/*:}
         echo >&2 "$src/$s => $d.new"
-        curl -s $src/$s >$d.new
+        fetch $src/$s >$d.new
 done
 
-curl -s $pfile >Packages.gz
+fetch $pfile >Packages.gz
 
 for p in $packages; do
         set +e
@@ -75,7 +80,7 @@ for p in $packages; do
         rc=$?
         set -e
         if [ $rc != 0 ]; then fail "package $p not found"; fi
-        curl -s "$site/$pkgfile" >$p.deb
+        fetch "$site/$pkgfile" >$p.deb
         rm -rf x
         dpkg-deb -x $p.deb x
         cd x; pax -x sv4cpio -s '%lib%/lib%' -w -M norm lib >../cpio; cd ..
@@ -91,7 +96,7 @@ if [ $arch = armhf ]; then
     bp="$sbase-backports"
     pfile=$bp/main/binary-armhf/Packages.gz
 
-    curl -s $pfile >Packages.gz
+    fetch $pfile >Packages.gz
 
     # Newer kernel often needs a newer initramfs-tools. Make that available
     echo >&2 "collecting backports initramfs-tools"
@@ -99,7 +104,7 @@ if [ $arch = armhf ]; then
     rc=$?
     set -e
     if [ $rc != 0 ]; then fail "initramfs-tools package not found"; fi
-    curl -s "$site/$pkgfile" >initramfs-tools.deb
+    fetch "$site/$pkgfile" >initramfs-tools.deb
 
     set +e
     echo >&2 "collecting armmp kernel"
@@ -109,7 +114,7 @@ if [ $arch = armhf ]; then
     rc=$?
     set -e
     if [ $rc != 0 ]; then fail "armmp kernel package not found"; fi
-    curl -s "$site/$pkgfile" >armmp.deb
+    fetch "$site/$pkgfile" >armmp.deb
     dpkg-deb -x armmp.deb x
     cp x/boot/vmlinuz-* linux.armmp
     # The full set of modules is pretty large and not that
-- 
2.1.1

Re: [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Jackson]

Ian Campbell writes ("[OSSTEST PATCH] mg-debian-installer-update: produce deterministic output"):
> Currently rerunning mg-debian-install-update when the external files
> have changed still produces differences in the local files produced
      ^
Missing "not" ?

> during post-processing.
> 
> Avoid these differences by:
> 
>   - Using gzip -n, which avoids storing a timestamp in the gzip
>     header (as well as the name, which we don't need).
>   - Using pax -M norm, which normalises all timestamps (among other
>     things, such as the owner, which we don't care about)
>   - Using tar --mtime, with a reference within the dpkg-deb created
>     hierarchy (which has timestamps from the package and is therefore
>     dependent only on the downloaded package revision)
> 
> With this the results of two invocations of
> mg-debian-installer-update(-all) are identical (assuming no changes to
> the downloaded files) as demonstrated by runnign this quick hack:

Excellent.  Apart from the missing word in the description,

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Thanks,
Ian.

Re: [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output"):
> We might like to consider something along these lines for the future:

A reasonable idea (although I wonder if it should be configurable).

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

Ian.

Re: [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Campbell]

On Mon, 2015-01-12 at 12:27 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output"):
> > We might like to consider something along these lines for the future:
> 
> A reasonable idea (although I wonder if it should be configurable).

I did consider just inserting a $CURL_OPTS which could be set to include
pragma no-cache from the command line. I mostly didn't because I was too
lazy to think about the correct shell quoting at the time...

> 
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
> 
> Ian.

Re: [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Jackson]

Ian Campbell writes ("Re: [Xen-devel] [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output"):
> On Mon, 2015-01-12 at 12:27 +0000, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [Xen-devel] [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output"):
> > > We might like to consider something along these lines for the future:
> > 
> > A reasonable idea (although I wonder if it should be configurable).
> 
> I did consider just inserting a $CURL_OPTS which could be set to include
> pragma no-cache from the command line. I mostly didn't because I was too
> lazy to think about the correct shell quoting at the time...

I have pushed this.  I edited a comment which talked about
"transparent proxies" (why is a lie) to refer to "intercepting
proxies" (which is not).

A "transparent proxy" is (as defined in the RFCs) one which refrains
from making certain changes to requests explicitly proxied through it.

A proxy which operates without awareness on the client side by
intercepting TCP streams at routers is an "intercepting proxy".

Ian.

Re: [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output

[Ian Campbell]

On Wed, 2015-02-04 at 15:40 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output"):
> > On Mon, 2015-01-12 at 12:27 +0000, Ian Jackson wrote:
> > > Ian Campbell writes ("Re: [Xen-devel] [OSSTEST PATCH] mg-debian-installer-update: produce deterministic output"):
> > > > We might like to consider something along these lines for the future:
> > > 
> > > A reasonable idea (although I wonder if it should be configurable).
> > 
> > I did consider just inserting a $CURL_OPTS which could be set to include
> > pragma no-cache from the command line. I mostly didn't because I was too
> > lazy to think about the correct shell quoting at the time...
> 
> I have pushed this.  I edited a comment which talked about
> "transparent proxies" (why is a lie) to refer to "intercepting
> proxies" (which is not).
> 
> A "transparent proxy" is (as defined in the RFCs) one which refrains
> from making certain changes to requests explicitly proxied through it.
> 
> A proxy which operates without awareness on the client side by
> intercepting TCP streams at routers is an "intercepting proxy".

Ack.