[xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar"

[xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar"

[Theodore Ts'o]

The gen-image script must not be run as root when generating a tar
file.  This is because it uses fakechroot and fakeroot, and if run as
root, the debootstrap script will end up damaging a large number of
devices in the /dev directory.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
 kvm-xfstests/test-appliance/gen-image | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
index dbd9b84..e7bdb35 100755
--- a/kvm-xfstests/test-appliance/gen-image
+++ b/kvm-xfstests/test-appliance/gen-image
@@ -76,6 +76,10 @@ fi
 
 if test -n "$OUT_TAR" ; then
     if test "$FAKECHROOT" != "true" ; then
+	if test "$EUID" -eq 0 ; then
+	    echo "This script MUST NOT be run as root with --out_tar"
+	    exit 1;
+	fi
 	set -- "${SAVE_ARGS[@]}"
 	exec fakechroot $0 "$@"
     fi
-- 
2.11.0.rc0.7.gbe5a750

[xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible

[Theodore Ts'o]

Use techniques from https://reproducible-builds.org so the builds will
be reproducible, assuming a fixed build environment.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
 build-all                             |  6 ++++++
 gen-tarball                           | 10 ++++++++--
 kvm-xfstests/test-appliance/gen-image | 12 +++++++++---
 3 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/build-all b/build-all
index 6ff5a48..570006e 100755
--- a/build-all
+++ b/build-all
@@ -148,6 +148,12 @@ else
     LCF="-g -O2"
 fi
 
+# To make the build more reproducible...
+export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
+export BUILD_PATH_PREFIX_MAP="xfstests-bld=$(pwd)"
+# GCC doesn't yet understand BUILD_PATH_PREFIX_MAP
+LCF="$LCF -fdebug-prefix-map=$(pwd)=xfstests-bld -gno-record-gcc-switches"
+
 if test -z "$SKIP_E2FSLIBS" ; then
     build_start "e2fslibs"
     (cd e2fsprogs-libs; \
diff --git a/gen-tarball b/gen-tarball
index 5f7230d..5aa0142 100755
--- a/gen-tarball
+++ b/gen-tarball
@@ -32,6 +32,10 @@ done
 
 which pigz &>/dev/null && GZIP=pigz || GZIP=gzip
 
+if test -z "$SOURCE_DATE_EPOCH" ; then
+    export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
+fi
+
 echo "GEN xfstests"
 rm -rf xfstests
 if test "$fast" = "yes"
@@ -47,6 +51,7 @@ cat *.ver > xfstests/git-versions
 mkdir -p xfstests/bin xfstests/lib
 cp bld/bin/* bld/sbin/* bld/usr/bin/* xfstests/bin
 cp bld/lib/client.txt xfstests/lib
+rm xfstests/bin/compile_et xfstests/bin/mk_cmds
 for i in ltp/fsstress ltp/fsx ltp/aio-stress src/godown
 do
     ln -s ../$i xfstests/bin
@@ -54,5 +59,6 @@ done
 echo "STRIP xfstests/*"
 find xfstests -mindepth 2 -type f -perm /0111 | xargs $STRIP 2> /dev/null
 echo "TAR xfstests"
-tar c --owner=root --group=root --mode=go+u-w xfstests | \
-	$GZIP -9 > xfstests.tar.gz
+find xfstests -print0 | LC_ALL=C sort -z | \
+    tar c --owner=0 --group=0 --numeric-owner --mtime="@${SOURCE_DATE_EPOCH}" \
+    --mode=go+u-w --null --no-recursion -T - | $GZIP -9n > xfstests.tar.gz
diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
index e7bdb35..2701353 100755
--- a/kvm-xfstests/test-appliance/gen-image
+++ b/kvm-xfstests/test-appliance/gen-image
@@ -66,6 +66,10 @@ while [ "$1" != "" ]; do
   shift
 done
 
+if test -z "$SOURCE_DATE_EPOCH" ; then
+    export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
+fi
+
 if test -n "$DO_LOG" ; then
     if test -z "$GEN_IMAGE_LOG" ; then
 	export GEN_IMAGE_LOG="$DO_LOG"
@@ -312,11 +316,13 @@ if test -n "$OUT_TAR"; then
     rmdir $ROOTDIR/debs
     rmdir $ROOTDIR/imgdir
     case "$OUT_TAR" in
-	*.gz)	ZFLAG="-z" ;;
-	*)	ZFLAG="" ;
+	*.gz)	ZCMD="gzip -9n" ;;
+	*)	ZCMD="cat" ;
     esac
     fix_symlinks
-    tar -C $ROOTDIR -c $ZFLAG -f "$OUT_TAR" .
+    (cd "$ROOTDIR" ; find . -print0 | LC_ALL=C sort -z |
+	 tar -c --null --no-recursion -T - --numeric-owner \
+	     --mtime="@${SOURCE_DATE_EPOCH}" -f - | $ZCMD > "$OUT_TAR")
     cleanup_rootdir
     exit 0
 fi
-- 
2.11.0.rc0.7.gbe5a750

Re: [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible

[Eric Biggers]

Hi Ted,

On Sun, Aug 13, 2017 at 01:32:24PM -0400, Theodore Ts'o wrote:
> +find xfstests -print0 | LC_ALL=C sort -z | \
> +    tar c --owner=0 --group=0 --numeric-owner --mtime="@${SOURCE_DATE_EPOCH}" \
> +    --mode=go+u-w --null --no-recursion -T - | $GZIP -9n > xfstests.tar.gz

This step isn't reproducible for me because earlier in the script $GZIP can be
set to pigz, which differs from gzip.  'gzip -n' omits the filename and
timestamp from the gz file, but 'pigz -n' only omits the filename, and the -T
option is needed to also omit the timestamp.  I think the following would work:

if type -P pigz >/dev/null; then
    GZIP="pigz -n -T"
else
    GZIP="gzip -n"
fi

Eric

Re: [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible

[Theodore Ts'o]

On Tue, Aug 15, 2017 at 08:45:43PM -0700, Eric Biggers wrote:
> On Sun, Aug 13, 2017 at 01:32:24PM -0400, Theodore Ts'o wrote:
> > +find xfstests -print0 | LC_ALL=C sort -z | \
> > +    tar c --owner=0 --group=0 --numeric-owner --mtime="@${SOURCE_DATE_EPOCH}" \
> > +    --mode=go+u-w --null --no-recursion -T - | $GZIP -9n > xfstests.tar.gz
> 
> This step isn't reproducible for me because earlier in the script $GZIP can be
> set to pigz, which differs from gzip.  'gzip -n' omits the filename and
> timestamp from the gz file, but 'pigz -n' only omits the filename, and the -T
> option is needed to also omit the timestamp.

Thanks for pointing this out.  I'll fix this in my tree before I push
out the commit.

I'll note that the use of pigz vs gzip does make a difference in the
reproducibility of the tar.gz file:

11904 -rw-r--r-- 1 tytso tytso 12188800 Aug 16 10:55 xfstests-pigz.tar.gz
11896 -rw-r--r-- 1 tytso tytso 12180273 Aug 16 10:50 xfstests-gzip.tar.gz

So eventually, if we really want to allow someone else to be reproduce
a binary build, we will need to distribute something like a
Debian-style "buildinfo" file where the exact version of the compiler,
and other parts of the build environment (including gzip vs pigz), are
documented.

					- Ted

Re: [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar"

[Eric Biggers]

On Sun, Aug 13, 2017 at 01:32:23PM -0400, Theodore Ts'o wrote:
> The gen-image script must not be run as root when generating a tar
> file.  This is because it uses fakechroot and fakeroot, and if run as
> root, the debootstrap script will end up damaging a large number of
> devices in the /dev directory.
> 
> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
> ---
>  kvm-xfstests/test-appliance/gen-image | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
> index dbd9b84..e7bdb35 100755
> --- a/kvm-xfstests/test-appliance/gen-image
> +++ b/kvm-xfstests/test-appliance/gen-image
> @@ -76,6 +76,10 @@ fi
>  
>  if test -n "$OUT_TAR" ; then
>      if test "$FAKECHROOT" != "true" ; then
> +	if test "$EUID" -eq 0 ; then
> +	    echo "This script MUST NOT be run as root with --out_tar"
> +	    exit 1;
> +	fi
>  	set -- "${SAVE_ARGS[@]}"
>  	exec fakechroot $0 "$@"
>      fi
> -- 
> 2.11.0.rc0.7.gbe5a750

Reviewed-by: Eric Biggers <ebiggers@google.com>