* [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar"
@ 2017-08-13 17:32 Theodore Ts'o
2017-08-13 17:32 ` [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible Theodore Ts'o
2017-08-16 3:40 ` [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar" Eric Biggers
0 siblings, 2 replies; 5+ messages in thread
From: Theodore Ts'o @ 2017-08-13 17:32 UTC (permalink / raw)
To: fstests; +Cc: Theodore Ts'o
The gen-image script must not be run as root when generating a tar
file. This is because it uses fakechroot and fakeroot, and if run as
root, the debootstrap script will end up damaging a large number of
devices in the /dev directory.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
kvm-xfstests/test-appliance/gen-image | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
index dbd9b84..e7bdb35 100755
--- a/kvm-xfstests/test-appliance/gen-image
+++ b/kvm-xfstests/test-appliance/gen-image
@@ -76,6 +76,10 @@ fi
if test -n "$OUT_TAR" ; then
if test "$FAKECHROOT" != "true" ; then
+ if test "$EUID" -eq 0 ; then
+ echo "This script MUST NOT be run as root with --out_tar"
+ exit 1;
+ fi
set -- "${SAVE_ARGS[@]}"
exec fakechroot $0 "$@"
fi
--
2.11.0.rc0.7.gbe5a750
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible
2017-08-13 17:32 [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar" Theodore Ts'o
@ 2017-08-13 17:32 ` Theodore Ts'o
2017-08-16 3:45 ` Eric Biggers
2017-08-16 3:40 ` [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar" Eric Biggers
1 sibling, 1 reply; 5+ messages in thread
From: Theodore Ts'o @ 2017-08-13 17:32 UTC (permalink / raw)
To: fstests; +Cc: Theodore Ts'o
Use techniques from https://reproducible-builds.org so the builds will
be reproducible, assuming a fixed build environment.
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
build-all | 6 ++++++
gen-tarball | 10 ++++++++--
kvm-xfstests/test-appliance/gen-image | 12 +++++++++---
3 files changed, 23 insertions(+), 5 deletions(-)
diff --git a/build-all b/build-all
index 6ff5a48..570006e 100755
--- a/build-all
+++ b/build-all
@@ -148,6 +148,12 @@ else
LCF="-g -O2"
fi
+# To make the build more reproducible...
+export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
+export BUILD_PATH_PREFIX_MAP="xfstests-bld=$(pwd)"
+# GCC doesn't yet understand BUILD_PATH_PREFIX_MAP
+LCF="$LCF -fdebug-prefix-map=$(pwd)=xfstests-bld -gno-record-gcc-switches"
+
if test -z "$SKIP_E2FSLIBS" ; then
build_start "e2fslibs"
(cd e2fsprogs-libs; \
diff --git a/gen-tarball b/gen-tarball
index 5f7230d..5aa0142 100755
--- a/gen-tarball
+++ b/gen-tarball
@@ -32,6 +32,10 @@ done
which pigz &>/dev/null && GZIP=pigz || GZIP=gzip
+if test -z "$SOURCE_DATE_EPOCH" ; then
+ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
+fi
+
echo "GEN xfstests"
rm -rf xfstests
if test "$fast" = "yes"
@@ -47,6 +51,7 @@ cat *.ver > xfstests/git-versions
mkdir -p xfstests/bin xfstests/lib
cp bld/bin/* bld/sbin/* bld/usr/bin/* xfstests/bin
cp bld/lib/client.txt xfstests/lib
+rm xfstests/bin/compile_et xfstests/bin/mk_cmds
for i in ltp/fsstress ltp/fsx ltp/aio-stress src/godown
do
ln -s ../$i xfstests/bin
@@ -54,5 +59,6 @@ done
echo "STRIP xfstests/*"
find xfstests -mindepth 2 -type f -perm /0111 | xargs $STRIP 2> /dev/null
echo "TAR xfstests"
-tar c --owner=root --group=root --mode=go+u-w xfstests | \
- $GZIP -9 > xfstests.tar.gz
+find xfstests -print0 | LC_ALL=C sort -z | \
+ tar c --owner=0 --group=0 --numeric-owner --mtime="@${SOURCE_DATE_EPOCH}" \
+ --mode=go+u-w --null --no-recursion -T - | $GZIP -9n > xfstests.tar.gz
diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
index e7bdb35..2701353 100755
--- a/kvm-xfstests/test-appliance/gen-image
+++ b/kvm-xfstests/test-appliance/gen-image
@@ -66,6 +66,10 @@ while [ "$1" != "" ]; do
shift
done
+if test -z "$SOURCE_DATE_EPOCH" ; then
+ export SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct)
+fi
+
if test -n "$DO_LOG" ; then
if test -z "$GEN_IMAGE_LOG" ; then
export GEN_IMAGE_LOG="$DO_LOG"
@@ -312,11 +316,13 @@ if test -n "$OUT_TAR"; then
rmdir $ROOTDIR/debs
rmdir $ROOTDIR/imgdir
case "$OUT_TAR" in
- *.gz) ZFLAG="-z" ;;
- *) ZFLAG="" ;
+ *.gz) ZCMD="gzip -9n" ;;
+ *) ZCMD="cat" ;
esac
fix_symlinks
- tar -C $ROOTDIR -c $ZFLAG -f "$OUT_TAR" .
+ (cd "$ROOTDIR" ; find . -print0 | LC_ALL=C sort -z |
+ tar -c --null --no-recursion -T - --numeric-owner \
+ --mtime="@${SOURCE_DATE_EPOCH}" -f - | $ZCMD > "$OUT_TAR")
cleanup_rootdir
exit 0
fi
--
2.11.0.rc0.7.gbe5a750
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar"
2017-08-13 17:32 [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar" Theodore Ts'o
2017-08-13 17:32 ` [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible Theodore Ts'o
@ 2017-08-16 3:40 ` Eric Biggers
1 sibling, 0 replies; 5+ messages in thread
From: Eric Biggers @ 2017-08-16 3:40 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: fstests
On Sun, Aug 13, 2017 at 01:32:23PM -0400, Theodore Ts'o wrote:
> The gen-image script must not be run as root when generating a tar
> file. This is because it uses fakechroot and fakeroot, and if run as
> root, the debootstrap script will end up damaging a large number of
> devices in the /dev directory.
>
> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
> ---
> kvm-xfstests/test-appliance/gen-image | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/kvm-xfstests/test-appliance/gen-image b/kvm-xfstests/test-appliance/gen-image
> index dbd9b84..e7bdb35 100755
> --- a/kvm-xfstests/test-appliance/gen-image
> +++ b/kvm-xfstests/test-appliance/gen-image
> @@ -76,6 +76,10 @@ fi
>
> if test -n "$OUT_TAR" ; then
> if test "$FAKECHROOT" != "true" ; then
> + if test "$EUID" -eq 0 ; then
> + echo "This script MUST NOT be run as root with --out_tar"
> + exit 1;
> + fi
> set -- "${SAVE_ARGS[@]}"
> exec fakechroot $0 "$@"
> fi
> --
> 2.11.0.rc0.7.gbe5a750
Reviewed-by: Eric Biggers <ebiggers@google.com>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible
2017-08-13 17:32 ` [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible Theodore Ts'o
@ 2017-08-16 3:45 ` Eric Biggers
2017-08-16 15:02 ` Theodore Ts'o
0 siblings, 1 reply; 5+ messages in thread
From: Eric Biggers @ 2017-08-16 3:45 UTC (permalink / raw)
To: Theodore Ts'o; +Cc: fstests
Hi Ted,
On Sun, Aug 13, 2017 at 01:32:24PM -0400, Theodore Ts'o wrote:
> +find xfstests -print0 | LC_ALL=C sort -z | \
> + tar c --owner=0 --group=0 --numeric-owner --mtime="@${SOURCE_DATE_EPOCH}" \
> + --mode=go+u-w --null --no-recursion -T - | $GZIP -9n > xfstests.tar.gz
This step isn't reproducible for me because earlier in the script $GZIP can be
set to pigz, which differs from gzip. 'gzip -n' omits the filename and
timestamp from the gz file, but 'pigz -n' only omits the filename, and the -T
option is needed to also omit the timestamp. I think the following would work:
if type -P pigz >/dev/null; then
GZIP="pigz -n -T"
else
GZIP="gzip -n"
fi
Eric
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible
2017-08-16 3:45 ` Eric Biggers
@ 2017-08-16 15:02 ` Theodore Ts'o
0 siblings, 0 replies; 5+ messages in thread
From: Theodore Ts'o @ 2017-08-16 15:02 UTC (permalink / raw)
To: Eric Biggers; +Cc: fstests
On Tue, Aug 15, 2017 at 08:45:43PM -0700, Eric Biggers wrote:
> On Sun, Aug 13, 2017 at 01:32:24PM -0400, Theodore Ts'o wrote:
> > +find xfstests -print0 | LC_ALL=C sort -z | \
> > + tar c --owner=0 --group=0 --numeric-owner --mtime="@${SOURCE_DATE_EPOCH}" \
> > + --mode=go+u-w --null --no-recursion -T - | $GZIP -9n > xfstests.tar.gz
>
> This step isn't reproducible for me because earlier in the script $GZIP can be
> set to pigz, which differs from gzip. 'gzip -n' omits the filename and
> timestamp from the gz file, but 'pigz -n' only omits the filename, and the -T
> option is needed to also omit the timestamp.
Thanks for pointing this out. I'll fix this in my tree before I push
out the commit.
I'll note that the use of pigz vs gzip does make a difference in the
reproducibility of the tar.gz file:
11904 -rw-r--r-- 1 tytso tytso 12188800 Aug 16 10:55 xfstests-pigz.tar.gz
11896 -rw-r--r-- 1 tytso tytso 12180273 Aug 16 10:50 xfstests-gzip.tar.gz
So eventually, if we really want to allow someone else to be reproduce
a binary build, we will need to distribute something like a
Debian-style "buildinfo" file where the exact version of the compiler,
and other parts of the build environment (including gzip vs pigz), are
documented.
- Ted
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-08-16 15:02 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-13 17:32 [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar" Theodore Ts'o
2017-08-13 17:32 ` [xfstests-bld PATCH 2/2] Make the build and creation of xfstests.tar.gz to be reproducible Theodore Ts'o
2017-08-16 3:45 ` Eric Biggers
2017-08-16 15:02 ` Theodore Ts'o
2017-08-16 3:40 ` [xfstest-bld PATCH 1/2] test-appliance: add safety check for "gen-image --out-tar" Eric Biggers
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.