All of lore.kernel.org
 help / color / mirror / Atom feed
From: Armin Kuster <akuster808@gmail.com>
To: openembedded-core@openembedded.org
Subject: [zeus 08/29] opkg-utils: Fix reproducibility issues in opkg-build
Date: Sun,  9 Feb 2020 08:09:36 -0800	[thread overview]
Message-ID: <7a222381df431a46dd0898c2b90a1d2dda2e0a43.1581264380.git.akuster808@gmail.com> (raw)
In-Reply-To: <cover.1581264380.git.akuster808@gmail.com>

From: Richard Purdie <richard.purdie@linuxfoundation.org>

There is a sorting problem with opkg-build where the ipk generated is depending
upon the order of files on disk. The reason is the --sort option to tar only
influences the orders of files tar reads, not those passed by the -T option.

Add in a sort call to resolve this issue. To ensure consistent sorting we
also need to force to a specific locale (C) else the results are still not
deterministic.

(From OE-Core rev: a9b8287984c63420e10329a69f7ac5125f1687f8)

(From OE-Core rev: b577a6d923042cfc04e67d470e0987488ea61412)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Armin Kuster <akuster808@gmail.com>
---
 .../opkg-utils/fix-reproducibility.patch      | 32 +++++++++++++++++++
 .../opkg-utils/opkg-utils_0.4.2.bb            |  1 +
 2 files changed, 33 insertions(+)
 create mode 100644 meta/recipes-devtools/opkg-utils/opkg-utils/fix-reproducibility.patch

diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils/fix-reproducibility.patch b/meta/recipes-devtools/opkg-utils/opkg-utils/fix-reproducibility.patch
new file mode 100644
index 0000000000..945979bc8a
--- /dev/null
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils/fix-reproducibility.patch
@@ -0,0 +1,32 @@
+Fix reproducibility issues in opkg-build
+
+There is a sorting problem with opkg-build where the ipk generated is depending
+upon the order of files on disk. The reason is the --sort option to tar only
+influences the orders of files tar reads, not those passed by the -T option.
+
+Add in a sort call to resolve this issue. To ensure consistent sorting we
+also need to force to a specific locale (C) else the results are still not
+deterministic.
+
+RP 2020/2/5
+
+Upstream-Status: Submitted [https://groups.google.com/forum/#!topic/opkg-devel/YttZ73NLrYQ]
+Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
+
+Index: opkg-utils-0.4.2/opkg-build
+===================================================================
+--- opkg-utils-0.4.2.orig/opkg-build
++++ opkg-utils-0.4.2/opkg-build
+@@ -305,8 +305,10 @@ if [ ! -z "$SOURCE_DATE_EPOCH"  ]; then
+     mtime_args="--mtime=@$build_date --clamp-mtime"
+ fi
+ 
+-( cd $pkg_dir/$CONTROL && find . -type f > $tmp_dir/control_list )
+-( cd $pkg_dir && find . -path ./$CONTROL -prune -o -path . -o -print  > $tmp_dir/file_list )
++export LANG=C
++export LC_ALL=C
++( cd $pkg_dir/$CONTROL && find . -type f | sort > $tmp_dir/control_list )
++( cd $pkg_dir && find . -path ./$CONTROL -prune -o -path . -o -print  | sort > $tmp_dir/file_list )
+ ( cd $pkg_dir && tar $ogargs $tsortargs --no-recursion $mtime_args -c $tarformat -T $tmp_dir/file_list | $compressor $compressorargs > $tmp_dir/data.tar.$cext )
+ ( cd $pkg_dir/$CONTROL && tar $ogargs $tsortargs --no-recursion --mtime=@$build_date -c $tarformat -T $tmp_dir/control_list | gzip $zipargs > $tmp_dir/control.tar.gz )
+ rm $tmp_dir/file_list
diff --git a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.2.bb b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.2.bb
index 6495726500..042eec7e0e 100644
--- a/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.2.bb
+++ b/meta/recipes-devtools/opkg-utils/opkg-utils_0.4.2.bb
@@ -8,6 +8,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=94d55d512a9ba36caa9b7df079bae19f \
 PROVIDES += "${@bb.utils.contains('PACKAGECONFIG', 'update-alternatives', 'virtual/update-alternatives', '', d)}"
 
 SRC_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/${BPN}/snapshot/${BPN}-${PV}.tar.gz \ 
+           file://fix-reproducibility.patch \
 "
 UPSTREAM_CHECK_URI = "http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/refs/"
 
-- 
2.17.1



  parent reply	other threads:[~2020-02-09 16:10 UTC|newest]

Thread overview: 34+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-02-09 16:09 [zeus 00/29] Patch review Armin Kuster
2020-02-09 16:09 ` [zeus 01/29] Revert "bzip2: Fix CVE-2019-12900" Armin Kuster
2020-02-09 16:09 ` [zeus 02/29] rsync: whitelist CVE-2017-16548 Armin Kuster
2020-02-09 16:09 ` [zeus 03/29] curl: fix CVE-2019-15601 Armin Kuster
2020-02-09 16:09 ` [zeus 04/29] cpio: fix CVE-2019-14866 Armin Kuster
2020-02-09 16:09 ` [zeus 05/29] devtool/standard.py: Allow recipe to disable menuconfig logic Armin Kuster
2020-02-09 16:09 ` [zeus 06/29] gcc-9.2: fix bug #91102 'aarch64 ICE on Linux kernel with -Os' Armin Kuster
2020-02-09 16:09 ` [zeus 07/29] opkg-utils: upgrade to version 0.4.2 Armin Kuster
2020-02-09 16:09 ` Armin Kuster [this message]
2020-02-09 16:09 ` [zeus 09/29] oeqa/reproducible: Improve test output and ensure deb+ipk compared Armin Kuster
2020-02-09 16:09 ` [zeus 10/29] classes/reproducible_build: Read SDE file later Armin Kuster
2020-02-09 16:09 ` [zeus 11/29] sudo: Set vardir deterministically Armin Kuster
2020-02-09 16:09 ` [zeus 12/29] libxshmfence: Set shm directory deterministically Armin Kuster
2020-02-09 16:09 ` [zeus 13/29] mc: Fix build reproducibility Armin Kuster
2020-02-09 16:09 ` [zeus 14/29] mc: Set zipinfo presence determinstically Armin Kuster
2020-02-09 16:09 ` [zeus 15/29] mc: Fix manpage date indeterminism Armin Kuster
2020-02-09 16:09 ` [zeus 16/29] tar: Fix build determinism, disable rsh Armin Kuster
2020-02-09 16:09 ` [zeus 17/29] patch: Extend to native/nativesdk and depend upon Armin Kuster
2020-02-09 16:09 ` [zeus 18/29] libidn2: Fix reproducibility issue Armin Kuster
2020-02-09 16:09 ` [zeus 19/29] gtk+3: sort resources for reproducible binaries Armin Kuster
2020-02-09 16:09 ` [zeus 20/29] perl: do not install files that contain build host specific data Armin Kuster
2020-02-09 16:09 ` [zeus 21/29] perl: Fix various reproducibile build issues Armin Kuster
2020-02-09 16:09 ` [zeus 22/29] openssl: Fix reproducibility issue Armin Kuster
2020-02-09 16:09 ` [zeus 23/29] iputils: Fix build determinism Armin Kuster
2020-02-09 16:09 ` [zeus 24/29] libinput: Fix determinism issue Armin Kuster
2020-02-09 16:09 ` [zeus 25/29] libgcrypt: " Armin Kuster
2020-02-09 16:09 ` [zeus 26/29] sudo: specify where target tools are Armin Kuster
2020-02-09 16:09 ` [zeus 27/29] sysvinit: Fix Reproducibility issue Armin Kuster
2020-02-09 16:09 ` [zeus 28/29] libevdev: Fix determinism issue Armin Kuster
2020-02-09 16:09 ` [zeus 29/29] ncurses: Fix reproducibility issue Armin Kuster
2020-02-10  8:07 ` [zeus 00/29] Patch review Schrempf Frieder
2020-02-10  9:55   ` Peter Kjellerstedt
2020-02-10 11:21     ` Richard Purdie
2020-02-10 16:01     ` akuster808

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7a222381df431a46dd0898c2b90a1d2dda2e0a43.1581264380.git.akuster808@gmail.com \
    --to=akuster808@gmail.com \
    --cc=openembedded-core@openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.