[PATCH 0/6] Reproducible: Find the git repo in WORKDIR/git or S first

[PATCH 0/6] Reproducible: Find the git repo in WORKDIR/git or S first

[Douglas Royds]

Change the search regime for find_git_folder():

1. WORKDIR/git: This is the default git fetcher unpack path
2. ${S}
3. Go looking for .git/ under the WORKDIR as a last resort.

linux-yocto:
We had an existing (silent) defect. The linux-yocto recipes all specify
two git SRC_URIs, one for the kernel source itself, the other for the
kmeta data (config fragments and friends). find_git_folder() was finding
the git checkout for the kmeta data, but due to a typo in the git log -1
--pretty=%ct line, we were (silently) reading the source_date_epoch from
the ${S} directory = STAGING_KERNEL_DIR, which is empty. If your
build/ happened to be inside a git checkout, git would walk up the
directory tree, and silently read the commit timestamp from this other
git checkout. The correct path to read the git commit timestamp from is
the "gitpath", being that found by find_git_folder(), though this
function was incorrectly finding the kmeta data checkout, not the kernel
source tree.

Non-kernel git recipes:
The default git fetcher clones and checks out the sources at
WORKDIR/git/ regardless of the setting of S (unless subpath or
destsuffix is set). find_git_folder() now looks for the
WORKDIR/git/.git/ directory first.

Non-yocto linux kernels:
Kernel recipes that don't inherit kernel-yocto should always set
S = ${WORKDIR}/git, so that when base_do_unpack_append() in
kernel.bbclass moves the checkout down to the STAGING_KERNEL_DIR and
symlinks it as WORKDIR/git, the build can still work by following the
symlink. We were previously failing to follow the symlink in the
os.walk(), but we now look first for WORKDIR/git/.git/, and find it due
to the symlink.

If none of the above mechanisms work for finding the git checkout,
perhaps there was a subpath or destsuffix specified in the SRC_URI.
We go looking for the git checkout under the WORKDIR as a last resort.

Douglas Royds (6):
  Reproducible: Tidy class description
  Reproducible: Refactor: get_source_date_epoch_from_youngest_file()
  Reproducible: Rename ambiguous "path" var to sourcedir or workdir as appropriate
  Reproducible: Find the git repo in WORKDIR/git or S first
  Reproducible: Don't enforce existence of ${S} dir
  Reproducible: Consistent debug logging

 meta/classes/reproducible_build.bbclass | 149 +++++++++++++-----------
 1 file changed, 81 insertions(+), 68 deletions(-)

-- 
2.17.1

[PATCH 1/6] Reproducible: Tidy class description

[Douglas Royds]

Tidy whitespace, delete commented-out lines

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 41 +++++++++++--------------
 1 file changed, 18 insertions(+), 23 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 268b5fb8f1..4e8850fb83 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -1,33 +1,32 @@
-#
 # reproducible_build.bbclass
 #
-# This bbclass is mainly responsible to determine SOURCE_DATE_EPOCH on a per recipe base.
-# We need to set a recipe specific SOURCE_DATE_EPOCH in each recipe environment for various tasks.
-# One way would be to modify all recipes one-by-one to specify SOURCE_DATE_EPOCH explicitly, 
-# but that is not realistic as there are hundreds (probably thousands) of recipes in various meta-layers.
-# Therefore we do it this class. 
-# After sources are unpacked but before they are patched, we try to determine the value for SOURCE_DATE_EPOCH.
+# Sets SOURCE_DATE_EPOCH in each component's build environment.
+# Upstream components (generally) respect this environment variable,
+# using it in place of the "current" date and time.
+# See https://reproducible-builds.org/specs/source-date-epoch/
+#
+# After sources are unpacked but before they are patched, we set a reproducible value for SOURCE_DATE_EPOCH.
+# This value should be reproducible for anyone who builds the same revision from the same sources.
 #
-# There are 4 ways to determine SOURCE_DATE_EPOCH:
+# There are 4 ways we determine SOURCE_DATE_EPOCH:
 #
-# 1. Use value from __source_date_epoch.txt file if this file exists. 
+# 1. Use the value from __source_date_epoch.txt file if this file exists.
 #    This file was most likely created in the previous build by one of the following methods 2,3,4.
-#    In principle, it could actually provided by a recipe via SRC_URI
+#    Alternatively, it can be provided by a recipe via SRC_URI.
 #
-# If the file does not exist, first try to determine the value for SOURCE_DATE_EPOCH:
+# If the file does not exist:
 #
-# 2. If we detected a folder .git, use .git last commit date timestamp, as git does not allow checking out
-#    files and preserving their timestamps.
+# 2. If there is a git checkout, use the last git commit timestamp.
+#    Git does not preserve file timestamps on checkout.
 #
 # 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
-#    This will work fine for any well kept repository distributed via tarballs.
+#    This works for well-kept repositories distributed via tarball.
 #
-# 4. If the above steps fail, we need to check all package source files and use the youngest file of the source tree.
+# 4. If the above steps fail, use the modification time of the youngest file in the source tree.
 #
-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe ${WORKDIR}/source_date_epoch folder
-# in a text file "__source_date_epoch.txt'. If this file is found by other recipe task, the value is exported in
-# the SOURCE_DATE_EPOCH variable in the task environment. This is done in an anonymous python function, 
-# so SOURCE_DATE_EPOCH is guaranteed to exist for all tasks the may use it (do_configure, do_compile, do_package, ...)
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# If this file is found by other tasks, the value is exported in the SOURCE_DATE_EPOCH variable.
+# SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...)
 
 BUILD_REPRODUCIBLE_BINARIES ??= '1'
 inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')}
@@ -67,7 +66,6 @@ def find_git_folder(path):
     for root, dirs, files in os.walk(path, topdown=True):
         dirs[:] = [d for d in dirs if d not in exclude]
         if '.git' in dirs:
-            #bb.warn("found root:%s" % (str(root)))
             return root
 
 def get_source_date_epoch_git(d, path):
@@ -79,10 +77,8 @@ def get_source_date_epoch_git(d, path):
             if os.path.isdir(os.path.join(gitpath,".git")):
                 try:
                     source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=path))
-                    #bb.warn("JB *** gitpath:%s sde: %d" % (gitpath,source_date_epoch))
                     bb.debug(1, "git repo path:%s sde: %d" % (gitpath,source_date_epoch))
                 except subprocess.CalledProcessError as grepexc:
-                    #bb.warn( "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
                     bb.debug(1, "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
         else:
             bb.warn("Failed to find a git repository for path:%s" % (path))
@@ -99,7 +95,6 @@ python do_create_source_date_epoch_stamp() {
         bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
         return
 
-    # Try to detect/find a git repository
     source_date_epoch = get_source_date_epoch_git(d, path)
     if source_date_epoch == 0:
         source_date_epoch = get_source_date_epoch_known_files(d, path)
-- 
2.17.1

[PATCH 2/6] Reproducible: Refactor: get_source_date_epoch_from_youngest_file()

[Douglas Royds]

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 59 ++++++++++++++-----------
 1 file changed, 33 insertions(+), 26 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 4e8850fb83..e118cfde01 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -49,15 +49,15 @@ do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
 addtask do_deploy_source_date_epoch_setscene
 addtask do_deploy_source_date_epoch before do_configure after do_patch
 
-def get_source_date_epoch_known_files(d, path):
-    source_date_epoch = 0
+def get_source_date_epoch_from_known_files(d, path):
+    source_date_epoch = None
     known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
     for file in known_files:
         filepath = os.path.join(path,file)
         if os.path.isfile(filepath):
             mtime = int(os.lstat(filepath).st_mtime)
             # There may be more than one "known_file" present, if so, use the youngest one
-            if mtime > source_date_epoch:
+            if not source_date_epoch or mtime > source_date_epoch:
                 source_date_epoch = mtime
     return source_date_epoch
 
@@ -68,8 +68,8 @@ def find_git_folder(path):
         if '.git' in dirs:
             return root
 
-def get_source_date_epoch_git(d, path):
-    source_date_epoch = 0
+def get_source_date_epoch_from_git(d, path):
+    source_date_epoch = None
     if "git://" in d.getVar('SRC_URI'):
         gitpath = find_git_folder(d.getVar('WORKDIR'))
         if gitpath != None:
@@ -84,24 +84,13 @@ def get_source_date_epoch_git(d, path):
             bb.warn("Failed to find a git repository for path:%s" % (path))
     return source_date_epoch
 
-python do_create_source_date_epoch_stamp() {
-    path = d.getVar('S')
-    if not os.path.isdir(path):
-        bb.warn("Unable to determine source_date_epoch! path:%s" % path)
-        return
-
-    epochfile = d.getVar('SDE_FILE')
-    if os.path.isfile(epochfile):
-        bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
-        return
-
-    source_date_epoch = get_source_date_epoch_git(d, path)
-    if source_date_epoch == 0:
-        source_date_epoch = get_source_date_epoch_known_files(d, path)
-    if source_date_epoch == 0:
+def get_source_date_epoch_from_youngest_file(d, path):
     # Do it the hard way: check all files and find the youngest one...
-        filename_dbg = None
-        exclude = set(["temp", "license-destdir", "patches", "recipe-sysroot-native", "recipe-sysroot", "pseudo", "build", "image", "sysroot-destdir"])
+    source_date_epoch = None
+    newest_file = None
+    # Just in case S = WORKDIR
+    exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
+                   "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
     for root, dirs, files in os.walk(path, topdown=True):
         files = [f for f in files if not f[0] == '.']
         dirs[:] = [d for d in dirs if d not in exclude]
@@ -112,13 +101,31 @@ python do_create_source_date_epoch_stamp() {
                 mtime = int(os.lstat(filename).st_mtime)
             except ValueError:
                 mtime = 0
-                if mtime > source_date_epoch:
+            if not source_date_epoch or mtime > source_date_epoch:
                 source_date_epoch = mtime
-                    filename_dbg = filename
+                newest_file = filename
+
+    if newest_file != None:
+        bb.debug(1," SOURCE_DATE_EPOCH %d derived from: %s" % (source_date_epoch, newest_file))
+    return source_date_epoch
 
-        if filename_dbg != None:
-            bb.debug(1," SOURCE_DATE_EPOCH %d derived from: %s" % (source_date_epoch, filename_dbg))
+python do_create_source_date_epoch_stamp() {
+    path = d.getVar('S')
+    if not os.path.isdir(path):
+        bb.warn("Unable to determine source_date_epoch! path:%s" % path)
+        return
+
+    epochfile = d.getVar('SDE_FILE')
+    if os.path.isfile(epochfile):
+        bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
+        return
 
+    source_date_epoch = (
+        get_source_date_epoch_from_git(d, path) or
+        get_source_date_epoch_from_known_files(d, path) or
+        get_source_date_epoch_from_youngest_file(d, path) or
+        0       # Last resort
+    )
     if source_date_epoch == 0:
         # empty folder, not a single file ...
         # kernel source do_unpack is special cased
-- 
2.17.1

[PATCH 3/6] Reproducible: Rename ambiguous "path" var to sourcedir or workdir as appropriate

[Douglas Royds]

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 37 +++++++++++++------------
 1 file changed, 19 insertions(+), 18 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index e118cfde01..3f3790dfe3 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -49,11 +49,11 @@ do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
 addtask do_deploy_source_date_epoch_setscene
 addtask do_deploy_source_date_epoch before do_configure after do_patch
 
-def get_source_date_epoch_from_known_files(d, path):
+def get_source_date_epoch_from_known_files(d, sourcedir):
     source_date_epoch = None
     known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
     for file in known_files:
-        filepath = os.path.join(path,file)
+        filepath = os.path.join(sourcedir, file)
         if os.path.isfile(filepath):
             mtime = int(os.lstat(filepath).st_mtime)
             # There may be more than one "known_file" present, if so, use the youngest one
@@ -61,37 +61,38 @@ def get_source_date_epoch_from_known_files(d, path):
                 source_date_epoch = mtime
     return source_date_epoch
 
-def find_git_folder(path):
+def find_git_folder(workdir):
     exclude = set(["temp", "license-destdir", "patches", "recipe-sysroot-native", "recipe-sysroot", "pseudo", "build", "image", "sysroot-destdir"])
-    for root, dirs, files in os.walk(path, topdown=True):
+    for root, dirs, files in os.walk(workdir, topdown=True):
         dirs[:] = [d for d in dirs if d not in exclude]
         if '.git' in dirs:
             return root
 
-def get_source_date_epoch_from_git(d, path):
+def get_source_date_epoch_from_git(d, sourcedir):
     source_date_epoch = None
     if "git://" in d.getVar('SRC_URI'):
-        gitpath = find_git_folder(d.getVar('WORKDIR'))
-        if gitpath != None:
+        workdir = d.getVar('WORKDIR')
+        gitpath = find_git_folder(workdir)
+        if gitpath:
             import subprocess
             if os.path.isdir(os.path.join(gitpath,".git")):
                 try:
-                    source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=path))
+                    source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=sourcedir))
                     bb.debug(1, "git repo path:%s sde: %d" % (gitpath,source_date_epoch))
                 except subprocess.CalledProcessError as grepexc:
                     bb.debug(1, "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
         else:
-            bb.warn("Failed to find a git repository for path:%s" % (path))
+            bb.warn("Failed to find a git repository for path:%s" % workdir)
     return source_date_epoch
 
-def get_source_date_epoch_from_youngest_file(d, path):
+def get_source_date_epoch_from_youngest_file(d, sourcedir):
     # Do it the hard way: check all files and find the youngest one...
     source_date_epoch = None
     newest_file = None
     # Just in case S = WORKDIR
     exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
                    "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
-    for root, dirs, files in os.walk(path, topdown=True):
+    for root, dirs, files in os.walk(sourcedir, topdown=True):
         files = [f for f in files if not f[0] == '.']
         dirs[:] = [d for d in dirs if d not in exclude]
 
@@ -110,9 +111,9 @@ def get_source_date_epoch_from_youngest_file(d, path):
     return source_date_epoch
 
 python do_create_source_date_epoch_stamp() {
-    path = d.getVar('S')
-    if not os.path.isdir(path):
-        bb.warn("Unable to determine source_date_epoch! path:%s" % path)
+    sourcedir = d.getVar('S')
+    if not os.path.isdir(sourcedir):
+        bb.warn("Unable to determine source_date_epoch! path:%s" % sourcedir)
         return
 
     epochfile = d.getVar('SDE_FILE')
@@ -121,16 +122,16 @@ python do_create_source_date_epoch_stamp() {
         return
 
     source_date_epoch = (
-        get_source_date_epoch_from_git(d, path) or
-        get_source_date_epoch_from_known_files(d, path) or
-        get_source_date_epoch_from_youngest_file(d, path) or
+        get_source_date_epoch_from_git(d, sourcedir) or
+        get_source_date_epoch_from_known_files(d, sourcedir) or
+        get_source_date_epoch_from_youngest_file(d, sourcedir) or
         0       # Last resort
     )
     if source_date_epoch == 0:
         # empty folder, not a single file ...
         # kernel source do_unpack is special cased
         if not bb.data.inherits_class('kernel', d):
-            bb.debug(1, "Unable to determine source_date_epoch! path:%s" % path)
+            bb.debug(1, "Unable to determine source_date_epoch! path:%s" % sourcedir)
 
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
     with open(epochfile, 'w') as f:
-- 
2.17.1

[PATCH 4/6] Reproducible: Find the git repo in WORKDIR/git or S first

[Douglas Royds]

Change the search regime for find_git_folder():

1. WORKDIR/git: This is the default git fetcher unpack path
2. ${S}
3. Go looking for .git/ under the WORKDIR as a last resort.

linux-yocto:
We had an existing (silent) defect. The linux-yocto recipes all specify
two git SRC_URIs, one for the kernel source itself, the other for the
kmeta data (config fragments and friends). find_git_folder() was finding
the git checkout for the kmeta data, but due to a typo in the git log -1
--pretty=%ct line, we were (silently) reading the source_date_epoch from
the ${S} directory = STAGING_KERNEL_DIR, which is empty. If your
build/ happened to be inside a git checkout, git would walk up the
directory tree, and silently read the commit timestamp from this other
git checkout. The correct path to read the git commit timestamp from is
the "gitpath", being that found by find_git_folder(), though this
function was incorrectly finding the kmeta data checkout, not the kernel
source tree.

Non-kernel git recipes:
The default git fetcher clones and checks out the sources at
WORKDIR/git/ regardless of the setting of S (unless subpath or
destsuffix is set). find_git_folder() now looks for the
WORKDIR/git/.git/ directory first.

Non-yocto linux kernels:
Kernel recipes that don't inherit kernel-yocto should always set
S = ${WORKDIR}/git, so that when base_do_unpack_append() in
kernel.bbclass moves the checkout down to the STAGING_KERNEL_DIR and
symlinks it as WORKDIR/git, the build can still work by following the
symlink. We were previously failing to follow the symlink in the
os.walk(), but we now look first for WORKDIR/git/.git/, and find it due
to the symlink.

If none of the above mechanisms work for finding the git checkout,
perhaps there was a subpath or destsuffix specified in the SRC_URI.
We go looking for the git checkout under the WORKDIR as a last resort.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 33 ++++++++++++++++---------
 1 file changed, 22 insertions(+), 11 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 3f3790dfe3..42cb37b042 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -61,28 +61,39 @@ def get_source_date_epoch_from_known_files(d, sourcedir):
                 source_date_epoch = mtime
     return source_date_epoch
 
-def find_git_folder(workdir):
-    exclude = set(["temp", "license-destdir", "patches", "recipe-sysroot-native", "recipe-sysroot", "pseudo", "build", "image", "sysroot-destdir"])
+def find_git_folder(d, sourcedir):
+    # First guess: WORKDIR/git
+    # This is the default git fetcher unpack path
+    workdir = d.getVar('WORKDIR')
+    gitpath = os.path.join(workdir, "git/.git")
+    if os.path.isdir(gitpath):
+        return gitpath
+
+    # Second guess: ${S}
+    gitpath = os.path.join(sourcedir, ".git")
+    if os.path.isdir(gitpath):
+        return gitpath
+
+    # Perhaps there was a subpath or destsuffix specified.
+    # Go looking in the WORKDIR
+    exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
+                   "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
     for root, dirs, files in os.walk(workdir, topdown=True):
         dirs[:] = [d for d in dirs if d not in exclude]
         if '.git' in dirs:
             return root
 
+    bb.warn("Failed to find a git repository in WORKDIR: %s" % workdir)
+    return None
+
 def get_source_date_epoch_from_git(d, sourcedir):
     source_date_epoch = None
     if "git://" in d.getVar('SRC_URI'):
-        workdir = d.getVar('WORKDIR')
-        gitpath = find_git_folder(workdir)
+        gitpath = find_git_folder(d, sourcedir)
         if gitpath:
             import subprocess
-            if os.path.isdir(os.path.join(gitpath,".git")):
-                try:
-                    source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=sourcedir))
+            source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=gitpath))
             bb.debug(1, "git repo path: %s sde: %d" % (gitpath, source_date_epoch))
-                except subprocess.CalledProcessError as grepexc:
-                    bb.debug(1, "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
-        else:
-            bb.warn("Failed to find a git repository for path:%s" % workdir)
     return source_date_epoch
 
 def get_source_date_epoch_from_youngest_file(d, sourcedir):
-- 
2.17.1

[PATCH 5/6] Reproducible: Don't enforce existence of ${S} dir

[Douglas Royds]

Unnecessary.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 42cb37b042..a81f249790 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -122,16 +122,12 @@ def get_source_date_epoch_from_youngest_file(d, sourcedir):
     return source_date_epoch
 
 python do_create_source_date_epoch_stamp() {
-    sourcedir = d.getVar('S')
-    if not os.path.isdir(sourcedir):
-        bb.warn("Unable to determine source_date_epoch! path:%s" % sourcedir)
-        return
-
     epochfile = d.getVar('SDE_FILE')
     if os.path.isfile(epochfile):
         bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
         return
 
+    sourcedir = d.getVar('S')
     source_date_epoch = (
         get_source_date_epoch_from_git(d, sourcedir) or
         get_source_date_epoch_from_known_files(d, sourcedir) or
-- 
2.17.1

[PATCH 6/6] Reproducible: Consistent debug logging

[Douglas Royds]

Log both the source of the source_date_epoch and the resulting
source_date_epoch in all cases.

Also, now that we are determining the source_date_epoch successfully for both
yocto and non-yocto kernels, remove the inherits_class('kernel') exception.
We will log a failure to find a source_date_epoch for kernels as well.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index a81f249790..4f3680a376 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -51,6 +51,7 @@ addtask do_deploy_source_date_epoch before do_configure after do_patch
 
 def get_source_date_epoch_from_known_files(d, sourcedir):
     source_date_epoch = None
+    newest_file = None
     known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
     for file in known_files:
         filepath = os.path.join(sourcedir, file)
@@ -59,6 +60,9 @@ def get_source_date_epoch_from_known_files(d, sourcedir):
             # There may be more than one "known_file" present, if so, use the youngest one
             if not source_date_epoch or mtime > source_date_epoch:
                 source_date_epoch = mtime
+                newest_file = filepath
+    if newest_file:
+        bb.debug(1, "SOURCE_DATE_EPOCH taken from: %s" % newest_file)
     return source_date_epoch
 
 def find_git_folder(d, sourcedir):
@@ -93,7 +97,7 @@ def get_source_date_epoch_from_git(d, sourcedir):
         if gitpath:
             import subprocess
             source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=gitpath))
-            bb.debug(1, "git repo path: %s sde: %d" % (gitpath, source_date_epoch))
+            bb.debug(1, "git repository: %s" % gitpath)
     return source_date_epoch
 
 def get_source_date_epoch_from_youngest_file(d, sourcedir):
@@ -117,14 +121,14 @@ def get_source_date_epoch_from_youngest_file(d, sourcedir):
                 source_date_epoch = mtime
                 newest_file = filename
 
-    if newest_file != None:
-        bb.debug(1," SOURCE_DATE_EPOCH %d derived from: %s" % (source_date_epoch, newest_file))
+    if newest_file:
+        bb.debug(1, "Newest file found: %s" % newest_file)
     return source_date_epoch
 
 python do_create_source_date_epoch_stamp() {
     epochfile = d.getVar('SDE_FILE')
     if os.path.isfile(epochfile):
-        bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
+        bb.debug(1, "Reusing SOURCE_DATE_EPOCH from: %s" % epochfile)
         return
 
     sourcedir = d.getVar('S')
@@ -136,10 +140,9 @@ python do_create_source_date_epoch_stamp() {
     )
     if source_date_epoch == 0:
         # empty folder, not a single file ...
-        # kernel source do_unpack is special cased
-        if not bb.data.inherits_class('kernel', d):
-            bb.debug(1, "Unable to determine source_date_epoch! path:%s" % sourcedir)
+        bb.debug(1, "No files found to determine SOURCE_DATE_EPOCH")
 
+    bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
     with open(epochfile, 'w') as f:
         f.write(str(source_date_epoch))
@@ -155,6 +158,6 @@ python () {
         if os.path.isfile(epochfile):
             with open(epochfile, 'r') as f:
                 source_date_epoch = f.read()
-            bb.debug(1, "source_date_epoch stamp found ---> stamp %s" % source_date_epoch)
+            bb.debug(1, "SOURCE_DATE_EPOCH: %s" % source_date_epoch)
         d.setVar('SOURCE_DATE_EPOCH', source_date_epoch)
 }
-- 
2.17.1

✗ patchtest: failure for Reproducible: Find the git repo in WORKDIR/git or S first

[Patchwork]

== Series Details ==

Series: Reproducible: Find the git repo in WORKDIR/git or S first
Revision: 1
URL   : https://patchwork.openembedded.org/series/14024/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at b7f3f7ecfd)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for Reproducible: Find the git repo in WORKDIR/git or S first

[Douglas Royds]

[-- Attachment #1: Type: text/plain, Size: 2277 bytes --]

The patch series already was applied on top of b7f3f7ecfd.

I observe at 
http://lists.openembedded.org/pipermail/openembedded-core/2018-September/thread.html 
that the patch set didn't hit the list threaded. Might this have been 
the reason for the Patchwork rejection?

git send-email appears (at my end) to have correctly set the headers, 
eg. from path 6/6:

    From: Douglas Royds <douglas.royds@taitradio.com>
    To: openembedded-core@lists.openembedded.org
    Cc: Douglas Royds <douglas.royds@taitradio.com>
    Subject: [PATCH 6/6] Reproducible: Consistent debug logging
    Date: Wed, 12 Sep 2018 13:27:07 +1200
    Message-Id: <20180912012707.30046-7-douglas.royds@taitradio.com>
    X-Mailer: git-send-email 2.17.1
    In-Reply-To: <20180912012707.30046-1-douglas.royds@taitradio.com>
    References: <20180912012707.30046-1-douglas.royds@taitradio.com>

Suggestions?


On 12/09/18 14:02, Patchwork wrote:

> == Series Details ==
>
> Series: Reproducible: Find the git repo in WORKDIR/git or S first
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/14024/
> State : failure
>
> == Summary ==
>
>
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
>
>
>
> * Issue             Series does not apply on top of target branch [test_series_merge_on_head]
>    Suggested fix    Rebase your series on top of targeted branch
>    Targeted branch  master (currently at b7f3f7ecfd)
>
>
>
> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
>
> ---
> Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
>


[-- Attachment #2: Type: text/html, Size: 3830 bytes --]

Re: ✗ patchtest: failure for Reproducible: Find the git repo in WORKDIR/git or S first

[Burton, Ross]

On 12 September 2018 at 03:15, Douglas Royds
<douglas.royds@taitradio.com> wrote:
> The patch series already was applied on top of b7f3f7ecfd.

Fails for me too:

Applying: Reproducible: Tidy class description
Using index info to reconstruct a base tree...
error: patch failed: meta/classes/reproducible_build.bbclass:1
error: meta/classes/reproducible_build.bbclass: patch does not apply
error: Did you hand edit your patch?
It does not apply to blobs recorded in its index.

[PATCH v2 0/6] Reproducible: Find the git repo in WORKDIR/git or S first

[Douglas Royds]

Change the search regime for find_git_folder():

1. WORKDIR/git: This is the default git fetcher unpack path
2. ${S}
3. Go looking for .git/ under the WORKDIR as a last resort.

linux-yocto:
We had an existing (silent) defect. The linux-yocto recipes all specify
two git SRC_URIs, one for the kernel source itself, the other for the
kmeta data (config fragments and friends). find_git_folder() was finding
the git checkout for the kmeta data, but due to a typo in the git log -1
--pretty=%ct line, we were (silently) reading the source_date_epoch from
the ${S} directory = STAGING_KERNEL_DIR, which is empty. If your
build/ happened to be inside a git checkout, git would walk up the
directory tree, and silently read the commit timestamp from this other
git checkout. The correct path to read the git commit timestamp from is
the "gitpath", being that found by find_git_folder(), though this
function was incorrectly finding the kmeta data checkout, not the kernel
source tree.

Non-kernel git recipes:
The default git fetcher clones and checks out the sources at
WORKDIR/git/ regardless of the setting of S (unless subpath or
destsuffix is set). find_git_folder() now looks for the
WORKDIR/git/.git/ directory first.

Non-yocto linux kernels:
Kernel recipes that don't inherit kernel-yocto should always set
S = ${WORKDIR}/git, so that when base_do_unpack_append() in
kernel.bbclass moves the checkout down to the STAGING_KERNEL_DIR and
symlinks it as WORKDIR/git, the build can still work by following the
symlink. We were previously failing to follow the symlink in the
os.walk(), but we now look first for WORKDIR/git/.git/, and find it due
to the symlink.

If none of the above mechanisms work for finding the git checkout,
perhaps there was a subpath or destsuffix specified in the SRC_URI.
We go looking for the git checkout under the WORKDIR as a last resort.

Douglas Royds (6):
  Reproducible: Tidy class description
  Reproducible: Refactor: get_source_date_epoch_from_youngest_file()
  Reproducible: Rename ambiguous "path" var to sourcedir or workdir as appropriate
  Reproducible: Find the git repo in WORKDIR/git or S first
  Reproducible: Don't enforce existence of ${S} dir
  Reproducible: Consistent debug logging

 meta/classes/reproducible_build.bbclass | 175 +++++++++++++-----------
 1 file changed, 94 insertions(+), 81 deletions(-)

-- 
2.17.1

[PATCH v2 1/6] Reproducible: Tidy class description

[Douglas Royds]

Tidy whitespace, delete commented-out lines

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 49 +++++++++++--------------
 1 file changed, 22 insertions(+), 27 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 268b5fb8f1..4e8850fb83 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -1,33 +1,32 @@
-#
 # reproducible_build.bbclass
 #
-# This bbclass is mainly responsible to determine SOURCE_DATE_EPOCH on a per recipe base.
-# We need to set a recipe specific SOURCE_DATE_EPOCH in each recipe environment for various tasks.
-# One way would be to modify all recipes one-by-one to specify SOURCE_DATE_EPOCH explicitly, 
-# but that is not realistic as there are hundreds (probably thousands) of recipes in various meta-layers.
-# Therefore we do it this class. 
-# After sources are unpacked but before they are patched, we try to determine the value for SOURCE_DATE_EPOCH.
+# Sets SOURCE_DATE_EPOCH in each component's build environment.
+# Upstream components (generally) respect this environment variable,
+# using it in place of the "current" date and time.
+# See https://reproducible-builds.org/specs/source-date-epoch/
+#
+# After sources are unpacked but before they are patched, we set a reproducible value for SOURCE_DATE_EPOCH.
+# This value should be reproducible for anyone who builds the same revision from the same sources.
 #
-# There are 4 ways to determine SOURCE_DATE_EPOCH:
+# There are 4 ways we determine SOURCE_DATE_EPOCH:
 #
-# 1. Use value from __source_date_epoch.txt file if this file exists. 
-#    This file was most likely created in the previous build by one of the following methods 2,3,4. 
-#    In principle, it could actually provided by a recipe via SRC_URI
+# 1. Use the value from __source_date_epoch.txt file if this file exists.
+#    This file was most likely created in the previous build by one of the following methods 2,3,4.
+#    Alternatively, it can be provided by a recipe via SRC_URI.
 #
-# If the file does not exist, first try to determine the value for SOURCE_DATE_EPOCH:
+# If the file does not exist:
 #
-# 2. If we detected a folder .git, use .git last commit date timestamp, as git does not allow checking out
-#    files and preserving their timestamps.
+# 2. If there is a git checkout, use the last git commit timestamp.
+#    Git does not preserve file timestamps on checkout.
 #
 # 3. Use the mtime of "known" files such as NEWS, CHANGLELOG, ...
-#    This will work fine for any well kept repository distributed via tarballs.
+#    This works for well-kept repositories distributed via tarball.
 #
-# 4. If the above steps fail, we need to check all package source files and use the youngest file of the source tree.
+# 4. If the above steps fail, use the modification time of the youngest file in the source tree.
 #
-# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe ${WORKDIR}/source_date_epoch folder
-# in a text file "__source_date_epoch.txt'. If this file is found by other recipe task, the value is exported in
-# the SOURCE_DATE_EPOCH variable in the task environment. This is done in an anonymous python function, 
-# so SOURCE_DATE_EPOCH is guaranteed to exist for all tasks the may use it (do_configure, do_compile, do_package, ...)
+# Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the recipe's ${SDE_FILE}.
+# If this file is found by other tasks, the value is exported in the SOURCE_DATE_EPOCH variable.
+# SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...)
 
 BUILD_REPRODUCIBLE_BINARIES ??= '1'
 inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')}
@@ -67,9 +66,8 @@ def find_git_folder(path):
     for root, dirs, files in os.walk(path, topdown=True):
         dirs[:] = [d for d in dirs if d not in exclude]
         if '.git' in dirs:
-            #bb.warn("found root:%s" % (str(root)))
             return root
-     
+
 def get_source_date_epoch_git(d, path):
     source_date_epoch = 0
     if "git://" in d.getVar('SRC_URI'):
@@ -79,15 +77,13 @@ def get_source_date_epoch_git(d, path):
             if os.path.isdir(os.path.join(gitpath,".git")):
                 try:
                     source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=path))
-                    #bb.warn("JB *** gitpath:%s sde: %d" % (gitpath,source_date_epoch))
                     bb.debug(1, "git repo path:%s sde: %d" % (gitpath,source_date_epoch))
                 except subprocess.CalledProcessError as grepexc:
-                    #bb.warn( "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
                     bb.debug(1, "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
         else:
             bb.warn("Failed to find a git repository for path:%s" % (path))
     return source_date_epoch
-            
+
 python do_create_source_date_epoch_stamp() {
     path = d.getVar('S')
     if not os.path.isdir(path):
@@ -98,8 +94,7 @@ python do_create_source_date_epoch_stamp() {
     if os.path.isfile(epochfile):
         bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
         return
- 
-    # Try to detect/find a git repository
+
     source_date_epoch = get_source_date_epoch_git(d, path)
     if source_date_epoch == 0:
         source_date_epoch = get_source_date_epoch_known_files(d, path)
-- 
2.17.1

[PATCH v2 2/6] Reproducible: Refactor: get_source_date_epoch_from_youngest_file()

[Douglas Royds]

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 73 ++++++++++++++-----------
 1 file changed, 40 insertions(+), 33 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 4e8850fb83..e118cfde01 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -49,15 +49,15 @@ do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
 addtask do_deploy_source_date_epoch_setscene
 addtask do_deploy_source_date_epoch before do_configure after do_patch
 
-def get_source_date_epoch_known_files(d, path):
-    source_date_epoch = 0
+def get_source_date_epoch_from_known_files(d, path):
+    source_date_epoch = None
     known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
     for file in known_files:
         filepath = os.path.join(path,file)
         if os.path.isfile(filepath):
             mtime = int(os.lstat(filepath).st_mtime)
             # There may be more than one "known_file" present, if so, use the youngest one
-            if mtime > source_date_epoch:
+            if not source_date_epoch or mtime > source_date_epoch:
                 source_date_epoch = mtime
     return source_date_epoch
 
@@ -68,8 +68,8 @@ def find_git_folder(path):
         if '.git' in dirs:
             return root
 
-def get_source_date_epoch_git(d, path):
-    source_date_epoch = 0
+def get_source_date_epoch_from_git(d, path):
+    source_date_epoch = None
     if "git://" in d.getVar('SRC_URI'):
         gitpath = find_git_folder(d.getVar('WORKDIR'))
         if gitpath != None:
@@ -84,6 +84,31 @@ def get_source_date_epoch_git(d, path):
             bb.warn("Failed to find a git repository for path:%s" % (path))
     return source_date_epoch
 
+def get_source_date_epoch_from_youngest_file(d, path):
+    # Do it the hard way: check all files and find the youngest one...
+    source_date_epoch = None
+    newest_file = None
+    # Just in case S = WORKDIR
+    exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
+                   "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
+    for root, dirs, files in os.walk(path, topdown=True):
+        files = [f for f in files if not f[0] == '.']
+        dirs[:] = [d for d in dirs if d not in exclude]
+
+        for fname in files:
+            filename = os.path.join(root, fname)
+            try:
+                mtime = int(os.lstat(filename).st_mtime)
+            except ValueError:
+                mtime = 0
+            if not source_date_epoch or mtime > source_date_epoch:
+                source_date_epoch = mtime
+                newest_file = filename
+
+    if newest_file != None:
+        bb.debug(1," SOURCE_DATE_EPOCH %d derived from: %s" % (source_date_epoch, newest_file))
+    return source_date_epoch
+
 python do_create_source_date_epoch_stamp() {
     path = d.getVar('S')
     if not os.path.isdir(path):
@@ -95,35 +120,17 @@ python do_create_source_date_epoch_stamp() {
         bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
         return
 
-    source_date_epoch = get_source_date_epoch_git(d, path)
+    source_date_epoch = (
+        get_source_date_epoch_from_git(d, path) or
+        get_source_date_epoch_from_known_files(d, path) or
+        get_source_date_epoch_from_youngest_file(d, path) or
+        0       # Last resort
+    )
     if source_date_epoch == 0:
-        source_date_epoch = get_source_date_epoch_known_files(d, path)
-    if source_date_epoch == 0:
-        # Do it the hard way: check all files and find the youngest one...
-        filename_dbg = None
-        exclude = set(["temp", "license-destdir", "patches", "recipe-sysroot-native", "recipe-sysroot", "pseudo", "build", "image", "sysroot-destdir"])
-        for root, dirs, files in os.walk(path, topdown=True):
-            files = [f for f in files if not f[0] == '.']
-            dirs[:] = [d for d in dirs if d not in exclude]
-
-            for fname in files:
-                filename = os.path.join(root, fname)
-                try:
-                    mtime = int(os.lstat(filename).st_mtime)
-                except ValueError:
-                    mtime = 0
-                if mtime > source_date_epoch:
-                    source_date_epoch = mtime
-                    filename_dbg = filename
-
-        if filename_dbg != None:
-            bb.debug(1," SOURCE_DATE_EPOCH %d derived from: %s" % (source_date_epoch, filename_dbg))
-
-        if source_date_epoch == 0:
-            # empty folder, not a single file ...
-            # kernel source do_unpack is special cased
-            if not bb.data.inherits_class('kernel', d):
-                bb.debug(1, "Unable to determine source_date_epoch! path:%s" % path)
+        # empty folder, not a single file ...
+        # kernel source do_unpack is special cased
+        if not bb.data.inherits_class('kernel', d):
+            bb.debug(1, "Unable to determine source_date_epoch! path:%s" % path)
 
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
     with open(epochfile, 'w') as f:
-- 
2.17.1

[PATCH v2 3/6] Reproducible: Rename ambiguous "path" var to sourcedir or workdir as appropriate

[Douglas Royds]

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 37 +++++++++++++------------
 1 file changed, 19 insertions(+), 18 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index e118cfde01..3f3790dfe3 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -49,11 +49,11 @@ do_deploy_source_date_epoch[sstate-plaindirs] = "${SDE_DIR}"
 addtask do_deploy_source_date_epoch_setscene
 addtask do_deploy_source_date_epoch before do_configure after do_patch
 
-def get_source_date_epoch_from_known_files(d, path):
+def get_source_date_epoch_from_known_files(d, sourcedir):
     source_date_epoch = None
     known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
     for file in known_files:
-        filepath = os.path.join(path,file)
+        filepath = os.path.join(sourcedir, file)
         if os.path.isfile(filepath):
             mtime = int(os.lstat(filepath).st_mtime)
             # There may be more than one "known_file" present, if so, use the youngest one
@@ -61,37 +61,38 @@ def get_source_date_epoch_from_known_files(d, path):
                 source_date_epoch = mtime
     return source_date_epoch
 
-def find_git_folder(path):
+def find_git_folder(workdir):
     exclude = set(["temp", "license-destdir", "patches", "recipe-sysroot-native", "recipe-sysroot", "pseudo", "build", "image", "sysroot-destdir"])
-    for root, dirs, files in os.walk(path, topdown=True):
+    for root, dirs, files in os.walk(workdir, topdown=True):
         dirs[:] = [d for d in dirs if d not in exclude]
         if '.git' in dirs:
             return root
 
-def get_source_date_epoch_from_git(d, path):
+def get_source_date_epoch_from_git(d, sourcedir):
     source_date_epoch = None
     if "git://" in d.getVar('SRC_URI'):
-        gitpath = find_git_folder(d.getVar('WORKDIR'))
-        if gitpath != None:
+        workdir = d.getVar('WORKDIR')
+        gitpath = find_git_folder(workdir)
+        if gitpath:
             import subprocess
             if os.path.isdir(os.path.join(gitpath,".git")):
                 try:
-                    source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=path))
+                    source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=sourcedir))
                     bb.debug(1, "git repo path:%s sde: %d" % (gitpath,source_date_epoch))
                 except subprocess.CalledProcessError as grepexc:
                     bb.debug(1, "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
         else:
-            bb.warn("Failed to find a git repository for path:%s" % (path))
+            bb.warn("Failed to find a git repository for path:%s" % workdir)
     return source_date_epoch
 
-def get_source_date_epoch_from_youngest_file(d, path):
+def get_source_date_epoch_from_youngest_file(d, sourcedir):
     # Do it the hard way: check all files and find the youngest one...
     source_date_epoch = None
     newest_file = None
     # Just in case S = WORKDIR
     exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
                    "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
-    for root, dirs, files in os.walk(path, topdown=True):
+    for root, dirs, files in os.walk(sourcedir, topdown=True):
         files = [f for f in files if not f[0] == '.']
         dirs[:] = [d for d in dirs if d not in exclude]
 
@@ -110,9 +111,9 @@ def get_source_date_epoch_from_youngest_file(d, path):
     return source_date_epoch
 
 python do_create_source_date_epoch_stamp() {
-    path = d.getVar('S')
-    if not os.path.isdir(path):
-        bb.warn("Unable to determine source_date_epoch! path:%s" % path)
+    sourcedir = d.getVar('S')
+    if not os.path.isdir(sourcedir):
+        bb.warn("Unable to determine source_date_epoch! path:%s" % sourcedir)
         return
 
     epochfile = d.getVar('SDE_FILE')
@@ -121,16 +122,16 @@ python do_create_source_date_epoch_stamp() {
         return
 
     source_date_epoch = (
-        get_source_date_epoch_from_git(d, path) or
-        get_source_date_epoch_from_known_files(d, path) or
-        get_source_date_epoch_from_youngest_file(d, path) or
+        get_source_date_epoch_from_git(d, sourcedir) or
+        get_source_date_epoch_from_known_files(d, sourcedir) or
+        get_source_date_epoch_from_youngest_file(d, sourcedir) or
         0       # Last resort
     )
     if source_date_epoch == 0:
         # empty folder, not a single file ...
         # kernel source do_unpack is special cased
         if not bb.data.inherits_class('kernel', d):
-            bb.debug(1, "Unable to determine source_date_epoch! path:%s" % path)
+            bb.debug(1, "Unable to determine source_date_epoch! path:%s" % sourcedir)
 
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
     with open(epochfile, 'w') as f:
-- 
2.17.1

[PATCH v2 4/6] Reproducible: Find the git repo in WORKDIR/git or S first

[Douglas Royds]

Change the search regime for find_git_folder():

1. WORKDIR/git: This is the default git fetcher unpack path
2. ${S}
3. Go looking for .git/ under the WORKDIR as a last resort.

linux-yocto:
We had an existing (silent) defect. The linux-yocto recipes all specify
two git SRC_URIs, one for the kernel source itself, the other for the
kmeta data (config fragments and friends). find_git_folder() was finding
the git checkout for the kmeta data, but due to a typo in the git log -1
--pretty=%ct line, we were (silently) reading the source_date_epoch from
the ${S} directory = STAGING_KERNEL_DIR, which is empty. If your
build/ happened to be inside a git checkout, git would walk up the
directory tree, and silently read the commit timestamp from this other
git checkout. The correct path to read the git commit timestamp from is
the "gitpath", being that found by find_git_folder(), though this
function was incorrectly finding the kmeta data checkout, not the kernel
source tree.

Non-kernel git recipes:
The default git fetcher clones and checks out the sources at
WORKDIR/git/ regardless of the setting of S (unless subpath or
destsuffix is set). find_git_folder() now looks for the
WORKDIR/git/.git/ directory first.

Non-yocto linux kernels:
Kernel recipes that don't inherit kernel-yocto should always set
S = ${WORKDIR}/git, so that when base_do_unpack_append() in
kernel.bbclass moves the checkout down to the STAGING_KERNEL_DIR and
symlinks it as WORKDIR/git, the build can still work by following the
symlink. We were previously failing to follow the symlink in the
os.walk(), but we now look first for WORKDIR/git/.git/, and find it due
to the symlink.

If none of the above mechanisms work for finding the git checkout,
perhaps there was a subpath or destsuffix specified in the SRC_URI.
We go looking for the git checkout under the WORKDIR as a last resort.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 35 ++++++++++++++++---------
 1 file changed, 23 insertions(+), 12 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 3f3790dfe3..42cb37b042 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -61,28 +61,39 @@ def get_source_date_epoch_from_known_files(d, sourcedir):
                 source_date_epoch = mtime
     return source_date_epoch
 
-def find_git_folder(workdir):
-    exclude = set(["temp", "license-destdir", "patches", "recipe-sysroot-native", "recipe-sysroot", "pseudo", "build", "image", "sysroot-destdir"])
+def find_git_folder(d, sourcedir):
+    # First guess: WORKDIR/git
+    # This is the default git fetcher unpack path
+    workdir = d.getVar('WORKDIR')
+    gitpath = os.path.join(workdir, "git/.git")
+    if os.path.isdir(gitpath):
+        return gitpath
+
+    # Second guess: ${S}
+    gitpath = os.path.join(sourcedir, ".git")
+    if os.path.isdir(gitpath):
+        return gitpath
+
+    # Perhaps there was a subpath or destsuffix specified.
+    # Go looking in the WORKDIR
+    exclude = set(["build", "image", "license-destdir", "patches", "pseudo",
+                   "recipe-sysroot", "recipe-sysroot-native", "sysroot-destdir", "temp"])
     for root, dirs, files in os.walk(workdir, topdown=True):
         dirs[:] = [d for d in dirs if d not in exclude]
         if '.git' in dirs:
             return root
 
+    bb.warn("Failed to find a git repository in WORKDIR: %s" % workdir)
+    return None
+
 def get_source_date_epoch_from_git(d, sourcedir):
     source_date_epoch = None
     if "git://" in d.getVar('SRC_URI'):
-        workdir = d.getVar('WORKDIR')
-        gitpath = find_git_folder(workdir)
+        gitpath = find_git_folder(d, sourcedir)
         if gitpath:
             import subprocess
-            if os.path.isdir(os.path.join(gitpath,".git")):
-                try:
-                    source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=sourcedir))
-                    bb.debug(1, "git repo path:%s sde: %d" % (gitpath,source_date_epoch))
-                except subprocess.CalledProcessError as grepexc:
-                    bb.debug(1, "Expected git repository not found, (path: %s) error:%d" % (gitpath, grepexc.returncode))
-        else:
-            bb.warn("Failed to find a git repository for path:%s" % workdir)
+            source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=gitpath))
+            bb.debug(1, "git repo path: %s sde: %d" % (gitpath, source_date_epoch))
     return source_date_epoch
 
 def get_source_date_epoch_from_youngest_file(d, sourcedir):
-- 
2.17.1

[PATCH v2 5/6] Reproducible: Don't enforce existence of ${S} dir

[Douglas Royds]

Unnecessary.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 42cb37b042..a81f249790 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -122,16 +122,12 @@ def get_source_date_epoch_from_youngest_file(d, sourcedir):
     return source_date_epoch
 
 python do_create_source_date_epoch_stamp() {
-    sourcedir = d.getVar('S')
-    if not os.path.isdir(sourcedir):
-        bb.warn("Unable to determine source_date_epoch! path:%s" % sourcedir)
-        return
-
     epochfile = d.getVar('SDE_FILE')
     if os.path.isfile(epochfile):
         bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
         return
 
+    sourcedir = d.getVar('S')
     source_date_epoch = (
         get_source_date_epoch_from_git(d, sourcedir) or
         get_source_date_epoch_from_known_files(d, sourcedir) or
-- 
2.17.1

[PATCH v2 6/6] Reproducible: Consistent debug logging

[Douglas Royds]

Log both the source of the source_date_epoch and the resulting
source_date_epoch in all cases.

Also, now that we are determining the source_date_epoch successfully for both
yocto and non-yocto kernels, remove the inherits_class('kernel') exception.
We will log a failure to find a source_date_epoch for kernels as well.

Signed-off-by: Douglas Royds <douglas.royds@taitradio.com>
---
 meta/classes/reproducible_build.bbclass | 19 +++++++++++--------
 1 file changed, 11 insertions(+), 8 deletions(-)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index a81f249790..4f3680a376 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -51,6 +51,7 @@ addtask do_deploy_source_date_epoch before do_configure after do_patch
 
 def get_source_date_epoch_from_known_files(d, sourcedir):
     source_date_epoch = None
+    newest_file = None
     known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
     for file in known_files:
         filepath = os.path.join(sourcedir, file)
@@ -59,6 +60,9 @@ def get_source_date_epoch_from_known_files(d, sourcedir):
             # There may be more than one "known_file" present, if so, use the youngest one
             if not source_date_epoch or mtime > source_date_epoch:
                 source_date_epoch = mtime
+                newest_file = filepath
+    if newest_file:
+        bb.debug(1, "SOURCE_DATE_EPOCH taken from: %s" % newest_file)
     return source_date_epoch
 
 def find_git_folder(d, sourcedir):
@@ -93,7 +97,7 @@ def get_source_date_epoch_from_git(d, sourcedir):
         if gitpath:
             import subprocess
             source_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct'], cwd=gitpath))
-            bb.debug(1, "git repo path: %s sde: %d" % (gitpath, source_date_epoch))
+            bb.debug(1, "git repository: %s" % gitpath)
     return source_date_epoch
 
 def get_source_date_epoch_from_youngest_file(d, sourcedir):
@@ -117,14 +121,14 @@ def get_source_date_epoch_from_youngest_file(d, sourcedir):
                 source_date_epoch = mtime
                 newest_file = filename
 
-    if newest_file != None:
-        bb.debug(1," SOURCE_DATE_EPOCH %d derived from: %s" % (source_date_epoch, newest_file))
+    if newest_file:
+        bb.debug(1, "Newest file found: %s" % newest_file)
     return source_date_epoch
 
 python do_create_source_date_epoch_stamp() {
     epochfile = d.getVar('SDE_FILE')
     if os.path.isfile(epochfile):
-        bb.debug(1, " path: %s reusing __source_date_epoch.txt" % epochfile)
+        bb.debug(1, "Reusing SOURCE_DATE_EPOCH from: %s" % epochfile)
         return
 
     sourcedir = d.getVar('S')
@@ -136,10 +140,9 @@ python do_create_source_date_epoch_stamp() {
     )
     if source_date_epoch == 0:
         # empty folder, not a single file ...
-        # kernel source do_unpack is special cased
-        if not bb.data.inherits_class('kernel', d):
-            bb.debug(1, "Unable to determine source_date_epoch! path:%s" % sourcedir)
+        bb.debug(1, "No files found to determine SOURCE_DATE_EPOCH")
 
+    bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
     bb.utils.mkdirhier(d.getVar('SDE_DIR'))
     with open(epochfile, 'w') as f:
         f.write(str(source_date_epoch))
@@ -155,6 +158,6 @@ python () {
         if os.path.isfile(epochfile):
             with open(epochfile, 'r') as f:
                 source_date_epoch = f.read()
-            bb.debug(1, "source_date_epoch stamp found ---> stamp %s" % source_date_epoch)
+            bb.debug(1, "SOURCE_DATE_EPOCH: %s" % source_date_epoch)
         d.setVar('SOURCE_DATE_EPOCH', source_date_epoch)
 }
-- 
2.17.1

Re: ✗ patchtest: failure for Reproducible: Find the git repo in WORKDIR/git or S first

[Douglas Royds]

On 14/09/18 04:11, Burton, Ross wrote:

> On 12 September 2018 at 03:15, Douglas Royds
> <douglas.royds@taitradio.com> wrote:
>> The patch series already was applied on top of b7f3f7ecfd.
> Fails for me too:
>
> Applying: Reproducible: Tidy class description
> Using index info to reconstruct a base tree...
> error: patch failed: meta/classes/reproducible_build.bbclass:1
> error: meta/classes/reproducible_build.bbclass: patch does not apply
> error: Did you hand edit your patch?
> It does not apply to blobs recorded in its index.


Lesson of the day: Don't use the -w (ignore whitespace) option to git 
format-patch