[PATCH 1/7] glibc-package: Avoid race sstate races with do_stash_locale

[PATCH 1/7] glibc-package: Avoid race sstate races with do_stash_locale

[Richard Purdie]

The change to make do_stash_locale an sstate task between do_install and
do_package has some unforeseen problems since the function deletes part of
${D} but may or may not run depending on whether the task is installed from
sstate.

This cleans up the current "pre packaging" function to be more deterministic
and result in the same set of files, whichever code path we end up reaching
that point by. Its not an ideal sitation but it should avoid the race
failures we've seen on some builds.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-core/glibc/glibc-package.inc | 17 +++++++++++------
 1 file changed, 11 insertions(+), 6 deletions(-)

diff --git a/meta/recipes-core/glibc/glibc-package.inc b/meta/recipes-core/glibc/glibc-package.inc
index 557b958..f0fc9c1 100644
--- a/meta/recipes-core/glibc/glibc-package.inc
+++ b/meta/recipes-core/glibc/glibc-package.inc
@@ -192,18 +192,23 @@ python do_stash_locale_setscene () {
 }
 addtask do_stash_locale_setscene
 
-PACKAGE_PREPROCESS_FUNCS += "glibc_package_preprocess"
-
-glibc_package_preprocess () {
+do_poststash_install_cleanup () {
+	# Remove all files which do_stash_locale would remove (mv)
+	# since that task could have come from sstate and not get run.
 	for i in ${bashscripts}; do
-	    rm -f ${PKGD}${bindir}/$i
+	    rm -f ${D}${bindir}/$i
 	done
-	rm -rf ${PKGD}/${localedir}
+	rm -f ${D}${bindir}/localedef
+	rm -rf ${D}${datadir}/i18n
+	rm -rf ${D}${libdir}/gconv
+	rm -rf ${D}/${localedir}
+	rm -rf ${D}${datadir}/locale
 	if [ "${libdir}" != "${exec_prefix}/lib" ]; then
 		# This dir only exists to hold locales
-		rm -rf ${PKGD}${exec_prefix}/lib
+		rm -rf ${D}${exec_prefix}/lib
 	fi
 }
+addtask do_poststash_install_cleanup after do_stash_locale do_install before do_populate_sysroot do_package
 
 pkg_postinst_nscd () {
 	if [ -z "$D" ]; then
-- 
2.7.4

[PATCH 2/7] python3-native: Remove tests to save copying uneeded files around

[Richard Purdie]

Removing these nearly halves the python3-native disk footprint meaning
more optimal sysroot operations. We don't need these tests for anything.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-devtools/python/python3-native_3.5.2.bb | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/recipes-devtools/python/python3-native_3.5.2.bb b/meta/recipes-devtools/python/python3-native_3.5.2.bb
index 7ac3942..bf4e686 100644
--- a/meta/recipes-devtools/python/python3-native_3.5.2.bb
+++ b/meta/recipes-devtools/python/python3-native_3.5.2.bb
@@ -98,4 +98,7 @@ do_install() {
 	for PYTHSCRIPT in `grep -rIl ${bindir}/${PN}/python ${D}${bindir}/${PN}`; do
 		sed -i -e '1s|^#!.*|#!/usr/bin/env python3|' $PYTHSCRIPT
 	done
+
+	# Tests are large and we don't need them in the native sysroot
+	rm ${D}${libdir}/python${PYTHON_MAJMIN}/test -rf
 }
-- 
2.7.4

[PATCH 3/7] gcc-cross-initial: Remove unneeded temporary sysroot

[Richard Purdie]

We used to need to build gcc-cross-initial against a bare sysroot to avoid
contamination. With RSS, we no longer need to do this since the recipe sysroot
is already bare. We can therefore simply point at that and drop this code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-devtools/gcc/gcc-cross-initial.inc | 13 +------------
 1 file changed, 1 insertion(+), 12 deletions(-)

diff --git a/meta/recipes-devtools/gcc/gcc-cross-initial.inc b/meta/recipes-devtools/gcc/gcc-cross-initial.inc
index dd35681..5c0208a 100644
--- a/meta/recipes-devtools/gcc/gcc-cross-initial.inc
+++ b/meta/recipes-devtools/gcc/gcc-cross-initial.inc
@@ -24,7 +24,7 @@ EXTRA_OECONF = "\
     --enable-languages=c \
     --program-prefix=${TARGET_PREFIX} \
     --with-sysroot=/not/exist \
-    --with-build-sysroot=${GCCCROSS_BUILDSYSROOT} \
+    --with-build-sysroot=${STAGING_DIR_TARGET} \
     ${EXTRA_OECONF_INITIAL} \
     ${@bb.utils.contains('DISTRO_FEATURES', 'ld-is-gold', '--with-ld=${STAGING_BINDIR_TOOLCHAIN}/${TARGET_PREFIX}ld.bfd', '', d)} \
     ${EXTRA_OECONF_GCC_FLOAT} \
@@ -33,17 +33,6 @@ EXTRA_OECONF = "\
 
 EXTRA_OECONF += "--with-native-system-header-dir=${SYSTEMHEADERS}"
 
-GCCCROSS_BUILDSYSROOT = "${B}/tmpsysroot"
-
-do_configure_prepend () {
-	sysr=${GCCCROSS_BUILDSYSROOT}${target_includedir}
-	mkdir -p $sysr
-	for t in linux asm asm-generic; do
-		rm -f $sysr/$t
-		ln -s ${STAGING_DIR_TARGET}${target_includedir}/$t $sysr/
-	done
-}
-
 do_compile () {
     oe_runmake all-gcc configure-target-libgcc
 }
-- 
2.7.4

[PATCH 4/7] lib/oe/path: Add replace_absolute_symlinks function

[Richard Purdie]

We need a function to walk a directory and replace absolute symlinks with
relative ones. Add such a function to the path module of lib/oe.

It does this relative to the directory being walked for files.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/lib/oe/path.py | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index d468540..a2a50c5 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -50,6 +50,27 @@ def make_relative_symlink(path):
     os.remove(path)
     os.symlink(base, path)
 
+def replace_absolute_symlinks(basedir, d):
+    """
+    Walk basedir looking for absolute symlinks and replacing them with relative ones.
+    The absolute links are assumed to be relative to basedir
+    (compared to make_relative_symlink above which tries to compute common ancestors
+    using pattern matching instead)
+    """
+    for walkroot, dirs, files in os.walk(basedir):
+        for file in files + dirs:
+            path = os.path.join(walkroot, file)
+            if not os.path.islink(path):
+                continue
+            link = os.readlink(path)
+            if not os.path.isabs(link):
+                continue
+            walkdir = os.path.dirname(path.rpartition(basedir)[2])
+            base = os.path.relpath(link, walkdir)
+            bb.debug(2, "Replacing absolute path %s with relative path %s" % (link, base))
+            os.remove(path)
+            os.symlink(base, path)
+
 def format_display(path, metadata):
     """ Prepare a path for display to the user. """
     rel = relative(metadata.getVar("TOPDIR"), path)
-- 
2.7.4

[PATCH 5/7] openssl/fontconfig/bzip2: Use relative symlinks instead of absolute ones (using a new class)

[Richard Purdie]

Absolute path symlinks are a bit of a pain for sstate and the native versions
of these recipes currently contain broken symlinks as a result. There are
only a small number of problematic recipes, at least in OE-Core, namely the
three here.

Rather than trying to make sstate handle this magically, which turns out to
be a harder problem than you'd first realise, simply make the symlinks relative
early in the process and avoid all the problems.

The alternative is adding new complexity to sstate which we could really
do without as without the complexity, you can't always tell where the
absolute symlink is relative to (due to prefixes used for native sstate).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/relative_symlinks.bbclass                | 5 +++++
 meta/recipes-connectivity/openssl/openssl.inc         | 2 +-
 meta/recipes-extended/bzip2/bzip2_1.0.6.bb            | 2 +-
 meta/recipes-graphics/fontconfig/fontconfig_2.12.1.bb | 2 +-
 4 files changed, 8 insertions(+), 3 deletions(-)
 create mode 100644 meta/classes/relative_symlinks.bbclass

diff --git a/meta/classes/relative_symlinks.bbclass b/meta/classes/relative_symlinks.bbclass
new file mode 100644
index 0000000..3157737
--- /dev/null
+++ b/meta/classes/relative_symlinks.bbclass
@@ -0,0 +1,5 @@
+do_install[postfuncs] += "install_relative_symlinks"
+
+python install_relative_symlinks () {
+    oe.path.replace_absolute_symlinks(d.getVar('D'), d)
+}
diff --git a/meta/recipes-connectivity/openssl/openssl.inc b/meta/recipes-connectivity/openssl/openssl.inc
index 5cca019..fc55925 100644
--- a/meta/recipes-connectivity/openssl/openssl.inc
+++ b/meta/recipes-connectivity/openssl/openssl.inc
@@ -30,7 +30,7 @@ export EX_LIBS = "-lgcc -ldl"
 export AS = "${CC} -c"
 EXTRA_OEMAKE = "-e MAKEFLAGS="
 
-inherit pkgconfig siteinfo multilib_header ptest
+inherit pkgconfig siteinfo multilib_header ptest relative_symlinks
 
 PACKAGES =+ "libcrypto libssl ${PN}-misc openssl-conf"
 FILES_libcrypto = "${libdir}/libcrypto${SOLIBS}"
diff --git a/meta/recipes-extended/bzip2/bzip2_1.0.6.bb b/meta/recipes-extended/bzip2/bzip2_1.0.6.bb
index ef7bc89..0512a75 100644
--- a/meta/recipes-extended/bzip2/bzip2_1.0.6.bb
+++ b/meta/recipes-extended/bzip2/bzip2_1.0.6.bb
@@ -25,7 +25,7 @@ PACKAGES =+ "libbz2"
 
 CFLAGS_append = " -fPIC -fpic -Winline -fno-strength-reduce -D_FILE_OFFSET_BITS=64"
 
-inherit autotools update-alternatives ptest
+inherit autotools update-alternatives ptest relative_symlinks
 
 ALTERNATIVE_PRIORITY = "100"
 ALTERNATIVE_${PN} = "bunzip2 bzcat"
diff --git a/meta/recipes-graphics/fontconfig/fontconfig_2.12.1.bb b/meta/recipes-graphics/fontconfig/fontconfig_2.12.1.bb
index 8616154..95b066c 100644
--- a/meta/recipes-graphics/fontconfig/fontconfig_2.12.1.bb
+++ b/meta/recipes-graphics/fontconfig/fontconfig_2.12.1.bb
@@ -37,7 +37,7 @@ RREPLACES_fontconfig-utils = "libfontconfig-utils"
 RCONFLICTS_fontconfig-utils = "libfontconfig-utils"
 DEBIAN_NOAUTONAME_fontconfig-utils = "1"
 
-inherit autotools pkgconfig
+inherit autotools pkgconfig relative_symlinks
 
 FONTCONFIG_CACHE_DIR ?= "${localstatedir}/cache/fontconfig"
 
-- 
2.7.4

[PATCH 6/7] sstate: Make absolute symlinks an error

[Richard Purdie]

The current relocation code is broken, at least in the native case. Fixing it
would mean trying pass in new data on sstate tasks about the relative positioning
of symlinks compared to the sstate relocation paths. Whilst we could do this,
right now I'm favouring making this an error and fixing the small number of
problematic recipes we have in OE-Core (3).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/sstate.bbclass | 42 ++++++++++++------------------------------
 1 file changed, 12 insertions(+), 30 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index ada6fe5..bd9c2ae 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -583,29 +583,6 @@ python sstate_hardcode_path () {
 def sstate_package(ss, d):
     import oe.path
 
-    def make_relative_symlink(path, outputpath, d):
-        # Replace out absolute TMPDIR paths in symlinks with relative ones
-        if not os.path.islink(path):
-            return
-        link = os.readlink(path)
-        if not os.path.isabs(link):
-            return
-        if not link.startswith(tmpdir):
-            return
-
-        #base = os.path.relpath(link, os.path.dirname(path))
-
-        depth = outputpath.rpartition(tmpdir)[2].count('/')
-        base = link.partition(tmpdir)[2].strip()
-        while depth > 1:
-            base = "/.." + base
-            depth -= 1
-        base = "." + base
-
-        bb.debug(2, "Replacing absolute path %s with relative path %s for %s" % (link, base, outputpath))
-        os.remove(path)
-        os.symlink(base, path)
-
     tmpdir = d.getVar('TMPDIR')
 
     sstatebuild = d.expand("${WORKDIR}/sstate-build-%s/" % ss['task'])
@@ -619,15 +596,20 @@ def sstate_package(ss, d):
         if d.getVar('SSTATE_SKIP_CREATION') == '1':
             continue
         srcbase = state[0].rstrip("/").rsplit('/', 1)[0]
+        # Find and error for absolute symlinks. We could attempt to relocate but its not
+        # clear where the symlink is relative to in this context. We could add that markup
+        # to sstate tasks but there aren't many of these so better just avoid them entirely.
         for walkroot, dirs, files in os.walk(state[1]):
-            for file in files:
+            for file in files + dirs:
                 srcpath = os.path.join(walkroot, file)
-                dstpath = srcpath.replace(state[1], state[2])
-                make_relative_symlink(srcpath, dstpath, d)
-            for dir in dirs:
-                srcpath = os.path.join(walkroot, dir)
-                dstpath = srcpath.replace(state[1], state[2])
-                make_relative_symlink(srcpath, dstpath, d)
+                if not os.path.islink(srcpath):
+                    continue
+                link = os.readlink(srcpath)
+                if not os.path.isabs(link):
+                    continue
+                if not link.startswith(tmpdir):
+                    continue
+                bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
         bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
         os.rename(state[1], sstatebuild + state[0])
 
-- 
2.7.4

[PATCH 7/7] nss: Fix nss-native so the checksum doesn't change with BUILD_ARCH

[Richard Purdie]

Switching between 32 and 64 bit BUILD_ARCH shows:

$ bitbake-diffsigs tmp-sstatesamehash*/stamps/*/nss-native/3.27.1-r0.do_compile.sigdata.*
basehash changed from 944cc4554a823ba966aeda0ac3d33b79 to 2475db3659c248d81d0e4dadb3c1b4cd
Variable SITEINFO_BITS value changed from '32' to '64'

We shouldn't have this dependency and it would fail oe-selftest test_sstate_32_64_same_hash
if nss-native were included, therefore exclude it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/recipes-support/nss/nss_3.27.1.bb | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-support/nss/nss_3.27.1.bb b/meta/recipes-support/nss/nss_3.27.1.bb
index 9aad5bd..c89470f 100644
--- a/meta/recipes-support/nss/nss_3.27.1.bb
+++ b/meta/recipes-support/nss/nss_3.27.1.bb
@@ -107,6 +107,7 @@ do_compile() {
         OS_TEST=${OS_TEST} \
         RPATH="${RPATH}"
 }
+do_compile[vardepsexclude] += "SITEINFO_BITS"
 
 
 do_install_prepend_class-nativesdk() {
@@ -180,6 +181,7 @@ do_install() {
         install -m 755 -t ${D}/${bindir} $binary
     done
 }
+do_install[vardepsexclude] += "SITEINFO_BITS"
 
 do_install_append() {
     # Create empty .chk files for the NSS libraries at build time. They could
-- 
2.7.4

✗ patchtest: failure for "glibc-package: Avoid race ssta..." and 6 more

[Patchwork]

== Series Details ==

Series: "glibc-package: Avoid race ssta..." and 6 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/5222/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [5/7] openssl/fontconfig/bzip2: Use relative symlinks instead of absolute ones (using a new class)
 Issue             Commit shortlog is too long [test_shortlog_length] 
  Suggested fix    Edit shortlog so that it is 90 characters or less (currently 92 characters)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe