[OE-core][dunfell 0/3] Patch review

[OE-core][dunfell 0/3] Patch review

[Steve Sakoman]

Please review this final set of patches for the 3.1.8 release and have
comments back by end of day Friday.

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/2172

The following changes since commit cc49591d84d241d90e3dccb3e174ddfd737de311:

  meta/lib/oe/rootfs.py: Fix typo "Restoreing" -> "Restoring" (2021-05-14 07:16:38 -1000)

are available in the Git repository at:

  git://git.openembedded.org/openembedded-core-contrib stable/dunfell-nut
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=stable/dunfell-nut

Richard Purdie (2):
  Revert "cml1.bbclass: Return sorted list of cfg files"
  sstate: Handle manifest 'corruption' issue

Stefan Ghinea (1):
  boost: fix do_fetch failure

 meta/classes/cml1.bbclass                   |  2 +-
 meta/classes/sstate.bbclass                 | 16 +++++++++++++++-
 meta/recipes-support/boost/boost-1.72.0.inc |  2 +-
 3 files changed, 17 insertions(+), 3 deletions(-)

-- 
2.25.1

[OE-core][dunfell 1/3] Revert "cml1.bbclass: Return sorted list of cfg files"

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

This reverts commit dee41e92f0efac7e453597bed4b4c02f867e3aa9.

This patch breaks cases where some config files make changes to earlier ones,
ordering is important. The reproducibility issue in busybox was elsewhere.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ab0a296607b58775e91948ba40956c666dbb1244)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/classes/cml1.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/cml1.bbclass b/meta/classes/cml1.bbclass
index e91452081c..8ab240589a 100644
--- a/meta/classes/cml1.bbclass
+++ b/meta/classes/cml1.bbclass
@@ -6,7 +6,7 @@ def find_cfgs(d):
         if s.endswith('.cfg'):
             sources_list.append(s)
 
-    return sorted(sources_list)
+    return sources_list
 
 cml1_do_configure() {
 	set -e
-- 
2.25.1

[OE-core][dunfell 2/3] boost: fix do_fetch failure

[Steve Sakoman]

From: Stefan Ghinea <stefan.ghinea@windriver.com>

Bintray service has been discontinued causing boost do_fetch to fail:
WARNING: boost-1.72.0-r0 do_fetch: Failed to fetch URL
https://dl.bintray.com/boostorg/release/1.76.0/source/boost_1_72_0.tar.bz2,
attempting MIRRORS if available

Signed-off-by: Stefan Ghinea <stefan.ghinea@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6a76da15ece9d27fca20ace12db4978092e042b7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/recipes-support/boost/boost-1.72.0.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-support/boost/boost-1.72.0.inc b/meta/recipes-support/boost/boost-1.72.0.inc
index 55a095bf1c..d152895f09 100644
--- a/meta/recipes-support/boost/boost-1.72.0.inc
+++ b/meta/recipes-support/boost/boost-1.72.0.inc
@@ -11,7 +11,7 @@ BOOST_VER = "${@"_".join(d.getVar("PV").split("."))}"
 BOOST_MAJ = "${@"_".join(d.getVar("PV").split(".")[0:2])}"
 BOOST_P = "boost_${BOOST_VER}"
 
-SRC_URI = "https://dl.bintray.com/boostorg/release/${PV}/source/${BOOST_P}.tar.bz2"
+SRC_URI = "https://boostorg.jfrog.io/artifactory/main/release/${PV}/source/${BOOST_P}.tar.bz2"
 SRC_URI[md5sum] = "cb40943d2a2cb8ce08d42bc48b0f84f0"
 SRC_URI[sha256sum] = "59c9b274bc451cf91a9ba1dd2c7fdcaf5d60b1b3aa83f2c9fa143417cc660722"
 
-- 
2.25.1

[OE-core][dunfell 3/3] sstate: Handle manifest 'corruption' issue

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Under certain build patterns, warnings about missing manifests can appear. These
are real issues where the manifest was removed and shouldn't have been.

Martin Jansa was able to find a reproducer of:

MACHINE=qemux86 bitbake zlib-native
echo 'PR = "r1"' >> meta/recipes-core/zlib/zlib_1.2.11.bb
MACHINE=qemux86-64 bitbake zlib-native
MACHINE=qemux86 bitbake zlib-native
<the zlib-native manifest is now removed along with the sysroot-components contents>

The code maintains a per machine list of stamps but a per PACAGE_ARCH list of
stamp/manifest/workdir mappings. The latter is only appended to for speed with
the assumption that once stamps are gone, the code wouldn't trigger.

The code only ever appends to the mapping list (for speed/efficency under lock)
meaning that multiple entries can result where the stamp/workdir differs due to
version changes but the manifest remains the same.

By switching MACHINE part way through the build, the older stamp is referenced
and the manifest is incorrectly removed as it matches an now obsolete entry in
the mapping file.

There are two possible fixes, one is to rewrite the mapping file every time
which means adding regexs, iterating and generally complicating that code. The
second option is to only use the last mapping entry in the file for a given
manifest and ignore any earlier ones. This patch implments the latter.

Also drop the stale entries if we are rewriting it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 63da9a4f889c5b0e41bc8ec08abe0acea1546479)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 meta/classes/sstate.bbclass | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index a8e169a10b..3c89c35ecf 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -319,6 +319,8 @@ def sstate_install(ss, d):
     if os.path.exists(i):
         with open(i, "r") as f:
             manifests = f.readlines()
+    # We append new entries, we don't remove older entries which may have the same
+    # manifest name but different versions from stamp/workdir. See below.
     if filedata not in manifests:
         with open(i, "a+") as f:
             f.write(filedata)
@@ -1175,11 +1177,21 @@ python sstate_eventhandler2() {
         i = d.expand("${SSTATE_MANIFESTS}/index-" + a)
         if not os.path.exists(i):
             continue
+        manseen = set()
+        ignore = []
         with open(i, "r") as f:
             lines = f.readlines()
-            for l in lines:
+            for l in reversed(lines):
                 try:
                     (stamp, manifest, workdir) = l.split()
+                    # The index may have multiple entries for the same manifest as the code above only appends
+                    # new entries and there may be an entry with matching manifest but differing version in stamp/workdir.
+                    # The last entry in the list is the valid one, any earlier entries with matching manifests
+                    # should be ignored.
+                    if manifest in manseen:
+                        ignore.append(l)
+                        continue
+                    manseen.add(manifest)
                     if stamp not in stamps and stamp not in preservestamps and stamp in machineindex:
                         toremove.append(l)
                         if stamp not in seen:
@@ -1210,6 +1222,8 @@ python sstate_eventhandler2() {
 
         with open(i, "w") as f:
             for l in lines:
+                if l in ignore:
+                    continue
                 f.write(l)
     machineindex |= set(stamps)
     with open(mi, "w") as f:
-- 
2.25.1