[PATCH 00/12 V2] Fixes for multilib and eSDK

[PATCH 00/12 V2] Fixes for multilib and eSDK

[Robert Yang]

* V2
  + Drop already merged patches

  + Fix RP's comments:
    - deltask populate_sdk and populate_sdk_ext for multilib
    - Drop "image.bbclass: remove depmodwrapper-cross from DEPENDS"
    - sstate.bbclass: sstate_hardcode_path(): fix for multilib
      > fix the grep command

* V1
  - Initial version

// Robert

The following changes since commit e9dfe7eb7f61b909ae7d034e80cfbebc1fad018b:

  icu-dbg: improve reproducibility (2018-01-08 08:45:33 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib oem1
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=oem1

Robert Yang (12):
  staging.bbclass: fix for multilib
  populate_sdk_ext.bbclass: fix for multilib
  sstate.bbclass: sstate_hardcode_path(): fix for multilib
  staging.bbclass: staging_populate_sysroot_dir(): fix for multilib
  staging.bbclass: print searched manifest when not found
  staging.bbclass: extend_recipe_sysroot(): search multilib manifest
  package_manager.py: reverse archs correctly
  multilib.bbclass: remove invalid PACKAGE_INSTALL
  multilib.bbclass: extend allarch recipes
  toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for
    image
  multilib.bbclass: deltask populate_sdk and populate_sdk_ext
  populate_sdk_ext.bbclass: remove the try...finally

 meta/classes/allarch.bbclass           |   3 -
 meta/classes/multilib.bbclass          |   8 +-
 meta/classes/multilib_global.bbclass   |   4 +-
 meta/classes/package.bbclass           |   9 ---
 meta/classes/populate_sdk_ext.bbclass  | 142 +++++++++++++++++++--------------
 meta/classes/sstate.bbclass            |   8 +-
 meta/classes/staging.bbclass           |  40 +++++++++-
 meta/classes/toolchain-scripts.bbclass |   6 ++
 meta/lib/oe/package_manager.py         |  57 +++++++++++--
 9 files changed, 186 insertions(+), 91 deletions(-)

-- 
2.7.4

[PATCH 01/12] staging.bbclass: fix for multilib

[Robert Yang]

Fixed:
MACHINE = "qemux86-64"
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake <image> -cpopulate_sdk_ext
[snip]
Exception: subprocess.CalledProcessError: Command 'sed -e [snip]'
[snip]

Subprocess output:
[snip]
sed: can't read /path/to/work/qemux86_64-wrs-linux/wrlinux-image-glibc-small/1.0-r1/recipe-sysroot/usr/lib/perl/5.24.1/ExtUtils/Liblist/Kid.pm: No such file or directory
[snip]

It was failed because "/usr/lib" is in qemux86_64-wrsmllib32-linux/, not in
qemux86_64-wrs-linux. The code has considered mutitlib, but seems not
completed, the multilib "variant" was not in fixme, so it wasn't handled
correctly, this patch fixes the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/staging.bbclass | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 1b9e84d..412e269 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -480,6 +480,13 @@ python extend_recipe_sysroot() {
                     multilibs[variant] = get_multilib_datastore(variant, d)
                 d2 = multilibs[variant]
                 destsysroot = d2.getVar("RECIPE_SYSROOT")
+                if variant not in fixme:
+                    fixme[variant] = []
+            # Clear variant when variant == current_variant since we
+            # don't need handle it as multilib in this case, just handle
+            # it as normal is OK.
+            else:
+                variant = ''
 
         native = False
         if c.endswith("-native"):
@@ -508,6 +515,9 @@ python extend_recipe_sysroot() {
             if native:
                 fm = fixme['native']
                 targetdir = recipesysrootnative
+            elif variant:
+                fm = fixme[variant]
+                targetdir = destsysroot
             else:
                 fm = fixme['']
                 targetdir = destsysroot
-- 
2.7.4

[PATCH 02/12] populate_sdk_ext.bbclass: fix for multilib

[Robert Yang]

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake <image> -cpopulate_sdk_ext
$ bitbake <image> -ctestsdkext
[snip]
Standard Output: /bin/sh: 1: i686-wrsmllib32-linux-gcc: not found
[snip]

It was failed because no lib32 toolchain or lib installed.

This patch fixes:
* Set SDK_TARGETS correctly
* Return multilib depends in get_ext_sdk_depends()
* Write information to all environment-setup-* scripts.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 72 +++++++++++++++++++++++------------
 1 file changed, 47 insertions(+), 25 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index eabc365..78cdfab 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -38,7 +38,7 @@ SDK_LOCAL_CONF_BLACKLIST ?= "CONF_VERSION \
 SDK_INHERIT_BLACKLIST ?= "buildhistory icecc"
 SDK_UPDATE_URL ?= ""
 
-SDK_TARGETS ?= "${PN}"
+SDK_TARGETS ?= "${@multilib_pkg_extend(d, d.getVar('BPN', True))}"
 
 def get_sdk_install_targets(d, images_only=False):
     sdk_install_targets = ''
@@ -55,9 +55,11 @@ def get_sdk_install_targets(d, images_only=False):
 
     if not images_only:
         if d.getVar('SDK_INCLUDE_PKGDATA') == '1':
-            sdk_install_targets += ' meta-world-pkgdata:do_allpackagedata'
+            for pn in multilib_pkg_extend(d, 'meta-world-pkgdata').split():
+                sdk_install_targets += ' %s:do_allpackagedata' % pn
         if d.getVar('SDK_INCLUDE_TOOLCHAIN') == '1':
-            sdk_install_targets += ' meta-extsdk-toolchain:do_populate_sysroot'
+            for pn in multilib_pkg_extend(d, 'meta-extsdk-toolchain').split():
+                sdk_install_targets += ' %s:do_populate_sysroot' % pn
 
     return sdk_install_targets
 
@@ -601,40 +603,55 @@ SDK_PRE_INSTALL_COMMAND_task-populate-sdk-ext = "${sdk_ext_preinst}"
 sdk_ext_postinst() {
 	printf "\nExtracting buildtools...\n"
 	cd $target_sdk_dir
-	env_setup_script="$target_sdk_dir/environment-setup-${REAL_MULTIMACH_TARGET_SYS}"
-	printf "buildtools\ny" | ./${SDK_BUILDTOOLS_INSTALLER} > buildtools.log || { printf 'ERROR: buildtools installation failed:\n' ; cat buildtools.log ; echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
+	env_setup_scripts="`ls $target_sdk_dir/environment-setup-*`"
+	./${SDK_BUILDTOOLS_INSTALLER} -d buildtools -y > buildtools.log
+	if [ $? -ne 0 ]; then
+		echo 'ERROR: buildtools installation failed:'
+		cat buildtools.log
+		for e in $env_setup_scripts; do
+			echo "echo 'ERROR: this SDK was not fully installed and needs reinstalling'" >> $e
+		done
+		exit 1
+	fi
 
 	# Delete the buildtools tar file since it won't be used again
 	rm -f ./${SDK_BUILDTOOLS_INSTALLER}
 	# We don't need the log either since it succeeded
 	rm -f buildtools.log
 
-	# Make sure when the user sets up the environment, they also get
-	# the buildtools-tarball tools in their path.
-	echo ". $target_sdk_dir/buildtools/environment-setup*" >> $env_setup_script
-
-	# Allow bitbake environment setup to be ran as part of this sdk.
-	echo "export OE_SKIP_SDK_CHECK=1" >> $env_setup_script
-	# Work around runqemu not knowing how to get this information within the eSDK
-	echo "export DEPLOY_DIR_IMAGE=$target_sdk_dir/tmp/${@os.path.relpath(d.getVar('DEPLOY_DIR_IMAGE'), d.getVar('TMPDIR'))}" >> $env_setup_script
+	for e in $env_setup_scripts; do
+		# Make sure when the user sets up the environment, they also get
+		# the buildtools-tarball tools in their path.
+		echo ". $target_sdk_dir/buildtools/environment-setup*" >> $e
 
-	# A bit of another hack, but we need this in the path only for devtool
-	# so put it at the end of $PATH.
-	echo "export PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $env_setup_script
+		# Allow bitbake environment setup to be ran as part of this sdk.
+		echo "export OE_SKIP_SDK_CHECK=1" >> $e
 
-	echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script
+		# Work around runqemu not knowing how to get this information within the eSDK
+		echo "export DEPLOY_DIR_IMAGE=$target_sdk_dir/tmp/${@os.path.relpath(d.getVar('DEPLOY_DIR_IMAGE'), d.getVar('TMPDIR'))}" >> $e
 
-	# Warn if trying to use external bitbake and the ext SDK together
-	echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $env_setup_script
+		# A bit of another hack, but we need this in the path only for devtool
+		# so put it at the end of $PATH.
+		echo "export PATH=$target_sdk_dir/sysroots/${SDK_SYS}${bindir_nativesdk}:\$PATH" >> $e
+		echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $e
+		# Warn if trying to use external bitbake and the ext SDK together
+		echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $e
+	done
 
 	if [ "$prepare_buildsystem" != "no" ]; then
-		printf "Preparing build system...\n"
+		echo "Preparing build system..."
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
 		LOGFILE="$target_sdk_dir/preparing_build_system.log"
-		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'" || { echo "printf 'ERROR: this SDK was not fully installed and needs reinstalling\n'" >> $env_setup_script ; exit 1 ; }
-		rm $target_sdk_dir/ext-sdk-prepare.py
+		sh -c ". buildtools/environment-setup* > $LOGFILE && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> $LOGFILE && python $target_sdk_dir/ext-sdk-prepare.py $LOGFILE '${SDK_INSTALL_TARGETS}'"
+		if [ $? -ne 0 ]; then
+			for e in $env_setup_scripts; do
+				echo "echo 'ERROR: this SDK was not fully installed and needs reinstalling'" >> $e
+			done
+			exit 1
+		fi
+		rm -f $target_sdk_dir/ext-sdk-prepare.py
 	fi
 	echo done
 }
@@ -662,12 +679,17 @@ fakeroot python do_populate_sdk_ext() {
 def get_ext_sdk_depends(d):
     # Note: the deps varflag is a list not a string, so we need to specify expand=False
     deps = d.getVarFlag('do_image_complete', 'deps', False)
-    pn = d.getVar('PN')
-    deplist = ['%s:%s' % (pn, dep) for dep in deps]
+    deplist = []
+    for pn in multilib_pkg_extend(d, d.getVar('BPN')).split():
+        deplist += ['%s:%s' % (pn, dep) for dep in deps]
     tasklist = bb.build.tasksbetween('do_image_complete', 'do_build', d)
     tasklist.append('do_rootfs')
     for task in tasklist:
         deplist.extend((d.getVarFlag(task, 'depends') or '').split())
+
+    for pn in multilib_pkg_extend(d, 'meta-extsdk-toolchain').split():
+         deplist += ['%s:do_populate_sysroot' % pn]
+
     return ' '.join(deplist)
 
 python do_sdk_depends() {
@@ -681,7 +703,7 @@ python do_sdk_depends() {
 addtask sdk_depends
 
 do_sdk_depends[dirs] = "${WORKDIR}"
-do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)} meta-extsdk-toolchain:do_populate_sysroot"
+do_sdk_depends[depends] = "${@get_ext_sdk_depends(d)}"
 do_sdk_depends[recrdeptask] = "${@d.getVarFlag('do_populate_sdk', 'recrdeptask', False)}"
 do_sdk_depends[recrdeptask] += "do_populate_lic do_package_qa do_populate_sysroot do_deploy ${SDK_RECRDEP_TASKS}"
 do_sdk_depends[rdepends] = "${@get_sdk_ext_rdepends(d)}"
-- 
2.7.4

[PATCH 03/12] sstate.bbclass: sstate_hardcode_path(): fix for multilib

[Robert Yang]

It only substituted staging_target for target recipe which didn't work
for multilib, for example, postinst-useradd-lib32-polkit:

* No multilib:
  PATH=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot-native/bin
  staging_target=/path/to/tmp-glibc/work/core2-64-wrs-linux/polkit/0.113-r0/recipe-sysroot
  The PATH would be substituted to:
  FIXMESTAGINGDIRTARGET-native/bin
  Not the funny "-native/bin", this works well.

* When multilib:
  PATH=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/recipe-sysroot-native/bin
  staging_target=/path/to/tmp-glibc/work/core2-32-wrsmllib32-linux/lib32-polkit/0.113-r0/lib32-recipe-sysroot
  Now staging_target endswith "/lib32-recipe-sysroot", so it can't
  replace '/recipe-sysroot-native' in PATH , so PATH can't be fixed, and
  there would be build errors when building multilib + rm_work, for
  example:
  chown: invalid user: ‘polkitd:root’

Substitute staging_host for target recipe can fix the problem, now all
of native, cross and target need substitute staging_host, so we can
simply the code a little.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/sstate.bbclass | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 6e41b56..2b25d27 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -538,15 +538,15 @@ python sstate_hardcode_path () {
     staging_host = d.getVar('RECIPE_SYSROOT_NATIVE')
     sstate_builddir = d.getVar('SSTATE_BUILDDIR')
 
+    sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRHOST:g'" % staging_host
     if bb.data.inherits_class('native', d) or bb.data.inherits_class('cross-canadian', d):
         sstate_grep_cmd = "grep -l -e '%s'" % (staging_host)
-        sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRHOST:g'" % (staging_host)
     elif bb.data.inherits_class('cross', d) or bb.data.inherits_class('crosssdk', d):
         sstate_grep_cmd = "grep -l -e '%s' -e '%s'" % (staging_target, staging_host)
-        sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRTARGET:g; s:%s:FIXMESTAGINGDIRHOST:g'" % (staging_target, staging_host)
+        sstate_sed_cmd += " -e 's:%s:FIXMESTAGINGDIRTARGET:g'" % staging_target
     else:
-        sstate_grep_cmd = "grep -l -e '%s'" % (staging_target)
-        sstate_sed_cmd = "sed -i -e 's:%s:FIXMESTAGINGDIRTARGET:g'" % (staging_target)
+        sstate_grep_cmd = "grep -l -e '%s' -e '%s'" % (staging_target, staging_host)
+        sstate_sed_cmd += " -e 's:%s:FIXMESTAGINGDIRTARGET:g'" % staging_target
 
     extra_staging_fixmes = d.getVar('EXTRA_STAGING_FIXMES') or ''
     for fixmevar in extra_staging_fixmes.split():
-- 
2.7.4

[PATCH 04/12] staging.bbclass: staging_populate_sysroot_dir(): fix for multilib

[Robert Yang]

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake core-image-sato lib32-core-image-sato
$ bitbake build-sysroots

All lib32 manifests which had been built should be installed, but only a few
such as qemux86_64-lib32-base-files were installed (it was installed because
MACHINE_ARCH was still qemux86-64 when multilib), but others such as
x86-lib32-zlib were not installed, this was incorrect. For multilib builds,
fix-up overrides to prepend :virtclass-multilib- and then again append
additional multilib arches from the PACKAGE_EXTRA_ARCHS list if needed can fix
the problem.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/staging.bbclass | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 412e269..455eb05 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -190,8 +190,23 @@ def staging_populate_sysroot_dir(targetsysroot, nativesysroot, native, d):
         pkgarchs = ['${MACHINE_ARCH}']
         pkgarchs = pkgarchs + list(reversed(d.getVar("PACKAGE_EXTRA_ARCHS").split()))
         pkgarchs.append('allarch')
+
+        # Handle multilib archs
+        variants = d.getVar("MULTILIB_VARIANTS", True) or ""
+        for variant in variants.split():
+            localdata = bb.data.createCopy(d)
+            overrides = localdata.getVar("OVERRIDES", False) + ":virtclass-multilib-" + variant
+            localdata.setVar("OVERRIDES", overrides)
+            bb.data.update_data(localdata)
+            pkgarchs_ml = localdata.getVar('PACKAGE_EXTRA_ARCHS').split()
+            for arch in pkgarchs_ml:
+                if arch not in pkgarchs:
+                    pkgarchs.append(arch)
+
         targetdir = targetsysroot
 
+    bb.debug(1, 'pkgarchs: %s' % pkgarchs)
+
     bb.utils.mkdirhier(targetdir)
     for pkgarch in pkgarchs:
         for manifest in glob.glob(d.expand("${SSTATE_MANIFESTS}/manifest-%s-*.populate_sysroot" % pkgarch)):
-- 
2.7.4

[PATCH 05/12] staging.bbclass: print searched manifest when not found

[Robert Yang]

The old warning was:
WARNING: lib32-wrlinux-image-glibc-small-1.0-r1 do_prepare_recipe_sysroot: Manifest /path/to/tmp/sstate-control/manifest-allarch-lib32-qemuwrapper-cross.populate_sysroot not found?

The message wasn't clear enough, it searched a few manifests, but only
reported the last one, which confused user.

Now the warning is:
WARNING: lib32-wrlinux-image-glibc-small-1.0-r1 do_rootfs: Manifest for lib32-qemuwrapper-cross not found, searched manifests:
/path/to/tmp/sstate-control/manifest-qemux86_64-lib32-qemuwrapper-cross.populate_sysroot
/path/to/tmp/sstate-control/manifest-core2-64-lib32-qemuwrapper-cross.populate_sysroot
/path/to/tmp/sstate-control/manifest-x86_64-lib32-qemuwrapper-cross.populate_sysroot
/path/to/tmp/sstate-control/manifest-allarch-lib32-qemuwrapper-cross.populate_sysroot

Whick makes debug easier.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/staging.bbclass | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 455eb05..75bec84 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -504,6 +504,7 @@ python extend_recipe_sysroot() {
                 variant = ''
 
         native = False
+        searched_manifest = []
         if c.endswith("-native"):
             manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-${BUILD_ARCH}-%s.populate_sysroot" % c)
             native = True
@@ -523,8 +524,11 @@ python extend_recipe_sysroot() {
                 manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.populate_sysroot" % (pkgarch, c))
                 if os.path.exists(manifest):
                     break
+                searched_manifest.append(manifest)
         if not os.path.exists(manifest):
-            bb.warn("Manifest %s not found?" % manifest)
+            if not searched_manifest:
+                searched_manifest.append(manifest)
+            bb.warn("Manifest for %s not found, searched manifests:\n%s" % (c, '\n'.join(searched_manifest)))
         else:
             newmanifest = collections.OrderedDict()
             if native:
-- 
2.7.4

[PATCH 06/12] staging.bbclass: extend_recipe_sysroot(): search multilib manifest

[Robert Yang]

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32"

$ bitbake lib32-wrlinux-image-glibc-small
WARNING: lib32-wrlinux-image-glibc-small-1.0-r1 do_rootfs: Manifest for lib32-qemuwrapper-cross not found, searched manifests:
/path/to/tmp/sstate-control/manifest-qemux86_64-lib32-qemuwrapper-cross.populate_sysroot
/path/to/tmp/sstate-control/manifest-core2-64-lib32-qemuwrapper-cross.populate_sysroot
/path/to/tmp/sstate-control/manifest-x86_64-lib32-qemuwrapper-cross.populate_sysroot
/path/to/tmp/sstate-control/manifest-allarch-lib32-qemuwrapper-cross.populate_sysroot

It was because image recipes' multilib PACKAGE_EXTRA_ARCHS are not changed by
mutlilib.bbclass, so extend_recipe_sysroot() didn't know anything about
mutlilib's PACKAGE_EXTRA_ARCHS, now fix it.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/staging.bbclass | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/meta/classes/staging.bbclass b/meta/classes/staging.bbclass
index 75bec84..91ae1e8 100644
--- a/meta/classes/staging.bbclass
+++ b/meta/classes/staging.bbclass
@@ -519,6 +519,15 @@ python extend_recipe_sysroot() {
         else:
             pkgarchs = ['${MACHINE_ARCH}']
             pkgarchs = pkgarchs + list(reversed(d2.getVar("PACKAGE_EXTRA_ARCHS").split()))
+            # Search multilib archs for multilib image recipe like lib32-core-image-minimal
+            mlprefix = d2.getVar('MLPREFIX')
+            if bb.data.inherits_class('image', d2) and mlprefix:
+                ml_variant = mlprefix.rstrip('-')
+                override = ":virtclass-multilib-" + ml_variant
+                localdata = bb.data.createCopy(d2)
+                overrides = localdata.getVar("OVERRIDES", False) + ":virtclass-multilib-" + ml_variant
+                localdata.setVar("OVERRIDES", overrides)
+                pkgarchs = pkgarchs + list(reversed(localdata.getVar("PACKAGE_EXTRA_ARCHS").split()))
             pkgarchs.append('allarch')
             for pkgarch in pkgarchs:
                 manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.populate_sysroot" % (pkgarch, c))
-- 
2.7.4

[PATCH 07/12] package_manager.py: reverse archs correctly

[Robert Yang]

It had reversed all the archs, which mixed multilib and common archs, e.g.:
"all any noarch x86_64 core2-64 qemux86_64 x86 i586 core2-32"
After reversed:
"core2-32 i586 x86 qemux86_64 core2-64 x86_64 noarch any all"
The core2-32 has a higher priority than core2-64 after reversed which is
incorrect. Don't mix with mulitlib when reverse can fix the problem, and let
multilib archs have a higher priority for multilib image.

Fixed:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32"
IMAGE_INSTALL_append = " ca-certificates"

$ bitbake wrlinux-image-glibc-small
Check rootfs.manifest, no core_32 packages is installed, this is
correct, but after we build lib32-bash, it will be incorrect:
$ bitbake lib32-bash
$ bitbake wrlinux-image-glibc-small
Check rootfs.manifest, a few lib32 packages are installed, such as
lib32-bash, this is incorrect. It was because ca-certificates is
allarch, and requires /bin/sh, which is provided by both bash and
lib32-bash, and lib32-bash has a higher priority than bash, so it would
be installed.

[YOCTO #12288]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/lib/oe/package_manager.py | 57 ++++++++++++++++++++++++++++++++++++++----
 1 file changed, 52 insertions(+), 5 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index ea99165..3ef6698 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -467,10 +467,10 @@ class RpmPM(PackageManager):
         self.target_rootfs = target_rootfs
         self.target_vendor = target_vendor
         self.task_name = task_name
-        if arch_var == None:
-            self.archs = self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS').replace("-","_")
-        else:
-            self.archs = self.d.getVar(arch_var).replace("-","_")
+
+        self.archs = self._determine_archs(arch_var)
+        bb.note("The archs used by package manager: %s" % self.archs)
+
         if task_name == "host":
             self.primary_arch = self.d.getVar('SDK_ARCH')
         else:
@@ -489,9 +489,56 @@ class RpmPM(PackageManager):
         if not os.path.exists(self.d.expand('${T}/saved')):
             bb.utils.mkdirhier(self.d.expand('${T}/saved'))
 
+    def _determine_archs(self, arch_var):
+        if arch_var:
+            archs = self.d.getVar(arch_var).replace("-","_").split()
+        else:
+            archs = self.d.getVar('ALL_MULTILIB_PACKAGE_ARCHS').replace("-","_").split()
+        # Reverse archs to ensure the -best- match is listed firstly, but need
+        # make sure not mixed with multilib archs, and let multilib archs have
+        # higher a priority for multilib image:
+        # For non-multilib image: (e.g., core-image-minimal)
+        # machine_arch:package_archs:multilib_archs:allarch_archs
+        # For multilib image: (e.g., lib32-core-image-minimal)
+        # machine_arch:multilib_archs:package_archs:allarch_archs
+        package_archs_var = self.d.getVar('PACKAGE_ARCHS').replace("-","_").split()
+        machine_arch_var = [self.d.getVar('MACHINE_ARCH').replace("-","_")]
+        package_archs = []
+        allarch_archs = []
+        ml_archs = []
+        machine_arch = []
+        for arch in archs:
+            if arch in machine_arch_var:
+                bb.note("Found MACHINE_ARCH: %s" % arch)
+                machine_arch.append(arch)
+            elif arch in package_archs_var:
+                bb.note("Found PACKAGE_ARCH: %s" % arch)
+                package_archs.append(arch)
+            elif arch in 'all any noarch'.split():
+                bb.note("Found allarch: %s" % arch)
+                allarch_archs.append(arch)
+            else:
+                bb.note("Found multilib arch: %s" % arch)
+                ml_archs.append(arch)
+        package_archs.reverse()
+        if machine_arch:
+            archs = machine_arch
+        mlprefix = self.d.getVar('MLPREFIX')
+        if ml_archs:
+            ml_archs.reverse()
+            if mlprefix:
+                archs += ml_archs + package_archs
+            else:
+                archs += package_archs + ml_archs
+        else:
+            archs += package_archs
+
+        archs += allarch_archs
+        return ' '.join(archs)
+
     def _configure_dnf(self):
         # libsolv handles 'noarch' internally, we don't need to specify it explicitly
-        archs = [i for i in reversed(self.archs.split()) if i not in ["any", "all", "noarch"]]
+        archs = [i for i in self.archs.split() if i not in ["any", "all", "noarch"]]
         # This prevents accidental matching against libsolv's built-in policies
         if len(archs) <= 1:
             archs = archs + ["bogusarch"]
-- 
2.7.4

[PATCH 08/12] multilib.bbclass: remove invalid PACKAGE_INSTALL

[Robert Yang]

The PACKAGE_INSTALL is only used by image recipe, the previous code had
handled it in "if bb.data.inherits_class('image', d)", handle it again
doesn't make any sense (there is no PACKAGE_INSTALL for non-image
recipe), so remove it.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/multilib.bbclass | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index 0570ea8..dc0868b 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -112,7 +112,6 @@ python __anonymous () {
 
     clsextend.map_packagevars()
     clsextend.map_regexp_variable("PACKAGES_DYNAMIC")
-    clsextend.map_variable("PACKAGE_INSTALL")
     clsextend.map_variable("INITSCRIPT_PACKAGES")
     clsextend.map_variable("USERADD_PACKAGES")
     clsextend.map_variable("SYSTEMD_PACKAGES")
-- 
2.7.4

[PATCH 09/12] multilib.bbclass: extend allarch recipes

[Robert Yang]

This can fix do_rootfs and depends chaos when multilib, for example
ca-certificates is an allarch recipe:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32"
IMAGE_INSTALL_append = " ca-certificates"
$ bitbake ca-certificates
$ rpm -qpR tmp-glibc/deploy/rpm/noarch/ca-certificates-20170717-r0.noarch.rpm
[snip]
openssl
[snip]

It requires openssl, but lib32-openssl doesn't proivide it:
$ rpm -qp --provides tmp-glibc/deploy/rpm/core2_32/lib32-openssl-1.0.2l-r0.core2_32.rpm
lib32-openssl = 1.0.2l-r0

So openssl-1.0.2l-r0.core2_64.rpm would be installed when install
ca-certificates, e.g.:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32"
IMAGE_INSTALL_append = " ca-certificates"
$ bitbake lib32-core-image-minimal
The openssl-1.0.2l-r0.core2_64.rpm and other 64 bit packages are installed,
this is incorrect.

We have two solutions on this:
1) Let lib32-openssl provide openssl, but this would cause more problems when
   building since more than one providers for openssl.

Or

2) Extend allarch recipes, there would be ca-certificates-20170717-r0.noarch.rpm
   and lib32-ca-certificates-20170717-r0.noarch.rpm, then ca-certificates
   requires openssl, and lib32-ca-certificates requires lib32-openssl, the
   packages will be installed correctly, this is the best solution that I can
   find.

The problem has existed for years for deb and ipk (rpm did work since it didn't
use lib32-BPN in rpm filename before changed to dnf, for example, there was no
lib32-openssl.core2_32.rpm, but openssl.core2_32.rpm), this patch can fix the
problem for rpm, ipk and deb.

[YOCTO #12288]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/allarch.bbclass         | 3 ---
 meta/classes/multilib.bbclass        | 4 ----
 meta/classes/multilib_global.bbclass | 4 +---
 meta/classes/package.bbclass         | 9 ---------
 4 files changed, 1 insertion(+), 19 deletions(-)

diff --git a/meta/classes/allarch.bbclass b/meta/classes/allarch.bbclass
index 51ba509..b4f694d 100644
--- a/meta/classes/allarch.bbclass
+++ b/meta/classes/allarch.bbclass
@@ -42,9 +42,6 @@ python () {
         d.setVar("INHIBIT_PACKAGE_DEBUG_SPLIT", "1")
         d.setVar("INHIBIT_PACKAGE_STRIP", "1")
 
-        # These multilib values shouldn't change allarch packages so exclude them
-        d.appendVarFlag("emit_pkgdata", "vardepsexclude", " MULTILIB_VARIANTS")
-        d.appendVarFlag("write_specfile", "vardepsexclude", " MULTILIBS")
     elif bb.data.inherits_class('packagegroup', d) and not bb.data.inherits_class('nativesdk', d):
         bb.error("Please ensure recipe %s sets PACKAGE_ARCH before inherit packagegroup" % d.getVar("FILE"))
 }
diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index dc0868b..770863f 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -46,10 +46,6 @@ python multilib_virtclass_handler () {
     if bb.data.inherits_class('nativesdk', e.data) or bb.data.inherits_class('crosssdk', e.data):
         raise bb.parse.SkipPackage("We can't extend nativesdk recipes")
 
-    if bb.data.inherits_class('allarch', e.data) and not bb.data.inherits_class('packagegroup', e.data):
-        raise bb.parse.SkipPackage("Don't extend allarch recipes which are not packagegroups")
-
-
     # Expand this since this won't work correctly once we set a multilib into place
     e.data.setVar("ALL_MULTILIB_PACKAGE_ARCHS", e.data.getVar("ALL_MULTILIB_PACKAGE_ARCHS"))
  
diff --git a/meta/classes/multilib_global.bbclass b/meta/classes/multilib_global.bbclass
index d2ec1ad..1bb6242 100644
--- a/meta/classes/multilib_global.bbclass
+++ b/meta/classes/multilib_global.bbclass
@@ -165,9 +165,7 @@ python multilib_virtclass_handler_global () {
         return
 
     if bb.data.inherits_class('kernel', e.data) or \
-            bb.data.inherits_class('module-base', e.data) or \
-            (bb.data.inherits_class('allarch', e.data) and\
-             not bb.data.inherits_class('packagegroup', e.data)):
+            bb.data.inherits_class('module-base', e.data):
             variants = (e.data.getVar("MULTILIB_VARIANTS") or "").split()
 
             import oe.classextend
diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7dc7596..ea0217c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -475,9 +475,6 @@ def get_package_mapping (pkg, basepkg, d):
     key = "PKG_%s" % pkg
 
     if key in data:
-        # Have to avoid undoing the write_extra_pkgs(global_variants...)
-        if bb.data.inherits_class('allarch', d) and data[key] == basepkg:
-            return pkg
         return data[key]
 
     return pkg
@@ -1368,9 +1365,6 @@ python emit_pkgdata() {
     if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d):
         write_extra_pkgs(variants, pn, packages, pkgdatadir)
 
-    if (bb.data.inherits_class('allarch', d) and not bb.data.inherits_class('packagegroup', d)):
-        write_extra_pkgs(global_variants, pn, packages, pkgdatadir)
-
     workdir = d.getVar('WORKDIR')
 
     for pkg in packages.split():
@@ -1455,9 +1449,6 @@ python emit_pkgdata() {
     if bb.data.inherits_class('kernel', d) or bb.data.inherits_class('module-base', d):
         write_extra_runtime_pkgs(variants, packages, pkgdatadir)
 
-    if bb.data.inherits_class('allarch', d) and not bb.data.inherits_class('packagegroup', d):
-        write_extra_runtime_pkgs(global_variants, packages, pkgdatadir)
-
     bb.utils.unlockfile(lf)
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
-- 
2.7.4

[PATCH 10/12] toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for image

[Robert Yang]

Install all MULTILIB_VARIANTS to sysroot and pack them only makes sense to
image recipe, so limit it to image recipe only, and it can fix:
MACHINE = "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "core2-32"

$ bitbake lib32-meta-ide-support
WARNING: lib32-meta-ide-support-1.0-r3 do_configure: Manifest for ncurses not found, searched manifests:
/workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-qemux86_64-ncurses.populate_sysroot
/workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-core2-32-ncurses.populate_sysroot
/workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-i586-ncurses.populate_sysroot
/workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-x86-ncurses.populate_sysroot
/workspace2/lyang1/test_mlwarn/tmp-glibc/sstate-control/manifest-allarch-ncurses.populate_sysroot

This is because core2-64 is not in lib32-meta-ide-support's
PACKAGE_ARCHS which doesn't like image recipe (lib32-image recipe's
DEFAULTTUNE is not overridded)

[YOCTO #12298]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/toolchain-scripts.bbclass | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/toolchain-scripts.bbclass b/meta/classes/toolchain-scripts.bbclass
index 9bcfe70..1721884 100644
--- a/meta/classes/toolchain-scripts.bbclass
+++ b/meta/classes/toolchain-scripts.bbclass
@@ -151,6 +151,12 @@ toolchain_create_sdk_siteconfig[vardepsexclude] = "TOOLCHAIN_CONFIGSITE_SYSROOTC
 
 python __anonymous () {
     import oe.classextend
+
+    # Install all MULTILIB_VARIANTS to sysroot and pack them only makes sense to
+    # image recipe.
+    if not bb.data.inherits_class('image', d):
+        return
+
     deps = ""
     for dep in (d.getVar('TOOLCHAIN_NEED_CONFIGSITE_CACHE') or "").split():
         deps += " %s:do_populate_sysroot" % dep
-- 
2.7.4

[PATCH 11/12] multilib.bbclass: deltask populate_sdk and populate_sdk_ext

[Robert Yang]

The "bitbake image -cpopulate_sdk/ext" generates SDK/eSDK for all multilib
variants, so "bitbake lib32-image -cpopulate_sdk/ext" is not needed, and it
doesn't work well, for example:

MACHINE ?= "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake lib32-core-image-minimal -cpopulate_sdk_ext
[snip]
Exception: FileExistsError: [Errno 17] File exists: '/buildarea/lyang1/test_q64/tmp/sysroots-components/core2-64/openssl/sysroot-providers/openssl10' -> '/buildarea/lyang1/test_q64/tmp/work/qemux86_64-pokymllib32-linux/lib32-core-image-minimal/1.0-r0/lib32-recipe-sysroot/sysroot-providers/openssl10'
[snip]

The problem is populate_sdk_ext installs all multilib variants, and
extend_recipe_sysroot() handles foo-image depends lib32-foo-image, but doesn't
handle lib32-foo-image depends foo-image, we can use a lot of trick ways to make
it work:
1) Get foo-image's RECIPE_SYSROOT when build lib32-foo-image
2) Handle conflicts with foo-image.do_rootfs
3) Handle conflicts when "bitbake lib32-foo-image foo-image -cpopulate_sdk_ext"

And maybe other potential problems, this looks painful, so just delete the task.

[YOCTO #12210]
---
 meta/classes/multilib.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index 770863f..13ba030 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -145,3 +145,6 @@ python do_package_qa_multilib() {
         check_mlprefix(pkg, 'RREPLACES', ml)
         check_mlprefix(pkg, 'RCONFLICTS', ml)
 }
+
+deltask do_populate_sdk
+deltask do_populate_sdk_ext
-- 
2.7.4

[PATCH 12/12] populate_sdk_ext.bbclass: remove the try...finally

[Robert Yang]

The "sdkbasepath + '/conf/local.conf.bak" doesn't exist when
"oe.copy_buildsystem.check_sstate_task_list()" fails, then os.replace() would
raise FileNotFoundError, which overcomes the real error. Keep the error status
makes debug easier, so remove the try..finally.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 70 +++++++++++++++++------------------
 1 file changed, 34 insertions(+), 36 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 78cdfab..c7a66d3 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -142,43 +142,41 @@ def create_filtered_tasklist(d, sdkbasepath, tasklistfile, conf_initpath):
 
     # Create a temporary build directory that we can pass to the env setup script
     shutil.copyfile(sdkbasepath + '/conf/local.conf', sdkbasepath + '/conf/local.conf.bak')
+    with open(sdkbasepath + '/conf/local.conf', 'a') as f:
+        # Force the use of sstate from the build system
+        f.write('\nSSTATE_DIR_forcevariable = "%s"\n' % d.getVar('SSTATE_DIR'))
+        f.write('SSTATE_MIRRORS_forcevariable = "file://universal/(.*) file://universal-4.9/\\1 file://universal-4.9/(.*) file://universal-4.8/\\1"\n')
+        # Ensure TMPDIR is the default so that clean_esdk_builddir() can delete it
+        f.write('TMPDIR_forcevariable = "${TOPDIR}/tmp"\n')
+        f.write('TCLIBCAPPEND_forcevariable = ""\n')
+        # Drop uninative if the build isn't using it (or else NATIVELSBSTRING will
+        # be different and we won't be able to find our native sstate)
+        if not bb.data.inherits_class('uninative', d):
+            f.write('INHERIT_remove = "uninative"\n')
+
+    # Unfortunately the default SDKPATH (or even a custom value) may contain characters that bitbake
+    # will not allow in its COREBASE path, so we need to rename the directory temporarily
+    temp_sdkbasepath = d.getVar('SDK_OUTPUT') + '/tmp-renamed-sdk'
+    # Delete any existing temp dir
     try:
-        with open(sdkbasepath + '/conf/local.conf', 'a') as f:
-            # Force the use of sstate from the build system
-            f.write('\nSSTATE_DIR_forcevariable = "%s"\n' % d.getVar('SSTATE_DIR'))
-            f.write('SSTATE_MIRRORS_forcevariable = "file://universal/(.*) file://universal-4.9/\\1 file://universal-4.9/(.*) file://universal-4.8/\\1"\n')
-            # Ensure TMPDIR is the default so that clean_esdk_builddir() can delete it
-            f.write('TMPDIR_forcevariable = "${TOPDIR}/tmp"\n')
-            f.write('TCLIBCAPPEND_forcevariable = ""\n')
-            # Drop uninative if the build isn't using it (or else NATIVELSBSTRING will
-            # be different and we won't be able to find our native sstate)
-            if not bb.data.inherits_class('uninative', d):
-                f.write('INHERIT_remove = "uninative"\n')
-
-        # Unfortunately the default SDKPATH (or even a custom value) may contain characters that bitbake
-        # will not allow in its COREBASE path, so we need to rename the directory temporarily
-        temp_sdkbasepath = d.getVar('SDK_OUTPUT') + '/tmp-renamed-sdk'
-        # Delete any existing temp dir
-        try:
-            shutil.rmtree(temp_sdkbasepath)
-        except FileNotFoundError:
-            pass
-        os.rename(sdkbasepath, temp_sdkbasepath)
-        cmdprefix = '. %s .; ' % conf_initpath
-        logfile = d.getVar('WORKDIR') + '/tasklist_bb_log.txt'
-        try:
-            oe.copy_buildsystem.check_sstate_task_list(d, get_sdk_install_targets(d), tasklistfile, cmdprefix=cmdprefix, cwd=temp_sdkbasepath, logfile=logfile)
-        except bb.process.ExecutionError as e:
-            msg = 'Failed to generate filtered task list for extensible SDK:\n%s' %  e.stdout.rstrip()
-            if 'attempted to execute unexpectedly and should have been setscened' in e.stdout:
-                msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n'
-            bb.fatal(msg)
-        os.rename(temp_sdkbasepath, sdkbasepath)
-        # Clean out residue of running bitbake, which check_sstate_task_list()
-        # will effectively do
-        clean_esdk_builddir(d, sdkbasepath)
-    finally:
-        os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf')
+        shutil.rmtree(temp_sdkbasepath)
+    except FileNotFoundError:
+        pass
+    os.rename(sdkbasepath, temp_sdkbasepath)
+    cmdprefix = '. %s .; ' % conf_initpath
+    logfile = d.getVar('WORKDIR') + '/tasklist_bb_log.txt'
+    try:
+        oe.copy_buildsystem.check_sstate_task_list(d, get_sdk_install_targets(d), tasklistfile, cmdprefix=cmdprefix, cwd=temp_sdkbasepath, logfile=logfile)
+    except bb.process.ExecutionError as e:
+        msg = 'Failed to generate filtered task list for extensible SDK:\n%s' %  e.stdout.rstrip()
+        if 'attempted to execute unexpectedly and should have been setscened' in e.stdout:
+            msg += '\n----------\n\nNOTE: "attempted to execute unexpectedly and should have been setscened" errors indicate this may be caused by missing sstate artifacts that were likely produced in earlier builds, but have been subsequently deleted for some reason.\n'
+        bb.fatal(msg)
+    os.rename(temp_sdkbasepath, sdkbasepath)
+    # Clean out residue of running bitbake, which check_sstate_task_list()
+    # will effectively do
+    clean_esdk_builddir(d, sdkbasepath)
+    os.replace(sdkbasepath + '/conf/local.conf.bak', sdkbasepath + '/conf/local.conf')
 
 python copy_buildsystem () {
     import re
-- 
2.7.4

✗ patchtest: failure for Fixes for multilib and eSDK (rev2)

[Patchwork]

== Series Details ==

Series: Fixes for multilib and eSDK (rev2)
Revision: 2
URL   : https://patchwork.openembedded.org/series/9702/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [11/12] multilib.bbclass: deltask populate_sdk and populate_sdk_ext
 Issue             Patch is missing Signed-off-by [test_signed_off_by_presence] 
  Suggested fix    Sign off the patch (either manually or with "git commit --amend -s")



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for Fixes for multilib and eSDK (rev2)

[Robert Yang]

Sorry, I added the SOB and updated in the repo.

// Robert

On 01/08/2018 06:04 PM, Patchwork wrote:
> == Series Details ==
> 
> Series: Fixes for multilib and eSDK (rev2)
> Revision: 2
> URL   : https://patchwork.openembedded.org/series/9702/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Patch            [11/12] multilib.bbclass: deltask populate_sdk and populate_sdk_ext
>   Issue             Patch is missing Signed-off-by [test_signed_off_by_presence]
>    Suggested fix    Sign off the patch (either manually or with "git commit --amend -s")
> 
> 
> 
> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
> 
> ---
> Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
> 
> 

Re: [PATCH 00/12 V2] Fixes for multilib and eSDK

[Robert Yang]

Two of the patches have been merged, so I rebased others and put them to:

   git://git.openembedded.org/openembedded-core-contrib rbt/multilib
   http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/multilib

Robert Yang (10):
   staging.bbclass: fix for multilib
   populate_sdk_ext.bbclass: fix for multilib
   staging.bbclass: staging_populate_sysroot_dir(): fix for multilib
   staging.bbclass: print searched manifest when not found
   staging.bbclass: extend_recipe_sysroot(): search multilib manifest
   package_manager.py: reverse archs correctly
   multilib.bbclass: extend allarch recipes
   toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for
     image
   multilib.bbclass: deltask populate_sdk and populate_sdk_ext
   populate_sdk_ext.bbclass: remove the try...finally

// Robert

On 01/08/2018 05:32 PM, Robert Yang wrote:
> * V2
>    + Drop already merged patches
> 
>    + Fix RP's comments:
>      - deltask populate_sdk and populate_sdk_ext for multilib
>      - Drop "image.bbclass: remove depmodwrapper-cross from DEPENDS"
>      - sstate.bbclass: sstate_hardcode_path(): fix for multilib
>        > fix the grep command
> 
> * V1
>    - Initial version
> 
> // Robert
> 
> The following changes since commit e9dfe7eb7f61b909ae7d034e80cfbebc1fad018b:
> 
>    icu-dbg: improve reproducibility (2018-01-08 08:45:33 +0000)
> 
> are available in the git repository at:
> 
>    git://git.openembedded.org/openembedded-core-contrib oem1
>    http://cgit.openembedded.org/openembedded-core-contrib/log/?h=oem1
> 
> Robert Yang (12):
>    staging.bbclass: fix for multilib
>    populate_sdk_ext.bbclass: fix for multilib
>    sstate.bbclass: sstate_hardcode_path(): fix for multilib
>    staging.bbclass: staging_populate_sysroot_dir(): fix for multilib
>    staging.bbclass: print searched manifest when not found
>    staging.bbclass: extend_recipe_sysroot(): search multilib manifest
>    package_manager.py: reverse archs correctly
>    multilib.bbclass: remove invalid PACKAGE_INSTALL
>    multilib.bbclass: extend allarch recipes
>    toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for
>      image
>    multilib.bbclass: deltask populate_sdk and populate_sdk_ext
>    populate_sdk_ext.bbclass: remove the try...finally
> 
>   meta/classes/allarch.bbclass           |   3 -
>   meta/classes/multilib.bbclass          |   8 +-
>   meta/classes/multilib_global.bbclass   |   4 +-
>   meta/classes/package.bbclass           |   9 ---
>   meta/classes/populate_sdk_ext.bbclass  | 142 +++++++++++++++++++--------------
>   meta/classes/sstate.bbclass            |   8 +-
>   meta/classes/staging.bbclass           |  40 +++++++++-
>   meta/classes/toolchain-scripts.bbclass |   6 ++
>   meta/lib/oe/package_manager.py         |  57 +++++++++++--
>   9 files changed, 186 insertions(+), 91 deletions(-)
> 

Re: [PATCH 00/12 V2] Fixes for multilib and eSDK

[Richard Purdie]

On Thu, 2018-01-18 at 17:35 +0800, Robert Yang wrote:
> Two of the patches have been merged, so I rebased others and put them
> to:
> 
>    git://git.openembedded.org/openembedded-core-contrib rbt/multilib
>    http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/
> multilib
> 
> Robert Yang (10):
>    staging.bbclass: fix for multilib
>    populate_sdk_ext.bbclass: fix for multilib
>    staging.bbclass: staging_populate_sysroot_dir(): fix for multilib
>    staging.bbclass: print searched manifest when not found
>    staging.bbclass: extend_recipe_sysroot(): search multilib manifest
>    package_manager.py: reverse archs correctly
>    multilib.bbclass: extend allarch recipes
>    toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for
>      image
>    multilib.bbclass: deltask populate_sdk and populate_sdk_ext
>    populate_sdk_ext.bbclass: remove the try...finally

Just to update, I also have a set of patches which alter the manifest
code in staging.bbclass, I wanted to compare what we're doing and
likely create a combination of those patches.

For the deltask commit, could you make it bb.build.deltask() only when
image.bbclass is inherited?

Cheers,

Richard

Re: [PATCH 00/12 V2] Fixes for multilib and eSDK

[Robert Yang]

On 01/18/2018 07:29 PM, Richard Purdie wrote:
> On Thu, 2018-01-18 at 17:35 +0800, Robert Yang wrote:
>> Two of the patches have been merged, so I rebased others and put them
>> to:
>>
>>     git://git.openembedded.org/openembedded-core-contrib rbt/multilib
>>     http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/
>> multilib
>>
>> Robert Yang (10):
>>     staging.bbclass: fix for multilib
>>     populate_sdk_ext.bbclass: fix for multilib
>>     staging.bbclass: staging_populate_sysroot_dir(): fix for multilib
>>     staging.bbclass: print searched manifest when not found
>>     staging.bbclass: extend_recipe_sysroot(): search multilib manifest
>>     package_manager.py: reverse archs correctly
>>     multilib.bbclass: extend allarch recipes
>>     toolchain-scripts.bbclass: only install all MULTILIB_VARIANTS for
>>       image
>>     multilib.bbclass: deltask populate_sdk and populate_sdk_ext
>>     populate_sdk_ext.bbclass: remove the try...finally
> 
> Just to update, I also have a set of patches which alter the manifest
> code in staging.bbclass, I wanted to compare what we're doing and
> likely create a combination of those patches.
> 
> For the deltask commit, could you make it bb.build.deltask() only when
> image.bbclass is inherited?

Thanks, I updated it in the repo:

diff --git a/meta/classes/multilib.bbclass b/meta/classes/multilib.bbclass
index 770863f..6a79d02 100644
--- a/meta/classes/multilib.bbclass
+++ b/meta/classes/multilib.bbclass
@@ -95,6 +95,8 @@ python __anonymous () {
          d.setVar("LINGUAS_INSTALL", "")
          # FIXME, we need to map this to something, not delete it!
          d.setVar("PACKAGE_INSTALL_ATTEMPTONLY", "")
+        bb.build.deltask('do_populate_sdk', d)
+        bb.build.deltask('do_populate_sdk_ext', d)
          return

      clsextend.map_depends_variable("DEPENDS")


But I think that there is no difference since only image recipe has
do_populate_sdk/_ext.

// Robert

> 
> Cheers,
> 
> Richard
>