[PATCH 0/1] meta-environment: fix site-config files identical for multilib

[PATCH 0/1] meta-environment: fix site-config files identical for multilib

[kai]

From: Kai Kang <kai.kang@windriver.com>

Test steps
1 build meta-environment for MACHINE qemuarm without this patch
2 build meta-environment for MACHINE qemuarm64 with multilib enabled

require conf/multilib.conf
MACHINE = "qemuarm64"
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "armv7vethf"

3 compare the 2 site-config files for 32-bit are identical
$ diff path/to/tmp/work/x86_64-nativesdk-pokysdk-linux/meta-environment-qemuarmv5/1.0-r8/sdk/image/opt/poky/3.1+snapshot/site-config-armv5e-poky-linux-gnueabi tmp/work/x86_64-nativesdk-pokysdk-linux/meta-environment-qemuarm64/1.0-r8/sdk/image/opt/poky/3.1+snapshot/site-config-armv7vet2hf-vfp-pokymllib32-linux-gnueabi


Kai Kang (1):
  meta-environment: fix site-config files identical for multilib

 meta/classes/toolchain-scripts.bbclass     | 47 ++++++++++++++++++++--
 meta/recipes-core/meta/meta-environment.bb |  2 +-
 2 files changed, 45 insertions(+), 4 deletions(-)

-- 
2.17.1

[PATCH 1/1] meta-environment: fix site-config files identical for multilib

[kai]

From: Kai Kang <kai.kang@windriver.com>

When multilib is enabled, such as:

require conf/multilib.conf
MACHINE = "qemuarm64"
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "armv7vethf"

the site-config files created by meta-environment are identical but
should not.

HOST_ARCH is overwritten when inherit cross-canadian, so it fails to get
right site files for meta-environment. There is a Yocto defect #10255 to
handle it. But multilib is not covered.

Save multilib related site files before inherit cross-canadian and
construct site-config-* file for multilib correctly.

Signed-off-by: Kai Kang <kai.kang@windriver.com>
---
 meta/classes/toolchain-scripts.bbclass     | 47 ++++++++++++++++++++--
 meta/recipes-core/meta/meta-environment.bb |  2 +-
 2 files changed, 45 insertions(+), 4 deletions(-)

diff --git a/meta/classes/toolchain-scripts.bbclass b/meta/classes/toolchain-scripts.bbclass
index db1d3215ef..8ce6489d10 100644
--- a/meta/classes/toolchain-scripts.bbclass
+++ b/meta/classes/toolchain-scripts.bbclass
@@ -163,16 +163,57 @@ TOOLCHAIN_CONFIGSITE_SYSROOTCACHE = "${STAGING_DIR}/${MLPREFIX}${MACHINE}/${targ
 TOOLCHAIN_NEED_CONFIGSITE_CACHE ??= "virtual/${MLPREFIX}libc ncurses"
 DEPENDS += "${TOOLCHAIN_NEED_CONFIGSITE_CACHE}"
 
+def get_siteinfo_files_ml(d):
+    siteinfo_files_ml = ''
+    variants = d.getVar("MULTILIB_VARIANTS") or ""
+
+    for item in variants.split():
+        localdata = bb.data.createCopy(d)
+        overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item
+        localdata.setVar("OVERRIDES", overrides)
+        site_files = siteinfo_get_files(localdata)
+        siteinfo_files_ml += item + '-:' + site_files + ';'
+
+    return siteinfo_files_ml
+
+SAVED_SITEINFO_FILES_MULTILIBS := "${@get_siteinfo_files_ml(d)}"
+
+def extract_siteinfo_files_ml(d):
+    mlprefix = d.getVar('MLPREFIX')
+    if not mlprefix:
+        return ''
+
+    siteinfo_files = d.getVar('SAVED_SITEINFO_FILES_MULTILIBS')
+
+    for item in siteinfo_files.split(';'):
+        item = item.strip()
+        if not item:
+            continue
+        l = item.split(':')
+        if mlprefix == l[0]:
+            return l[1].strip()
+
+    bb.warn('Not extract siteinfo files for multilib variant {}'.format(mlprefix[:-1]))
+    return ''
+
+
 #This function create a site config file
 toolchain_create_sdk_siteconfig () {
 	local siteconfig=$1
+	local mlprefix=$2
 
 	rm -f $siteconfig
 	touch $siteconfig
 
-	for sitefile in ${TOOLCHAIN_CONFIGSITE_NOCACHE} ; do
-		cat $sitefile >> $siteconfig
-	done
+	if [ -z "$mlprefix" ]; then
+		for sitefile in ${TOOLCHAIN_CONFIGSITE_NOCACHE} ; do
+			cat $sitefile >> $siteconfig
+		done
+	else
+		for sitefile in ${@extract_siteinfo_files_ml(d)}; do
+			cat $sitefile >> $siteconfig
+		done
+	fi
 
 	#get cached site config
 	for sitefile in ${TOOLCHAIN_NEED_CONFIGSITE_CACHE}; do
diff --git a/meta/recipes-core/meta/meta-environment.bb b/meta/recipes-core/meta/meta-environment.bb
index da1230bead..af9b8ca162 100644
--- a/meta/recipes-core/meta/meta-environment.bb
+++ b/meta/recipes-core/meta/meta-environment.bb
@@ -49,7 +49,7 @@ addtask generate_content before do_install after do_compile
 
 create_sdk_files() {
 	# Setup site file for external use
-	toolchain_create_sdk_siteconfig ${SDK_OUTPUT}/${SDKPATH}/site-config-${REAL_MULTIMACH_TARGET_SYS}
+	toolchain_create_sdk_siteconfig ${SDK_OUTPUT}/${SDKPATH}/site-config-${REAL_MULTIMACH_TARGET_SYS} ${@d.getVar("MLPREFIX") or ''}
 
 	toolchain_create_sdk_env_script ${SDK_OUTPUT}/${SDKPATH}/environment-setup-${REAL_MULTIMACH_TARGET_SYS}
 
-- 
2.17.1

Re: [OE-core] [PATCH 1/1] meta-environment: fix site-config files identical for multilib

[Richard Purdie]

On Wed, 2020-10-21 at 15:32 +0800, kai wrote:
> From: Kai Kang <kai.kang@windriver.com>
> 
> When multilib is enabled, such as:
> 
> require conf/multilib.conf
> MACHINE = "qemuarm64"
> MULTILIBS = "multilib:lib32"
> DEFAULTTUNE_virtclass-multilib-lib32 = "armv7vethf"
> 
> the site-config files created by meta-environment are identical but
> should not.
> 
> HOST_ARCH is overwritten when inherit cross-canadian, so it fails to get
> right site files for meta-environment. There is a Yocto defect #10255 to
> handle it. But multilib is not covered.
> 
> Save multilib related site files before inherit cross-canadian and
> construct site-config-* file for multilib correctly.
> 
> Signed-off-by: Kai Kang <kai.kang@windriver.com>
> ---
>  meta/classes/toolchain-scripts.bbclass     | 47 ++++++++++++++++++++--
>  meta/recipes-core/meta/meta-environment.bb |  2 +-
>  2 files changed, 45 insertions(+), 4 deletions(-)
> 
> diff --git a/meta/classes/toolchain-scripts.bbclass b/meta/classes/toolchain-scripts.bbclass
> index db1d3215ef..8ce6489d10 100644
> --- a/meta/classes/toolchain-scripts.bbclass
> +++ b/meta/classes/toolchain-scripts.bbclass
> @@ -163,16 +163,57 @@ TOOLCHAIN_CONFIGSITE_SYSROOTCACHE = "${STAGING_DIR}/${MLPREFIX}${MACHINE}/${targ
>  TOOLCHAIN_NEED_CONFIGSITE_CACHE ??= "virtual/${MLPREFIX}libc ncurses"
>  DEPENDS += "${TOOLCHAIN_NEED_CONFIGSITE_CACHE}"
>  
> +def get_siteinfo_files_ml(d):
> +    siteinfo_files_ml = ''
> +    variants = d.getVar("MULTILIB_VARIANTS") or ""
> +
> +    for item in variants.split():
> +        localdata = bb.data.createCopy(d)
> +        overrides = d.getVar("OVERRIDES", False) + ":virtclass-multilib-" + item
> +        localdata.setVar("OVERRIDES", overrides)
> +        site_files = siteinfo_get_files(localdata)
> +        siteinfo_files_ml += item + '-:' + site_files + ';'
> +
> +    return siteinfo_files_ml
> +
> +SAVED_SITEINFO_FILES_MULTILIBS := "${@get_siteinfo_files_ml(d)}"

I'm a bit worried that we're doing immediate expansion here which is
going to be very order dependent on how/where this class is used.

We also seem to be duplicating functionality into multilib and non
multilib functions which means one path could easily be updated and the
other missed.

I agree there is a bug here but I'm hoping there is some neater way to
fix this which isn't so order dependent and avoids the duplication. I'd
have to spend time looking at what is going on to provide better more
specific feedback though...

Cheers,

Richard