[PATCH] bash: nativesdk-sdk-provides-dummy RPROVIDES bash for nativesdk already so usrmerge should not

[PATCH] bash: nativesdk-sdk-provides-dummy RPROVIDES bash for nativesdk already so usrmerge should not

[Maanya Goenka]

From: Maanya Goenka <maanyagoenka@microsoft.com>

The package nativesdk-sdk-provides-dummy already provides /bin/sh and /bin/bash
so we prevent re-including them with the usrmerge distro feature to avoid conflicts.

Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com>
Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com>
---
 meta/recipes-extended/bash/bash.inc | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-extended/bash/bash.inc b/meta/recipes-extended/bash/bash.inc
index 18874a0e24..94972294d3 100644
--- a/meta/recipes-extended/bash/bash.inc
+++ b/meta/recipes-extended/bash/bash.inc
@@ -128,4 +128,6 @@ PACKAGE_BEFORE_PN += "${PN}-loadable"
 RDEPENDS:${PN}-loadable += "${PN}"
 FILES:${PN}-loadable += "${libdir}/bash/*"
 
-RPROVIDES:${PN} += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '/bin/sh /bin/bash', '', d)}"
+# Limit the RPROVIDES here to class target so that if usrmerge is added to the host, it does not also include bash and sh;
+# this is needed because the package 'nativesdk-sdk-provides-dummy' already provides /bin/sh and /bin/bash and we do not want any conflicting RPROVIDES
+RPROVIDES:${PN}:append:class-target = " ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', '/bin/sh /bin/bash', '', d)}"
-- 
2.25.1

[PATCH] usrmerge: add the distro feature to nativesdk builds when added to target and enable symlink creation for both

[Maanya Goenka]

From: Maanya Goenka <maanyagoenka@microsoft.com>

Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency),
since we have come across cases where the distro feature would be required by both contexts.
As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage,
these need to be created for both the target and native sysroots within the SDK.

Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com>
Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com>
---
 meta/classes-recipe/image.bbclass | 2 ++
 meta/conf/bitbake.conf            | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index 7231fad940..df59133f77 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -653,8 +653,10 @@ create_merged_usr_symlinks_rootfs() {
     create_merged_usr_symlinks ${IMAGE_ROOTFS}
 }
 
+# Create the usrmerge symlink for the host in addition to the target build
 create_merged_usr_symlinks_sdk() {
     create_merged_usr_symlinks ${SDK_OUTPUT}${SDKTARGETSYSROOT}
+    create_merged_usr_symlinks ${SDK_OUTPUT}${SDKPATHNATIVE}
 }
 
 ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks_rootfs', '',d)}"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 6f180d18b0..ff9f7ab407 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -920,8 +920,8 @@ DISTRO_FEATURES_NATIVESDK ?= "x11"
 
 # Normally target distro features will not be applied to native builds:
 # Native distro features on this list will use the target feature value
-DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl wayland"
-DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl wayland"
+DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl usrmerge wayland"
+DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl usrmerge wayland"
 
 DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit gobject-introspection-data ldconfig"
 MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"
-- 
2.25.1

[PATCH] toolchain-shar-relocate: allow 'find' access to libraries in symlinked directories

[Maanya Goenka]

From: Maanya Goenka <maanyagoenka@microsoft.com>

If the '/' is not added to the end of the dynamic loader path, only directories
that are not symlinks will be looked into for libraries such as ld-linux*,
so the slash is added to allow the 'find' command to have symlinked directory access too.

Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com>
Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com>
---
 meta/files/toolchain-shar-relocate.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/files/toolchain-shar-relocate.sh b/meta/files/toolchain-shar-relocate.sh
index cee9adbf39..e8ab357717 100644
--- a/meta/files/toolchain-shar-relocate.sh
+++ b/meta/files/toolchain-shar-relocate.sh
@@ -4,8 +4,9 @@ if ! xargs --version > /dev/null 2>&1; then
 fi
 
 # fix dynamic loader paths in all ELF SDK binaries
+# allow symlinks to be accessed via the find command too
 native_sysroot=$($SUDO_EXEC cat $env_setup_script |grep 'OECORE_NATIVE_SYSROOT='|cut -d'=' -f2|tr -d '"')
-dl_path=$($SUDO_EXEC find $native_sysroot/lib -maxdepth 1 -name "ld-linux*")
+dl_path=$($SUDO_EXEC find $native_sysroot/lib/ -maxdepth 1 -name "ld-linux*")
 if [ "$dl_path" = "" ] ; then
 	echo "SDK could not be set up. Relocate script unable to find ld-linux.so. Abort!"
 	exit 1
-- 
2.25.1

Re: [OE-core] [PATCH] usrmerge: add the distro feature to nativesdk builds when added to target and enable symlink creation for both

[Alexandre Belloni]

Hello,

This causes failures on the autobuilders:

https://autobuilder.yoctoproject.org/typhoon/#/builders/40/builds/8540/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/108/builds/5665/steps/12/logs/stdio



On 01/02/2024 02:56:22+0000, Maanya Goenka wrote:
> From: Maanya Goenka <maanyagoenka@microsoft.com>
> 
> Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency),
> since we have come across cases where the distro feature would be required by both contexts.
> As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage,
> these need to be created for both the target and native sysroots within the SDK.
> 
> Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com>
> Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com>
> ---
>  meta/classes-recipe/image.bbclass | 2 ++
>  meta/conf/bitbake.conf            | 4 ++--
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
> index 7231fad940..df59133f77 100644
> --- a/meta/classes-recipe/image.bbclass
> +++ b/meta/classes-recipe/image.bbclass
> @@ -653,8 +653,10 @@ create_merged_usr_symlinks_rootfs() {
>      create_merged_usr_symlinks ${IMAGE_ROOTFS}
>  }
>  
> +# Create the usrmerge symlink for the host in addition to the target build
>  create_merged_usr_symlinks_sdk() {
>      create_merged_usr_symlinks ${SDK_OUTPUT}${SDKTARGETSYSROOT}
> +    create_merged_usr_symlinks ${SDK_OUTPUT}${SDKPATHNATIVE}
>  }
>  
>  ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks_rootfs', '',d)}"
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 6f180d18b0..ff9f7ab407 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -920,8 +920,8 @@ DISTRO_FEATURES_NATIVESDK ?= "x11"
>  
>  # Normally target distro features will not be applied to native builds:
>  # Native distro features on this list will use the target feature value
> -DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl wayland"
> -DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl wayland"
> +DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opengl usrmerge wayland"
> +DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opengl usrmerge wayland"
>  
>  DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit gobject-introspection-data ldconfig"
>  MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"
> -- 
> 2.25.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#194548): https://lists.openembedded.org/g/openembedded-core/message/194548
> Mute This Topic: https://lists.openembedded.org/mt/104089708/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] usrmerge: add the distro feature to nativesdk builds when added to target and enable symlink creation for both

[Richard Purdie]

On Thu, 2024-02-01 at 02:56 +0000, Maanya Goenka wrote:
> From: Maanya Goenka <maanyagoenka@microsoft.com>
> 
> Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency),
> since we have come across cases where the distro feature would be required by both contexts.
> As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage,
> these need to be created for both the target and native sysroots within the SDK.
> 
> Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com>
> Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com>
> ---
>  meta/classes-recipe/image.bbclass | 2 ++
>  meta/conf/bitbake.conf            | 4 ++--
>  2 files changed, 4 insertions(+), 2 deletions(-)

This doesn't quite follow. "target", "nativesdk" and "native" are all
different separate filesystems and not connected. Yes, you can use
usrmerge on some (or all?) of them but you don't have to have them
match.

Forcing them to match is likely just working around an underlying issue
which needs to be fixed.

What were the real failures?

For example, this would stop native or nativesdk sstate being reused
between differently configured builds when it could be.

Cheers,

Richard

Re: [OE-core] [PATCH] usrmerge: add the distro feature to nativesdk builds when added to target and enable symlink creation for both

[Alejandro Enedino Hernandez Samaniego]

[-- Attachment #1: Type: text/plain, Size: 3478 bytes --]

On 2/6/24 03:27, Richard Purdie wrote:
> On Thu, 2024-02-01 at 02:56 +0000, Maanya Goenka wrote:
>> From: Maanya Goenka<maanyagoenka@microsoft.com>
>>
>> Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency),
>> since we have come across cases where the distro feature would be required by both contexts.
>> As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage,
>> these need to be created for both the target and native sysroots within the SDK.
>>
>> Signed-off-by: Maanya Goenka<maanyagoenka@microsoft.com>
>> Signed-off-by : Alejandro Hernandez Samaniego<alhe@linux.microsoft.com>
>> ---
>>   meta/classes-recipe/image.bbclass | 2 ++
>>   meta/conf/bitbake.conf            | 4 ++--
>>   2 files changed, 4 insertions(+), 2 deletions(-)
> This doesn't quite follow. "target", "nativesdk" and "native" are all
> different separate filesystems and not connected. Yes, you can use
> usrmerge on some (or all?) of them but you don't have to have them
> match.
>
> Forcing them to match is likely just working around an underlying issue
> which needs to be fixed.
>
> What were the real failures?

Hello Richard,

Apologies for the late response.

I do agree that target, native and nativesdk are all separate and that you could use usrmerge on some
or all. I understand your point and I'm to blame for the suggestion to add usrmerge to the NATIVE case
for consistency.

The proposed patch is trying to fix an issue for a use case related to systemd.

Newer versions of systemd do not function properly without usrmerge because upstream systemd is dropping
support for non-usrmerge cases, which is typically fine because usrmerge works well for the TARGET.

https://lists.freedesktop.org/archives/systemd-devel/2022-September/048352.html

However, we are using some NATIVESDK systemd components that require usrmerge to work as well hence why
we needed to add usrmerge to the NATIVESDK filter since systemd components where expecting usr/foo locations
in the nativesdk filesystem.

While I understand that using nativesdk systemd components may not be widely common, I do believe its a
valid usecase.

I'd like to clarify that I'm not aware of any native systemd usecase and that my suggestion to add usrmerge
to the NATIVE filter was only for consistency, but not required (apologies for that again).

I've reproduced the autobuilder failures locally and I see that taking out usrmerge from the NATIVE filter does
get rid of the errors, if you agree with my statement that nativesdk systemd components are a valid usecase,
we can send a v2 that adds usrmerge to the NATIVESDK filter only to fix such cases.

If you prefer, we could also go one step above that and create a condition to add usrmerge to the NATIVESDK filter
only if systemd is being used or something similar.

Cheers,

Alejandro


>
> For example, this would stop native or nativesdk sstate being reused
> between differently configured builds when it could be.
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#194983):https://lists.openembedded.org/g/openembedded-core/message/194983
> Mute This Topic:https://lists.openembedded.org/mt/104089708/4354175
> Group Owner:openembedded-core+owner@lists.openembedded.org
> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [alhe@linux.microsoft.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

[-- Attachment #2: Type: text/html, Size: 5133 bytes --]

Re: [OE-core] [PATCH] usrmerge: add the distro feature to nativesdk builds when added to target and enable symlink creation for both

[Richard Purdie]

Hi Alejandro,

On Thu, 2024-02-15 at 12:10 -0700, Alejandro Enedino Hernandez
Samaniego wrote:
> On 2/6/24 03:27, Richard Purdie wrote:
> > 
> > On Thu, 2024-02-01 at 02:56 +0000, Maanya Goenka wrote:
> > > From: Maanya Goenka <maanyagoenka@microsoft.com>
> > > 
> > > Add usrmerge to DISTRO_FEATURES_FILTER_NATIVESDK (and DISTRO_FEATURES_FILTER_NATIVE for consistency),
> > > since we have come across cases where the distro feature would be required by both contexts.
> > > As a supplement to this change, when creating the merged usr symlinks at the do_populate_sdk stage,
> > > these need to be created for both the target and native sysroots within the SDK.
> > > 
> > > Signed-off-by: Maanya Goenka <maanyagoenka@microsoft.com>
> > > Signed-off-by : Alejandro Hernandez Samaniego <alhe@linux.microsoft.com>
> > > ---
> > >  meta/classes-recipe/image.bbclass | 2 ++
> > >  meta/conf/bitbake.conf            | 4 ++--
> > >  2 files changed, 4 insertions(+), 2 deletions(-)
> > 
> > This doesn't quite follow. "target", "nativesdk" and "native" are all
> > different separate filesystems and not connected. Yes, you can use
> > usrmerge on some (or all?) of them but you don't have to have them
> > match.
> > 
> > Forcing them to match is likely just working around an underlying issue
> > which needs to be fixed.
> > 
> > What were the real failures?
> 
>
> Apologies for the late response.
> 
> I do agree that target, native and nativesdk are all separate and that you could use usrmerge on some
> or all. I understand your point and I'm to blame for the suggestion to add usrmerge to the NATIVE case
> for consistency.
> 
> The proposed patch is trying to fix an issue for a use case related to systemd.
> 
> Newer versions of systemd do not function properly without usrmerge because upstream systemd is dropping
> support for non-usrmerge cases, which is typically fine because usrmerge works well for the TARGET.
> 
> https://lists.freedesktop.org/archives/systemd-devel/2022-September/048352.html
> 
> However, we are using some NATIVESDK systemd components that require usrmerge to work as well hence why
> we needed to add usrmerge to the NATIVESDK filter since systemd components where expecting usr/foo locations
> in the nativesdk filesystem.
> 
> While I understand that using nativesdk systemd components may not be widely common, I do believe its a
> valid usecase.
> 
> I'd like to clarify that I'm not aware of any native systemd usecase and that my suggestion to add usrmerge
> to the NATIVE filter was only for consistency, but not required (apologies for that again).
> 
> I've reproduced the autobuilder failures locally and I see that taking out usrmerge from the NATIVE filter does
> get rid of the errors, if you agree with my statement that nativesdk systemd components are a valid usecase,
> we can send a v2 that adds usrmerge to the NATIVESDK filter only to fix such cases.
> 
> If you prefer, we could also go one step above that and create a condition to add usrmerge to the NATIVESDK filter
> only if systemd is being used or something similar.

I think there is a piece of this that is still missing. Changing
usrmerge in nativesdk fundamentally alters the layout of the SDK. The
layout of the SDK is meant to be independent of the target (think
multilibs and so on as well).

What that means is that the tools are meant to work with any target
system and not need to "match" their layout. This effectively means
that systemd's tools aren't working properly for environments which
have cross layouts.

The SDK layout is already drastically different from that of the target
system in that everything is installed into a deep prefix path. How
long will it be before systemd drops support for that too?

I don't really buy into this "systemd should dictate the layout of
everything" situation we're ending up in. It does mean any given SDK
can only support systemd or not systemd but not both so it breaks
multiconfig too.

What is systemd's position on supporting cross environments? That is
effectively the real question here and it sounds like they don't
support that? We already have big issues with this in the "native"
tooling side of things since that cross support doesn't exist and we
had to do our own emulation.

So whilst I understand your usecase, I do believe this is a sign of a
much deeper problem unfortunately, namely that systemd doesn't support
cross environments or usage. We could probably hack around it like this
for now but I'm far from sure it won't just break again in the future
in worse ways.

Cheers,

Richard