[PATCH] qemu: access host pkg-config via HOSTTOOLS

[PATCH] qemu: access host pkg-config via HOSTTOOLS

[Taras Kondratiuk]

On some build hosts qemu-system-native fails to build with errors:
like:
| build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/qemu-keymap.c:16:10: fatal error: xkbcommon/xkbcommon.h: No such file or directory
|    16 | #include <xkbcommon/xkbcommon.h>
|       |          ^~~~~~~~~~~~~~~~~~~~~~~
| compilation terminated.
| make: *** [build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/rules.mak:69: qemu-keymap.o] Error 1

This is a combination of two factors:
1. In our setup we use native toolchain from buildtools-tarball because
   default host GCC on some machines is too old.
2. Qemu recipe adds host pkg-config paths to PKG_CONFIG_PATH using
   hardcoded pkg-config tool's location (/usr/bin:/bin).  Added by
   commit 68a5ed337f8f ("qemu: fix qemu-native pkg-config paths")

Host xkbcommon package is detected in configure. But then the toolchain
can't find headers because it uses headers from buildtools-tarball only.

Instead of using hardcoded location, use pkg-config via HOSTTOOLS. It
will pick up correct pkg-config from PATH and avoid configuration
discrepancies.

Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
---
 meta/conf/bitbake.conf              | 2 +-
 meta/recipes-devtools/qemu/qemu.inc | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 353caacef986..fae1147d397c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -488,7 +488,7 @@ HOSTTOOLS += " \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
     head hostname iconv id install ld ldd ln ls make md5sum mkdir mknod \
-    mktemp mv nm objcopy objdump od patch perl pr printf pwd \
+    mktemp mv nm objcopy objdump od patch perl pkg-config pr printf pwd \
     python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
     sha1sum sha224sum sha256sum sha384sum sha512sum \
     sleep sort split stat strings strip tail tar tee test touch tr true uname \
diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc
index d41cc8f200b3..881a5ba65025 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -94,7 +94,7 @@ EXTRA_OECONF_append = " --python=${HOSTTOOLS_DIR}/python3"
 
 do_configure_prepend_class-native() {
 	# Append build host pkg-config paths for native target since the host may provide sdl
-	BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "")
+	BHOST_PKGCONFIG_PATH=$(${HOSTTOOLS_DIR}/pkg-config --variable pc_path pkg-config || echo "")
 	if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
 		export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
 	fi
-- 
2.25.1

Re: [PATCH] qemu: access host pkg-config via HOSTTOOLS

[Richard Purdie]

On Fri, 2020-07-10 at 10:48 -0700, Taras Kondratiuk wrote:
> On some build hosts qemu-system-native fails to build with errors:
> like:
> > build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/qemu-keymap.c:16:10: fatal error: xkbcommon/xkbcommon.h: No such file or directory
> >    16 | #include <xkbcommon/xkbcommon.h>
> >       |          ^~~~~~~~~~~~~~~~~~~~~~~
> > compilation terminated.
> > make: *** [build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/rules.mak:69: qemu-keymap.o] Error 1
> 
> This is a combination of two factors:
> 1. In our setup we use native toolchain from buildtools-tarball because
>    default host GCC on some machines is too old.
> 2. Qemu recipe adds host pkg-config paths to PKG_CONFIG_PATH using
>    hardcoded pkg-config tool's location (/usr/bin:/bin).  Added by
>    commit 68a5ed337f8f ("qemu: fix qemu-native pkg-config paths")
> 
> Host xkbcommon package is detected in configure. But then the toolchain
> can't find headers because it uses headers from buildtools-tarball only.
> 
> Instead of using hardcoded location, use pkg-config via HOSTTOOLS. It
> will pick up correct pkg-config from PATH and avoid configuration
> discrepancies.
> 
> Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
> ---
>  meta/conf/bitbake.conf              | 2 +-
>  meta/recipes-devtools/qemu/qemu.inc | 2 +-
>  2 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 353caacef986..fae1147d397c 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -488,7 +488,7 @@ HOSTTOOLS += " \
>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
>      head hostname iconv id install ld ldd ln ls make md5sum mkdir mknod \
> -    mktemp mv nm objcopy objdump od patch perl pr printf pwd \
> +    mktemp mv nm objcopy objdump od patch perl pkg-config pr printf pwd \
>      python3 ranlib readelf readlink realpath rm rmdir rpcgen sed seq sh \
>      sha1sum sha224sum sha256sum sha384sum sha512sum \
>      sleep sort split stat strings strip tail tar tee test touch tr true uname \
> diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc
> index d41cc8f200b3..881a5ba65025 100644
> --- a/meta/recipes-devtools/qemu/qemu.inc
> +++ b/meta/recipes-devtools/qemu/qemu.inc
> @@ -94,7 +94,7 @@ EXTRA_OECONF_append = " --python=${HOSTTOOLS_DIR}/python3"
>  
>  do_configure_prepend_class-native() {
>  	# Append build host pkg-config paths for native target since the host may provide sdl
> -	BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "")
> +	BHOST_PKGCONFIG_PATH=$(${HOSTTOOLS_DIR}/pkg-config --variable pc_path pkg-config || echo "")
>  	if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
>  		export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
>  	fi

qemu is a little special, its about the only thing we let look into the
system for things like this. We really don't want pkg-config available
generally in HOSTTOOLS.

I'm not 100% sure I understand how this is fixing things either. Is
this using the pkg-config from your buildtools tarball instead?

Cheers,

Richard

Re: [PATCH] qemu: access host pkg-config via HOSTTOOLS

[Taras Kondratiuk]

On Fri, 2020-07-10 at 18:56 +0100, Richard Purdie wrote:
> On Fri, 2020-07-10 at 10:48 -0700, Taras Kondratiuk wrote:
> > On some build hosts qemu-system-native fails to build with errors:
> > like:
> > > build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-
> > > 4.1.0/qemu-keymap.c:16:10: fatal error: xkbcommon/xkbcommon.h: No
> > > such file or directory
> > >    16 | #include <xkbcommon/xkbcommon.h>
> > >       |          ^~~~~~~~~~~~~~~~~~~~~~~
> > > compilation terminated.
> > > make: *** [build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-
> > > r0/qemu-4.1.0/rules.mak:69: qemu-keymap.o] Error 1
> > 
> > This is a combination of two factors:
> > 1. In our setup we use native toolchain from buildtools-tarball
> > because
> >    default host GCC on some machines is too old.
> > 2. Qemu recipe adds host pkg-config paths to PKG_CONFIG_PATH using
> >    hardcoded pkg-config tool's location (/usr/bin:/bin).  Added by
> >    commit 68a5ed337f8f ("qemu: fix qemu-native pkg-config paths")
> > 
> > Host xkbcommon package is detected in configure. But then the
> > toolchain
> > can't find headers because it uses headers from buildtools-tarball
> > only.
> > 
> > Instead of using hardcoded location, use pkg-config via HOSTTOOLS.
> > It
> > will pick up correct pkg-config from PATH and avoid configuration
> > discrepancies.
> > 
> > Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
> > ---
> >  meta/conf/bitbake.conf              | 2 +-
> >  meta/recipes-devtools/qemu/qemu.inc | 2 +-
> >  2 files changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 353caacef986..fae1147d397c 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -488,7 +488,7 @@ HOSTTOOLS += " \
> >      cpp cut date dd diff diffstat dirname du echo egrep env expand
> > expr false \
> >      fgrep file find flock g++ gawk gcc getconf getopt git grep
> > gunzip gzip \
> >      head hostname iconv id install ld ldd ln ls make md5sum mkdir
> > mknod \
> > -    mktemp mv nm objcopy objdump od patch perl pr printf pwd \
> > +    mktemp mv nm objcopy objdump od patch perl pkg-config pr
> > printf pwd \
> >      python3 ranlib readelf readlink realpath rm rmdir rpcgen sed
> > seq sh \
> >      sha1sum sha224sum sha256sum sha384sum sha512sum \
> >      sleep sort split stat strings strip tail tar tee test touch tr
> > true uname \
> > diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-
> > devtools/qemu/qemu.inc
> > index d41cc8f200b3..881a5ba65025 100644
> > --- a/meta/recipes-devtools/qemu/qemu.inc
> > +++ b/meta/recipes-devtools/qemu/qemu.inc
> > @@ -94,7 +94,7 @@ EXTRA_OECONF_append = " --
> > python=${HOSTTOOLS_DIR}/python3"
> >  
> >  do_configure_prepend_class-native() {
> >  	# Append build host pkg-config paths for native target since
> > the host may provide sdl
> > -	BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable 
> > pc_path pkg-config || echo "")
> > +	BHOST_PKGCONFIG_PATH=$(${HOSTTOOLS_DIR}/pkg-config --variable
> > pc_path pkg-config || echo "")
> >  	if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
> >  		export
> > PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
> >  	fi
> 
> qemu is a little special, its about the only thing we let look into
> the
> system for things like this. We really don't want pkg-config
> available
> generally in HOSTTOOLS.
> 
> I'm not 100% sure I understand how this is fixing things either. Is
> this using the pkg-config from your buildtools tarball instead?
> 

Yes, it is using pkg-config from buildtools tarball.

As per my understanding other parts of the build will still use
pkgconfig-native and should not be affected. Or the concern is that
packages without proper pkgconfig-native dependency will find this
pkg-config in HOSTTOOLS?

Re: [PATCH] qemu: access host pkg-config via HOSTTOOLS

[Richard Purdie]

On Fri, 2020-07-10 at 18:18 +0000, Taras Kondratiuk (takondra) wrote:
> On Fri, 2020-07-10 at 18:56 +0100, Richard Purdie wrote:
> > qemu is a little special, its about the only thing we let look into
> > the
> > system for things like this. We really don't want pkg-config
> > available
> > generally in HOSTTOOLS.
> > 
> > I'm not 100% sure I understand how this is fixing things either. Is
> > this using the pkg-config from your buildtools tarball instead?
> > 
> 
> Yes, it is using pkg-config from buildtools tarball.
> 
> As per my understanding other parts of the build will still use
> pkgconfig-native and should not be affected. Or the concern is that
> packages without proper pkgconfig-native dependency will find this
> pkg-config in HOSTTOOLS?

Yes, that is one concern.

Another is that if its using nativesdk-pkgconfig from buildtools, does
this break the reason we have this code in the first place (to allow
qemu to use the host GL for virtgl)? Presumably that is not in
buildtools...

Cheers,

Richard

Re: [OE-core] [PATCH] qemu: access host pkg-config via HOSTTOOLS

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On Fri, 10 Jul 2020 at 21:11, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> Another is that if its using nativesdk-pkgconfig from buildtools, does
> this break the reason we have this code in the first place (to allow
> qemu to use the host GL for virtgl)? Presumably that is not in
> buildtools...
>

I think fall-through to host pkg-config was there to allow usage of host
libsdl? I think you wanted to keep that in as an example of how to do such
a thing, when I previously proposed removal of it.

Alex

[-- Attachment #2: Type: text/html, Size: 863 bytes --]

Re: [PATCH] qemu: access host pkg-config via HOSTTOOLS

[Taras Kondratiuk]

On Fri, 2020-07-10 at 20:11 +0100, Richard Purdie wrote:
> On Fri, 2020-07-10 at 18:18 +0000, Taras Kondratiuk (takondra) wrote:
> > On Fri, 2020-07-10 at 18:56 +0100, Richard Purdie wrote:
> > > qemu is a little special, its about the only thing we let look
> > > into
> > > the
> > > system for things like this. We really don't want pkg-config
> > > available
> > > generally in HOSTTOOLS.
> > > 
> > > I'm not 100% sure I understand how this is fixing things either.
> > > Is
> > > this using the pkg-config from your buildtools tarball instead?
> > > 
> > 
> > Yes, it is using pkg-config from buildtools tarball.
> > 
> > As per my understanding other parts of the build will still use
> > pkgconfig-native and should not be affected. Or the concern is that
> > packages without proper pkgconfig-native dependency will find this
> > pkg-config in HOSTTOOLS?
> 
> Yes, that is one concern.
> 
> Another is that if its using nativesdk-pkgconfig from buildtools,
> does
> this break the reason we have this code in the first place (to allow
> qemu to use the host GL for virtgl)? Presumably that is not in
> buildtools...

I see. IIUC this host pkg-config part should be used only
if ASSUME_PROVIDED has libsdl2-native
https://git.openembedded.org/openembedded-core/tree/meta/conf/local.conf.sample?h=2020-04.1-dunfell#n213

Does it make sense to make this part conditional on ASSUME_PROVIDED
containing libsdl2-native?

[PATCH] qemu: use host pkg-config only when host provides sdl

[Taras Kondratiuk]

Qemu recipe adds host pkg-config paths to PKG_CONFIG_PATH using
hardcoded pkg-config tool's location (/usr/bin:/bin). This may cause
qemu build failures on systems that use external native toolchain (from
buildtools tarball): a host package is detected during do_configure via
host PKG_CONFIG_PATH, but then the toolchain can't find the package
because it uses headers from buildtools-tarball only.

For example if xkbcommon is present on host, but not in
buildtools-tarball:
| build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/qemu-keymap.c:16:10: fatal error: xkbcommon/xkbcommon.h: No such file or directory
|    16 | #include <xkbcommon/xkbcommon.h>
|       |          ^~~~~~~~~~~~~~~~~~~~~~~
| compilation terminated.
| make: *** [build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/rules.mak:69: qemu-keymap.o] Error 1

Using hardcoded pkg-config path seems like a hack. It was added to
handle a case when host sdl is provided by host. So limit usage of
host pkg-config only to the case when ASSUME_PROVIDED contains
libsdl2-native.

Signed-off-by: Taras Kondratiuk <takondra@cisco.com>
---
 meta/recipes-devtools/qemu/qemu.inc | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-devtools/qemu/qemu.inc b/meta/recipes-devtools/qemu/qemu.inc
index d41cc8f200b3..bfa69c402c8b 100644
--- a/meta/recipes-devtools/qemu/qemu.inc
+++ b/meta/recipes-devtools/qemu/qemu.inc
@@ -93,10 +93,12 @@ B = "${WORKDIR}/build"
 EXTRA_OECONF_append = " --python=${HOSTTOOLS_DIR}/python3"
 
 do_configure_prepend_class-native() {
-	# Append build host pkg-config paths for native target since the host may provide sdl
-	BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "")
-	if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
-		export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
+	# Append build host pkg-config paths for native target if the host provides sdl
+	if ${@bb.utils.contains('ASSUME_PROVIDED', 'libsdl2-native', 'true', 'false', d)}; then
+		BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "")
+		if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
+			export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
+		fi
 	fi
 }
 
-- 
2.25.1

Re: [OE-core] [PATCH] qemu: use host pkg-config only when host provides sdl

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

On Sun, 12 Jul 2020 at 23:35, Taras Kondratiuk via lists.openembedded.org
<takondra=cisco.com@lists.openembedded.org> wrote:

> For example if xkbcommon is present on host, but not in
> buildtools-tarball:
> |
> build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/qemu-keymap.c:16:10:
> fatal error: xkbcommon/xkbcommon.h: No such file or directory
> |    16 | #include <xkbcommon/xkbcommon.h>
> |       |          ^~~~~~~~~~~~~~~~~~~~~~~
> | compilation terminated.
> | make: ***
> [build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-4.1.0/rules.mak:69:
> qemu-keymap.o] Error 1
>

Qemu's xkbcommon support is disabled by default via PACKAGECONFIG:
PACKAGECONFIG[xkbcommon] =
"--enable-xkbcommon,--disable-xkbcommon,libxkbcommon"

So why is it then re-enabled? Can you take a closer look at what happens at
configure step? The way I read it, it should skip the whole pkg-config
thing.
(please check on latest master poky)

Alex

[-- Attachment #2: Type: text/html, Size: 1422 bytes --]

Re: [OE-core] [PATCH] qemu: use host pkg-config only when host provides sdl

[Taras Kondratiuk]

On Wed, 2020-07-15 at 11:15 +0200, Alexander Kanavin wrote:
> On Sun, 12 Jul 2020 at 23:35, Taras Kondratiuk via
> lists.openembedded.org <takondra=cisco.com@lists.openembedded.org>
> wrote:
> > For example if xkbcommon is present on host, but not in
> > buildtools-tarball:
> > | build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-r0/qemu-
> > 4.1.0/qemu-keymap.c:16:10: fatal error: xkbcommon/xkbcommon.h: No
> > such file or directory
> > |    16 | #include <xkbcommon/xkbcommon.h>
> > |       |          ^~~~~~~~~~~~~~~~~~~~~~~
> > | compilation terminated.
> > | make: *** [build/tmp/work/x86_64-linux/qemu-system-native/4.1.0-
> > r0/qemu-4.1.0/rules.mak:69: qemu-keymap.o] Error 1
> 
> Qemu's xkbcommon support is disabled by default via PACKAGECONFIG:
> PACKAGECONFIG[xkbcommon] = "--enable-xkbcommon,--disable-
> xkbcommon,libxkbcommon"
> 
> So why is it then re-enabled? Can you take a closer look at what
> happens at configure step? The way I read it, it should skip the
> whole pkg-config thing.
> (please check on latest master poky)

This specific xkbcommon issue was hit on Zeus branch. You are right, on
Dunfell and master the commit 3f1fd15cf0df ("qemu-system-native:
disable options not included in extended tarball") will hide this
issue. But the issue is still there. One will have to keep adding
PACKAGECONFIG options for each configuration that can be out of sync
between host and buildtools tarball.

Re: [OE-core] [PATCH] qemu: use host pkg-config only when host provides sdl

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 855 bytes --]

On Wed, 15 Jul 2020 at 19:09, Taras Kondratiuk (takondra) <
takondra@cisco.com> wrote:

> This specific xkbcommon issue was hit on Zeus branch. You are right, on
> Dunfell and master the commit 3f1fd15cf0df ("qemu-system-native:
> disable options not included in extended tarball") will hide this
> issue. But the issue is still there. One will have to keep adding
> PACKAGECONFIG options for each configuration that can be out of sync
> between host and buildtools tarball.
>

But that means we have a non-determinism problem: if qemu is picking up
things from the host without being explicitly told to do so, then the right
fix is indeed to add the missing PACKAGECONFIGs so those things are either
explicitly enabled or explicitly disabled.  This patch seems to be working
around the non-determinism, rather than addressing it.

Alex

[-- Attachment #2: Type: text/html, Size: 1170 bytes --]

Re: [OE-core] [PATCH] qemu: use host pkg-config only when host provides sdl

[Taras Kondratiuk]

On Wed, 2020-07-15 at 19:23 +0200, Alexander Kanavin wrote:
> On Wed, 15 Jul 2020 at 19:09, Taras Kondratiuk (takondra) <
> takondra@cisco.com> wrote:
> > This specific xkbcommon issue was hit on Zeus branch. You are
> > right, on
> > Dunfell and master the commit 3f1fd15cf0df ("qemu-system-native:
> > disable options not included in extended tarball") will hide this
> > issue. But the issue is still there. One will have to keep adding
> > PACKAGECONFIG options for each configuration that can be out of
> > sync
> > between host and buildtools tarball.
> 
> But that means we have a non-determinism problem: if qemu is picking
> up things from the host without being explicitly told to do so, then
> the right fix is indeed to add the missing PACKAGECONFIGs so those
> things are either explicitly enabled or explicitly disabled.  This
> patch seems to be working around the non-determinism, rather than
> addressing it.

PACKAGECONFIG improves determinism, but it is not sufficient. It is not
feasible to make sure that PACKAGECONFIGs cover all configurations.
There will be always something that slips through. I think
acknowledging this fact was one of the reasons for recipe-specific
sysroots.

We use buildtools tarball to better isolate OE build from host
environment and improve determinism. Host pkg-config opens a big hole
for non-determinism and plugging it with PACKAGECONFIGs seems fragile.

Re: [OE-core] [PATCH] qemu: use host pkg-config only when host provides sdl

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On Wed, 15 Jul 2020 at 20:54, Taras Kondratiuk (takondra) <
takondra@cisco.com> wrote:

>
> PACKAGECONFIG improves determinism, but it is not sufficient. It is not
> feasible to make sure that PACKAGECONFIGs cover all configurations.
> There will be always something that slips through. I think
> acknowledging this fact was one of the reasons for recipe-specific
> sysroots.
>
> We use buildtools tarball to better isolate OE build from host
> environment and improve determinism. Host pkg-config opens a big hole
> for non-determinism and plugging it with PACKAGECONFIGs seems fragile.
>

Right, with this I agree.

Alex

[-- Attachment #2: Type: text/html, Size: 959 bytes --]

Re: [OE-core] [PATCH] qemu: use host pkg-config only when host provides sdl

[Richard Purdie]

On Wed, 2020-07-15 at 18:54 +0000, Taras Kondratiuk (takondra) wrote:
> On Wed, 2020-07-15 at 19:23 +0200, Alexander Kanavin wrote:
> > On Wed, 15 Jul 2020 at 19:09, Taras Kondratiuk (takondra) <
> > takondra@cisco.com> wrote:
> > > This specific xkbcommon issue was hit on Zeus branch. You are
> > > right, on
> > > Dunfell and master the commit 3f1fd15cf0df ("qemu-system-native:
> > > disable options not included in extended tarball") will hide this
> > > issue. But the issue is still there. One will have to keep adding
> > > PACKAGECONFIG options for each configuration that can be out of
> > > sync
> > > between host and buildtools tarball.
> > 
> > But that means we have a non-determinism problem: if qemu is picking
> > up things from the host without being explicitly told to do so, then
> > the right fix is indeed to add the missing PACKAGECONFIGs so those
> > things are either explicitly enabled or explicitly disabled.  This
> > patch seems to be working around the non-determinism, rather than
> > addressing it.
> 
> PACKAGECONFIG improves determinism, but it is not sufficient. It is not
> feasible to make sure that PACKAGECONFIGs cover all configurations.
> There will be always something that slips through. I think
> acknowledging this fact was one of the reasons for recipe-specific
> sysroots.

We are supposed to have PACKAGECONFIG for all the options a given bit
of software has. Setting those options is meant to be deterministic. I
appreciate that some recipes don't have complete coverage but they
should have.

Recipe specific sysroots is there as an extra layer of protection but
it doesn't mean PACKAGECONFIG shouldn't be specified or working. RSS
was added for multiple reasons, this is only a small piece of it.

> We use buildtools tarball to better isolate OE build from host
> environment and improve determinism. Host pkg-config opens a big hole
> for non-determinism and plugging it with PACKAGECONFIGs seems fragile.

You make this sound like we do that a lot, we don't. There are some
isolated and very specific cases. Ensuring the PACKAGECONFIG handling
in those cases is correct should be managable.

The question is therefore whether there is a problem with PACKAGECONFIG
in master. I appreciate zeus may have a bigger issue.

Cheers,

Richard

Re: [OE-core] [PATCH] qemu: use host pkg-config only when host provides sdl

[Taras Kondratiuk]

On Wed, 2020-07-15 at 21:17 +0100, Richard Purdie wrote:
> On Wed, 2020-07-15 at 18:54 +0000, Taras Kondratiuk (takondra) wrote:
> > On Wed, 2020-07-15 at 19:23 +0200, Alexander Kanavin wrote:
> > > On Wed, 15 Jul 2020 at 19:09, Taras Kondratiuk (takondra) <
> > > takondra@cisco.com> wrote:
> > PACKAGECONFIG improves determinism, but it is not sufficient. It is
> > not
> > feasible to make sure that PACKAGECONFIGs cover all configurations.
> > There will be always something that slips through. I think
> > acknowledging this fact was one of the reasons for recipe-specific
> > sysroots.
> 
> We are supposed to have PACKAGECONFIG for all the options a given bit
> of software has. Setting those options is meant to be deterministic.
> I
> appreciate that some recipes don't have complete coverage but they
> should have.
> 
> Recipe specific sysroots is there as an extra layer of protection but
> it doesn't mean PACKAGECONFIG shouldn't be specified or working. RSS
> was added for multiple reasons, this is only a small piece of it.

I think we are on the same page here. Ideally recipes should have all
configs covered by PACKAGECONFIG, but in practice the extra layer of
protection is needed.

> > We use buildtools tarball to better isolate OE build from host
> > environment and improve determinism. Host pkg-config opens a big
> > hole
> > for non-determinism and plugging it with PACKAGECONFIGs seems
> > fragile.
> 
> You make this sound like we do that a lot, we don't. There are some
> isolated and very specific cases. Ensuring the PACKAGECONFIG handling
> in those cases is correct should be managable.
> 
> The question is therefore whether there is a problem with
> PACKAGECONFIG
> in master. I appreciate zeus may have a bigger issue.

Right now master should not have this specific issue with xkbcommon.
But I'm not sure I've got your point. Do you want to rely on
PACKAGECONFIGs being perfect in this case?