bitbake-devel.lists.openembedded.org archive mirror
 help / color / mirror / Atom feed
* Re: [bitbake-devel] [RFC][PATCH] providers.sh: versionVariableMatch: use the name of provided item instead of pn
       [not found] <1748E9A04D5EFD06.26202@lists.openembedded.org>
@ 2023-03-03 13:25 ` Martin Jansa
       [not found] ` <1748EB9FF7582344.26202@lists.openembedded.org>
  1 sibling, 0 replies; 2+ messages in thread
From: Martin Jansa @ 2023-03-03 13:25 UTC (permalink / raw)
  To: Martin.Jansa; +Cc: bitbake-devel, khem.raj

[-- Attachment #1: Type: text/plain, Size: 7117 bytes --]

Please do not merge, there are couple of typos in commit message and I'm
starting to prefer to change this in meta-clang instead as in:
https://github.com/shr-project/meta-clang/commit/61e8b5635db25e55757157b55ad63a66e4286d96

On Fri, Mar 3, 2023 at 1:49 PM Martin Jansa via lists.openembedded.org
<Martin.Jansa=gmail.com@lists.openembedded.org> wrote:

> * when checking for PREFERRED_VERSION and REQUIRED_VERSION use
>   the name of the preferred item, not the pn of the provider
>
> * sending as RFC, because there might be some other uses of
>   this function where the old behavior was better
>
> * the case where this seems to work incorrectly is meta-clang clang recipe
>   which correctly says:
>
>   conf/layer.conf:LLVMVERSION = "15.0.7"
>   conf/layer.conf:PREFERRED_PROVIDER_llvm-native = "clang-native"
>   recipes-devtools/clang/clang_git.bb:PROVIDES:append:class-native = "
> llvm-native"
>
>   LLVMVERSION is used by oe-core as:
>   meta/conf/distro/include/tcmode-default.inc:LLVMVERSION ?= "15.%"
>   meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_llvm =
> "${LLVMVERSION}"
>
> meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_llvm-native =
> "${LLVMVERSION}"
>
> meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_nativesdk-llvm
> = "${LLVMVERSION}"
>
>   and now this code will cause a warning about 15.0.7 version not being
> available
>   even when in the end it will use clang-native as a provider with this
> version:
>
>   $ bitbake-getvar -r llvm-native PV 2>&1 | tee
> log.bitbake-getvar.llvm-native-4
>   WARNING: preferred version 15.0.7 of llvm-native not available (for item
> llvm-native)
>   WARNING: versions of llvm-native available: 15.0.6
>   #
>   # $PV [4 operations]
>   #   set oe-core/meta/conf/bitbake.conf:239
>   #     "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or
> '1.0'}"
>   #   set oe-core/meta/conf/documentation.conf:340
>   #     [doc] "The version of the recipe. The version is normally
> extracted from the recipe filename."
>   #   set oe-core/meta/classes-global/sstate.bbclass:51
>   #     [vardepvalue] "${PV}"
>   #   set meta-clang/recipes-devtools/clang/clang.inc:13
>   #     "${MAJOR_VER}.${MINOR_VER}.${PATCH_VER}"
>   # pre-expansion value:
>   #   "${MAJOR_VER}.${MINOR_VER}.${PATCH_VER}"
>   PV="15.0.7"
>
>   the problem is that both llvm-native from oe-core and clang-native are
>   found as possible providers for item llvm-native, but then we call
>   findPreferredProvider for pkg_pn clang-native, we'll check for
>   PREFERRED_VERSION_clang_native not PREFERRED_VERSION_llvm-native
>
>   then when running findPreferredProvider for pkg_pn llvm-native
>   we find the PREFERRED_VERSION_llvm-native and if it doesn't
>   match with PV we issue the warning, be aware that this issue
>   is reproducible only when they don't happen to match, e.g. with
>   master before:
>
> https://git.openembedded.org/openembedded-core/commit/?id=7438f9a9efaf0826f1be13acbc51f0a1134cf175
>   in dunfell it's triggered as well, with different versions:
>

this was with kirkstone not dunfell


>
>   WARNING: preferred version 14.0.3 of llvm-native not available (for item
> llvm-native)
>   WARNING: versions of llvm-native available: 13.0.1
>
> * if we agree that this is something worth changing, then the
>   search for available versions should be similarly changed
>   as well, because now in dunfell it shows only 13.0.1
>   while clang-native-14.0.3 is also available provider for
>   it (and actually gets used in the end).
>
> * this might make some sense for llvm-native and clang-native
>   but I can imagine counter example when you have 2 very different
>   providers for something and you want to control their versions
>   independently from which provider is preferred e.g. by MACHINE.conf
>

The counter example could be something like

# Prefer latest-greatest from fancy layer instead of boring 22.3.5 from
oe-core
PREFERRED_VERSION_virtual/libgles3 ?= "23.0.0"

and some MACHINEs using
PREFERRED_PROVIDER_virtual/libgles3 ?= "mesa"
while others (breaking multi-machine builds if they happen to use the same
TUNE_PKGARCH):
PREFERRED_PROVIDER_virtual/libgles3 ?= "some-ugly-blobs-from-vendor"

With some-ugly-blobs-from-vendor recipe having PV = "0.0.1" because that's
what vendors do.

The current behavior where you can independently set
PREFERRED_VERSION_mesa and PREFERRED_VERSION_some-ugly-blobs-from-vendor

and leave PREFERRED_PROVIDER to choose between them is cleaner than setting
PREFERRED_VERSION_virtual/libgles3 conditionally based on which provider is
preferred.


> * this change didn't cause any new warnings in my builds, but other
>   layers might be more creative with P_V/P_P
>
> * alternative solution might be to use different variable name
>   in meta-clang/conf/layer.conf to avoid changing llvm-native P_V
>   and depend on P_P to always select clang recipes to provide llvm
>   which might be less controversial than this P_V behavoir change
>
>   I have a local commit which changes LLVMVERSION to CLANGLLVMVERSION
>   in these 3 meta-clang places:
>   conf/layer.conf:LLVMVERSION = "14.0.3"
>   dynamic-layers/openembedded-layer/recipes-devtools/bcc/bcc_0.24.0.bb:
>   -DLLVM_PACKAGE_VERSION=${LLVMVERSION} \
>   dynamic-layers/openembedded-layer/recipes-devtools/bpftrace/
> bpftrace_0.14.1.bb:    pvsplit = d.getVar('LLVMVERSION').split('.')
>

 Pushed to my meta-clang fork:
https://github.com/shr-project/meta-clang/commit/61e8b5635db25e55757157b55ad63a66e4286d96
will send PR to Khem if we agree that this is indeed better solution (at
least until LLVMVERSION gets used for other things than P_V in oe-core and
we'll need to be able to easily change it from other layers).

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  lib/bb/providers.py | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/lib/bb/providers.py b/lib/bb/providers.py
> index e11a4637d..1aa0d1282 100644
> --- a/lib/bb/providers.py
> +++ b/lib/bb/providers.py
> @@ -124,8 +124,8 @@ def findPreferredProvider(pn, cfgData, dataCache,
> pkg_pn = None, item = None):
>      preferred_ver = None
>      required = False
>
> -    required_v = versionVariableMatch(cfgData, "REQUIRED", pn)
> -    preferred_v = versionVariableMatch(cfgData, "PREFERRED", pn)
> +    required_v = versionVariableMatch(cfgData, "REQUIRED", item)
> +    preferred_v = versionVariableMatch(cfgData, "PREFERRED", item)
>
>      itemstr = ""
>      if item:
> --
> 2.39.2
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14521):
> https://lists.openembedded.org/g/bitbake-devel/message/14521
> Mute This Topic: https://lists.openembedded.org/mt/97360853/3617156
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> Martin.Jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 9748 bytes --]

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: [bitbake-devel] [RFC][PATCH] providers.sh: versionVariableMatch: use the name of provided item instead of pn
       [not found] ` <1748EB9FF7582344.26202@lists.openembedded.org>
@ 2023-03-06 16:04   ` Martin Jansa
  0 siblings, 0 replies; 2+ messages in thread
From: Martin Jansa @ 2023-03-06 16:04 UTC (permalink / raw)
  To: Martin.Jansa; +Cc: bitbake-devel, khem.raj

[-- Attachment #1: Type: text/plain, Size: 8780 bytes --]

FWIW: the suggested solution for meta-clang works with master/mickledore,
but doesn't work for langdale and kirkstone as those were using LLVMVERSION
for other things as well:

meta/classes-recipe/meson.bbclass:llvm-config = 'llvm-config${LLVMVERSION}'
meta/conf/distro/include/tcmode-default.inc:LLVMVERSION ?= "14.0.6"
meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_llvm =
"${LLVMVERSION}"
meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_llvm-native =
"${LLVMVERSION}"
meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_nativesdk-llvm
= "${LLVMVERSION}"
meta/recipes-graphics/mesa/mesa.inc:MESA_LLVM_RELEASE ?= "${LLVMVERSION}"

So e.g. with kirkstone when meta-clang provides llvm-config14.0.6 mesa will
still search for llvm-config13.0.1 as in meson.bbclass and fails with:

WARNING: Ignoring LLVM CMake dependency because dynamic was requested
llvm-config binary for 1 specified from cross file, native file, or env var
as ['llvm-config13.0.1']
llvm-config found: NO need ['>= 3.9.0']

MESA_LLVM_RELEASE can be easily set in meta-clang with CLANGLLVMVERSION,
but we would need separate variable for llvm-config used by meson.bbclass
and at that point it might be easier to just drop P_V_*llvm* to avoid
confusing warning about unavailable version.

On Fri, Mar 3, 2023 at 2:25 PM Martin Jansa via lists.openembedded.org
<Martin.Jansa=gmail.com@lists.openembedded.org> wrote:

> Please do not merge, there are couple of typos in commit message and I'm
> starting to prefer to change this in meta-clang instead as in:
>
> https://github.com/shr-project/meta-clang/commit/61e8b5635db25e55757157b55ad63a66e4286d96
>
> On Fri, Mar 3, 2023 at 1:49 PM Martin Jansa via lists.openembedded.org
> <Martin.Jansa=gmail.com@lists.openembedded.org> wrote:
>
>> * when checking for PREFERRED_VERSION and REQUIRED_VERSION use
>>   the name of the preferred item, not the pn of the provider
>>
>> * sending as RFC, because there might be some other uses of
>>   this function where the old behavior was better
>>
>> * the case where this seems to work incorrectly is meta-clang clang recipe
>>   which correctly says:
>>
>>   conf/layer.conf:LLVMVERSION = "15.0.7"
>>   conf/layer.conf:PREFERRED_PROVIDER_llvm-native = "clang-native"
>>   recipes-devtools/clang/clang_git.bb:PROVIDES:append:class-native = "
>> llvm-native"
>>
>>   LLVMVERSION is used by oe-core as:
>>   meta/conf/distro/include/tcmode-default.inc:LLVMVERSION ?= "15.%"
>>   meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_llvm =
>> "${LLVMVERSION}"
>>
>> meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_llvm-native =
>> "${LLVMVERSION}"
>>
>> meta/conf/distro/include/tcmode-default.inc:PREFERRED_VERSION_nativesdk-llvm
>> = "${LLVMVERSION}"
>>
>>   and now this code will cause a warning about 15.0.7 version not being
>> available
>>   even when in the end it will use clang-native as a provider with this
>> version:
>>
>>   $ bitbake-getvar -r llvm-native PV 2>&1 | tee
>> log.bitbake-getvar.llvm-native-4
>>   WARNING: preferred version 15.0.7 of llvm-native not available (for
>> item llvm-native)
>>   WARNING: versions of llvm-native available: 15.0.6
>>   #
>>   # $PV [4 operations]
>>   #   set oe-core/meta/conf/bitbake.conf:239
>>   #     "${@bb.parse.vars_from_file(d.getVar('FILE', False),d)[1] or
>> '1.0'}"
>>   #   set oe-core/meta/conf/documentation.conf:340
>>   #     [doc] "The version of the recipe. The version is normally
>> extracted from the recipe filename."
>>   #   set oe-core/meta/classes-global/sstate.bbclass:51
>>   #     [vardepvalue] "${PV}"
>>   #   set meta-clang/recipes-devtools/clang/clang.inc:13
>>   #     "${MAJOR_VER}.${MINOR_VER}.${PATCH_VER}"
>>   # pre-expansion value:
>>   #   "${MAJOR_VER}.${MINOR_VER}.${PATCH_VER}"
>>   PV="15.0.7"
>>
>>   the problem is that both llvm-native from oe-core and clang-native are
>>   found as possible providers for item llvm-native, but then we call
>>   findPreferredProvider for pkg_pn clang-native, we'll check for
>>   PREFERRED_VERSION_clang_native not PREFERRED_VERSION_llvm-native
>>
>>   then when running findPreferredProvider for pkg_pn llvm-native
>>   we find the PREFERRED_VERSION_llvm-native and if it doesn't
>>   match with PV we issue the warning, be aware that this issue
>>   is reproducible only when they don't happen to match, e.g. with
>>   master before:
>>
>> https://git.openembedded.org/openembedded-core/commit/?id=7438f9a9efaf0826f1be13acbc51f0a1134cf175
>>   in dunfell it's triggered as well, with different versions:
>>
>
> this was with kirkstone not dunfell
>
>
>>
>>   WARNING: preferred version 14.0.3 of llvm-native not available (for
>> item llvm-native)
>>   WARNING: versions of llvm-native available: 13.0.1
>>
>> * if we agree that this is something worth changing, then the
>>   search for available versions should be similarly changed
>>   as well, because now in dunfell it shows only 13.0.1
>>   while clang-native-14.0.3 is also available provider for
>>   it (and actually gets used in the end).
>>
>> * this might make some sense for llvm-native and clang-native
>>   but I can imagine counter example when you have 2 very different
>>   providers for something and you want to control their versions
>>   independently from which provider is preferred e.g. by MACHINE.conf
>>
>
> The counter example could be something like
>
> # Prefer latest-greatest from fancy layer instead of boring 22.3.5 from
> oe-core
> PREFERRED_VERSION_virtual/libgles3 ?= "23.0.0"
>
> and some MACHINEs using
> PREFERRED_PROVIDER_virtual/libgles3 ?= "mesa"
> while others (breaking multi-machine builds if they happen to use the same
> TUNE_PKGARCH):
> PREFERRED_PROVIDER_virtual/libgles3 ?= "some-ugly-blobs-from-vendor"
>
> With some-ugly-blobs-from-vendor recipe having PV = "0.0.1" because that's
> what vendors do.
>
> The current behavior where you can independently set
> PREFERRED_VERSION_mesa and PREFERRED_VERSION_some-ugly-blobs-from-vendor
>
> and leave PREFERRED_PROVIDER to choose between them is cleaner than
> setting PREFERRED_VERSION_virtual/libgles3 conditionally based on which
> provider is preferred.
>
>
>> * this change didn't cause any new warnings in my builds, but other
>>   layers might be more creative with P_V/P_P
>>
>> * alternative solution might be to use different variable name
>>   in meta-clang/conf/layer.conf to avoid changing llvm-native P_V
>>   and depend on P_P to always select clang recipes to provide llvm
>>   which might be less controversial than this P_V behavoir change
>>
>>   I have a local commit which changes LLVMVERSION to CLANGLLVMVERSION
>>   in these 3 meta-clang places:
>>   conf/layer.conf:LLVMVERSION = "14.0.3"
>>   dynamic-layers/openembedded-layer/recipes-devtools/bcc/bcc_0.24.0.bb:
>>   -DLLVM_PACKAGE_VERSION=${LLVMVERSION} \
>>   dynamic-layers/openembedded-layer/recipes-devtools/bpftrace/
>> bpftrace_0.14.1.bb:    pvsplit = d.getVar('LLVMVERSION').split('.')
>>
>
>  Pushed to my meta-clang fork:
>
> https://github.com/shr-project/meta-clang/commit/61e8b5635db25e55757157b55ad63a66e4286d96
> will send PR to Khem if we agree that this is indeed better solution (at
> least until LLVMVERSION gets used for other things than P_V in oe-core and
> we'll need to be able to easily change it from other layers).
>
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
>> ---
>>  lib/bb/providers.py | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/bb/providers.py b/lib/bb/providers.py
>> index e11a4637d..1aa0d1282 100644
>> --- a/lib/bb/providers.py
>> +++ b/lib/bb/providers.py
>> @@ -124,8 +124,8 @@ def findPreferredProvider(pn, cfgData, dataCache,
>> pkg_pn = None, item = None):
>>      preferred_ver = None
>>      required = False
>>
>> -    required_v = versionVariableMatch(cfgData, "REQUIRED", pn)
>> -    preferred_v = versionVariableMatch(cfgData, "PREFERRED", pn)
>> +    required_v = versionVariableMatch(cfgData, "REQUIRED", item)
>> +    preferred_v = versionVariableMatch(cfgData, "PREFERRED", item)
>>
>>      itemstr = ""
>>      if item:
>> --
>> 2.39.2
>>
>>
>>
>>
>>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14522):
> https://lists.openembedded.org/g/bitbake-devel/message/14522
> Mute This Topic: https://lists.openembedded.org/mt/97360853/3617156
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> Martin.Jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 11807 bytes --]

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2023-03-06 16:04 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
     [not found] <1748E9A04D5EFD06.26202@lists.openembedded.org>
2023-03-03 13:25 ` [bitbake-devel] [RFC][PATCH] providers.sh: versionVariableMatch: use the name of provided item instead of pn Martin Jansa
     [not found] ` <1748EB9FF7582344.26202@lists.openembedded.org>
2023-03-06 16:04   ` Martin Jansa

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).