how often would one use "VAR_someoverride_append = ..."?

how often would one use "VAR_someoverride_append = ..."?

[Robert P. J. Day]

  bitbake manual, chapter 3, examples of conditional syntax:

https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#examples

correctly distinguishes between A_foo_append and A_append_foo, but how
often would one use that first form, anyway?

  most uses of conditional appending are either just straight
appending:

  VAR_append = "fubar"

or used with an override thusly:

  VAR_append_x86 = "snafu"

is there an actual practical usage of, say:

  VAR_x86_append = "huh"

i can't remember the last time i saw something of that form and,
while it might be worth explaining, it seems that the reader might be
warned that that form is almost certainly *not* what they want.

  thoughts?

rday

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Quentin Schulz]

Hi Robert,

On Tue, Mar 09, 2021 at 09:39:14AM -0500, Robert P. J. Day wrote:
> 
>   bitbake manual, chapter 3, examples of conditional syntax:
> 
> https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#examples
> 
> correctly distinguishes between A_foo_append and A_append_foo, but how
> often would one use that first form, anyway?
> 
>   most uses of conditional appending are either just straight
> appending:
> 
>   VAR_append = "fubar"
> 
> or used with an override thusly:
> 
>   VAR_append_x86 = "snafu"
> 
> is there an actual practical usage of, say:
> 
>   VAR_x86_append = "huh"
> 
> i can't remember the last time i saw something of that form and,
> while it might be worth explaining, it seems that the reader might be
> warned that that form is almost certainly *not* what they want.
> 

Yes, in 99% of the cases, you want VAR_append_foo and not VAR_foo_append.

VAR_foo_append makes sense when you want to append to VAR_foo which is a
way to override completely VAR for builds matching the foo override.
This happens in kernel-yocto recipes where branches and defconfigs are
different per machine for example.

The sneaky thing about VAR_foo_append is that it creates VAR_foo even if
it didn't exist beforehand, meaning you though you *appended* something
to VAR while you actually replaced VAR entirely with what VAR_foo_append
is set too.

Quentin

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Robert P. J. Day]

On Tue, 9 Mar 2021, Quentin Schulz wrote:

> Hi Robert,
>
> On Tue, Mar 09, 2021 at 09:39:14AM -0500, Robert P. J. Day wrote:
> >
> >   bitbake manual, chapter 3, examples of conditional syntax:
> >
> > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#examples
> >
> > correctly distinguishes between A_foo_append and A_append_foo, but how
> > often would one use that first form, anyway?
> >
> >   most uses of conditional appending are either just straight
> > appending:
> >
> >   VAR_append = "fubar"
> >
> > or used with an override thusly:
> >
> >   VAR_append_x86 = "snafu"
> >
> > is there an actual practical usage of, say:
> >
> >   VAR_x86_append = "huh"
> >
> > i can't remember the last time i saw something of that form and,
> > while it might be worth explaining, it seems that the reader might be
> > warned that that form is almost certainly *not* what they want.
> >
>
> Yes, in 99% of the cases, you want VAR_append_foo and not
> VAR_foo_append.
>
> VAR_foo_append makes sense when you want to append to VAR_foo which
> is a way to override completely VAR for builds matching the foo
> override. This happens in kernel-yocto recipes where branches and
> defconfigs are different per machine for example.
>
> The sneaky thing about VAR_foo_append is that it creates VAR_foo
> even if it didn't exist beforehand, meaning you though you
> *appended* something to VAR while you actually replaced VAR entirely
> with what VAR_foo_append is set too.

  i remember testing this a few years back, and making sure i
appreciated the subtleties. i might take a shot at rewriting that
section, since i think it has potential for confusion. mostly, warning
the reader that the first form is almost always what they want.

rday

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Leon Woestenberg]

Hello Robert, all,

On Tue, Mar 9, 2021 at 3:39 PM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> is there an actual practical usage of, say:
>
>   VAR_x86_append = "huh"
>
$grep -re '[A-Z_]\+_[a-z0-9]\+_append' poky/meta*
poky/meta/recipes-support/gmp/gmp_6.2.0.bb:EXTRA_OECONF_mipsarchr6_append
= " --disable-assembly"
poky/meta/recipes-core/images/build-appliance-image_15.0.0.bb:IMAGE_CMD_ext4_append
() {
poky/meta/recipes-devtools/python/python3_3.8.2.bb:RDEPENDS_libpython3_append_libc-glibc
= " libgcc"
poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdb_append_linux
= " glibc-thread-db "
poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdb_append_linux-gnueabi
= " glibc-thread-db "
poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdbserver_append_linux
= " glibc-thread-db "
poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdbserver_append_linux-gnueabi
= " glibc-thread-db "

which is appending an override variable conditionally with another
override condition. (If I put this correctly.) Is this a valid use
case where += would not work?

Other appearances without an override condition:
$ grep -re '[A-Z_]\+_[a-z0-9]\+_append' meta-*
meta-openembedded/<..>/libnma_1.8.28.bb:EXTRA_OEMESON_mipsarchn32_append
= " -Dvapi=false"
meta-openembedded/<..>/libnma_1.8.28.bb:EXTRA_OEMESON_riscv32_append =
" -Dvapi=false"

Could these have been replaced with the following?

EXTRA_OEMESON_mipsarchn32 += "-Dvapi=false"
EXTRA_OEMESON_riscv32 += "-Dvapi=false"

Seems like a good test case indeed.

Regards,

Leon.

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Quentin Schulz]

Hi Leon,

On Tue, Mar 09, 2021 at 03:55:03PM +0100, Leon Woestenberg wrote:
> Hello Robert, all,
> 
> On Tue, Mar 9, 2021 at 3:39 PM Robert P. J. Day <rpjday@crashcourse.ca> wrote:
> > is there an actual practical usage of, say:
> >
> >   VAR_x86_append = "huh"
> >
> $grep -re '[A-Z_]\+_[a-z0-9]\+_append' poky/meta*
> poky/meta/recipes-support/gmp/gmp_6.2.0.bb:EXTRA_OECONF_mipsarchr6_append
> = " --disable-assembly"
> poky/meta/recipes-core/images/build-appliance-image_15.0.0.bb:IMAGE_CMD_ext4_append
> () {
> poky/meta/recipes-devtools/python/python3_3.8.2.bb:RDEPENDS_libpython3_append_libc-glibc
> = " libgcc"
> poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdb_append_linux
> = " glibc-thread-db "
> poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdb_append_linux-gnueabi
> = " glibc-thread-db "
> poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdbserver_append_linux
> = " glibc-thread-db "
> poky/meta/recipes-devtools/gdb/gdb-common.inc:RRECOMMENDS_gdbserver_append_linux-gnueabi
> = " glibc-thread-db "
> 
> which is appending an override variable conditionally with another
> override condition. (If I put this correctly.) Is this a valid use
> case where += would not work?
> 

Define what you mean by "where += would not work", I'm not sure to have
enough context to answer this question.

gdb/gdbserver above would be replaced with ${PN} in many/most recipes.
I am not sure how ${PN} "overrides" are handled but that would be a
valid use case for VAR_foo_append.

I think Robert and I were talking about foo being part of OVERRIDES
variable meaning we (I at least) oversaw this, thanks for the heads up.

> Other appearances without an override condition:
> $ grep -re '[A-Z_]\+_[a-z0-9]\+_append' meta-*
> meta-openembedded/<..>/libnma_1.8.28.bb:EXTRA_OEMESON_mipsarchn32_append
> = " -Dvapi=false"
> meta-openembedded/<..>/libnma_1.8.28.bb:EXTRA_OEMESON_riscv32_append =
> " -Dvapi=false"
> 
> Could these have been replaced with the following?
> 
> EXTRA_OEMESON_mipsarchn32 += "-Dvapi=false"
> EXTRA_OEMESON_riscv32 += "-Dvapi=false"
> 

Considering there are no EXTRA_OEMESON_mipsarchn32 in any of the classes
included and that EXTRA_OEMESON expects a space-separated list of
strings, it'd probably be ok.

However, this seems fishy... It is possible that this is a mistake and
one should have written EXTRA_OEMESON_append_mipsarchn32...

If EXTRA_OEMESON was intended to be overwritten completly for
mipsarchn32 to disable gobject-introspection, having
GI_DATA_ENABLED_mipsarchn32 = "False"  and
EXTRA_OEMESON_append_mipsarchn32 = " -Dvapi=false"  would be more
suitable.

If I were you, I would ask on the ML if it was intended and adding the
committer in Cc :)

Quentin

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Robert P. J. Day]

On Tue, 9 Mar 2021, Quentin Schulz wrote:

> Hi Robert,
>
> On Tue, Mar 09, 2021 at 09:39:14AM -0500, Robert P. J. Day wrote:
> >
> >   bitbake manual, chapter 3, examples of conditional syntax:
> >
> > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#examples
> >
> > correctly distinguishes between A_foo_append and A_append_foo, but how
> > often would one use that first form, anyway?
> >
> >   most uses of conditional appending are either just straight
> > appending:
> >
> >   VAR_append = "fubar"
> >
> > or used with an override thusly:
> >
> >   VAR_append_x86 = "snafu"
> >
> > is there an actual practical usage of, say:
> >
> >   VAR_x86_append = "huh"
> >
> > i can't remember the last time i saw something of that form and,
> > while it might be worth explaining, it seems that the reader might be
> > warned that that form is almost certainly *not* what they want.
> >
>
> Yes, in 99% of the cases, you want VAR_append_foo and not VAR_foo_append.
>
> VAR_foo_append makes sense when you want to append to VAR_foo which
> is a way to override completely VAR for builds matching the foo
> override. This happens in kernel-yocto recipes where branches and
> defconfigs are different per machine for example.

  can you point at an actual example of that? i took a look and all
the yocto kernel recipes i see use the first form. am i just looking
in the wrong place?

rday

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Quentin Schulz]

Hi Robert,

On Wed, Mar 10, 2021 at 06:16:44AM -0500, Robert P. J. Day wrote:
> On Tue, 9 Mar 2021, Quentin Schulz wrote:
> 
> > Hi Robert,
> >
> > On Tue, Mar 09, 2021 at 09:39:14AM -0500, Robert P. J. Day wrote:
> > >
> > >   bitbake manual, chapter 3, examples of conditional syntax:
> > >
> > > https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#examples
> > >
> > > correctly distinguishes between A_foo_append and A_append_foo, but how
> > > often would one use that first form, anyway?
> > >
> > >   most uses of conditional appending are either just straight
> > > appending:
> > >
> > >   VAR_append = "fubar"
> > >
> > > or used with an override thusly:
> > >
> > >   VAR_append_x86 = "snafu"
> > >
> > > is there an actual practical usage of, say:
> > >
> > >   VAR_x86_append = "huh"
> > >
> > > i can't remember the last time i saw something of that form and,
> > > while it might be worth explaining, it seems that the reader might be
> > > warned that that form is almost certainly *not* what they want.
> > >
> >
> > Yes, in 99% of the cases, you want VAR_append_foo and not VAR_foo_append.
> >
> > VAR_foo_append makes sense when you want to append to VAR_foo which
> > is a way to override completely VAR for builds matching the foo
> > override. This happens in kernel-yocto recipes where branches and
> > defconfigs are different per machine for example.
> 
>   can you point at an actual example of that? i took a look and all
> the yocto kernel recipes i see use the first form. am i just looking
> in the wrong place?
> 

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n12
for example.

This is an example of a "valid" use case (not that there are invalid
ones) for VAR_foo.

Would probably a better example:
https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n50

So, you might decide that a machine is so much different than others
that KERNEL_FEATURES should be overridden for said machine.

Then in a bbappend, one might want to add another feature for this
machine, they'll therefore need to use KERNEL_FEATURES_foo_append.

I do not have examples at hand of VAR_foo_append except the ones Leon
sent in another mail. Which should show how rare it is :)

Cheers,
Quentin

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Robert P. J. Day]

On Wed, 10 Mar 2021, Quentin Schulz wrote:
... snip ...

> https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n12
> for example.
>
> This is an example of a "valid" use case (not that there are invalid
> ones) for VAR_foo.
>
> Would probably a better example:
> https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n50
>
> So, you might decide that a machine is so much different than others
> that KERNEL_FEATURES should be overridden for said machine.
>
> Then in a bbappend, one might want to add another feature for this
> machine, they'll therefore need to use KERNEL_FEATURES_foo_append.
>
> I do not have examples at hand of VAR_foo_append except the ones
> Leon sent in another mail. Which should show how rare it is :)

  i will ramble for just a bit as i think i settled on a way to
perhaps explain the difference here to some students.

  in the overwhelmingly common case, we normally see overrides at the
end of either assignments or appends:

  VAR = "snafu"
  VAR_qemux86 = "more snafu"
  VAR_append_qemux86 = "qemux86_specific_fubar"

it seems that the proper way to interpret the above is that, in the
end, all you would be after is the *final* value of the single
variable VAR, and all the appending and overriding is leading you in
that direction -- whatever "VAR" contains after it's all over is all
you care about.

  on the other hand, with the example from the kernel recipe:

  KBRANCH_qemuarm  ?= "v5.10/standard/arm-versatile-926ejs"
  KBRANCH_qemuarm64 ?= "v5.10/standard/qemuarm64"
  KBRANCH_qemumips ?= "v5.10/standard/mti-malta32"
  KBRANCH_qemuppc  ?= "v5.10/standard/qemuppc"
  KBRANCH_qemuriscv64  ?= "v5.10/standard/base"
  KBRANCH_qemux86  ?= "v5.10/standard/base"
  KBRANCH_qemux86-64 ?= "v5.10/standard/base"
  KBRANCH_qemumips64 ?= "v5.10/standard/mti-malta64"

it seems that the right way to interpret the above is that you are,
simultaneously, creating *multiple* versions of the variable KBRANCH,
so that if you subsequently see, for example:

  KBRANCH_qemux86_append = " whatever"

you're appending to the variable KBRANCH_qemux86 because the ultimate
result you're after is the *set* of KBRANCH-related variables, perhaps
because you will actually need them all at once.

  and given that most situations don't require access to all of those
variations of that variable, that's why the first approach is by far
the most common.

  that might be an over-simplification, but it seems to at least give
an idea of why that second approach would be used in very particular
situations.

  thoughts?

rday

Re: [yocto] how often would one use "VAR_someoverride_append = ..."?

[Quentin Schulz]

Hi Robert,

On Thu, Mar 11, 2021 at 04:27:46AM -0500, Robert P. J. Day wrote:
> On Wed, 10 Mar 2021, Quentin Schulz wrote:
> ... snip ...
> 
> > https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n12
> > for example.
> >
> > This is an example of a "valid" use case (not that there are invalid
> > ones) for VAR_foo.
> >
> > Would probably a better example:
> > https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-kernel/linux/linux-yocto_5.10.bb#n50
> >
> > So, you might decide that a machine is so much different than others
> > that KERNEL_FEATURES should be overridden for said machine.
> >
> > Then in a bbappend, one might want to add another feature for this
> > machine, they'll therefore need to use KERNEL_FEATURES_foo_append.
> >
> > I do not have examples at hand of VAR_foo_append except the ones
> > Leon sent in another mail. Which should show how rare it is :)
> 
>   i will ramble for just a bit as i think i settled on a way to
> perhaps explain the difference here to some students.
> 
>   in the overwhelmingly common case, we normally see overrides at the
> end of either assignments or appends:
> 
>   VAR = "snafu"
>   VAR_qemux86 = "more snafu"
>   VAR_append_qemux86 = "qemux86_specific_fubar"
> 
> it seems that the proper way to interpret the above is that, in the
> end, all you would be after is the *final* value of the single
> variable VAR, and all the appending and overriding is leading you in
> that direction -- whatever "VAR" contains after it's all over is all
> you care about.
> 

For that particular example, I'm not entirely sure what the actual value
for VAR would be.

Also not sure what:
VAR = "snafu"
VAR_append = " more"
VAR_qemux86 = "qemu"

would mean for VAR. I don't know exactly when _append is applied to
VAR... after or before the override is applied?

Gut feeling would say before, in which case for non-qemux86 one would
have VAR set to "snafu more" and for qemux86 it'd be "qemu".

I'd anyway use bitbake -e trick to check that, which IMO should really
be given in that section of the docs, to explain how one can debug the
assignment of the variable thanks to the history of the variable :)

>   on the other hand, with the example from the kernel recipe:
> 
>   KBRANCH_qemuarm  ?= "v5.10/standard/arm-versatile-926ejs"
>   KBRANCH_qemuarm64 ?= "v5.10/standard/qemuarm64"
>   KBRANCH_qemumips ?= "v5.10/standard/mti-malta32"
>   KBRANCH_qemuppc  ?= "v5.10/standard/qemuppc"
>   KBRANCH_qemuriscv64  ?= "v5.10/standard/base"
>   KBRANCH_qemux86  ?= "v5.10/standard/base"
>   KBRANCH_qemux86-64 ?= "v5.10/standard/base"
>   KBRANCH_qemumips64 ?= "v5.10/standard/mti-malta64"
> 
> it seems that the right way to interpret the above is that you are,
> simultaneously, creating *multiple* versions of the variable KBRANCH,
> so that if you subsequently see, for example:
> 
>   KBRANCH_qemux86_append = " whatever"
> 
> you're appending to the variable KBRANCH_qemux86 because the ultimate
> result you're after is the *set* of KBRANCH-related variables, perhaps
> because you will actually need them all at once.
> 
>   and given that most situations don't require access to all of those
> variations of that variable, that's why the first approach is by far
> the most common.
> 
>   that might be an over-simplification, but it seems to at least give
> an idea of why that second approach would be used in very particular
> situations.
> 
>   thoughts?
> 

I like to think that:

KBRANCH_qemux86

is for OVERRIDING, while

VAR_append_qemux86

is for CONDITIONAL appending.

In which case,

KBRANCH_qemux86_append

is UNCONDITIONALLY APPENDING to OVERRIDDEN KBRANCH for qemux86 override.

Only _append and _prepend would be a case of CONDITIONAL
appending/prepending. All others are OVERRIDING.

It's a topic I've long been interested in describing with simple words
but couldn't come with anything yet.

Because you kind of need to explain also the difference between:
KBRANCH_qemux86_class-native
and
KBRANCH_class-native_qemux86

which are completely different.

You also have many, many, many different kinds of overrides, from
pn-recipe, to class-target, virtclass-lib32, task-compile, ${PN},
remove, poky, _X.Y.Z when BBVERSIONS is set, etc...

If you omit _append and _prepend, it's simpler to explain that the
overrides happen from right to left. But once you have _append and
_prepend in the mix, it's a bit confusing and hard to explain.

IMO it would warrant some kind of drawings/schemes to explain step by
step what's happening, couldn't think of a proper one though. Thought of
using multiple FIFOs but couldn't come up with something less confusing
that what it already is :/

I'd be very happy to find a way to explain those (IMO) difficult
notions and make it part of the docs.

Quentin