[PATCH v3] bitbake-user-manual: Document override :append, :prepend, :remove order

[PATCH v3] bitbake-user-manual: Document override :append, :prepend, :remove order

[Marek Vasut]

The application order of override syntax :append, :prepend, :remove is
inobvious and undocumented. The order does not match variable parse
history, i.e. output of "bitbake -e", either.

Add note documenting the exact order in which :append, :prepend, :remove
are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
that this order does not match the variable parse history order. This
is mostly to prevent others from running into this and scratching their
heads, trying to find out what is going on.

Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
Signed-off-by: Marek Vasut <marex@denx.de>
Cc: Martin Jansa <martin.jansa@gmail.com>
Cc: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Cc: Quentin Schulz <foss+yocto@0leil.net>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
---
V2: - Add RB from Quentin
    - Mention un-remove implementation
    - Reference :remove order in append/prepend override syntax
V3: - Reword the note per suggestion from Quentin
    - Use explicit FOOREMOVE per suggestion from Peter
---
 .../bitbake-user-manual-metadata.rst          | 26 +++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
index 33782161..587c9120 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
@@ -319,6 +319,10 @@ The variable ``D`` becomes "dvaladditional data".
 
    You must control all spacing when you use the override syntax.
 
+.. note::
+
+   The overrides are applied in this order, ":append", ":prepend", ":remove".
+
 It is also possible to append and prepend to shell functions and
 BitBake-style Python functions. See the ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:shell functions`" and ":ref:`bitbake-user-manual/bitbake-user-manual-metadata:bitbake-style python functions`"
 sections for examples.
@@ -351,6 +355,28 @@ The variable ``FOO`` becomes
 Like ":append" and ":prepend", ":remove" is applied at variable
 expansion time.
 
+.. note::
+
+   The overrides are applied in this order, ":append", ":prepend", ":remove".
+   This implies it is not possible to re-append previously removed strings.
+   However, one can undo a ":remove" by using an intermediate variable whose
+   content is passed to the ":remove" so that modifying the intermediate
+   variable equals to keeping the string in::
+
+     FOOREMOVE = "123 456 789"
+     FOO:remove = "${FOOREMOVE}"
+     ...
+     FOOREMOVE = "123 789"
+
+   This expands to ``FOO:remove = "123 789"``.
+
+.. note::
+
+   Override application order may not match variable parse history, i.e.
+   the output of ``bitbake -e`` may contain ":remove" before ":append",
+   but the result will be removed string, because ":remove" is handled
+   last.
+
 Override Style Operation Advantages
 -----------------------------------
 
-- 
2.35.1

Re: [bitbake-devel] [PATCH v3] bitbake-user-manual: Document override :append, :prepend, :remove order

[Enrico Scholz]

"Marek Vasut" <marex@denx.de> writes:

>  
> +.. note::
> +
> +   The overrides are applied in this order, ":append", ":prepend", ":remove".
> +   This implies it is not possible to re-append previously removed
>     strings.

fwiw, this is possible by anonymous python functions which are evaluated
after ":remove"

| FOO = "123 abc XYZ"
| 
| python () {
|     v = d.getVar('FOO')
|     d.setVar('FOO', v + ' abc 789')
| }
| 
| FOO:remove = "abc XYZ"

results in

| FOO="123   abc 789"



Enrico

Re: [bitbake-devel] [PATCH v3] bitbake-user-manual: Document override :append, :prepend, :remove order

[Marek Vasut]

On 7/25/22 20:14, Enrico Scholz wrote:
> "Marek Vasut" <marex@denx.de> writes:
> 
>>   
>> +.. note::
>> +
>> +   The overrides are applied in this order, ":append", ":prepend", ":remove".
>> +   This implies it is not possible to re-append previously removed
>>      strings.
> 
> fwiw, this is possible by anonymous python functions which are evaluated
> after ":remove"
> 
> | FOO = "123 abc XYZ"
> |
> | python () {
> |     v = d.getVar('FOO')
> |     d.setVar('FOO', v + ' abc 789')
> | }
> |
> | FOO:remove = "abc XYZ"
> 
> results in
> 
> | FOO="123   abc 789"
> 

IIRC anonymous python functions are frowned upon, aren't they ?

Re: [docs] [bitbake-devel] [PATCH v3] bitbake-user-manual: Document override :append, :prepend, :remove order

[Richard Purdie]

On Mon, 2022-07-25 at 20:14 +0200, Enrico Scholz via
lists.yoctoproject.org wrote:
> "Marek Vasut" <marex@denx.de> writes:
> 
> >  
> > +.. note::
> > +
> > +   The overrides are applied in this order, ":append", ":prepend", ":remove".
> > +   This implies it is not possible to re-append previously removed
> >     strings.
> 
> fwiw, this is possible by anonymous python functions which are evaluated
> after ":remove"
> 
> > FOO = "123 abc XYZ"
> > 
> > python () {
> >     v = d.getVar('FOO')
> >     d.setVar('FOO', v + ' abc 789')
> > }
> > 
> > FOO:remove = "abc XYZ"
> 
> results in
> 
> > FOO="123   abc 789"

The reason for this isn't perhaps obvious and worth documenting - use
of setVar clears all append/prepend/remove operations against a
variable.

FOO = "A"
FOO:append = " B"

python () {
    d.setVar('FOO', 'C')
}

results in C.

Cheers,

Richard

Re: [PATCH v3] bitbake-user-manual: Document override :append, :prepend, :remove order

[Marek Vasut]

On 7/25/22 17:56, Marek Vasut wrote:
> The application order of override syntax :append, :prepend, :remove is
> inobvious and undocumented. The order does not match variable parse
> history, i.e. output of "bitbake -e", either.
> 
> Add note documenting the exact order in which :append, :prepend, :remove
> are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
> that this order does not match the variable parse history order. This
> is mostly to prevent others from running into this and scratching their
> heads, trying to find out what is going on.

Can this be applied now or is there something more to change ?