[PATCH] bitbake-user-manual: Document override :append, :prepend, :remove order

[PATCH] bitbake-user-manual: Document override :append, :prepend, :remove order

[Marek Vasut]

The application order of override syntax :append, :prepend, :remove is
inobvious and undocumented. The order does not match variable parse
history, i.e. output of "bitbake -e", either.

Add note documenting the exact order in which :append, :prepend, :remove
are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
that this order does not match the variable parse history order. This
is mostly to prevent others from running into this and scratching their
heads, trying to find out what is going on.

Signed-off-by: Marek Vasut <marex@denx.de>
Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 .../bitbake-user-manual-metadata.rst                 | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
index 33782161..651116ce 100644
--- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
+++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
@@ -351,6 +351,18 @@ The variable ``FOO`` becomes
 Like ":append" and ":prepend", ":remove" is applied at variable
 expansion time.
 
+.. note::
+
+   The overrides are applied in this order, ":append", ":prepend", ":remove".
+   This implies it is not possible to re-append previously removed strings.
+
+.. note::
+
+   Override application order may not match variable parse history, i.e.
+   the output of ``bitbake -e`` may contain ":remove" before ":append",
+   but the result will be removed string, because ":remove" is handled
+   last.
+
 Override Style Operation Advantages
 -----------------------------------
 
-- 
2.35.1

Re: [bitbake-devel] [PATCH] bitbake-user-manual: Document override :append, :prepend, :remove order

[Quentin Schulz]

Hi Marek,

On 7/23/22 03:13, Marek Vasut wrote:
> The application order of override syntax :append, :prepend, :remove is
> inobvious and undocumented. The order does not match variable parse
> history, i.e. output of "bitbake -e", either.
> 
> Add note documenting the exact order in which :append, :prepend, :remove
> are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
> that this order does not match the variable parse history order. This
> is mostly to prevent others from running into this and scratching their
> heads, trying to find out what is going on.
> 
> Signed-off-by: Marek Vasut <marex@denx.de>
> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>   .../bitbake-user-manual-metadata.rst                 | 12 ++++++++++++
>   1 file changed, 12 insertions(+)
> 
> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> index 33782161..651116ce 100644
> --- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> +++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> @@ -351,6 +351,18 @@ The variable ``FOO`` becomes
>   Like ":append" and ":prepend", ":remove" is applied at variable
>   expansion time.
>   
> +.. note::
> +
> +   The overrides are applied in this order, ":append", ":prepend", ":remove".
> +   This implies it is not possible to re-append previously removed strings.
> +
> +.. note::
> +
> +   Override application order may not match variable parse history, i.e.
> +   the output of ``bitbake -e`` may contain ":remove" before ":append",
> +   but the result will be removed string, because ":remove" is handled
> +   last.
> +

I can suggest a more concise note, since I don't think we care about the 
actual order of append and prepend:
```
``:remove`` is final. Any string appearing in ``:remove`` cannot be 
added back regardless of when ``:remove`` was parsed.
```
Up to you :)

Otherwise, welcome addition thanks :)
Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>

I'm surprised this was not mentioned in 
https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#removal-override-style-syntax 
and 
https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#appending-and-prepending-override-style-syntax, 
so adding a few words there would be great too :)

Cheers,
Quentin

>   Override Style Operation Advantages
>   -----------------------------------
>   
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13837): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_message_13837&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=rAdWPF-nAT2ooJM9hyieW5jHTZRB2rlSJfhuxJeBaS29jNG06J_Qzv4xTP81zwUH&s=SzPrWDNzoIr-MjT-x6vXbW8rxAt0l0odSFygeF4NjTE&e=
> Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_92560243_6293953&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=rAdWPF-nAT2ooJM9hyieW5jHTZRB2rlSJfhuxJeBaS29jNG06J_Qzv4xTP81zwUH&s=UaYe1Ue7jye5Pir7dZS1iHXBrTN_mWaZkOBIxibHJHA&e=
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_unsub&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=rAdWPF-nAT2ooJM9hyieW5jHTZRB2rlSJfhuxJeBaS29jNG06J_Qzv4xTP81zwUH&s=qYGfFmF6YZwURBbgyGUOFrulcscOXAZUk22WrexXNA8&e=  [quentin.schulz@theobroma-systems.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [bitbake-devel] [PATCH] bitbake-user-manual: Document override :append, :prepend, :remove order

[Marek Vasut]

On 7/25/22 10:36, Quentin Schulz wrote:
> Hi Marek,

Hi,

> On 7/23/22 03:13, Marek Vasut wrote:
>> The application order of override syntax :append, :prepend, :remove is
>> inobvious and undocumented. The order does not match variable parse
>> history, i.e. output of "bitbake -e", either.
>>
>> Add note documenting the exact order in which :append, :prepend, :remove
>> are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
>> that this order does not match the variable parse history order. This
>> is mostly to prevent others from running into this and scratching their
>> heads, trying to find out what is going on.
>>
>> Signed-off-by: Marek Vasut <marex@denx.de>
>> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
>> ---
>>   .../bitbake-user-manual-metadata.rst                 | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst 
>> b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
>> index 33782161..651116ce 100644
>> --- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
>> +++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
>> @@ -351,6 +351,18 @@ The variable ``FOO`` becomes
>>   Like ":append" and ":prepend", ":remove" is applied at variable
>>   expansion time.
>> +.. note::
>> +
>> +   The overrides are applied in this order, ":append", ":prepend", 
>> ":remove".
>> +   This implies it is not possible to re-append previously removed 
>> strings.
>> +
>> +.. note::
>> +
>> +   Override application order may not match variable parse history, i.e.
>> +   the output of ``bitbake -e`` may contain ":remove" before ":append",
>> +   but the result will be removed string, because ":remove" is handled
>> +   last.
>> +
> 
> I can suggest a more concise note, since I don't think we care about the 
> actual order of append and prepend:
> ```
> ``:remove`` is final. Any string appearing in ``:remove`` cannot be 
> added back regardless of when ``:remove`` was parsed.
> ```
> Up to you :)

I'd argue it's better to keep the information full and complete, esp. 
since its not easy to find within bitbake sources.


> Otherwise, welcome addition thanks :)
> Reviewed-by: Quentin Schulz <foss+yocto@0leil.net>
> 
> I'm surprised this was not mentioned in 
> https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#removal-override-style-syntax 
> and 
> https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#appending-and-prepending-override-style-syntax, 
> so adding a few words there would be great too :)

For appending and prepending, this makes little difference. It's the 
remove which has this nasty catch.

Re: [bitbake-devel] [PATCH] bitbake-user-manual: Document override :append, :prepend, :remove order

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

On Mon, Jul 25, 2022 at 11:40 AM Marek Vasut <marex@denx.de> wrote:

> On 7/25/22 10:36, Quentin Schulz wrote:
> > Hi Marek,
>
> Hi,
>
> > On 7/23/22 03:13, Marek Vasut wrote:
> >> The application order of override syntax :append, :prepend, :remove is
> >> inobvious and undocumented. The order does not match variable parse
> >> history, i.e. output of "bitbake -e", either.
> >>
> >> Add note documenting the exact order in which :append, :prepend, :remove
> >> are applied to variable in lib/bb/data_smart.py getVarFlag(). Add note
> >> that this order does not match the variable parse history order. This
> >> is mostly to prevent others from running into this and scratching their
> >> heads, trying to find out what is going on.
> >>
> >> Signed-off-by: Marek Vasut <marex@denx.de>
> >> Cc: Richard Purdie <richard.purdie@linuxfoundation.org>
> >> ---
> >>   .../bitbake-user-manual-metadata.rst                 | 12 ++++++++++++
> >>   1 file changed, 12 insertions(+)
> >>
> >> diff --git a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> >> b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> >> index 33782161..651116ce 100644
> >> --- a/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> >> +++ b/doc/bitbake-user-manual/bitbake-user-manual-metadata.rst
> >> @@ -351,6 +351,18 @@ The variable ``FOO`` becomes
> >>   Like ":append" and ":prepend", ":remove" is applied at variable
> >>   expansion time.
> >> +.. note::
> >> +
> >> +   The overrides are applied in this order, ":append", ":prepend",
> >> ":remove".
> >> +   This implies it is not possible to re-append previously removed
> >> strings.
> >> +
> >> +.. note::
> >> +
> >> +   Override application order may not match variable parse history,
> i.e.
> >> +   the output of ``bitbake -e`` may contain ":remove" before ":append",
> >> +   but the result will be removed string, because ":remove" is handled
> >> +   last.
> >> +
> >
> > I can suggest a more concise note, since I don't think we care about the
> > actual order of append and prepend:
> > ```
> > ``:remove`` is final. Any string appearing in ``:remove`` cannot be
> > added back regardless of when ``:remove`` was parsed.
> > ```
> > Up to you :)
>
> I'd argue it's better to keep the information full and complete, esp.
> since its not easy to find within bitbake sources.
>

Then I would suggest also to mention a way to undo :remove with separate
variable as discussed on IRC.

Cheers,

[-- Attachment #2: Type: text/html, Size: 3462 bytes --]