[PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[Wes Lindauer]

Thanks to wildcards in bbappend filenames, it's possible to have
multiple bbappends that apply to the same recipe in the same directory.
In order to get sstate hits between different workspaces, we want to
apply those bbappend files in a consistent order.  Since readdir()
returns files in a non-deterministic order between workspaces (based on
inode number and/or time of creation), we'll need to sort its result in
order to have any consistency.

Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
---
 bitbake/lib/bb/cooker.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 0607fcc708..a9ee142aa5 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
             (bbappend, filename) = b
             if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
                 filelist.append(filename)
+        filelist.sort()
         return filelist
 
     def collection_priorities(self, pkgfns, d):
-- 
2.14.5

✗ patchtest: failure for bitbake: cooker: Ensure bbappends are found in stable order

[Patchwork]

== Series Details ==

Series: bitbake: cooker: Ensure bbappends are found in stable order
Revision: 1
URL   : https://patchwork.openembedded.org/series/19547/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series sent to the wrong mailing list or some patches from the series correspond to different mailing lists [test_target_mailing_list] 
  Suggested fix    Send the series again to the correct mailing list (ML)
  Suggested ML     bitbake-devel@lists.openembedded.org [http://git.openembedded.org/bitbake/]
  Patch's path:    bitbake/lib/bb/cooker.py

* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 684fc36402)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[Mark Hatle]

On 9/11/19 12:07 PM, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
>> core-bounces@lists.openembedded.org> On Behalf Of Wes Lindauer
>> Sent: den 28 augusti 2019 23:38
>> To: openembedded-core@lists.openembedded.org
>> Subject: [OE-core] [PATCH] bitbake: cooker: Ensure bbappends are found
>> in stable order
>>
>> Thanks to wildcards in bbappend filenames, it's possible to have
>> multiple bbappends that apply to the same recipe in the same directory.
>> In order to get sstate hits between different workspaces, we want to
>> apply those bbappend files in a consistent order.  Since readdir()
>> returns files in a non-deterministic order between workspaces (based on
>> inode number and/or time of creation), we'll need to sort its result in
>> order to have any consistency.
>>
>> Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
>> ---
>>  bitbake/lib/bb/cooker.py | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index 0607fcc708..a9ee142aa5 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
>>              (bbappend, filename) = b
>>              if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
>>                  filelist.append(filename)
>> +        filelist.sort()
>>          return filelist
>>
>>      def collection_priorities(self, pkgfns, d):
>> --
>> 2.14.5
> 
> I don't think this was a good idea at all. Before, bbappends were applied 
> in the priority order of the layers, but this change throws the sorting 
> based on priority out the window and might cause bbappend files for higher 
> layers to be applied before lower layers' bbappend files.
> 
> Please revert this change, and rethink how to maintain a stable order for 
> bbappend files from the same directory without affecting bbappend files 
> from different layers.

I had assumed that the sorting was only within each layer's bbappends.  If it's
global, you are absolutely right that priority order matters.

So maybe:

Within a layer (multiple .bbappends) 'sort' (alphabetic)

Between layers first by priority, then by order in the bblayers (for two layers
with the same priority)?

--Mark

> //Peter
> 

Re: [PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Wes Lindauer
> Sent: den 28 augusti 2019 23:38
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH] bitbake: cooker: Ensure bbappends are found
> in stable order
> 
> Thanks to wildcards in bbappend filenames, it's possible to have
> multiple bbappends that apply to the same recipe in the same directory.
> In order to get sstate hits between different workspaces, we want to
> apply those bbappend files in a consistent order.  Since readdir()
> returns files in a non-deterministic order between workspaces (based on
> inode number and/or time of creation), we'll need to sort its result in
> order to have any consistency.
> 
> Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
> ---
>  bitbake/lib/bb/cooker.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 0607fcc708..a9ee142aa5 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
>              (bbappend, filename) = b
>              if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
>                  filelist.append(filename)
> +        filelist.sort()
>          return filelist
> 
>      def collection_priorities(self, pkgfns, d):
> --
> 2.14.5

I don't think this was a good idea at all. Before, bbappends were applied 
in the priority order of the layers, but this change throws the sorting 
based on priority out the window and might cause bbappend files for higher 
layers to be applied before lower layers' bbappend files.

Please revert this change, and rethink how to maintain a stable order for 
bbappend files from the same directory without affecting bbappend files 
from different layers.

//Peter

[PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[Wes Lindauer]

Thanks to wildcards in bbappend filenames, it's possible to have
multiple bbappends that apply to the same recipe in the same directory.
In order to get sstate hits between different workspaces, we want to
apply those bbappend files in a consistent order.  Since readdir()
returns files in a non-deterministic order between workspaces (based on
inode number and/or time of creation), we'll need to sort its result in
order to have any consistency.

Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
---
 bitbake/lib/bb/cooker.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 0607fcc708..a9ee142aa5 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
             (bbappend, filename) = b
             if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
                 filelist.append(filename)
+        filelist.sort()
         return filelist
 
     def collection_priorities(self, pkgfns, d):
-- 
2.14.5