[PATCH] cookerdata: Fix cache/reparsing issue

[PATCH] cookerdata: Fix cache/reparsing issue

[Richard Purdie]

When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES variables,
we need to be deterministic. The random ordering from the sets was causing
unexpected reparses.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cookerdata.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index b4e0c4216b..8198f509e2 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -425,7 +425,7 @@ class CookerDataBuilder(object):
             data.delVar('LAYERDIR_RE')
             data.delVar('LAYERDIR')
             for c in compat_entries:
-                data.setVar("LAYERSERIES_COMPAT_%s" % c, compat_entries[c])
+                data.setVar("LAYERSERIES_COMPAT_%s" % c, sorted(compat_entries[c]))
 
             bbfiles_dynamic = (data.getVar('BBFILES_DYNAMIC') or "").split()
             collections = (data.getVar('BBFILE_COLLECTIONS') or "").split()
@@ -461,7 +461,7 @@ class CookerDataBuilder(object):
                 elif not compat and not data.getVar("BB_WORKERCONTEXT"):
                     bb.warn("Layer %s should set LAYERSERIES_COMPAT_%s in its conf/layer.conf file to list the core layer names it is compatible with." % (c, c))
 
-            data.setVar("LAYERSERIES_CORENAMES", " ".join(layerseries))
+            data.setVar("LAYERSERIES_CORENAMES", " ".join(sorted(layerseries)))
 
         if not data.getVar("BBPATH"):
             msg = "The BBPATH variable is not set"
-- 
2.37.2

Re: [bitbake-devel] [PATCH] cookerdata: Fix cache/reparsing issue

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 980 bytes --]

On Jan 1, 2023 at 7:25 AM -0700, Richard Purdie <richard.purdie@linuxfoundation.org>, wrote:
> When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES variables,
> we need to be deterministic. The random ordering from the sets was causing
> unexpected reparses.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> lib/bb/cookerdata.py | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
> index b4e0c4216b..8198f509e2 100644
> --- a/lib/bb/cookerdata.py
> +++ b/lib/bb/cookerdata.py
> @@ -425,7 +425,7 @@ class CookerDataBuilder(object):
> data.delVar('LAYERDIR_RE')
> data.delVar('LAYERDIR')
> for c in compat_entries:
> - data.setVar("LAYERSERIES_COMPAT_%s" % c, compat_entries[c])
> + data.setVar("LAYERSERIES_COMPAT_%s" % c, sorted(compat_entries[c]))

Missed a “ “.join() here, it’s currently setting the variable to a generator object, afaik :)

[-- Attachment #2: Type: text/html, Size: 1416 bytes --]

Re: [bitbake-devel] [PATCH] cookerdata: Fix cache/reparsing issue

[Richard Purdie]

On Sun, 2023-01-01 at 09:55 -0700, Christopher Larson wrote:
> On Jan 1, 2023 at 7:25 AM -0700, Richard Purdie
> <richard.purdie@linuxfoundation.org>, wrote:
> > When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES
> > variables,
> > we need to be deterministic. The random ordering from the sets was
> > causing
> > unexpected reparses.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > ---
> > lib/bb/cookerdata.py | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
> > index b4e0c4216b..8198f509e2 100644
> > --- a/lib/bb/cookerdata.py
> > +++ b/lib/bb/cookerdata.py
> > @@ -425,7 +425,7 @@ class CookerDataBuilder(object):
> > data.delVar('LAYERDIR_RE')
> > data.delVar('LAYERDIR')
> > for c in compat_entries:
> > - data.setVar("LAYERSERIES_COMPAT_%s" % c, compat_entries[c])
> > + data.setVar("LAYERSERIES_COMPAT_%s" % c,
> > sorted(compat_entries[c]))
> 
> 
> Missed a “ “.join() here, it’s currently setting the variable to a
> generator object, afaik :)

Well spotted, fix pushed, thanks.

Cheers,

Richard

Re: [bitbake-devel] [PATCH] cookerdata: Fix cache/reparsing issue

[Quentin Schulz]

Hi Richard,

On 1/2/23 01:50, Richard Purdie wrote:
> On Sun, 2023-01-01 at 09:55 -0700, Christopher Larson wrote:
>> On Jan 1, 2023 at 7:25 AM -0700, Richard Purdie
>> <richard.purdie@linuxfoundation.org>, wrote:
>>> When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES
>>> variables,
>>> we need to be deterministic. The random ordering from the sets was
>>> causing
>>> unexpected reparses.
>>>
>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>> ---
>>> lib/bb/cookerdata.py | 4 ++--
>>> 1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
>>> index b4e0c4216b..8198f509e2 100644
>>> --- a/lib/bb/cookerdata.py
>>> +++ b/lib/bb/cookerdata.py
>>> @@ -425,7 +425,7 @@ class CookerDataBuilder(object):
>>> data.delVar('LAYERDIR_RE')
>>> data.delVar('LAYERDIR')
>>> for c in compat_entries:
>>> - data.setVar("LAYERSERIES_COMPAT_%s" % c, compat_entries[c])
>>> + data.setVar("LAYERSERIES_COMPAT_%s" % c,
>>> sorted(compat_entries[c]))
>>

compat_entries and layerseries are sets set just a few lines above. 
Would it make sense to actually have them sorted directly there instead 
of only when they are used?

https://git.openembedded.org/bitbake/tree/lib/bb/cookerdata.py#n418
https://git.openembedded.org/bitbake/tree/lib/bb/cookerdata.py#n421

Cheers,
Quentin

Re: [bitbake-devel] [PATCH] cookerdata: Fix cache/reparsing issue

[Richard Purdie]

On Mon, 2023-01-02 at 11:01 +0100, Quentin Schulz wrote:
> Hi Richard,
> 
> On 1/2/23 01:50, Richard Purdie wrote:
> > On Sun, 2023-01-01 at 09:55 -0700, Christopher Larson wrote:
> > > On Jan 1, 2023 at 7:25 AM -0700, Richard Purdie
> > > <richard.purdie@linuxfoundation.org>, wrote:
> > > > When setting the LAYERSERIES_COMPAT and LAYERSERIES_CORENAMES
> > > > variables,
> > > > we need to be deterministic. The random ordering from the sets was
> > > > causing
> > > > unexpected reparses.
> > > > 
> > > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > > ---
> > > > lib/bb/cookerdata.py | 4 ++--
> > > > 1 file changed, 2 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
> > > > index b4e0c4216b..8198f509e2 100644
> > > > --- a/lib/bb/cookerdata.py
> > > > +++ b/lib/bb/cookerdata.py
> > > > @@ -425,7 +425,7 @@ class CookerDataBuilder(object):
> > > > data.delVar('LAYERDIR_RE')
> > > > data.delVar('LAYERDIR')
> > > > for c in compat_entries:
> > > > - data.setVar("LAYERSERIES_COMPAT_%s" % c, compat_entries[c])
> > > > + data.setVar("LAYERSERIES_COMPAT_%s" % c,
> > > > sorted(compat_entries[c]))
> > > 
> 
> compat_entries and layerseries are sets set just a few lines above. 
> Would it make sense to actually have them sorted directly there instead 
> of only when they are used?
> 
> https://git.openembedded.org/bitbake/tree/lib/bb/cookerdata.py#n418
> https://git.openembedded.org/bitbake/tree/lib/bb/cookerdata.py#n421

They're used as sets elsewhere and that is used to de-duplicate them so
whilst I did wonder too, I ended up just sorting as we wrote strings
back...

Cheers,

Richard