[PATCH 1/2] license.bbclass: Improve parsing time when INCOMPATIBLE_LICENSES is big

[PATCH 1/2] license.bbclass: Improve parsing time when INCOMPATIBLE_LICENSES is big

[Peter Kjellerstedt]

The commit 08cbf1748 (licenses: Update INCOMPATIBLE_LICENSE for
'or-later' handling) increased the parsing time considerably if there
are many licenses in INCOMPATIBLE_LICENSE. Reorganize the code to get
almost all the time back.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/license.bbclass | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index bcea0b3cb5..f7978e266b 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -282,16 +282,15 @@ def expand_wildcard_licenses(d, wildcard_licenses):
     """
     import fnmatch
 
-    # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
-    for lic in wildcard_licenses[:]:
-        if not lic.endswith(("-or-later", "-only", "*")):
-            wildcard_licenses.append(lic + "+")
-
     licenses = wildcard_licenses[:]
     spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
     for wld_lic in wildcard_licenses:
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
         licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
+        # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
+        if not wld_lic.endswith(("-or-later", "-only", "*", "+")):
+            spdxflags = fnmatch.filter(spdxmapkeys, wld_lic + "+")
+            licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
 
     spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
     for wld_lic in wildcard_licenses:

[PATCH 2/2] metadata_scm.bbclass: Use immediate expansion for the METADATA_* variables

[Peter Kjellerstedt]

Define METADATA_BRANCH and METADATA_REVISION using immediate expansion.
This avoids running `git rev-parse HEAD` multiple times during recipe
parsing.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/metadata_scm.bbclass | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/classes/metadata_scm.bbclass b/meta/classes/metadata_scm.bbclass
index 2608a7ef7b..47cb969b8d 100644
--- a/meta/classes/metadata_scm.bbclass
+++ b/meta/classes/metadata_scm.bbclass
@@ -1,8 +1,3 @@
-METADATA_BRANCH ?= "${@base_detect_branch(d)}"
-METADATA_BRANCH[vardepvalue] = "${METADATA_BRANCH}"
-METADATA_REVISION ?= "${@base_detect_revision(d)}"
-METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"
-
 def base_detect_revision(d):
     path = base_get_scmbasepath(d)
     return base_get_metadata_git_revision(path, d)
@@ -42,3 +37,8 @@ def base_get_metadata_git_revision(path, d):
     except bb.process.ExecutionError:
         rev = '<unknown>'
     return rev.strip()
+
+METADATA_BRANCH := "${@base_detect_branch(d)}"
+METADATA_BRANCH[vardepvalue] = "${METADATA_BRANCH}"
+METADATA_REVISION := "${@base_detect_revision(d)}"
+METADATA_REVISION[vardepvalue] = "${METADATA_REVISION}"

Re: [bitbake-devel] [PATCH 1/2] license.bbclass: Improve parsing time when INCOMPATIBLE_LICENSES is big

[Richard Purdie]

On Fri, 2021-03-12 at 04:20 +0100, Peter Kjellerstedt wrote:
> The commit 08cbf1748 (licenses: Update INCOMPATIBLE_LICENSE for
> 'or-later' handling) increased the parsing time considerably if there
> are many licenses in INCOMPATIBLE_LICENSE. Reorganize the code to get
> almost all the time back.
> 
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
>  meta/classes/license.bbclass | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
> 
> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> index bcea0b3cb5..f7978e266b 100644
> --- a/meta/classes/license.bbclass
> +++ b/meta/classes/license.bbclass
> @@ -282,16 +282,15 @@ def expand_wildcard_licenses(d, wildcard_licenses):
>      """
>      import fnmatch
>  
> 
> 
> 
> -    # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
> -    for lic in wildcard_licenses[:]:
> -        if not lic.endswith(("-or-later", "-only", "*")):
> -            wildcard_licenses.append(lic + "+")
> -
>      licenses = wildcard_licenses[:]
>      spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
>      for wld_lic in wildcard_licenses:
>          spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
>          licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
> +        # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
> +        if not wld_lic.endswith(("-or-later", "-only", "*", "+")):
> +            spdxflags = fnmatch.filter(spdxmapkeys, wld_lic + "+")
> +            licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
>  
> 
> 
> 
>      spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
>      for wld_lic in wildcard_licenses:


As far as I can tell, this changes behaviour since the variants are no 
longer added to licenses in this second loop over wildcard_licenses. I 
think this means you get the licences matching from SPDXLICENSEMAP
but not the matching licenses from AVAILABLE_LICENSES?

Cheers,

Richard

Re: [bitbake-devel] [PATCH 1/2] license.bbclass: Improve parsing time when INCOMPATIBLE_LICENSES is big

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 12 mars 2021 10:28
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; bitbake-devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH 1/2] license.bbclass: Improve parsing time when INCOMPATIBLE_LICENSES is big
> 
> On Fri, 2021-03-12 at 04:20 +0100, Peter Kjellerstedt wrote:
> > The commit 08cbf1748 (licenses: Update INCOMPATIBLE_LICENSE for
> > 'or-later' handling) increased the parsing time considerably if there
> > are many licenses in INCOMPATIBLE_LICENSE. Reorganize the code to get
> > almost all the time back.
> >
> > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > ---
> >  meta/classes/license.bbclass | 9 ++++-----
> >  1 file changed, 4 insertions(+), 5 deletions(-)
> >
> > diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> > index bcea0b3cb5..f7978e266b 100644
> > --- a/meta/classes/license.bbclass
> > +++ b/meta/classes/license.bbclass
> > @@ -282,16 +282,15 @@ def expand_wildcard_licenses(d, wildcard_licenses):
> >      """
> >      import fnmatch
> >
> > -    # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
> > -    for lic in wildcard_licenses[:]:
> > -        if not lic.endswith(("-or-later", "-only", "*")):
> > -            wildcard_licenses.append(lic + "+")
> > -
> >      licenses = wildcard_licenses[:]
> >      spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
> >      for wld_lic in wildcard_licenses:
> >          spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
> >          licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
> > +        # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
> > +        if not wld_lic.endswith(("-or-later", "-only", "*", "+")):
> > +            spdxflags = fnmatch.filter(spdxmapkeys, wld_lic + "+")
> > +            licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
> >
> >      spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
> >      for wld_lic in wildcard_licenses:
> 
> As far as I can tell, this changes behaviour since the variants are no
> longer added to licenses in this second loop over wildcard_licenses. I
> think this means you get the licences matching from SPDXLICENSEMAP
> but not the matching licenses from AVAILABLE_LICENSES?

That is correct. My assumption is that there are no "foo+" licenses in 
AVAILABLE_LICENSES. At least this is the case for poky and meta-openembedded, 
and I do not see why one would add such a license file, when the "+" is 
used to refer to another license, which typically means that even if you 
add some other license than the already existing GPL licenses that has a 
similar "or later" use case, you would add it to the SPDXLICENSEMAP to 
not have to duplicate the license texts.

Anyways, I have now tried adding code to the second loop as well to 
handle the '+', and it is still better than having a separate loop 
before the two we have now.

Here are some timings for the different cases:

Dunfell/Gatesgarth:					38 s
Support for '+' in the SPDXLICENSEMAP loop:			41 s
Support for '+' also in the AVAILABLE_LICENSES loop:		45 s
Separate loop before the other two (original solution):	56 s

(This assumes the patch to define METADATA_REVISION using immediate 
expansion has also been applied.)

So, based on the numbers above and the expected lack of need for 
support in the second loop, I would prefer the solution as suggested 
in this patch.

> Cheers,
> 
> Richard

//Peter

[PATCH 1/2] license.bbclass: Improve parsing time when INCOMPATIBLE_LICENSES is big

[Peter Kjellerstedt]

The commit 08cbf1748 (licenses: Update INCOMPATIBLE_LICENSE for
'or-later' handling) increased the parsing time considerably if there
are many licenses in INCOMPATIBLE_LICENSE. Reorganize the code to get
almost all the time back.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/license.bbclass | 9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index bcea0b3cb5..f7978e266b 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -282,16 +282,15 @@ def expand_wildcard_licenses(d, wildcard_licenses):
     """
     import fnmatch
 
-    # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
-    for lic in wildcard_licenses[:]:
-        if not lic.endswith(("-or-later", "-only", "*")):
-            wildcard_licenses.append(lic + "+")
-
     licenses = wildcard_licenses[:]
     spdxmapkeys = d.getVarFlags('SPDXLICENSEMAP').keys()
     for wld_lic in wildcard_licenses:
         spdxflags = fnmatch.filter(spdxmapkeys, wld_lic)
         licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
+        # Assume if we're passed "GPLv3" or "*GPLv3" it means -or-later as well
+        if not wld_lic.endswith(("-or-later", "-only", "*", "+")):
+            spdxflags = fnmatch.filter(spdxmapkeys, wld_lic + "+")
+            licenses += [d.getVarFlag('SPDXLICENSEMAP', flag) for flag in spdxflags]
 
     spdx_lics = d.getVar('AVAILABLE_LICENSES').split()
     for wld_lic in wildcard_licenses: