[poky-contrib][RFC PATCH 0/5] Suggestions for improvements?

[poky-contrib][RFC PATCH 0/5] Suggestions for improvements?

[Meh Mbeh Ida Delphine]

Hello everyone,

The following patchset is aimed at improving the Yocto Project license tracing.

Initially, after a build I will come across 130 license warnings with some false positives. I got rid of these by replacing "or-later" with "+" for source licenses (computedpkglics) and also passing the recipe license through canonical_license of license.py(). As a result, some of the licenses will match during comparism hence some warnings eliminate.

Though some false positive warnings were eliminated, lots of warnings were ouputed as a result of several source licenses having "WITH Linux-syscall-note". With the help of a variable being set in local.conf, one can chooses whether or not to get lid of the "WITH Linux-syscall-note" string from the source licenses. Just by doing this, several warnings are dropped and now left with 118.

I later on added some logic depending on "source_spdx_license" variable whether or not for the user to allow the warnings being outputed during the build. It reduces the extra processing and reduces time if the user chooses not to display warnings. It uses bb.utils.contains() to find out if that particular vaiable is present. Also, I added package_qa_handle_error() to allow the user decide whether they want the issues treated as warnings, errors or not at all.

Some of the license-related logic I added to take care of splitting recipe licences before canonicalizing, getting rid of "or-later" warnings, I moved it to license.bbclass and put it in functions. I also plan on fixing the variable naming since some have underscores whereas others do not have. However, I will love to move more license logic in package.bbclass to license.bbclass but I need help with the right way to go about it. Also, general suggestions on any improvements to this are highly welcomed.

Cheers,
Ida.

Ida Delphine (5):
  package: Remove false positive lic warnings
  package: Remove false positive lic warnings
  package: Remove false positive lic warnings
  license.bbclass: Add functions to split and canonicalise license
    strings
  package.bbclass: Remove false positive license warnings

 meta/classes/license.bbclass |  27 ++++++++
 meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
 2 files changed, 96 insertions(+), 58 deletions(-)

-- 
2.25.1

[poky-contrib][RFC PATCH 1/5] package: Remove false positive lic warnings

[Meh Mbeh Ida Delphine]

The recipe LICENSE strings are split into their individual license and the canonicalised to easy enable matching with licenses of sources.
License strings with "or-later" are replaced with "+" before comparism.
The warnings showup if and only if licenses in the sources don't match LICENSE value of the recipe.
Signed-off-by: Ida Delphine <idadelm@gmail.com>
---
 meta/classes/package.bbclass | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index ab96f141ae..c3259146b6 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1545,6 +1545,7 @@ PKGDESTWORK = "${WORKDIR}/pkgdata"
 PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY RDEPENDS RPROVIDES RRECOMMENDS RSUGGESTS RREPLACES RCONFLICTS SECTION PKG ALLOW_EMPTY FILES CONFFILES FILES_INFO PACKAGE_ADD_METADATA pkg_postinst pkg_postrm pkg_preinst pkg_prerm"
 
 python emit_pkgdata() {
+    import oe.license
     from glob import glob
     import json
     import subprocess
@@ -1762,7 +1763,21 @@ fi
             lic = d.getVar('LICENSE_%s' % (pkg))
             if not lic:
                 lic = d.getVar('LICENSE')
-            bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], lic))
+
+            # Splits the LICENSE values and canonicalise each license
+            # in the set of split license(s)    
+            split_lic = oe.license.list_licenses(lic)
+            spdx_lic = set([canonical_license(d, l) for l in split_lic])
+            if computedpkglics[pkg]:
+                computedpkglicsperpkg = set([])
+                for l in computedpkglics[pkg]:
+                    if l.endswith('-or-later'):
+                        lic_ = l.replace('-or-later', '+')
+                        computedpkglicsperpkg.add(lic_)
+                    else:
+                        computedpkglicsperpkg.add(l)
+            if spdx_lic - computedpkglicsperpkg:
+                bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglicsperpkg, spdx_lic))
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
 
-- 
2.25.1

Re: [OE-core] [poky-contrib][RFC PATCH 1/5] package: Remove false positive lic warnings

[Richard Purdie]

On Wed, 2021-02-17 at 05:00 +0100, Meh Mbeh Ida Delphine wrote:
The recipe LICENSE strings are split into their individual license and the canonicalised to easy enable matching with licenses of sources.
License strings with "or-later" are replaced with "+" before comparism.
The warnings showup if and only if licenses in the sources don't match LICENSE value of the recipe.
Signed-off-by: Ida Delphine <idadelm@gmail.com>
---
 meta/classes/package.bbclass | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index ab96f141ae..c3259146b6 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1545,6 +1545,7 @@ PKGDESTWORK = "${WORKDIR}/pkgdata"
 PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY RDEPENDS RPROVIDES RRECOMMENDS RSUGGESTS RREPLACES RCONFLICTS SECTION PKG ALLOW_EMPTY FILES CONFFILES FILES_INFO PACKAGE_ADD_METADATA pkg_postinst pkg_postrm pkg_preinst pkg_prerm"
 

 python emit_pkgdata() {
+    import oe.license
     from glob import glob
     import json
     import subprocess
@@ -1762,7 +1763,21 @@ fi
             lic = d.getVar('LICENSE_%s' % (pkg))
             if not lic:
                 lic = d.getVar('LICENSE')
-            bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglics[pkg], lic))
+
+            # Splits the LICENSE values and canonicalise each license
+            # in the set of split license(s)    
+            split_lic = oe.license.list_licenses(lic)
+            spdx_lic = set([canonical_license(d, l) for l in split_lic])
+            if computedpkglics[pkg]:
+                computedpkglicsperpkg = set([])
+                for l in computedpkglics[pkg]:
+                    if l.endswith('-or-later'):
+                        lic_ = l.replace('-or-later', '+')
+                        computedpkglicsperpkg.add(lic_)
+                    else:
+                        computedpkglicsperpkg.add(l)
+            if spdx_lic - computedpkglicsperpkg:
+                bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglicsperpkg, spdx_lic))
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"

I added your license changes from back in October into master-next.
With those changes, the "-or-later" piece should be uneeded and happen
automatically?

Cheers,

Richard

[poky-contrib][RFC PATCH 2/5] package: Remove false positive lic warnings

[Meh Mbeh Ida Delphine]

With the IGNOREWITHLINUXSYSCALLNOTE variable set to "1" in local.conf, it removes the "WITH Linux syscall note" string attached to the license value for sources before comparism.
This is to get more recipe LICENSES to match with those of the sources and reduce the number of warnings outputed.

Signed-off-by: Ida Delphine <idadelm@gmail.com>
---
 meta/classes/package.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index c3259146b6..14c8475cfa 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1774,6 +1774,10 @@ fi
                     if l.endswith('-or-later'):
                         lic_ = l.replace('-or-later', '+')
                         computedpkglicsperpkg.add(lic_)
+                    elif l.endswith(' WITH Linux-syscall-note'):
+                        if d.getVar("IGNOREWITHLINUXSYSCALLNOTE") == "1":
+                            lic_ = l.replace(' WITH Linux-syscall-note', '')
+                            computedpkglicsperpkg.add(lic_)
                     else:
                         computedpkglicsperpkg.add(l)
             if spdx_lic - computedpkglicsperpkg:
-- 
2.25.1

Re: [OE-core] [poky-contrib][RFC PATCH 2/5] package: Remove false positive lic warnings

[Richard Purdie]

On Wed, 2021-02-17 at 05:00 +0100, Meh Mbeh Ida Delphine wrote:
> With the IGNOREWITHLINUXSYSCALLNOTE variable set to "1" in local.conf, it removes the "WITH Linux syscall note" string attached to the license value for sources before comparism.
> This is to get more recipe LICENSES to match with those of the sources and reduce the number of warnings outputed.
> 
> Signed-off-by: Ida Delphine <idadelm@gmail.com>
> ---
>  meta/classes/package.bbclass | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index c3259146b6..14c8475cfa 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1774,6 +1774,10 @@ fi
>                      if l.endswith('-or-later'):
>                          lic_ = l.replace('-or-later', '+')
>                          computedpkglicsperpkg.add(lic_)
> +                    elif l.endswith(' WITH Linux-syscall-note'):
> +                        if d.getVar("IGNOREWITHLINUXSYSCALLNOTE") == "1":
> +                            lic_ = l.replace(' WITH Linux-syscall-note', '')
> +                            computedpkglicsperpkg.add(lic_)

I think this is going to break things and it should be entirely
removing this license entry.

Why? Headers under GPLv2 WITH Linux-syscall-note can be used in a non-
GPLv2 application and that application doesn't then become under GPLv2.

What we probably want is a "safe" list of licenses which don't have
implications and entries matching "* WITH Linux-syscall-note" can be
ignored, as long as they're only in header files. Perhaps we also need
a check to ensure we only see "* WITH Linux-syscall-note" in header
files?

Cheers,

Richard

Re: [OE-core] [poky-contrib][RFC PATCH 2/5] package: Remove false positive lic warnings

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 17 februari 2021 11:40
> To: Meh Mbeh Ida Delphine <idadelm@gmail.com>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [poky-contrib][RFC PATCH 2/5] package: Remove false
> positive lic warnings
> 
> On Wed, 2021-02-17 at 05:00 +0100, Meh Mbeh Ida Delphine wrote:
> > With the IGNOREWITHLINUXSYSCALLNOTE variable set to "1" in local.conf,
> it removes the "WITH Linux syscall note" string attached to the license
> value for sources before comparism.
> > This is to get more recipe LICENSES to match with those of the sources
> and reduce the number of warnings outputed.
> >
> > Signed-off-by: Ida Delphine <idadelm@gmail.com>
> > ---
> >  meta/classes/package.bbclass | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> > index c3259146b6..14c8475cfa 100644
> > --- a/meta/classes/package.bbclass
> > +++ b/meta/classes/package.bbclass
> > @@ -1774,6 +1774,10 @@ fi
> >                      if l.endswith('-or-later'):
> >                          lic_ = l.replace('-or-later', '+')
> >                          computedpkglicsperpkg.add(lic_)
> > +                    elif l.endswith(' WITH Linux-syscall-note'):
> > +                        if d.getVar("IGNOREWITHLINUXSYSCALLNOTE") == "1":
> > +                            lic_ = l.replace(' WITH Linux-syscall-note', '')
> > +                            computedpkglicsperpkg.add(lic_)
> 
> I think this is going to break things and it should be entirely
> removing this license entry.
> 
> Why? Headers under GPLv2 WITH Linux-syscall-note can be used in a non-
> GPLv2 application and that application doesn't then become under GPLv2.
> 
> What we probably want is a "safe" list of licenses which don't have
> implications and entries matching "* WITH Linux-syscall-note" can be
> ignored, as long as they're only in header files. Perhaps we also need
> a check to ensure we only see "* WITH Linux-syscall-note" in header
> files?
> 
> Cheers,
> 
> Richard

Isn't the problem here really the multiple (sometimes contradictory) use 
cases for the LICENSE variable. 

1) It is used to document the licenses that cover the code.
2) It is used to determine if the recipe can be built.
3) It is used (together with LIC_FILES_CHKSUM) to populate the licenses that 
   go into, e.g., ${PN}-lic.
4) It is used by archiver.bbclass to determine what code to export.

To comply with 1) and 3), you want to include, e.g., "GPL-3.0 WITH 
Linux-syscall-note", but for 2) and 4), you probably do not want it. 

Related to this is also recipes that have a LICENSE such as "Foo | Bar" 
where there is a choice as to which license we as a publisher of binaries 
choose when distributing the code. Since this choice may vary from 
distributor to distributor it is not as simple as to just list them both.

//Peter

Re: [OE-core] [poky-contrib][RFC PATCH 2/5] package: Remove false positive lic warnings

[Richard Purdie]

On Wed, 2021-02-17 at 14:55 +0000, Peter Kjellerstedt wrote:
> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Richard Purdie
> On Wed, 2021-02-17 at 05:00 +0100, Meh Mbeh Ida Delphine wrote:
> 
> > 
> > diff --git a/meta/classes/package.bbclass
> > b/meta/classes/package.bbclass
> > index c3259146b6..14c8475cfa 100644
> > --- a/meta/classes/package.bbclass
> > +++ b/meta/classes/package.bbclass
> > @@ -1774,6 +1774,10 @@ fi
> >                      if l.endswith('-or-later'):
> >                          lic_ = l.replace('-or-later', '+')
> >                          computedpkglicsperpkg.add(lic_)
> > +                    elif l.endswith(' WITH Linux-syscall-note'):
> > +                        if d.getVar("IGNOREWITHLINUXSYSCALLNOTE")
> > == "1":
> > +                            lic_ = l.replace(' WITH Linux-syscall-
> > note', '')
> > +                            computedpkglicsperpkg.add(lic_)
> 
> I think this is going to break things and it should be entirely
> removing this license entry.
> 
> Why? Headers under GPLv2 WITH Linux-syscall-note can be used in a
> non-
> GPLv2 application and that application doesn't then become under
> GPLv2.
> 
> What we probably want is a "safe" list of licenses which don't have
> implications and entries matching "* WITH Linux-syscall-note" can be
> ignored, as long as they're only in header files. Perhaps we also
> need
> a check to ensure we only see "* WITH Linux-syscall-note" in header
> files?
> 
> Cheers,
> 
> Richard

Isn't the problem here really the multiple (sometimes contradictory)
use 
cases for the LICENSE variable. 

1) It is used to document the licenses that cover the code.
2) It is used to determine if the recipe can be built.
3) It is used (together with LIC_FILES_CHKSUM) to populate the licenses
that 
   go into, e.g., ${PN}-lic.
4) It is used by archiver.bbclass to determine what code to export.

I guess I have more specific context here since I know Ida is looking
at comparing what we have in LICENSE with what we can learn from
license headers being built into the code, e.g. to find mismatches.

In that context, trimming to "GPLv2" and them comparing with a recipe
saying "APL-2.0" is bad when in fact its fine.

Cheers,

Richard

[poky-contrib][RFC PATCH 3/5] package: Remove false positive lic warnings

[Meh Mbeh Ida Delphine]

Added text to distinguish licenses from sources and those from recipes when warnings are outputed.

Signed-off-by: Ida Delphine <idadelm@gmail.com>
---
 meta/classes/package.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 14c8475cfa..782d9d00b3 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1780,8 +1780,9 @@ fi
                             computedpkglicsperpkg.add(lic_)
                     else:
                         computedpkglicsperpkg.add(l)
+            # Displays warnings for licenses found in the recipes and not sources
             if spdx_lic - computedpkglicsperpkg:
-                bb.warn("License for package %s is %s vs %s" % (pkg, computedpkglicsperpkg, spdx_lic))
+                bb.warn("License for package %s is %s (source SPDX headers) vs %s (LICENSE)" % (pkg, computedpkglicsperpkg, spdx_lic))
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
 
-- 
2.25.1

[poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Meh Mbeh Ida Delphine]

These functions that will later be used in package.bbclass simply make the source and recipe licenses in the same format so that they can easily be compared and the ouput warnings filtered accordingly.
split_spdx_lic() splits the license strings and returns a set of the canonicalised licenses.
rem_false_lics() does two things:
- Converts '-or-later' licenses to their canonicalised form
- Gets rid of "WITH Linux-syscall-note" from license string if specified in local.conf

Signed-off-by: Ida Delphine <idadelm@gmail.com>
---
 meta/classes/license.bbclass | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index dc91118340..576464cb26 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -435,3 +435,30 @@ python do_populate_lic_setscene () {
     sstate_setscene(d)
 }
 addtask do_populate_lic_setscene
+
+
+def split_spdx_lic(d, licensestr):
+    """
+    Split the license strings and returns a set of the
+    canonicalised licenses.
+    """
+    import oe.license
+    split_lic = oe.license.list_licenses(licensestr)
+    spdx_lic = set([canonical_license(d, l) for l in split_lic])
+    return spdx_lic
+
+def rem_false_lics(d, pkglic):
+    pkglicsperpkg = set([])
+    for l in pkglic:
+        if l.endswith('-or-later'):
+            # Converts '-or-later' licenses to their canonicalised form
+            lic_ = l.replace('-or-later', '+')
+            pkglicsperpkg.add(lic_)
+        elif l.endswith(' WITH Linux-syscall-note'):
+        # Gets rid of "WITH Linux-syscall-note from license sring"
+            if d.getVar("LICENSE_WITH_LINUX_SYS") == "1":
+                lic_ = l.replace(' WITH Linux-syscall-note', '')
+                pkglicsperpkg.add(lic_)
+        else:
+            pkglicsperpkg.add(l)
+    return pkglicsperpkg
\ No newline at end of file
-- 
2.25.1

Re: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Richard Purdie]

On Wed, 2021-02-17 at 05:00 +0100, Meh Mbeh Ida Delphine wrote:
> These functions that will later be used in package.bbclass simply make the source and recipe licenses in the same format so that they can easily be compared and the ouput warnings filtered accordingly.
> split_spdx_lic() splits the license strings and returns a set of the canonicalised licenses.
> rem_false_lics() does two things:
> - Converts '-or-later' licenses to their canonicalised form
> - Gets rid of "WITH Linux-syscall-note" from license string if specified in local.conf
> 
> Signed-off-by: Ida Delphine <idadelm@gmail.com>
> ---
>  meta/classes/license.bbclass | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> index dc91118340..576464cb26 100644
> --- a/meta/classes/license.bbclass
> +++ b/meta/classes/license.bbclass
> @@ -435,3 +435,30 @@ python do_populate_lic_setscene () {
>      sstate_setscene(d)
>  }
>  addtask do_populate_lic_setscene
> +
> +
> +def split_spdx_lic(d, licensestr):
> +    """
> +    Split the license strings and returns a set of the
> +    canonicalised licenses.
> +    """
> +    import oe.license
> +    split_lic = oe.license.list_licenses(licensestr)
> +    spdx_lic = set([canonical_license(d, l) for l in split_lic])
> +    return spdx_lic
> +
> +def rem_false_lics(d, pkglic):
> +    pkglicsperpkg = set([])
> +    for l in pkglic:
> +        if l.endswith('-or-later'):
> +            # Converts '-or-later' licenses to their canonicalised form
> +            lic_ = l.replace('-or-later', '+')
> +            pkglicsperpkg.add(lic_)
> +        elif l.endswith(' WITH Linux-syscall-note'):
> +        # Gets rid of "WITH Linux-syscall-note from license sring"
> +            if d.getVar("LICENSE_WITH_LINUX_SYS") == "1":
> +                lic_ = l.replace(' WITH Linux-syscall-note', '')
> +                pkglicsperpkg.add(lic_)
> +        else:
> +            pkglicsperpkg.add(l)
> +    return pkglicsperpkg


Creating these functions is good. We should put them into the function
library in meta/lib/oe/license.py. I already commented about a couple
of elements of this second function.

Cheers,

Richard

Re: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Meh Mbeh Ida Delphine
> Sent: den 17 februari 2021 05:01
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add
> functions to split and canonicalise license strings
> 
> These functions that will later be used in package.bbclass simply make the
> source and recipe licenses in the same format so that they can easily be
> compared and the ouput warnings filtered accordingly.
> split_spdx_lic() splits the license strings and returns a set of the
> canonicalised licenses.
> rem_false_lics() does two things:
> - Converts '-or-later' licenses to their canonicalised form
> - Gets rid of "WITH Linux-syscall-note" from license string if specified
> in local.conf
> 
> Signed-off-by: Ida Delphine <idadelm@gmail.com>
> ---
>  meta/classes/license.bbclass | 27 +++++++++++++++++++++++++++
>  1 file changed, 27 insertions(+)
> 
> diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> index dc91118340..576464cb26 100644
> --- a/meta/classes/license.bbclass
> +++ b/meta/classes/license.bbclass
> @@ -435,3 +435,30 @@ python do_populate_lic_setscene () {
>      sstate_setscene(d)
>  }
>  addtask do_populate_lic_setscene
> +
> +
> +def split_spdx_lic(d, licensestr):
> +    """
> +    Split the license strings and returns a set of the
> +    canonicalised licenses.
> +    """
> +    import oe.license
> +    split_lic = oe.license.list_licenses(licensestr)
> +    spdx_lic = set([canonical_license(d, l) for l in split_lic])
> +    return spdx_lic
> +
> +def rem_false_lics(d, pkglic):
> +    pkglicsperpkg = set([])
> +    for l in pkglic:
> +        if l.endswith('-or-later'):
> +            # Converts '-or-later' licenses to their canonicalised form
> +            lic_ = l.replace('-or-later', '+')

Given that licenses such as "GPL-2.0+" are deprecated by SPDX, shouldn't we 
instead introduce the canonical "GPL-2.0-only" and "GPL-2.0-or-later"?
And then add mappings for, e.g., "GPL-2.0" to "GPL-2.0-only" and "GPL-2.0+" 
to "GPL-2.0-or-later".

> +            pkglicsperpkg.add(lic_)
> +        elif l.endswith(' WITH Linux-syscall-note'):
> +        # Gets rid of "WITH Linux-syscall-note from license sring"
> +            if d.getVar("LICENSE_WITH_LINUX_SYS") == "1":
> +                lic_ = l.replace(' WITH Linux-syscall-note', '')

Looking at https://spdx.org/licenses/exceptions-index.html, there is a long 
list of predefined exceptions and "Linux-syscall-note" is just one of them. 
We probably want a more generic solution for how to handle exceptions.

> +                pkglicsperpkg.add(lic_)
> +        else:
> +            pkglicsperpkg.add(l)
> +    return pkglicsperpkg
> \ No newline at end of file
> --
> 2.25.1

//Peter

Re: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Richard Purdie]

On Wed, 2021-02-17 at 15:04 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Meh Mbeh Ida Delphine
> > Sent: den 17 februari 2021 05:01
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add
> > functions to split and canonicalise license strings
> > 
> > These functions that will later be used in package.bbclass simply make the
> > source and recipe licenses in the same format so that they can easily be
> > compared and the ouput warnings filtered accordingly.
> > split_spdx_lic() splits the license strings and returns a set of the
> > canonicalised licenses.
> > rem_false_lics() does two things:
> > - Converts '-or-later' licenses to their canonicalised form
> > - Gets rid of "WITH Linux-syscall-note" from license string if specified
> > in local.conf
> > 
> > Signed-off-by: Ida Delphine <idadelm@gmail.com>
> > ---
> >  meta/classes/license.bbclass | 27 +++++++++++++++++++++++++++
> >  1 file changed, 27 insertions(+)
> > 
> > diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
> > index dc91118340..576464cb26 100644
> > --- a/meta/classes/license.bbclass
> > +++ b/meta/classes/license.bbclass
> > @@ -435,3 +435,30 @@ python do_populate_lic_setscene () {
> >      sstate_setscene(d)
> >  }
> >  addtask do_populate_lic_setscene
> > +
> > +
> > +def split_spdx_lic(d, licensestr):
> > +    """
> > +    Split the license strings and returns a set of the
> > +    canonicalised licenses.
> > +    """
> > +    import oe.license
> > +    split_lic = oe.license.list_licenses(licensestr)
> > +    spdx_lic = set([canonical_license(d, l) for l in split_lic])
> > +    return spdx_lic
> > +
> > +def rem_false_lics(d, pkglic):
> > +    pkglicsperpkg = set([])
> > +    for l in pkglic:
> > +        if l.endswith('-or-later'):
> > +            # Converts '-or-later' licenses to their canonicalised form
> > +            lic_ = l.replace('-or-later', '+')
> 
> Given that licenses such as "GPL-2.0+" are deprecated by SPDX, shouldn't we 
> instead introduce the canonical "GPL-2.0-only" and "GPL-2.0-or-later"?
> And then add mappings for, e.g., "GPL-2.0" to "GPL-2.0-only" and "GPL-2.0+" 
> to "GPL-2.0-or-later".

Something like:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=1c2ce4565bc85915ab970e42bf7af3a78ec2dd6a
?

:)

> > +            pkglicsperpkg.add(lic_)
> > +        elif l.endswith(' WITH Linux-syscall-note'):
> > +        # Gets rid of "WITH Linux-syscall-note from license sring"
> > +            if d.getVar("LICENSE_WITH_LINUX_SYS") == "1":
> > +                lic_ = l.replace(' WITH Linux-syscall-note', '')
> 
> Looking at https://spdx.org/licenses/exceptions-index.html, there is a long 
> list of predefined exceptions and "Linux-syscall-note" is just one of them. 
> We probably want a more generic solution for how to handle exceptions.

Agreed, I was trying to make that point in one of my replies!

Cheers,

Richard
> 

Re: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Meh Mbeh Ida Delphine]

[-- Attachment #1: Type: text/plain, Size: 3950 bytes --]

Thanks for the feedback. I will get rid of the logic to take care of those
licenses with "or-later" and move the functions to license.py. Does this
also mean I shouldn't try removing "WITH-Linux-syscall-note" from the
licences? The point of me doing that was for there to be more matches and
the number of warnings reduced.

Also there is more license-related logic in package.bbclass. Will it be
okay if I move it as well to meta/lib/oe/license.py putting it in functions?

Cheers,
Ida.

On Wed, Feb 17, 2021 at 5:36 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2021-02-17 at 15:04 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: openembedded-core@lists.openembedded.org <openembedded-
> > > core@lists.openembedded.org> On Behalf Of Meh Mbeh Ida Delphine
> > > Sent: den 17 februari 2021 05:01
> > > To: openembedded-core@lists.openembedded.org
> > > Subject: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add
> > > functions to split and canonicalise license strings
> > >
> > > These functions that will later be used in package.bbclass simply make
> the
> > > source and recipe licenses in the same format so that they can easily
> be
> > > compared and the ouput warnings filtered accordingly.
> > > split_spdx_lic() splits the license strings and returns a set of the
> > > canonicalised licenses.
> > > rem_false_lics() does two things:
> > > - Converts '-or-later' licenses to their canonicalised form
> > > - Gets rid of "WITH Linux-syscall-note" from license string if
> specified
> > > in local.conf
> > >
> > > Signed-off-by: Ida Delphine <idadelm@gmail.com>
> > > ---
> > >  meta/classes/license.bbclass | 27 +++++++++++++++++++++++++++
> > >  1 file changed, 27 insertions(+)
> > >
> > > diff --git a/meta/classes/license.bbclass
> b/meta/classes/license.bbclass
> > > index dc91118340..576464cb26 100644
> > > --- a/meta/classes/license.bbclass
> > > +++ b/meta/classes/license.bbclass
> > > @@ -435,3 +435,30 @@ python do_populate_lic_setscene () {
> > >      sstate_setscene(d)
> > >  }
> > >  addtask do_populate_lic_setscene
> > > +
> > > +
> > > +def split_spdx_lic(d, licensestr):
> > > +    """
> > > +    Split the license strings and returns a set of the
> > > +    canonicalised licenses.
> > > +    """
> > > +    import oe.license
> > > +    split_lic = oe.license.list_licenses(licensestr)
> > > +    spdx_lic = set([canonical_license(d, l) for l in split_lic])
> > > +    return spdx_lic
> > > +
> > > +def rem_false_lics(d, pkglic):
> > > +    pkglicsperpkg = set([])
> > > +    for l in pkglic:
> > > +        if l.endswith('-or-later'):
> > > +            # Converts '-or-later' licenses to their canonicalised
> form
> > > +            lic_ = l.replace('-or-later', '+')
> >
> > Given that licenses such as "GPL-2.0+" are deprecated by SPDX, shouldn't
> we
> > instead introduce the canonical "GPL-2.0-only" and "GPL-2.0-or-later"?
> > And then add mappings for, e.g., "GPL-2.0" to "GPL-2.0-only" and
> "GPL-2.0+"
> > to "GPL-2.0-or-later".
>
> Something like:
>
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?h=master-next&id=1c2ce4565bc85915ab970e42bf7af3a78ec2dd6a
> ?
>
> :)
>
> > > +            pkglicsperpkg.add(lic_)
> > > +        elif l.endswith(' WITH Linux-syscall-note'):
> > > +        # Gets rid of "WITH Linux-syscall-note from license sring"
> > > +            if d.getVar("LICENSE_WITH_LINUX_SYS") == "1":
> > > +                lic_ = l.replace(' WITH Linux-syscall-note', '')
> >
> > Looking at https://spdx.org/licenses/exceptions-index.html, there is a
> long
> > list of predefined exceptions and "Linux-syscall-note" is just one of
> them.
> > We probably want a more generic solution for how to handle exceptions.
>
> Agreed, I was trying to make that point in one of my replies!
>
> Cheers,
>
> Richard
> >
>
>

[-- Attachment #2: Type: text/html, Size: 5662 bytes --]

Re: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Richard Purdie]

On Wed, 2021-02-17 at 17:55 +0100, Ida Delphine wrote:
> Thanks for the feedback. I will get rid of the logic to take care of 
> those licenses with "or-later" and move the functions to license.py. 
> Does this also mean I shouldn't try removing "WITH-Linux-syscall-note" 
> from the licences?

I think the idea of what you're looking at is to remove "noise" from the 
output and see if we can identify real licence issues. Removing the 
"XXX WITH-Linux-syscall-note" where XXX is removed is key to that but 
XXX must me removed too.

> The point of me doing that was for there to be more matches and the 
> number of warnings reduced.


> Also there is more license-related logic in package.bbclass. Will it 
> be okay if I move it as well to meta/lib/oe/license.py putting it 
> in functions?

Possibly, but please do it in a separate patch which we can evaluate 
on its own merit, I haven't looked at which code you mean so its hard 
to tell offhand.

Cheers,

Richard

Re: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Meh Mbeh Ida Delphine]

[-- Attachment #1: Type: text/plain, Size: 1507 bytes --]

Hi Richard,

About moving the functions to license.py I have some trouble importing a
.bbclass file to the python file. In one of the functions I made use of a
function found in license.bbclass. Just wondering why it's better to move
the functions to license.py instead of leaving them at license.bbclass and
how I will deal with this import issue

Cheers,
Ida

On Wed, Feb 17, 2021 at 9:51 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2021-02-17 at 17:55 +0100, Ida Delphine wrote:
> > Thanks for the feedback. I will get rid of the logic to take care of
> > those licenses with "or-later" and move the functions to license.py.
> > Does this also mean I shouldn't try removing "WITH-Linux-syscall-note"
> > from the licences?
>
> I think the idea of what you're looking at is to remove "noise" from the
> output and see if we can identify real licence issues. Removing the
> "XXX WITH-Linux-syscall-note" where XXX is removed is key to that but
> XXX must me removed too.
>
> > The point of me doing that was for there to be more matches and the
> > number of warnings reduced.
>
>
> > Also there is more license-related logic in package.bbclass. Will it
> > be okay if I move it as well to meta/lib/oe/license.py putting it
> > in functions?
>
> Possibly, but please do it in a separate patch which we can evaluate
> on its own merit, I haven't looked at which code you mean so its hard
> to tell offhand.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 2023 bytes --]

Re: [OE-core] [poky-contrib][RFC PATCH 4/5] license.bbclass: Add functions to split and canonicalise license strings

[Richard Purdie]

On Sun, 2021-02-21 at 04:04 +0100, Ida Delphine wrote:
> About moving the functions to license.py I have some trouble importing 
> a .bbclass file to the python file. In one of the functions I made use
> of a function found in license.bbclass. Just wondering why it's better
>  to move the functions to license.py instead of leaving them at 
> license.bbclass and how I will deal with this import issue

The functions in license.py will need to be standalone, they can't easily
call code within a bblcass, only other functions within that module. That 
suggests you may need to move other functions, or maybe not move them at
all, it depends what the code is doing really.

We also need to be mindful about dependency tracking. Code in bbclass
files is scanned for variable dependencies, so a getVar("XXX") will
automatically add a dependency on XXX for the generation of task hashes.
Once the code moves to license.py, this does not happen so we have to
account for it manually if needed.

Cheers,

Richard

[poky-contrib][RFC PATCH 5/5] package.bbclass: Remove false positive license warnings

[Meh Mbeh Ida Delphine]

- Splits license strings and canonicalise using split_spdx_lic() from license.bbclass. This is so that they can be easily compared with the computedpkglics. The computedpkglics are passed into rem_false_lics() in order so that more warnings can be filtered during comparism with the source licenses.
- After the comparism between computedpkglics and recipe lics, the warnings are displayed if and only if there are licenses in the recipes not found in the sources.
- package_qa_handle_error() from insane.bbclass handles display of warnings by allowing the user decide whether they want the issues treated as warnings, errors or hidden completely. This is done by setting a variable "license_source_spdx"

Signed-off-by: Ida Delphine <idadelm@gmail.com>
---
 meta/classes/package.bbclass | 145 ++++++++++++++++-------------------
 1 file changed, 68 insertions(+), 77 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 782d9d00b3..08e8fa1694 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1546,6 +1546,7 @@ PKGDATA_VARS = "PN PE PV PR PKGE PKGV PKGR LICENSE DESCRIPTION SUMMARY RDEPENDS
 
 python emit_pkgdata() {
     import oe.license
+    import bb.utils
     from glob import glob
     import json
     import subprocess
@@ -1706,83 +1707,73 @@ fi
         write_extra_runtime_pkgs(global_variants, packages, pkgdatadir)
 
     sourceresult = d.getVar('TEMPDBGSRCMAPPING', False)
-    sources = {}
-    if sourceresult:
-        for r in sourceresult:
-            sources[r[0]] = r[1]
-        with open(data_file + ".srclist", 'w') as f:
-            f.write(json.dumps(sources, sort_keys=True))
-
-        filelics = {}
-        for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]:
-            p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent)
-            out, err = p.communicate()
-            if p.returncode == 0:
-                for l in out.decode("utf-8").split("\n"):
-                    l = l.strip()
-                    if not l:
-                        continue
-                    l = l.split(":")
-                    if len(l) < 3:
-                        bb.warn(str(l))
-                        continue
-                    fn = "/" + l[0]
-                    lic = l[2].strip()
-                    if lic.endswith("*/"):
-                        lic = lic[:-2]
-                    lic = lic.strip()
-                    filelics[fn] = lic
-        with open(data_file + ".filelics", 'w') as f:
-            f.write(json.dumps(filelics, sort_keys=True))
-
-        computedlics = {}
-        computedpkglics = {}
-        for r in sourceresult:
-            for subf in r[1]:
-                if subf in filelics:
-                    if r[0] not in computedlics:
-                        computedlics[r[0]] = set()
-                    computedlics[r[0]].add(filelics[subf])
-        #if computedlics:
-        #    bb.warn(str(computedlics))
-        dvar = d.getVar('PKGD')
-        for f in computedlics:
-            shortf = f.replace(dvar, "")
-            found = False
-            for pkg in filemap:
-                if shortf in filemap[pkg]:
-                    found = True
-                    if pkg not in computedpkglics:
-                        computedpkglics[pkg] = set()
-                    computedpkglics[pkg].update(computedlics[f])
-            if not found:
-                bb.warn("%s not in %s" % (f, str(filemap)))
-        #if computedpkglics:
-        #    bb.warn(str(computedpkglics))
-        for pkg in computedpkglics:
-            lic = d.getVar('LICENSE_%s' % (pkg))
-            if not lic:
-                lic = d.getVar('LICENSE')
-
-            # Splits the LICENSE values and canonicalise each license
-            # in the set of split license(s)    
-            split_lic = oe.license.list_licenses(lic)
-            spdx_lic = set([canonical_license(d, l) for l in split_lic])
-            if computedpkglics[pkg]:
-                computedpkglicsperpkg = set([])
-                for l in computedpkglics[pkg]:
-                    if l.endswith('-or-later'):
-                        lic_ = l.replace('-or-later', '+')
-                        computedpkglicsperpkg.add(lic_)
-                    elif l.endswith(' WITH Linux-syscall-note'):
-                        if d.getVar("IGNOREWITHLINUXSYSCALLNOTE") == "1":
-                            lic_ = l.replace(' WITH Linux-syscall-note', '')
-                            computedpkglicsperpkg.add(lic_)
-                    else:
-                        computedpkglicsperpkg.add(l)
-            # Displays warnings for licenses found in the recipes and not sources
-            if spdx_lic - computedpkglicsperpkg:
-                bb.warn("License for package %s is %s (source SPDX headers) vs %s (LICENSE)" % (pkg, computedpkglicsperpkg, spdx_lic))
+    if bb.utils.contains("license_source_spdx", "1", True, False, d):
+        sources = {}
+        if sourceresult:
+            for r in sourceresult:
+                sources[r[0]] = r[1]
+            with open(data_file + ".srclist", 'w') as f:
+                f.write(json.dumps(sources, sort_keys=True))
+
+            filelics = {}
+            for dirent in [d.getVar('PKGD'), d.getVar('STAGING_DIR_TARGET')]:
+                p = subprocess.Popen(["grep", 'SPDX-License-Identifier:', '-R', '-I'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=dirent)
+                out, err = p.communicate()
+                if p.returncode == 0:
+                    for l in out.decode("utf-8").split("\n"):
+                        l = l.strip()
+                        if not l:
+                            continue
+                        l = l.split(":")
+                        if len(l) < 3:
+                            bb.warn(str(l))
+                            continue
+                        fn = "/" + l[0]
+                        lic = l[2].strip()
+                        if lic.endswith("*/"):
+                            lic = lic[:-2]
+                        lic = lic.strip()
+                        filelics[fn] = lic
+            with open(data_file + ".filelics", 'w') as f:
+                f.write(json.dumps(filelics, sort_keys=True))
+
+            computedlics = {}
+            computedpkglics = {}
+            for r in sourceresult:
+                for subf in r[1]:
+                    if subf in filelics:
+                        if r[0] not in computedlics:
+                            computedlics[r[0]] = set()
+                        computedlics[r[0]].add(filelics[subf])
+            # if computedlics:
+            #    bb.warn(str(computedlics))
+            dvar = d.getVar('PKGD')
+            for f in computedlics:
+                shortf = f.replace(dvar, "")
+                found = False
+                for pkg in filemap:
+                    if shortf in filemap[pkg]:
+                        found = True
+                        if pkg not in computedpkglics:
+                            computedpkglics[pkg] = set()
+                        computedpkglics[pkg].update(computedlics[f])
+                if not found:
+                    bb.warn("%s not in %s" % (f, str(filemap)))
+            # if computedpkglics:
+            #    bb.warn(str(computedpkglics))
+            for pkg in computedpkglics:
+                lic = d.getVar('LICENSE_%s' % (pkg))    
+                if not lic:
+                    lic = d.getVar('LICENSE')
+
+                # Splits the LICENSE values and canonicalise each license
+                # in the set of split license(s)
+                spdx_lic = split_spdx_lic(d, lic)
+                computedpkglicsperpkg = rem_false_lics(d, computedpkglics[pkg])    
+               
+                # Displays warnings for licenses found in the recipes and not sources
+                if spdx_lic - computedpkglicsperpkg:
+                    package_qa_handle_error("license_source_spdx", f'License for package {pkg} is {computedpkglicsperpkg} (source SPDX headers) vs {spdx_lic} (LICENSE)', d)
 }
 emit_pkgdata[dirs] = "${PKGDESTWORK}/runtime ${PKGDESTWORK}/runtime-reverse ${PKGDESTWORK}/runtime-rprovides"
 
-- 
2.25.1

Re: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for improvements?

[Peter Kjellerstedt]

I tried to apply these patches, but the context does not match. Are 
there other patches I need to apply before these can be applied?

//Peter

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Meh Mbeh Ida Delphine
> Sent: den 17 februari 2021 05:00
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for
> improvements?
> 
> Hello everyone,
> 
> The following patchset is aimed at improving the Yocto Project license
> tracing.
> 
> Initially, after a build I will come across 130 license warnings with some
> false positives. I got rid of these by replacing "or-later" with "+" for
> source licenses (computedpkglics) and also passing the recipe license
> through canonical_license of license.py(). As a result, some of the
> licenses will match during comparism hence some warnings eliminate.
> 
> Though some false positive warnings were eliminated, lots of warnings were
> ouputed as a result of several source licenses having "WITH Linux-syscall-
> note". With the help of a variable being set in local.conf, one can
> chooses whether or not to get lid of the "WITH Linux-syscall-note" string
> from the source licenses. Just by doing this, several warnings are dropped
> and now left with 118.
> 
> I later on added some logic depending on "source_spdx_license" variable
> whether or not for the user to allow the warnings being outputed during
> the build. It reduces the extra processing and reduces time if the user
> chooses not to display warnings. It uses bb.utils.contains() to find out
> if that particular vaiable is present. Also, I added
> package_qa_handle_error() to allow the user decide whether they want the
> issues treated as warnings, errors or not at all.
> 
> Some of the license-related logic I added to take care of splitting recipe
> licences before canonicalizing, getting rid of "or-later" warnings, I
> moved it to license.bbclass and put it in functions. I also plan on fixing
> the variable naming since some have underscores whereas others do not
> have. However, I will love to move more license logic in package.bbclass
> to license.bbclass but I need help with the right way to go about it.
> Also, general suggestions on any improvements to this are highly welcomed.
> 
> Cheers,
> Ida.
> 
> Ida Delphine (5):
>   package: Remove false positive lic warnings
>   package: Remove false positive lic warnings
>   package: Remove false positive lic warnings
>   license.bbclass: Add functions to split and canonicalise license
>     strings
>   package.bbclass: Remove false positive license warnings
> 
>  meta/classes/license.bbclass |  27 ++++++++
>  meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
>  2 files changed, 96 insertions(+), 58 deletions(-)
> 
> --
> 2.25.1

Re: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for improvements?

[Meh Mbeh Ida Delphine]

[-- Attachment #1: Type: text/plain, Size: 3223 bytes --]

Please what exactly didn't match?

Ida

On Wed, Feb 17, 2021 at 4:50 PM Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:

> I tried to apply these patches, but the context does not match. Are
> there other patches I need to apply before these can be applied?
>
> //Peter
>
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Meh Mbeh Ida Delphine
> > Sent: den 17 februari 2021 05:00
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for
> > improvements?
> >
> > Hello everyone,
> >
> > The following patchset is aimed at improving the Yocto Project license
> > tracing.
> >
> > Initially, after a build I will come across 130 license warnings with
> some
> > false positives. I got rid of these by replacing "or-later" with "+" for
> > source licenses (computedpkglics) and also passing the recipe license
> > through canonical_license of license.py(). As a result, some of the
> > licenses will match during comparism hence some warnings eliminate.
> >
> > Though some false positive warnings were eliminated, lots of warnings
> were
> > ouputed as a result of several source licenses having "WITH
> Linux-syscall-
> > note". With the help of a variable being set in local.conf, one can
> > chooses whether or not to get lid of the "WITH Linux-syscall-note" string
> > from the source licenses. Just by doing this, several warnings are
> dropped
> > and now left with 118.
> >
> > I later on added some logic depending on "source_spdx_license" variable
> > whether or not for the user to allow the warnings being outputed during
> > the build. It reduces the extra processing and reduces time if the user
> > chooses not to display warnings. It uses bb.utils.contains() to find out
> > if that particular vaiable is present. Also, I added
> > package_qa_handle_error() to allow the user decide whether they want the
> > issues treated as warnings, errors or not at all.
> >
> > Some of the license-related logic I added to take care of splitting
> recipe
> > licences before canonicalizing, getting rid of "or-later" warnings, I
> > moved it to license.bbclass and put it in functions. I also plan on
> fixing
> > the variable naming since some have underscores whereas others do not
> > have. However, I will love to move more license logic in package.bbclass
> > to license.bbclass but I need help with the right way to go about it.
> > Also, general suggestions on any improvements to this are highly
> welcomed.
> >
> > Cheers,
> > Ida.
> >
> > Ida Delphine (5):
> >   package: Remove false positive lic warnings
> >   package: Remove false positive lic warnings
> >   package: Remove false positive lic warnings
> >   license.bbclass: Add functions to split and canonicalise license
> >     strings
> >   package.bbclass: Remove false positive license warnings
> >
> >  meta/classes/license.bbclass |  27 ++++++++
> >  meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
> >  2 files changed, 96 insertions(+), 58 deletions(-)
> >
> > --
> > 2.25.1
>
>

[-- Attachment #2: Type: text/html, Size: 4151 bytes --]

Re: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for improvements?

[Peter Kjellerstedt]

[-- Attachment #1: Type: text/plain, Size: 3735 bytes --]

None of the context for the second hunk in the first patch exists. E.g., if I checkout master of poky and grep for “License for package %s” (part of the line that the first patch replaces), there is no match in any file.

//Peter

From: Ida Delphine <idadelm@gmail.com>
Sent: den 17 februari 2021 17:59
To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Cc: openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for improvements?

Please what exactly didn't match?

Ida

On Wed, Feb 17, 2021 at 4:50 PM Peter Kjellerstedt <peter.kjellerstedt@axis.com<mailto:peter.kjellerstedt@axis.com>> wrote:
I tried to apply these patches, but the context does not match. Are
there other patches I need to apply before these can be applied?

//Peter

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org> <openembedded-
> core@lists.openembedded.org<mailto:core@lists.openembedded.org>> On Behalf Of Meh Mbeh Ida Delphine
> Sent: den 17 februari 2021 05:00
> To: openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org>
> Subject: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for
> improvements?
>
> Hello everyone,
>
> The following patchset is aimed at improving the Yocto Project license
> tracing.
>
> Initially, after a build I will come across 130 license warnings with some
> false positives. I got rid of these by replacing "or-later" with "+" for
> source licenses (computedpkglics) and also passing the recipe license
> through canonical_license of license.py(). As a result, some of the
> licenses will match during comparism hence some warnings eliminate.
>
> Though some false positive warnings were eliminated, lots of warnings were
> ouputed as a result of several source licenses having "WITH Linux-syscall-
> note". With the help of a variable being set in local.conf, one can
> chooses whether or not to get lid of the "WITH Linux-syscall-note" string
> from the source licenses. Just by doing this, several warnings are dropped
> and now left with 118.
>
> I later on added some logic depending on "source_spdx_license" variable
> whether or not for the user to allow the warnings being outputed during
> the build. It reduces the extra processing and reduces time if the user
> chooses not to display warnings. It uses bb.utils.contains() to find out
> if that particular vaiable is present. Also, I added
> package_qa_handle_error() to allow the user decide whether they want the
> issues treated as warnings, errors or not at all.
>
> Some of the license-related logic I added to take care of splitting recipe
> licences before canonicalizing, getting rid of "or-later" warnings, I
> moved it to license.bbclass and put it in functions. I also plan on fixing
> the variable naming since some have underscores whereas others do not
> have. However, I will love to move more license logic in package.bbclass
> to license.bbclass but I need help with the right way to go about it.
> Also, general suggestions on any improvements to this are highly welcomed.
>
> Cheers,
> Ida.
>
> Ida Delphine (5):
>   package: Remove false positive lic warnings
>   package: Remove false positive lic warnings
>   package: Remove false positive lic warnings
>   license.bbclass: Add functions to split and canonicalise license
>     strings
>   package.bbclass: Remove false positive license warnings
>
>  meta/classes/license.bbclass |  27 ++++++++
>  meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
>  2 files changed, 96 insertions(+), 58 deletions(-)
>
> --
> 2.25.1

[-- Attachment #2: Type: text/html, Size: 7465 bytes --]

Re: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for improvements?

[Meh Mbeh Ida Delphine]

[-- Attachment #1: Type: text/plain, Size: 3984 bytes --]

I'm using poky-contrib; checkout paule/rpurdie-license-experiments-osls

Cheers,
Ida

On Thu, Feb 18, 2021 at 12:17 PM Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:

> None of the context for the second hunk in the first patch exists. E.g.,
> if I checkout master of poky and grep for “License for package %s” (part of
> the line that the first patch replaces), there is no match in any file.
>
>
>
> //Peter
>
>
>
> *From:* Ida Delphine <idadelm@gmail.com>
> *Sent:* den 17 februari 2021 17:59
> *To:* Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> *Cc:* openembedded-core@lists.openembedded.org
> *Subject:* Re: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for
> improvements?
>
>
>
> Please what exactly didn't match?
>
>
>
> Ida
>
>
>
> On Wed, Feb 17, 2021 at 4:50 PM Peter Kjellerstedt <
> peter.kjellerstedt@axis.com> wrote:
>
> I tried to apply these patches, but the context does not match. Are
> there other patches I need to apply before these can be applied?
>
> //Peter
>
> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Meh Mbeh Ida Delphine
> > Sent: den 17 februari 2021 05:00
> > To: openembedded-core@lists.openembedded.org
> > Subject: [OE-core] [poky-contrib][RFC PATCH 0/5] Suggestions for
> > improvements?
> >
> > Hello everyone,
> >
> > The following patchset is aimed at improving the Yocto Project license
> > tracing.
> >
> > Initially, after a build I will come across 130 license warnings with
> some
> > false positives. I got rid of these by replacing "or-later" with "+" for
> > source licenses (computedpkglics) and also passing the recipe license
> > through canonical_license of license.py(). As a result, some of the
> > licenses will match during comparism hence some warnings eliminate.
> >
> > Though some false positive warnings were eliminated, lots of warnings
> were
> > ouputed as a result of several source licenses having "WITH
> Linux-syscall-
> > note". With the help of a variable being set in local.conf, one can
> > chooses whether or not to get lid of the "WITH Linux-syscall-note" string
> > from the source licenses. Just by doing this, several warnings are
> dropped
> > and now left with 118.
> >
> > I later on added some logic depending on "source_spdx_license" variable
> > whether or not for the user to allow the warnings being outputed during
> > the build. It reduces the extra processing and reduces time if the user
> > chooses not to display warnings. It uses bb.utils.contains() to find out
> > if that particular vaiable is present. Also, I added
> > package_qa_handle_error() to allow the user decide whether they want the
> > issues treated as warnings, errors or not at all.
> >
> > Some of the license-related logic I added to take care of splitting
> recipe
> > licences before canonicalizing, getting rid of "or-later" warnings, I
> > moved it to license.bbclass and put it in functions. I also plan on
> fixing
> > the variable naming since some have underscores whereas others do not
> > have. However, I will love to move more license logic in package.bbclass
> > to license.bbclass but I need help with the right way to go about it.
> > Also, general suggestions on any improvements to this are highly
> welcomed.
> >
> > Cheers,
> > Ida.
> >
> > Ida Delphine (5):
> >   package: Remove false positive lic warnings
> >   package: Remove false positive lic warnings
> >   package: Remove false positive lic warnings
> >   license.bbclass: Add functions to split and canonicalise license
> >     strings
> >   package.bbclass: Remove false positive license warnings
> >
> >  meta/classes/license.bbclass |  27 ++++++++
> >  meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
> >  2 files changed, 96 insertions(+), 58 deletions(-)
> >
> > --
> > 2.25.1
>
>

[-- Attachment #2: Type: text/html, Size: 6762 bytes --]