[PATCH 0/2][V2] License compliance fixes

[PATCH 0/2][V2] License compliance fixes

[Ross Burton]

Hi,

If INCOMPATIBLE_LICENSE=GPLv3.0 but the recipe sets LICENSE=GPLv3, the current
code won't trigger because they're different strings.

Fix this by attempting to canonicalise every license name to a SPDX name, so
both names in this example become GPL-3.0.

The whitelists also have to be updated.  This means using variable names that
could be considered non-standard (or "crazy"), such as WHITELIST_GPL-3.0.  This
does look strange and if the community agrees we can take the compatibility hit
I'll change the whitelists to use flags, i.e HOSTTOOLS_WHITELIST[GPL-3.0].

Ross

The following changes since commit 1dcdd877c7946be4c0b1203deb14e2f842f9d0c2:

  bitbake: toasterui: fix build - project identification (2014-07-14 14:10:03 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ross/license

for you to fetch changes up to 934289b8ba6a8b7a0f1eec8aabf723f823d515d8:

  default-distrovars: update license whitelists to use canonical names (2014-07-16 21:02:45 +0100)

----------------------------------------------------------------
Ross Burton (2):
      license.bbclass: canonicalise licenses when dealing with INCOMPATIBLE_LICENSE
      default-distrovars: update license whitelists to use canonical names

 meta/classes/base.bbclass                       |    2 ++
 meta/classes/license.bbclass                    |   14 +++++++++++---
 meta/conf/distro/include/default-distrovars.inc |    6 +++---
 3 files changed, 16 insertions(+), 6 deletions(-)

Ross Burton (2):
  license.bbclass: canonicalise licenses when dealing with
    INCOMPATIBLE_LICENSE
  default-distrovars: update license whitelists to use canonical names

 meta/classes/base.bbclass                       |    2 ++
 meta/classes/license.bbclass                    |   14 +++++++++++---
 meta/conf/distro/include/default-distrovars.inc |    6 +++---
 3 files changed, 16 insertions(+), 6 deletions(-)

-- 
1.7.10.4

[PATCH 1/2] license.bbclass: canonicalise licenses when dealing with INCOMPATIBLE_LICENSE

[Ross Burton]

If INCOMPATIBLE_LICENSE=GPLv3.0 but the recipe sets LICENSE=GPLv3, the current
code won't trigger because they're different strings.

Fix this by attempting to canonicalise every license name to a SPDX name, so
both names in this example become GPL-3.0.

[ YOCTO #5622 ]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/classes/base.bbclass    |    2 ++
 meta/classes/license.bbclass |   14 +++++++++++---
 2 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index c0d2c8ec8..8114cf6 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -540,6 +540,8 @@ python () {
                 check_license = False
 
         if check_license and bad_licenses:
+            bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses)
+
             whitelist = []
             for lic in bad_licenses:
                 for w in ["HOSTTOOLS_WHITELIST_", "LGPLv2_WHITELIST_", "WHITELIST_"]:
diff --git a/meta/classes/license.bbclass b/meta/classes/license.bbclass
index 08f0665..973a95b 100644
--- a/meta/classes/license.bbclass
+++ b/meta/classes/license.bbclass
@@ -264,10 +264,18 @@ def return_spdx(d, license):
      """
     return d.getVarFlag('SPDXLICENSEMAP', license, True)
 
+def canonical_license(d, license):
+    """
+    Return the canonical (SPDX) form of the license if available (so GPLv3
+    becomes GPL-3.0), or the passed license if there is no canonical form.
+    """
+    return d.getVarFlag('SPDXLICENSEMAP', license, True) or license
+
 def incompatible_license(d, dont_want_licenses, package=None):
     """
-    This function checks if a recipe has only incompatible licenses. It also take into consideration 'or'
-    operand.
+    This function checks if a recipe has only incompatible licenses. It also
+    take into consideration 'or' operand.  dont_want_licenses should be passed
+    as canonical (SPDX) names.
     """
     import re
     import oe.license
@@ -298,7 +306,7 @@ def incompatible_license(d, dont_want_licenses, package=None):
         licenses = oe.license.flattened_licenses(license, choose_lic_set)
     except oe.license.LicenseError as exc:
         bb.fatal('%s: %s' % (d.getVar('P', True), exc))
-    return any(not license_ok(l) for l in licenses)
+    return any(not license_ok(canonical_license(d, l)) for l in licenses)
 
 def check_license_flags(d):
     """
-- 
1.7.10.4

[PATCH 2/2] default-distrovars: update license whitelists to use canonical names

[Ross Burton]

Now that all licenses are canonicalised to SPDX names when processing, we need
to rename the whitelists to the match.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/conf/distro/include/default-distrovars.inc |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/conf/distro/include/default-distrovars.inc b/meta/conf/distro/include/default-distrovars.inc
index 1bc8e46..8d24d66 100644
--- a/meta/conf/distro/include/default-distrovars.inc
+++ b/meta/conf/distro/include/default-distrovars.inc
@@ -23,9 +23,9 @@ IMAGE_FEATURES ?= ""
 
 # This is a list of packages that are used by the build system to build the distribution, they are not
 # directly part of the distribution. 
-HOSTTOOLS_WHITELIST_GPLv3 ?= ""
-WHITELIST_GPLv3 ?= "less"
-LGPLv2_WHITELIST_GPLv3 ?= "libassuan gnutls libtasn1 libidn libgcc libgcc-initial gcc-runtime" 
+HOSTTOOLS_WHITELIST_GPL-3.0 ?= ""
+WHITELIST_GPL-3.0 ?= "less"
+LGPLv2_WHITELIST_GPL-3.0 ?= "libassuan gnutls libtasn1 libidn libgcc libgcc-initial gcc-runtime"
 
 COMMERCIAL_AUDIO_PLUGINS ?= ""
 # COMMERCIAL_AUDIO_PLUGINS ?= "gst-plugins-ugly-mad gst-plugins-ugly-mpegaudioparse"
-- 
1.7.10.4

Re: [PATCH 1/2] license.bbclass: canonicalise licenses when dealing with INCOMPATIBLE_LICENSE

[Paul Eggleton]

On Wednesday 16 July 2014 22:08:13 Ross Burton wrote:
> If INCOMPATIBLE_LICENSE=GPLv3.0 but the recipe sets LICENSE=GPLv3, the

Pedantic, but INCOMPATIBLE_LICENSE = "GPL-3.0" surely?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/2] license.bbclass: canonicalise licenses when dealing with INCOMPATIBLE_LICENSE

[Burton, Ross]

On 17 July 2014 15:28, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> On Wednesday 16 July 2014 22:08:13 Ross Burton wrote:
>> If INCOMPATIBLE_LICENSE=GPLv3.0 but the recipe sets LICENSE=GPLv3, the
>
> Pedantic, but INCOMPATIBLE_LICENSE = "GPL-3.0" surely?

Yes, sigh.

Ross

Re: [PATCH 2/2] default-distrovars: update license whitelists to use canonical names

[Saul Wold]

[-- Attachment #1: Type: text/plain, Size: 1403 bytes --]

On 07/16/2014 02:08 PM, Ross Burton wrote:
> Now that all licenses are canonicalised to SPDX names when processing, we need
> to rename the whitelists to the match.
>
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>   meta/conf/distro/include/default-distrovars.inc |    6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/meta/conf/distro/include/default-distrovars.inc b/meta/conf/distro/include/default-distrovars.inc
> index 1bc8e46..8d24d66 100644
> --- a/meta/conf/distro/include/default-distrovars.inc
> +++ b/meta/conf/distro/include/default-distrovars.inc
> @@ -23,9 +23,9 @@ IMAGE_FEATURES ?= ""
>
>   # This is a list of packages that are used by the build system to build the distribution, they are not
>   # directly part of the distribution.
> -HOSTTOOLS_WHITELIST_GPLv3 ?= ""
> -WHITELIST_GPLv3 ?= "less"
> -LGPLv2_WHITELIST_GPLv3 ?= "libassuan gnutls libtasn1 libidn libgcc libgcc-initial gcc-runtime"
> +HOSTTOOLS_WHITELIST_GPL-3.0 ?= ""
> +WHITELIST_GPL-3.0 ?= "less"
> +LGPLv2_WHITELIST_GPL-3.0 ?= "libassuan gnutls libtasn1 libidn libgcc libgcc-initial gcc-runtime"
>

This change seems to make multilib builds unhappy, both on the 
Autobuilder and locally, see attached output.

Sau!

>   COMMERCIAL_AUDIO_PLUGINS ?= ""
>   # COMMERCIAL_AUDIO_PLUGINS ?= "gst-plugins-ugly-mad gst-plugins-ugly-mpegaudioparse"
>

[-- Attachment #2: cranky_multilib_build --]
[-- Type: text/plain, Size: 11291 bytes --]

Parsing recipes...ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xaa61110>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xc2d9bd0>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x2eec3d0>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Unable to parse /srv/ssd/poky/meta-yocto/recipes-core/tiny-init/tiny-init.bb
Traceback (most recent call last):
  File "/srv/ssd/poky/bitbake/lib/bb/parse/ast.py", line 320, in finalize(fn='/srv/ssd/poky/meta-yocto/recipes-core/tiny-init/tiny-init.bb', d=<bb.data_smart.DataSmart object at 0x4eb9690>, variant='multilib:lib32'):
     
    >    bb.event.fire(bb.event.RecipePreFinalise(fn), d)
     
  File "/srv/ssd/poky/bitbake/lib/bb/event.py", line 158, in fire(event=<bb.event.RecipePreFinalise object at 0xaa61110>, d=<bb.data_smart.DataSmart object at 0x4eb9690>):
     
    >    fire_class_handlers(event, d)
         if worker_fire:
  File "/srv/ssd/poky/bitbake/lib/bb/event.py", line 97, in fire_class_handlers(event=<bb.event.RecipePreFinalise object at 0xaa61110>, d=<bb.data_smart.DataSmart object at 0x4eb9690>):
             if name in _catchall_handlers or name in evt_hmap:
    >            execute_handler(name, handler, event, d)
     
  File "/srv/ssd/poky/bitbake/lib/bb/event.py", line 74, in execute_handler(name='multilib_virtclass_handler', handler=<function multilib_virtclass_handler at 0x2eff140>, event=<bb.event.RecipePreFinalise object at 0xaa61110>, d=<bb.data_smart.DataSmart object at 0x4eb9690>):
         try:
    >        ret = handler(event)
         except (bb.parse.SkipRecipe, bb.BBHandledException):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xaa61110>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x92ef390>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xa0bdb10>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xa704050>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x251b110>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xc2bf550>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x9fa5c10>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x251b110>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xc2d00d0>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xc2d9150>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x92efc90>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x2ede8d0>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xc2d0ed0>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x4eccd90>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x4eb9c10>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x2eecb10>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x9205fd0>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xc2eab10>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x92ef650>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x2ed4c90>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x251b110>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x92efad0>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0xc2e6510>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x2ede890>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x9b6f390>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x920a850>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x9d6e590>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x4eb4f10>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x2edea90>)
AttributeError: 'NoneType' object has no attribute 'split'

ERROR: Execution of event handler 'multilib_virtclass_handler' failed
Traceback (most recent call last):
  File "multilib_virtclass_handler(e)", line 55, in multilib_virtclass_handler(e=<bb.event.RecipePreFinalise object at 0x2ee7d90>)
AttributeError: 'NoneType' object has no attribute 'split'


Summary: There were 33 ERROR messages shown, returning a non-zero exit code.

Re: [PATCH 2/2] default-distrovars: update license whitelists to use canonical names

[Richard Purdie]

On Fri, 2014-07-18 at 15:35 -0700, Saul Wold wrote:
> On 07/16/2014 02:08 PM, Ross Burton wrote:
> > Now that all licenses are canonicalised to SPDX names when processing, we need
> > to rename the whitelists to the match.
> >
> > Signed-off-by: Ross Burton <ross.burton@intel.com>
> > ---
> >   meta/conf/distro/include/default-distrovars.inc |    6 +++---
> >   1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/meta/conf/distro/include/default-distrovars.inc b/meta/conf/distro/include/default-distrovars.inc
> > index 1bc8e46..8d24d66 100644
> > --- a/meta/conf/distro/include/default-distrovars.inc
> > +++ b/meta/conf/distro/include/default-distrovars.inc
> > @@ -23,9 +23,9 @@ IMAGE_FEATURES ?= ""
> >
> >   # This is a list of packages that are used by the build system to build the distribution, they are not
> >   # directly part of the distribution.
> > -HOSTTOOLS_WHITELIST_GPLv3 ?= ""
> > -WHITELIST_GPLv3 ?= "less"
> > -LGPLv2_WHITELIST_GPLv3 ?= "libassuan gnutls libtasn1 libidn libgcc libgcc-initial gcc-runtime"
> > +HOSTTOOLS_WHITELIST_GPL-3.0 ?= ""
> > +WHITELIST_GPL-3.0 ?= "less"
> > +LGPLv2_WHITELIST_GPL-3.0 ?= "libassuan gnutls libtasn1 libidn libgcc libgcc-initial gcc-runtime"
> >
> 
> This change seems to make multilib builds unhappy, both on the 
> Autobuilder and locally, see attached output.

FWIW I've fixed this before I merged it, it needed a tweak to
multilib.bbclass...

Cheers,

Richard