[Buildroot] [PATCH 0/4 v6] legal-info: extract packages to get license files (branch yem/legal)

[Buildroot] [PATCH 0/4 v6] legal-info: extract packages to get license files (branch yem/legal)

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Hello All!

Here is a series that fixes the legal-info infra:
  - extracts sources before copying license files
  - saves license files even for no-redistribute packages
  - cleans up to account for the changes (comment and helper rename)


Changes v5 -> v6:
  - fix commit log about what is actually fixed  (Luca)
  - use a proper name for the renamed macro  (Luca)
  - fix rebasing after the $$ mess ;-)
  - drop patch 5, it's no longer needed

Changes v4 -> v5:
  - don't warn about missing license fioles for packages bundled
    with Buildroot  (Luca)
  - new patch 5/5 to display a proper error message for missing license
    files
  - typoes

Changes v3 -> v4:
  - save licens files for no-redistribute packages  (Luca)
  - download sources before copying to legal-info/sources/
  - rename a legal-info helper

Changes v2 -> v3:
  - fix manifest for no-redistribute, or overriden, or local packages


Regards,
Yann E. MORIN.


The following changes since commit 8d24201f4b943b7a1edd9a16c97155f1a2efda59:

  imagemagick: bump to version 6.8.9-3 (2014-06-21 19:38:07 +0200)

are available in the git repository at:

  git://gitorious.org/buildroot/buildroot.git yem/legal

for you to fetch changes up to 1b7669d8f5b489b6c8b2b3394ba1207c903a9e68:

  legal-info: rename legal-warning-pkg-savednothing helper (2014-06-22 11:19:31 +0200)

----------------------------------------------------------------
Yann E. MORIN (4):
      legal-info: extract even no-redistribute packages
      legal-info: save license files even for local or overridden packages
      legal-info: add a comment about what packages we save the tarballs of
      legal-info: rename legal-warning-pkg-savednothing helper

 package/pkg-generic.mk | 39 +++++++++++++++++++++++++++------------
 package/pkg-utils.mk   |  4 ++--
 2 files changed, 29 insertions(+), 14 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] legal-info: extract even no-redistribute packages

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Currently, if a package is marked _REDISTRIBUTE = NO, then legal-info
will not try to extract it first.

If that package also declares some _LICENSE_FILES, legal-info fails
if it is the only action we're trying to run:

    $ cat defconfig
    BR2_arm=y
    BR2_TOOLCHAIN_BUILDROOT_EGLIBC=y
    BR2_PACKAGE_LIBFSLCODEC=y
    $ make BR2_DEFCONFIG=$(pwd)/defconfig defconfig
    $ make legal-info
    [--SNIP--]
    cat: /home/ymorin/dev/buildroot/O/build/libfslcodec-3.5.7-1.0.0/EULA: No such file or directory

Fix this by always having legal-info extract the archives if one or
more _LICENSE_FILES are specified.

We do this for all types of packages: overridden, local or 'normal'
remote packages. Even though we do not save the sources for the
overridden or local packages, we need to save their licensing info,
so we need to extract them.

This implies that we now need only PKG-source, not PKG-extract anymore,
as a dependency of legal-info for packages we want to save (ie.
redistributable, non-local and non-overriden packages.)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

---
Chamges v5 -> v6:
  - fix commit log about the actual case that is being solved  (Luca)
  - fix the comment about needing only PKG-source  (Luca)

Changes v4 -> v5:
  - my usual s/pacakge/package/

Changes v3 -> v4:
  - legal-info needs to depend on PKG-source when it needs to save a
    package's tarball

Changes v2 -> v3:
  - don't include source URL of no-redistribute, or overriden, or local
    packages in the manifest

Changes v1 -> v2:
  - this is not fixing the autobuilders failure it was written to fix,
    so remove the references to such build failures  (Thomas P)
  - also extract overriden and local packages  (Fabio)
---
 package/pkg-generic.mk | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 457d873..5f2a27e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -620,11 +620,18 @@ $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
 endif
 $(2)_MANIFEST_LICENSE_FILES ?= not saved
 
+# If the package declares _LICENSE_FILES, we need to extract it,
+# for overriden, local or normal remote packages alike, whether
+# we want to redistribute it or not.
+ifneq ($$($(2)_LICENSE_FILES),)
+$(1)-legal-info: $(1)-extract
+endif
+
 ifeq ($$($(2)_REDISTRIBUTE),YES)
 ifneq ($$($(2)_SITE_METHOD),local)
 ifneq ($$($(2)_SITE_METHOD),override)
-# Packages that have a tarball need it downloaded and extracted beforehand
-$(1)-legal-info: $(1)-extract $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
+# Packages that have a tarball need it downloaded beforehand
+$(1)-legal-info: $(1)-source $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
 $(2)_MANIFEST_TARBALL = $$($(2)_SOURCE)
 endif
 endif
-- 
1.8.3.2

[Buildroot] [PATCH 2/4] legal-info: save license files even for local or overridden packages

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

Even if we do not save the sources for local or overridden packages because
it is too complex, we can still quite easily save the license files.

Also, having the license files is a very important part of complying with
the licenses.

Move the copy of license files out of the non-local, non-overridden package
case, but still in the case where packages have a _SOURCE defined, to
avoid catching packages bundled in Buildroot (eg. mkpasswd et al.)

Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

---
Changes v5 -> v6:
  - typoes  (Luca)
  - fix the rebasing mess after the $$ conversion

Changes v4 -> v5:
  - avoid warning for packages bundled in Buildroot  (Luca)
  - the change is not about non-redistributable packages, but about
    local and overriden packages
---
 package/pkg-generic.mk | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5f2a27e..e8f2f9e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -644,6 +644,19 @@ $(1)-legal-info:
 	$$(foreach hook,$$($(2)_PRE_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
 ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 
+# Save license files if defined
+# We save the license files for any kind of package: normal, local,
+# overridden, or non-redistributable alike.
+# The reason to save license files even for no-redistribute packages
+# is that the license still applies to the files distributed as part
+# of the rootfs, even if the sources are not themselves redistributed.
+ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),)
+	@$$(call legal-license-nofiles,$$($(2)_RAWNAME),$$(call UPPERCASE,$(4)))
+	@$$(call legal-warning-pkg,$$($(2)_RAWNAME),cannot save license ($(2)_LICENSE_FILES not defined))
+else
+	@$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep))
+endif # license files
+
 ifeq ($$($(2)_SITE_METHOD),local)
 # Packages without a tarball: don't save and warn
 	@$$(call legal-warning-pkg-savednothing,$$($(2)_RAWNAME),local)
@@ -654,14 +667,6 @@ else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 else
 # Other packages
 
-# Save license files if defined
-ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),)
-	@$$(call legal-license-nofiles,$$($(2)_RAWNAME),$$(call UPPERCASE,$(4)))
-	@$$(call legal-warning-pkg,$$($(2)_RAWNAME),cannot save license ($(2)_LICENSE_FILES not defined))
-else
-	@$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep))
-endif # license files
-
 ifeq ($$($(2)_REDISTRIBUTE),YES)
 # Copy the source tarball (just hardlink if possible)
 	@cp -l $$(DL_DIR)/$$($(2)_SOURCE) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) 2>/dev/null || \
-- 
1.8.3.2

[Buildroot] [PATCH 3/4] legal-info: add a comment about what packages we save the tarballs of

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

As the legal-info infra only (rightfully) saves the tarballs of packages
that:
  - we want to redistribute,
  - and are not local,
  - and are not overriden,

add a comment stating so.

This should clarify the code-block, which although trivial to read,
was not easy to interpret without thinking thouroughly about it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
---
 package/pkg-generic.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index e8f2f9e..55936d2 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -627,6 +627,9 @@ ifneq ($$($(2)_LICENSE_FILES),)
 $(1)-legal-info: $(1)-extract
 endif
 
+# We only save the sources of packages we want to redistribute, that are
+# non-local, and non-overriden. So only store, in the manifest, the tarball
+# name of those packages.
 ifeq ($$($(2)_REDISTRIBUTE),YES)
 ifneq ($$($(2)_SITE_METHOD),local)
 ifneq ($$($(2)_SITE_METHOD),override)
-- 
1.8.3.2

[Buildroot] [PATCH 4/4] legal-info: rename legal-warning-pkg-savednothing helper

[Yann E. MORIN]

From: "Yann E. MORIN" <yann.morin.1998@free.fr>

This helper was called when none of the sources or license
files were saved.

Now we handle license files separately from the sources,
this is no longer the case: they are only called when the
sources are not saved.

Rename the handler and change the warning message accordingly.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Fabio Porcedda <fabio.porcedda@gmail.com>

---
Changes v5 -> v6:
  - fix the silly new name of the macro  (Luca)
---
 package/pkg-generic.mk | 4 ++--
 package/pkg-utils.mk   | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 55936d2..97c8416 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -662,10 +662,10 @@ endif # license files
 
 ifeq ($$($(2)_SITE_METHOD),local)
 # Packages without a tarball: don't save and warn
-	@$$(call legal-warning-pkg-savednothing,$$($(2)_RAWNAME),local)
+	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),local)
 
 else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
-	@$$(call legal-warning-pkg-savednothing,$$($(2)_RAWNAME),override)
+	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),override)
 
 else
 # Other packages
diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 163ef8a..59a5ab2 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -101,8 +101,8 @@ define legal-warning-pkg # pkg, text
 	echo "WARNING: $(1): $(2)" >>$(LEGAL_WARNINGS)
 endef
 
-define legal-warning-pkg-savednothing # pkg, {local|override}
-	$(call legal-warning-pkg,$(1),sources and license files not saved ($(2) packages not handled))
+define legal-warning-nosource # pkg, {local|override}
+	$(call legal-warning-pkg,$(1),sources not saved ($(2) packages not handled))
 endef
 
 define legal-manifest # pkg, version, license, license-files, source, {HOST|TARGET}
-- 
1.8.3.2

[Buildroot] [PATCH 1/4] legal-info: extract even no-redistribute packages

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 22 Jun 2014 14:41:12 +0200, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> Currently, if a package is marked _REDISTRIBUTE = NO, then legal-info
> will not try to extract it first.
> 
> If that package also declares some _LICENSE_FILES, legal-info fails
> if it is the only action we're trying to run:
> 
>     $ cat defconfig
>     BR2_arm=y
>     BR2_TOOLCHAIN_BUILDROOT_EGLIBC=y
>     BR2_PACKAGE_LIBFSLCODEC=y
>     $ make BR2_DEFCONFIG=$(pwd)/defconfig defconfig
>     $ make legal-info
>     [--SNIP--]
>     cat: /home/ymorin/dev/buildroot/O/build/libfslcodec-3.5.7-1.0.0/EULA: No such file or directory
> 
> Fix this by always having legal-info extract the archives if one or
> more _LICENSE_FILES are specified.
> 
> We do this for all types of packages: overridden, local or 'normal'
> remote packages. Even though we do not save the sources for the
> overridden or local packages, we need to save their licensing info,
> so we need to extract them.
> 
> This implies that we now need only PKG-source, not PKG-extract anymore,
> as a dependency of legal-info for packages we want to save (ie.
> redistributable, non-local and non-overriden packages.)
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 2/4] legal-info: save license files even for local or overridden packages

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 22 Jun 2014 14:41:13 +0200, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> Even if we do not save the sources for local or overridden packages because
> it is too complex, we can still quite easily save the license files.
> 
> Also, having the license files is a very important part of complying with
> the licenses.
> 
> Move the copy of license files out of the non-local, non-overridden package
> case, but still in the case where packages have a _SOURCE defined, to
> avoid catching packages bundled in Buildroot (eg. mkpasswd et al.)
> 
> Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 3/4] legal-info: add a comment about what packages we save the tarballs of

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 22 Jun 2014 14:41:14 +0200, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> As the legal-info infra only (rightfully) saves the tarballs of packages
> that:
>   - we want to redistribute,
>   - and are not local,
>   - and are not overriden,
> 
> add a comment stating so.
> 
> This should clarify the code-block, which although trivial to read,
> was not easy to interpret without thinking thouroughly about it.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Fabio Porcedda <fabio.porcedda@gmail.com>
> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
> ---
>  package/pkg-generic.mk | 3 +++
>  1 file changed, 3 insertions(+)

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 4/4] legal-info: rename legal-warning-pkg-savednothing helper

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Sun, 22 Jun 2014 14:41:15 +0200, Yann E. MORIN wrote:
> From: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> This helper was called when none of the sources or license
> files were saved.
> 
> Now we handle license files separately from the sources,
> this is no longer the case: they are only called when the
> sources are not saved.
> 
> Rename the handler and change the warning message accordingly.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Fabio Porcedda <fabio.porcedda@gmail.com>

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com