[Buildroot] [PATCH 0/16 v3] legal-info improvements and completeness (branch yem/legal-2)

[Buildroot] [PATCH 0/16 v3] legal-info improvements and completeness (branch yem/legal-2)

[Yann E. MORIN]

Hello All!

This series brings improvements to the legal-info infrastructure, so
that we provide the most complete and correct content in the output of
legal-info.

Currently, our legal-info ouput is missing two types of files that might
be important to have:
  - patches
  - extra downloads


This series is split in 6 consecutive parts, each depending on the
previous ones:

    - patches 1-2 fix the legal-info header to account for the fact that
      the sources for some external toolchains may get installed in the
      legal-info output; hashes for those are added to the .hash file;

    - patches 3-6 make sure legal-info will work in off-line mode;

    - pathces 7-9 reorganise the legal-info directory structure to
      accomodate for the fact that more than one source archive/file may
      be saved for each package;

    - patches 10-13 actually save the patches and extra downloads in the
      legal-info output;

    - patch 14 adds a list of hashes for all files in the legal-info
      output;

    - patches 15-16 add the possibility to ignore packages from the
      legal-info output; this is then used to ignore virtual packages.


Why save patches?
-----------------

So far, we've shuffled the patches under the rag, assuming the user
would provide the Buildroot source tree with the compliance delivery, so
that our bundled patches would automatically be included.

However, that's not enough, as not all patches may be in the Buildroot
source tree. That's the case for at least two types of patches:
  - patches that are downloaded,
  - patches from a global patch directory.

In either case, those patches must be provided in the output of
legal-info, because they are not part of Buildroot, so distributing
Buidlroot would not be enough.

Patches that are referenced from Buidlroot (like patches retrieved at
download time from a http://-or-such scheme to a publicly-reachable
location) would probably be OK-ish, even if not to the letter of the
compliance requirements.

That's not so much the case for patches from a global patch dir, since
those would be completely ignored and usually unreachable from a
recipient of the compliance delivery.

So we must save those two types of patches in the output of legal-info.
Because it would be a bit silly to only save the non-bundled patches, we
just save all of them, whether bundled in Buildroot, downloaded or from
a global patch dir alike.


Why save extra downloads?
-------------------------

Some packages are using extra-downloads to complement the content of the
main archive. That's the case for Perl, for which the cross-compilation
"enabler" is downloaded as a secondary archive via extra downloads. The
Blackfin external toolchains also use extra downloads to download a
secondary archive with the sysroot.

Even though the Blackfin sysroot archive is not really a source, we
still need to provide it along with the main archive, otherwise it's
completely impossible to compile with just the "main" toolchain.

As for the Perl case, however, we're "only" downloading a buildsystem
infrastructure (AFAIU), but without it, it is completely impossible to
cross-compile Perl at all.

So, in both cases, we also need to save the extra downloads.


Changes v2 -> v3:
  - re-order variables in their own patch  (Arnout)
  - update legal-info header about the patches  (Luca)
  - add hashes for external toolchains sources  (Luca, Arnout)
  - misc and typoes  (Arnout, Luca)
  - enhance the hardlink-or-copy macro

Changes v1 -> v2:
  - keep only the core legal-info patches, drop the gcc/binutils/gdb
    changes (they'll be reworked later, let's focus on the important and
    easier parts first)
  - drop the tristate REDISTRIBUTE, introduce another boolean
    _LEGAL_IGNORE  (Thomas, Peter, Luca)
  - drop the post-legal-info Perl hook, it's no longer needed thanks to
    saving extra downloads  (Thomas, Luca)
  - compute the rawname-version tuple only once, instead of five times
    (Luca)
  - reorder patches  (Luca)
  - slight commit log rephrasing and corrections  (Luca)


Regards,
Yann E. MORIN.


The following changes since commit 5929b1bdbaafa274d2183a28bea07bf53003065c:

  linux-headers: bump 3.12.x series (2016-01-28 17:47:44 +0100)

are available in the git repository at:

  git://git.busybox.net/~ymorin/git/buildroot yem/legal-2

for you to fetch changes up to bee63e49a6e51196ca55bab2698c8e69d38e2512:

  core/pkg-virtual: ignore from legal-info output (2016-01-28 18:53:52 +0100)

----------------------------------------------------------------
Yann E. MORIN (16):
      core/legal-info: update the legal-info report header
      toolchain/external: add hashes for actual sources
      core/pkg-utils: add macro to hardlink-or-copy
      core/legal-info: use the macro to install source archives
      core/pkg-generic: reorder variables definitions for legal-info
      core/legal-info: ensure legal-info works in off-line mode
      core/pkg-generic: add variable to store the package rawname-version
      core/legal-info: install source archives in their own sub-dir
      core/legal-info: add package version to license directory
      core/apply-patches: store full path of applied patches
      core/legal-info: also save patches
      core/legal-info: renumber saved patches
      core/legal-info: also save extra downloads
      core/legal-info: generate a hash of all saved files
      core/legal-info: allow ignoring packages from the legal-info
      core/pkg-virtual: ignore from legal-info output

 Makefile                                           |  8 +-
 docs/manual/adding-packages-generic.txt            |  9 +++
 docs/manual/adding-packages-virtual.txt            |  3 +
 package/pkg-generic.mk                             | 89 +++++++++++++++++-----
 package/pkg-utils.mk                               | 25 ++++++
 package/pkg-virtual.mk                             |  2 +
 support/legal-info/README.header                   | 13 ++--
 support/scripts/apply-patches.sh                   | 11 ++-
 .../toolchain-external/toolchain-external.hash     |  5 ++
 9 files changed, 134 insertions(+), 31 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 01/16 v3] core/legal-info: update the legal-info report header

[Yann E. MORIN]

In some cases, the toolchain sources are now recovered and available in
the legal-info output. So, adapt the header to use conditional instead
of an definitive negation.

Also update the part about saving the sources: it's not the license list
that defines whether sources are installed, but rather whether the
package is redistributable or not.

Update the header accordingly.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

---
Changes v2 -> v3:
  - typoes  (Arnout)

Changes v1 -> v2:
  - slight rephrasing  (Luca)
  - move early in the series  (Luca)
---
 support/legal-info/README.header | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/support/legal-info/README.header b/support/legal-info/README.header
index 3321adb..d07c45d 100644
--- a/support/legal-info/README.header
+++ b/support/legal-info/README.header
@@ -12,12 +12,12 @@ This material is composed of the following items.
  * The Buildroot configuration file; this has been saved in buildroot.config.
  * The toolchain (cross-compiler and related tools) used to generate all the
    compiled programs.
-   Note: this has not been saved due to technical limitations, you must
-   collect it manually.
+   Note: this may have not been saved due to technical limitations, you may
+   need to collect it manually.
  * The source code for all packages; this has been saved in the sources/
-   subdirectory (except for the proprietary packages, which have not been
-   saved); patches applied to some packages by Buildroot are included in the
-   Buildroot sources and were not duplicated in the sources/ subdirectory.
+   subdirectory (except for the non-redistributable packages, which have not
+   been saved); patches applied to some packages by Buildroot are included in
+   the Buildroot sources and were not duplicated in the sources/ subdirectory.
  * A manifest file listing the configured packages and related information.
  * The license text of the packages; they have been saved in the licenses/
    subdirectory.
-- 
1.9.1

[Buildroot] [PATCH 01/16 v3] core/legal-info: update the legal-info report header

[Thomas Petazzoni]

Dear Yann E. MORIN,

On Thu, 28 Jan 2016 19:14:59 +0100, Yann E. MORIN wrote:
> In some cases, the toolchain sources are now recovered and available in
> the legal-info output. So, adapt the header to use conditional instead
> of an definitive negation.
> 
> Also update the part about saving the sources: it's not the license list
> that defines whether sources are installed, but rather whether the
> package is redistributable or not.
> 
> Update the header accordingly.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
> ---
> Changes v2 -> v3:
>   - typoes  (Arnout)

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 02/16 v3] toolchain/external: add hashes for actual sources

[Yann E. MORIN]

As we currently download the actual sources as part of saving the
legal-info, we do not check the hashes of those downloads.

That's because, during legal-info, there is not package involved, and
thus there's no path to an actual .hash file.

However, this precludes legal-info from working in off-line mode. A
subsequent patch will make it possible to do so, and actual sources will
be downloaded as another classical package download.

This will have two consequences:

  - first, we will be able to add hashes for actual sources, so we can
    ensure their integrity,

  - second, and as a direct consequence of the above, when a .hash file
    is present, it would have to list all the hashes for that package,
    or that would be treated as an error.

Currently, the only package that falls in this case is the external-
toolchain, for which we have means to retrieve the sources for some of
the toolchains.

So we just add hashes for those actual external-toolchain sources we may
have to download.

Those hashes are not used for now, but they'll come into play a few
patches down.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 toolchain/toolchain-external/toolchain-external.hash | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/toolchain/toolchain-external/toolchain-external.hash b/toolchain/toolchain-external/toolchain-external.hash
index dd7073f..22bbe51 100644
--- a/toolchain/toolchain-external/toolchain-external.hash
+++ b/toolchain/toolchain-external/toolchain-external.hash
@@ -12,6 +12,7 @@ sha256 c65b1b4b918d5185349d62a3b7bf43b4b21e1175c52598ec047ca56b3f11d857  blackfi
 # Mentor's Sourcery CodeBench Lite toolchains
 # ARM
 sha256 39ee0e789034334ecc89af94e838e3a4815400ac5ff980f808f466b04778532e  arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
+sha256 e16a5b1e41d7ff1e74161f9405182001bc8d1360d89564e73911032e6966cc0d  arm-2014.05-29-arm-none-linux-gnueabi.src.tar.bz2
 # NiosII
 sha256 cc47745dc1264fcb8fb98fb1315ab772ab98691396021c455229b58abaf887f5  sourceryg++-2015.11-27-nios2-linux-gnu-i686-pc-linux-gnu.tar.bz2
 # PowerPC
@@ -29,13 +30,17 @@ sha256 8ea78c5988b2bb507534f1ad46aa46659f66b39d55f2fc40e163a90b4195e70f  aarch64
 # ARM toolchains from Texas Instrument's Arago project
 sha256 f2febf3b3c565536461ad4405f1bcb835d75a6afb2a8bec958a1248cb4b81fc7  arago-2011.09-armv7a-linux-gnueabi-sdk.tar.bz2
 sha256 254af7d02eb3bcc8345c78e131700bc995d65b68232caaed21150a5fd1456070  arago-2011.09-armv5te-linux-gnueabi-sdk.tar.bz2
+sha256 25fbf0513ad7322b15cbaae964cafadcbb4c939f2708f57f40b8f9f2d601122b  arago-toolchain-2011.09-sources.tar.bz2
 
 # ARM and Aarch64 toolchains from Linaro
 sha256 0cffac0caea0eb3c8bdddfa14be011ce366680f40aeddbefc7cf23cb6d4f1891  gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
+sha256 eafeb3a5247e9ce31aa35d812e296fba5d5f1443e106d9bef9e38d3ee3ade006  gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_src.tar.bz2
 sha256 34812c5d0556db86259ac6eb3f8bcf4ce8eca3fa8d7180875958492a42e9853f  gcc-linaro-5.1-2015.08-x86_64_arm-linux-gnueabihf.tar.xz
 sha256 4bc9d86390f8fa67a693ba4768ba5b12faaf7dd37c706c05ccd9321e765226e4  gcc-linaro-armeb-linux-gnueabihf-4.9-2014.09_linux.tar.xz
+sha256 bf5d3111dad5aa9aef0e955875fb7fc9e918cb24519af7014dd67a9e581a49b1  gcc-linaro-armeb-linux-gnueabihf-4.9-2014.09_src.tar.bz2
 sha256 24b86799a6c64380c740bf31a700b46e854fc0a821da2341e9868f0196c864de  gcc-linaro-5.1-2015.08-x86_64_armeb-linux-gnueabihf.tar.xz
 sha256 3954f496ab01de67241109e82abfaa9b7625fdab4f05e79e7902e9814a07b832  gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
+sha256 a7b8f842fdc9d9be22ca4e0c999429780fc6f16ea798b032421b5ec0cfa53b3e  gcc-linaro-aarch64-linux-gnu-4.9-2014.09_src.tar.bz2
 sha256 b9137008744d9009877f662dbac7481d673cdcb1798e727e325a37c98a0f63da  gcc-linaro-5.1-2015.08-x86_64_aarch64-linux-gnu.tar.xz
 
 # Codescape toolchains from Imagination Technologies
-- 
1.9.1

[Buildroot] [PATCH 02/16 v3] toolchain/external: add hashes for actual sources

[Arnout Vandecappelle]

On 28-01-16 19:15, Yann E. MORIN wrote:
> As we currently download the actual sources as part of saving the
> legal-info, we do not check the hashes of those downloads.
> 
> That's because, during legal-info, there is not package involved, and
> thus there's no path to an actual .hash file.
> 
> However, this precludes legal-info from working in off-line mode. A
> subsequent patch will make it possible to do so, and actual sources will
> be downloaded as another classical package download.
> 
> This will have two consequences:
> 
>   - first, we will be able to add hashes for actual sources, so we can
>     ensure their integrity,
> 
>   - second, and as a direct consequence of the above, when a .hash file
>     is present, it would have to list all the hashes for that package,
>     or that would be treated as an error.
> 
> Currently, the only package that falls in this case is the external-
> toolchain, for which we have means to retrieve the sources for some of
> the toolchains.
> 
> So we just add hashes for those actual external-toolchain sources we may
> have to download.
> 
> Those hashes are not used for now, but they'll come into play a few
> patches down.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  toolchain/toolchain-external/toolchain-external.hash | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/toolchain/toolchain-external/toolchain-external.hash b/toolchain/toolchain-external/toolchain-external.hash
> index dd7073f..22bbe51 100644
> --- a/toolchain/toolchain-external/toolchain-external.hash
> +++ b/toolchain/toolchain-external/toolchain-external.hash
> @@ -12,6 +12,7 @@ sha256 c65b1b4b918d5185349d62a3b7bf43b4b21e1175c52598ec047ca56b3f11d857  blackfi
>  # Mentor's Sourcery CodeBench Lite toolchains
>  # ARM
>  sha256 39ee0e789034334ecc89af94e838e3a4815400ac5ff980f808f466b04778532e  arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
> +sha256 e16a5b1e41d7ff1e74161f9405182001bc8d1360d89564e73911032e6966cc0d  arm-2014.05-29-arm-none-linux-gnueabi.src.tar.bz2

 Why only this one, and not the six other Sourcery toolchains?

 Regards,
 Arnout

>  # NiosII
>  sha256 cc47745dc1264fcb8fb98fb1315ab772ab98691396021c455229b58abaf887f5  sourceryg++-2015.11-27-nios2-linux-gnu-i686-pc-linux-gnu.tar.bz2
>  # PowerPC
> @@ -29,13 +30,17 @@ sha256 8ea78c5988b2bb507534f1ad46aa46659f66b39d55f2fc40e163a90b4195e70f  aarch64
>  # ARM toolchains from Texas Instrument's Arago project
>  sha256 f2febf3b3c565536461ad4405f1bcb835d75a6afb2a8bec958a1248cb4b81fc7  arago-2011.09-armv7a-linux-gnueabi-sdk.tar.bz2
>  sha256 254af7d02eb3bcc8345c78e131700bc995d65b68232caaed21150a5fd1456070  arago-2011.09-armv5te-linux-gnueabi-sdk.tar.bz2
> +sha256 25fbf0513ad7322b15cbaae964cafadcbb4c939f2708f57f40b8f9f2d601122b  arago-toolchain-2011.09-sources.tar.bz2
>  
>  # ARM and Aarch64 toolchains from Linaro
>  sha256 0cffac0caea0eb3c8bdddfa14be011ce366680f40aeddbefc7cf23cb6d4f1891  gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
> +sha256 eafeb3a5247e9ce31aa35d812e296fba5d5f1443e106d9bef9e38d3ee3ade006  gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_src.tar.bz2
>  sha256 34812c5d0556db86259ac6eb3f8bcf4ce8eca3fa8d7180875958492a42e9853f  gcc-linaro-5.1-2015.08-x86_64_arm-linux-gnueabihf.tar.xz
>  sha256 4bc9d86390f8fa67a693ba4768ba5b12faaf7dd37c706c05ccd9321e765226e4  gcc-linaro-armeb-linux-gnueabihf-4.9-2014.09_linux.tar.xz
> +sha256 bf5d3111dad5aa9aef0e955875fb7fc9e918cb24519af7014dd67a9e581a49b1  gcc-linaro-armeb-linux-gnueabihf-4.9-2014.09_src.tar.bz2
>  sha256 24b86799a6c64380c740bf31a700b46e854fc0a821da2341e9868f0196c864de  gcc-linaro-5.1-2015.08-x86_64_armeb-linux-gnueabihf.tar.xz
>  sha256 3954f496ab01de67241109e82abfaa9b7625fdab4f05e79e7902e9814a07b832  gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
> +sha256 a7b8f842fdc9d9be22ca4e0c999429780fc6f16ea798b032421b5ec0cfa53b3e  gcc-linaro-aarch64-linux-gnu-4.9-2014.09_src.tar.bz2
>  sha256 b9137008744d9009877f662dbac7481d673cdcb1798e727e325a37c98a0f63da  gcc-linaro-5.1-2015.08-x86_64_aarch64-linux-gnu.tar.xz
>  
>  # Codescape toolchains from Imagination Technologies
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 02/16 v3] toolchain/external: add hashes for actual sources

[Yann E. MORIN]

Arnout, All,

On 2016-01-31 23:30 +0100, Arnout Vandecappelle spake thusly:
> On 28-01-16 19:15, Yann E. MORIN wrote:
> > As we currently download the actual sources as part of saving the
> > legal-info, we do not check the hashes of those downloads.
[--SNIP--]
> > So we just add hashes for those actual external-toolchain sources we may
> > have to download.
[--SNIP--]
> > diff --git a/toolchain/toolchain-external/toolchain-external.hash b/toolchain/toolchain-external/toolchain-external.hash
> > index dd7073f..22bbe51 100644
> > --- a/toolchain/toolchain-external/toolchain-external.hash
> > +++ b/toolchain/toolchain-external/toolchain-external.hash
> > @@ -12,6 +12,7 @@ sha256 c65b1b4b918d5185349d62a3b7bf43b4b21e1175c52598ec047ca56b3f11d857  blackfi
> >  # Mentor's Sourcery CodeBench Lite toolchains
> >  # ARM
> >  sha256 39ee0e789034334ecc89af94e838e3a4815400ac5ff980f808f466b04778532e  arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
> > +sha256 e16a5b1e41d7ff1e74161f9405182001bc8d1360d89564e73911032e6966cc0d  arm-2014.05-29-arm-none-linux-gnueabi.src.tar.bz2
> 
>  Why only this one, and not the six other Sourcery toolchains?

Because I did not add them. ;-)

Seriously, I just forgot. And it takes quite some time to get them all.

I've added hashes for all the CS toolchains now, but maybe there are
other toolchains for which we're still missing the hashes for the source
tarball. Maybe we can just add them whenever someone stumbles on it?

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 02/16 v3] toolchain/external: add hashes for actual sources

[Arnout Vandecappelle]

On 01-02-16 14:54, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2016-01-31 23:30 +0100, Arnout Vandecappelle spake thusly:
>> On 28-01-16 19:15, Yann E. MORIN wrote:
>>> As we currently download the actual sources as part of saving the
>>> legal-info, we do not check the hashes of those downloads.
> [--SNIP--]
>>> So we just add hashes for those actual external-toolchain sources we may
>>> have to download.
> [--SNIP--]
>>> diff --git a/toolchain/toolchain-external/toolchain-external.hash b/toolchain/toolchain-external/toolchain-external.hash
>>> index dd7073f..22bbe51 100644
>>> --- a/toolchain/toolchain-external/toolchain-external.hash
>>> +++ b/toolchain/toolchain-external/toolchain-external.hash
>>> @@ -12,6 +12,7 @@ sha256 c65b1b4b918d5185349d62a3b7bf43b4b21e1175c52598ec047ca56b3f11d857  blackfi
>>>  # Mentor's Sourcery CodeBench Lite toolchains
>>>  # ARM
>>>  sha256 39ee0e789034334ecc89af94e838e3a4815400ac5ff980f808f466b04778532e  arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
>>> +sha256 e16a5b1e41d7ff1e74161f9405182001bc8d1360d89564e73911032e6966cc0d  arm-2014.05-29-arm-none-linux-gnueabi.src.tar.bz2
>>
>>  Why only this one, and not the six other Sourcery toolchains?
> 
> Because I did not add them. ;-)
> 
> Seriously, I just forgot. And it takes quite some time to get them all.
> 
> I've added hashes for all the CS toolchains now, but maybe there are
> other toolchains for which we're still missing the hashes for the source
> tarball. Maybe we can just add them whenever someone stumbles on it?

 I checked, and we only have source for Sourcery, Arago and Linaro toolchains.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 03/16 v3] core/pkg-utils: add macro to hardlink-or-copy

[Yann E. MORIN]

This macro will try to copy a source file into a destination fdirectory,
by first atempting a hard-link, and falling back to a plain copy.

In some situations, it will be necessary that the destination file is
named differently than the source (e.g. due to a re-numbering), so if a
third argument is specified, it is treated as the basename of the
destination file.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>

---
Changes v2 -> v3;
  - use "ln" instead of "cp -l"
  - accept third argument, as the basename of the destination file
  - drop reviewed-by and tested-by tags given in v2, due to the above
    two changes

Changes RFC -> v1:
  - move to pkg-utils  (Luca)
---
 package/pkg-utils.mk | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
index 44bd2c9..6479cd8 100644
--- a/package/pkg-utils.mk
+++ b/package/pkg-utils.mk
@@ -113,6 +113,31 @@ $$(error Package error: use $(2) instead of $(1). Please fix your .mk file)
 endif
 endef
 
+################################################################################
+# hardlink-copy -- hardlink source and destination if possible, otherwise
+# do a simple copy
+#
+# argument 1 is the source *file*
+# argument 2 is the destination *directory*
+# argument 3 is the basename of the destination file (optional, defaults to
+#            the basename of the source file.
+#
+# examples:
+#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/)
+#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/,new-name)
+#
+# Note: we make that a single command, so se can:
+#  - use '$(Q)' in front of it and properly silence the whole macro,
+#  - use '|| exit 1' after it, so we can exit on error in compound commands.
+################################################################################
+define hardlink-copy
+	{ mkdir -p $(2) && \
+		{ ln $(1) $(2)$(if $(3),/$(strip $(3))) 2>/dev/null || \
+		  cp $(1) $(2)$(if $(3),/$(strip $(3))); \
+		} \
+	}
+endef
+
 #
 # legal-info helper functions
 #
-- 
1.9.1

[Buildroot] [PATCH 03/16 v3] core/pkg-utils: add macro to hardlink-or-copy

[Luca Ceresoli]

Dear Yann,

Yann E. MORIN wrote:
> This macro will try to copy a source file into a destination fdirectory,

fdirectory -> directory

> by first atempting a hard-link, and falling back to a plain copy.

atempting a -> attempting to

>
> In some situations, it will be necessary that the destination file is
> named differently than the source (e.g. due to a re-numbering), so if a
> third argument is specified, it is treated as the basename of the
> destination file.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
>
> ---
> Changes v2 -> v3;
>    - use "ln" instead of "cp -l"
>    - accept third argument, as the basename of the destination file
>    - drop reviewed-by and tested-by tags given in v2, due to the above
>      two changes
>
> Changes RFC -> v1:
>    - move to pkg-utils  (Luca)
> ---
>   package/pkg-utils.mk | 25 +++++++++++++++++++++++++
>   1 file changed, 25 insertions(+)
>
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 44bd2c9..6479cd8 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -113,6 +113,31 @@ $$(error Package error: use $(2) instead of $(1). Please fix your .mk file)
>   endif
>   endef
>
> +################################################################################
> +# hardlink-copy -- hardlink source and destination if possible, otherwise
> +# do a simple copy
> +#
> +# argument 1 is the source *file*
> +# argument 2 is the destination *directory*
> +# argument 3 is the basename of the destination file (optional, defaults to
> +#            the basename of the source file.
> +#
> +# examples:
> +#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/)
> +#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/,new-name)
> +#
> +# Note: we make that a single command, so se can:

se can -> we can

> +#  - use '$(Q)' in front of it and properly silence the whole macro,
> +#  - use '|| exit 1' after it, so we can exit on error in compound commands.
> +################################################################################
> +define hardlink-copy
> +	{ mkdir -p $(2) && \
> +		{ ln $(1) $(2)$(if $(3),/$(strip $(3))) 2>/dev/null || \
> +		  cp $(1) $(2)$(if $(3),/$(strip $(3))); \
> +		} \
> +	}
> +endef

With typos fixed:
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>

[Ran 'make legal-info' with output dir on {the same,another} fs]
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 03/16 v3] core/pkg-utils: add macro to hardlink-or-copy

[Yann E. MORIN]

Luca, All,

On 2016-01-30 12:29 +0100, Luca Ceresoli spake thusly:
> Yann E. MORIN wrote:
> >This macro will try to copy a source file into a destination fdirectory,
> 
> fdirectory -> directory
> 
> >by first atempting a hard-link, and falling back to a plain copy.
> 
> atempting a -> attempting to

No matter how many times I'll try to proof-read my prose, I'll always
leave typoes lying here and there... Seems like it is easier to
proff-read others'...

[--SNIP--]
> With typos fixed:

Done.

> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
> 
> [Ran 'make legal-info' with output dir on {the same,another} fs]
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 03/16 v3] core/pkg-utils: add macro to hardlink-or-copy

[Arnout Vandecappelle]

On 28-01-16 19:15, Yann E. MORIN wrote:
> This macro will try to copy a source file into a destination fdirectory,
> by first atempting a hard-link, and falling back to a plain copy.
> 
> In some situations, it will be necessary that the destination file is
> named differently than the source (e.g. due to a re-numbering), so if a
> third argument is specified, it is treated as the basename of the
> destination file.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> 
> ---
> Changes v2 -> v3;
>   - use "ln" instead of "cp -l"
>   - accept third argument, as the basename of the destination file
>   - drop reviewed-by and tested-by tags given in v2, due to the above
>     two changes
> 
> Changes RFC -> v1:
>   - move to pkg-utils  (Luca)
> ---
>  package/pkg-utils.mk | 25 +++++++++++++++++++++++++
>  1 file changed, 25 insertions(+)
> 
> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
> index 44bd2c9..6479cd8 100644
> --- a/package/pkg-utils.mk
> +++ b/package/pkg-utils.mk
> @@ -113,6 +113,31 @@ $$(error Package error: use $(2) instead of $(1). Please fix your .mk file)
>  endif
>  endef
>  
> +################################################################################
> +# hardlink-copy -- hardlink source and destination if possible, otherwise
> +# do a simple copy
> +#
> +# argument 1 is the source *file*
> +# argument 2 is the destination *directory*
> +# argument 3 is the basename of the destination file (optional, defaults to
> +#            the basename of the source file.
> +#
> +# examples:
> +#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/)
> +#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/,new-name)
> +#
> +# Note: we make that a single command, so se can:
> +#  - use '$(Q)' in front of it and properly silence the whole macro,
> +#  - use '|| exit 1' after it, so we can exit on error in compound commands.
> +################################################################################
> +define hardlink-copy
> +	{ mkdir -p $(2) && \
> +		{ ln $(1) $(2)$(if $(3),/$(strip $(3))) 2>/dev/null || \

 Shouldn't this be ln -f (and maybe cp -f) to make sure it behaves the same when
you run it a second time?


 Regards,
 Arnout

> +		  cp $(1) $(2)$(if $(3),/$(strip $(3))); \
> +		} \
> +	}
> +endef
> +
>  #
>  # legal-info helper functions
>  #
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 03/16 v3] core/pkg-utils: add macro to hardlink-or-copy

[Luca Ceresoli]

Hi,

Arnout Vandecappelle wrote:
> On 28-01-16 19:15, Yann E. MORIN wrote:
>> This macro will try to copy a source file into a destination fdirectory,
>> by first atempting a hard-link, and falling back to a plain copy.
>>
>> In some situations, it will be necessary that the destination file is
>> named differently than the source (e.g. due to a re-numbering), so if a
>> third argument is specified, it is treated as the basename of the
>> destination file.
>>
>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>> Cc: Luca Ceresoli <luca@lucaceresoli.net>
>>
>> ---
>> Changes v2 -> v3;
>>    - use "ln" instead of "cp -l"
>>    - accept third argument, as the basename of the destination file
>>    - drop reviewed-by and tested-by tags given in v2, due to the above
>>      two changes
>>
>> Changes RFC -> v1:
>>    - move to pkg-utils  (Luca)
>> ---
>>   package/pkg-utils.mk | 25 +++++++++++++++++++++++++
>>   1 file changed, 25 insertions(+)
>>
>> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
>> index 44bd2c9..6479cd8 100644
>> --- a/package/pkg-utils.mk
>> +++ b/package/pkg-utils.mk
>> @@ -113,6 +113,31 @@ $$(error Package error: use $(2) instead of $(1). Please fix your .mk file)
>>   endif
>>   endef
>>
>> +################################################################################
>> +# hardlink-copy -- hardlink source and destination if possible, otherwise
>> +# do a simple copy
>> +#
>> +# argument 1 is the source *file*
>> +# argument 2 is the destination *directory*
>> +# argument 3 is the basename of the destination file (optional, defaults to
>> +#            the basename of the source file.
>> +#
>> +# examples:
>> +#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/)
>> +#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/,new-name)
>> +#
>> +# Note: we make that a single command, so se can:
>> +#  - use '$(Q)' in front of it and properly silence the whole macro,
>> +#  - use '|| exit 1' after it, so we can exit on error in compound commands.
>> +################################################################################
>> +define hardlink-copy
>> +	{ mkdir -p $(2) && \
>> +		{ ln $(1) $(2)$(if $(3),/$(strip $(3))) 2>/dev/null || \
>
>   Shouldn't this be ln -f (and maybe cp -f) to make sure it behaves the same when
> you run it a second time?

Indeed it's true. Not an issue for 'make legal-info' since it wipes the
output dir before copying file in it, but the macro is generic so it
has to handle this case.

This definitely applies to ln. cp normally doesn't need it, but it does
not hurt.

-- 
Luca

[Buildroot] [PATCH 03/16 v3] core/pkg-utils: add macro to hardlink-or-copy

[Arnout Vandecappelle]

On 01-02-16 11:13, Luca Ceresoli wrote:
> Hi,
> 
> Arnout Vandecappelle wrote:
>> On 28-01-16 19:15, Yann E. MORIN wrote:
>>> This macro will try to copy a source file into a destination fdirectory,
>>> by first atempting a hard-link, and falling back to a plain copy.
>>>
>>> In some situations, it will be necessary that the destination file is
>>> named differently than the source (e.g. due to a re-numbering), so if a
>>> third argument is specified, it is treated as the basename of the
>>> destination file.
>>>
>>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>> Cc: Luca Ceresoli <luca@lucaceresoli.net>
>>>
>>> ---
>>> Changes v2 -> v3;
>>>    - use "ln" instead of "cp -l"
>>>    - accept third argument, as the basename of the destination file
>>>    - drop reviewed-by and tested-by tags given in v2, due to the above
>>>      two changes
>>>
>>> Changes RFC -> v1:
>>>    - move to pkg-utils  (Luca)
>>> ---
>>>   package/pkg-utils.mk | 25 +++++++++++++++++++++++++
>>>   1 file changed, 25 insertions(+)
>>>
>>> diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
>>> index 44bd2c9..6479cd8 100644
>>> --- a/package/pkg-utils.mk
>>> +++ b/package/pkg-utils.mk
>>> @@ -113,6 +113,31 @@ $$(error Package error: use $(2) instead of $(1). Please
>>> fix your .mk file)
>>>   endif
>>>   endef
>>>
>>> +################################################################################
>>>
>>> +# hardlink-copy -- hardlink source and destination if possible, otherwise
>>> +# do a simple copy
>>> +#
>>> +# argument 1 is the source *file*
>>> +# argument 2 is the destination *directory*
>>> +# argument 3 is the basename of the destination file (optional, defaults to
>>> +#            the basename of the source file.
>>> +#
>>> +# examples:
>>> +#   $(call hardlink-copy,/path/to/source/file,/path/to/destination/dir/)
>>> +#   $(call
>>> hardlink-copy,/path/to/source/file,/path/to/destination/dir/,new-name)
>>> +#
>>> +# Note: we make that a single command, so se can:
>>> +#  - use '$(Q)' in front of it and properly silence the whole macro,
>>> +#  - use '|| exit 1' after it, so we can exit on error in compound commands.
>>> +################################################################################
>>>
>>> +define hardlink-copy
>>> +    { mkdir -p $(2) && \
>>> +        { ln $(1) $(2)$(if $(3),/$(strip $(3))) 2>/dev/null || \
>>
>>   Shouldn't this be ln -f (and maybe cp -f) to make sure it behaves the same when
>> you run it a second time?
> 
> Indeed it's true. Not an issue for 'make legal-info' since it wipes the
> output dir before copying file in it, but the macro is generic so it
> has to handle this case.
> 
> This definitely applies to ln. cp normally doesn't need it, but it does
> not hurt.

 With -f, cp will indeed only unlink if it can't write; if the target is a
symlink, it will still not remove the symlink but instead it will overwrite the
file pointed to. So an explicit rm -f is probably better.

 Regards,
 Arnout

> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 04/16 v3] core/legal-info: use the macro to install source archives

[Yann E. MORIN]

.. and use $(Q) instead of a hard-coded @.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

---
Change v2 -> v3;
  - comment the @ -> $(Q) change  (Arnout)
---
 package/pkg-generic.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1e024d3..893ee07 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -804,9 +804,10 @@ ifeq ($$($(2)_REDISTRIBUTE),YES)
 ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE))
 	$$(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL))
 endif
-# Copy the source tarball (just hardlink if possible)
-	@cp -l $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) 2>/dev/null || \
-	    cp $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
+# Copy the source tarball
+	$$(Q)$$(call hardlink-copy,\
+		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
+		     $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))))
 endif # redistribute
 
 endif # other packages
-- 
1.9.1

[Buildroot] [PATCH 05/16 v3] core/pkg-generic: reorder variables definitions for legal-info

[Yann E. MORIN]

Move the declarations of _ACTUAL_SOURCE and _ACTUAL_SITE earlier, so
that they are close to where _SOURCE and _SITE are handled.

This looks so far like a purely cosmetic change, but makes more sense
with the follow-up patch, where we'll need them earlier in the file.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>

---
Changes v2 -> v3:
  - move that to its own patch  (Arnout)
---
 package/pkg-generic.mk | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 893ee07..5958029 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -427,6 +427,14 @@ ifndef $(2)_SOURCE
  endif
 endif
 
+# If FOO_ACTUAL_SOURCE_TARBALL is explicitly defined, it means FOO_SOURCE is
+# indeed a binary (e.g. external toolchain) and FOO_ACTUAL_SOURCE_TARBALL/_SITE
+# point to the actual sources tarball. Use the actual sources for legal-info.
+# For most packages the FOO_SITE/FOO_SOURCE pair points to real source code,
+# so these are the defaults for FOO_ACTUAL_*.
+$(2)_ACTUAL_SOURCE_TARBALL ?= $$($(2)_SOURCE)
+$(2)_ACTUAL_SOURCE_SITE    ?= $$(call qstrip,$$($(2)_SITE))
+
 ifndef $(2)_PATCH
  ifdef $(3)_PATCH
   $(2)_PATCH = $$($(3)_PATCH)
@@ -763,14 +771,6 @@ endif
 endif
 endif
 
-# If FOO_ACTUAL_SOURCE_TARBALL is explicitly defined, it means FOO_SOURCE is
-# indeed a binary (e.g. external toolchain) and FOO_ACTUAL_SOURCE_TARBALL/_SITE
-# point to the actual sources tarball. Use the actual sources for legal-info.
-# For most packages the FOO_SITE/FOO_SOURCE pair points to real source code,
-# so these are the defaults for FOO_ACTUAL_*.
-$(2)_ACTUAL_SOURCE_TARBALL ?= $$($(2)_SOURCE)
-$(2)_ACTUAL_SOURCE_SITE    ?= $$(call qstrip,$$($(2)_SITE))
-
 # legal-info: produce legally relevant info.
 $(1)-legal-info:
 # Packages without a source are assumed to be part of Buildroot, skip them.
-- 
1.9.1

[Buildroot] [PATCH 05/16 v3] core/pkg-generic: reorder variables definitions for legal-info

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Move the declarations of _ACTUAL_SOURCE and _ACTUAL_SITE earlier, so
> that they are close to where _SOURCE and _SITE are handled.
>
> This looks so far like a purely cosmetic change, but makes more sense
> with the follow-up patch, where we'll need them earlier in the file.

Not sure in which patch you need this... but it's nice to have
$(2)_SOURCE and $(2)_ACTUAL_SOURCE_* defined at the same place, thus:

Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 05/16 v3] core/pkg-generic: reorder variables definitions for legal-info

[Yann E. MORIN]

Luca, All,

On 2016-01-30 16:02 +0100, Luca Ceresoli spake thusly:
> Yann E. MORIN wrote:
> >Move the declarations of _ACTUAL_SOURCE and _ACTUAL_SITE earlier, so
> >that they are close to where _SOURCE and _SITE are handled.
> >
> >This looks so far like a purely cosmetic change, but makes more sense
> >with the follow-up patch, where we'll need them earlier in the file.
> 
> Not sure in which patch you need this...

In patch 6, since we use _ACTUAL_SOURCE_TARBALL on line 725 (in an
ifeq-block) while it was declared on line 771.

> but it's nice to have
> $(2)_SOURCE and $(2)_ACTUAL_SOURCE_* defined at the same place, thus:
> 
> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 05/16 v3] core/pkg-generic: reorder variables definitions for legal-info

[Arnout Vandecappelle]

On 28-01-16 19:15, Yann E. MORIN wrote:
> Move the declarations of _ACTUAL_SOURCE and _ACTUAL_SITE earlier, so
> that they are close to where _SOURCE and _SITE are handled.
> 
> This looks so far like a purely cosmetic change, but makes more sense
> with the follow-up patch, where we'll need them earlier in the file.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


 Regards,
 Arnout

> 
> ---
> Changes v2 -> v3:
>   - move that to its own patch  (Arnout)
> ---
>  package/pkg-generic.mk | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 893ee07..5958029 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -427,6 +427,14 @@ ifndef $(2)_SOURCE
>   endif
>  endif
>  
> +# If FOO_ACTUAL_SOURCE_TARBALL is explicitly defined, it means FOO_SOURCE is
> +# indeed a binary (e.g. external toolchain) and FOO_ACTUAL_SOURCE_TARBALL/_SITE
> +# point to the actual sources tarball. Use the actual sources for legal-info.
> +# For most packages the FOO_SITE/FOO_SOURCE pair points to real source code,
> +# so these are the defaults for FOO_ACTUAL_*.
> +$(2)_ACTUAL_SOURCE_TARBALL ?= $$($(2)_SOURCE)
> +$(2)_ACTUAL_SOURCE_SITE    ?= $$(call qstrip,$$($(2)_SITE))
> +
>  ifndef $(2)_PATCH
>   ifdef $(3)_PATCH
>    $(2)_PATCH = $$($(3)_PATCH)
> @@ -763,14 +771,6 @@ endif
>  endif
>  endif
>  
> -# If FOO_ACTUAL_SOURCE_TARBALL is explicitly defined, it means FOO_SOURCE is
> -# indeed a binary (e.g. external toolchain) and FOO_ACTUAL_SOURCE_TARBALL/_SITE
> -# point to the actual sources tarball. Use the actual sources for legal-info.
> -# For most packages the FOO_SITE/FOO_SOURCE pair points to real source code,
> -# so these are the defaults for FOO_ACTUAL_*.
> -$(2)_ACTUAL_SOURCE_TARBALL ?= $$($(2)_SOURCE)
> -$(2)_ACTUAL_SOURCE_SITE    ?= $$(call qstrip,$$($(2)_SITE))
> -
>  # legal-info: produce legally relevant info.
>  $(1)-legal-info:
>  # Packages without a source are assumed to be part of Buildroot, skip them.
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 06/16 v3] core/legal-info: ensure legal-info works in off-line mode

[Yann E. MORIN]

Almost all packages which are saved for legal-info have their source
archives downloaded as part of 'make source', which makes an off-line
build completely possible [0].

However, for the pre-configured external toolchains, the source tarball
is different, as the main tarball is a binary package. And that source
tarball is only downloaded during the legal-info phase, which makes it
inconvenient for full off-line builds.

We fix that by adding a new rule, $(1)-legal-source which only
$(1)-all-source depends on, so that we only download it for a top-level
'make source', not as part of the standard download mechanism (i.e. only
what is really needed to build).

This way, we can do a complete [0] off-line build and are still able to
generate legal-info, while at the same time we do not incur any download
overhead during a simple build.

Also, we previously downloaded the _ACTUAL_SOURCE_TARBALL when it was
not empty. However, since _ACTUAL_SOURCE_TARBALL defaults to the value
of _SOURCE, it can not be empty when _SOURCE is not. Thus, we'd get a
spurious report of a missing hash for the tarball, since it was not in
a standard package rule (configure, build, install..) and thus would
miss the PKG and PKGDIR variables to find the .hash file.

We fix that in this commit as well, by:

  - setting PKG and PKGDIR just for the -legal-source rule;

  - only downloading _ACTUAL_SOURCE_TARBALL if it is not empty *and* not
    the same as _SOURCE (to avoid a second report about the hash).

[0] Save for nodejs which invarriably wants to download stuff at build
time. Sigh... :-( Fixing that is work for another time...

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>

---
Changes v2 -> v3:
  - re-order the PHONY targets  (Arnout)
  - don't reorder setting _ACTUAL_SOURCE/_SITE, it's done in its own
    patch now  (Arnout)
  - adapt the commit log accordingly  (Arnout)
  - typo

Changes v1 -> v2:
  - drop the 'redistribute == ignore', it does not exist yet
---
 package/pkg-generic.mk | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 5958029..f8ff6f1 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -634,6 +634,10 @@ $(1)-depends:		$$($(2)_FINAL_DEPENDENCIES)
 
 $(1)-source:		$$($(2)_TARGET_SOURCE)
 
+$(1)-all-source:	$(1)-legal-source
+$(1)-legal-info:	$(1)-legal-source
+$(1)-legal-source:	$(1)-source
+
 $(1)-source-check:
 	$$(foreach p,$$($(2)_ALL_DOWNLOADS),$$(call SOURCE_CHECK,$$(p))$$(sep))
 
@@ -771,6 +775,20 @@ endif
 endif
 endif
 
+# Download actual sources if they differ from the extracted sources
+# (e.g. for external toolchains)
+#
+# We need to provide PKG and PKGDIR, because there's no .stamp file for
+# the legal-info step.
+$(1)-legal-source:	PKG=$(2)
+$(1)-legal-source:	PKGDIR=$(pkgdir)
+$(1)-legal-source:
+ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),)
+ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE))
+	$$(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL))
+endif # actual sources != sources
+endif # actual sources != ""
+
 # legal-info: produce legally relevant info.
 $(1)-legal-info:
 # Packages without a source are assumed to be part of Buildroot, skip them.
@@ -801,9 +819,6 @@ else
 # Other packages
 
 ifeq ($$($(2)_REDISTRIBUTE),YES)
-ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE))
-	$$(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL))
-endif
 # Copy the source tarball
 	$$(Q)$$(call hardlink-copy,\
 		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
@@ -898,6 +913,7 @@ endif
 	$(1)-install-staging \
 	$(1)-install-target \
 	$(1)-legal-info \
+	$(1)-legal-source \
 	$(1)-patch \
 	$(1)-rebuild \
 	$(1)-reconfigure \
-- 
1.9.1

[Buildroot] [PATCH 06/16 v3] core/legal-info: ensure legal-info works in off-line mode

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Almost all packages which are saved for legal-info have their source
> archives downloaded as part of 'make source', which makes an off-line
> build completely possible [0].
>
> However, for the pre-configured external toolchains, the source tarball
> is different, as the main tarball is a binary package. And that source
> tarball is only downloaded during the legal-info phase, which makes it
> inconvenient for full off-line builds.
>
> We fix that by adding a new rule, $(1)-legal-source which only
> $(1)-all-source depends on, so that we only download it for a top-level
> 'make source', not as part of the standard download mechanism (i.e. only
> what is really needed to build).
>
> This way, we can do a complete [0] off-line build and are still able to
> generate legal-info, while at the same time we do not incur any download
> overhead during a simple build.
>
> Also, we previously downloaded the _ACTUAL_SOURCE_TARBALL when it was
> not empty. However, since _ACTUAL_SOURCE_TARBALL defaults to the value
> of _SOURCE, it can not be empty when _SOURCE is not. Thus, we'd get a
> spurious report of a missing hash for the tarball, since it was not in
> a standard package rule (configure, build, install..) and thus would
> miss the PKG and PKGDIR variables to find the .hash file.
>
> We fix that in this commit as well, by:
>
>    - setting PKG and PKGDIR just for the -legal-source rule;
>
>    - only downloading _ACTUAL_SOURCE_TARBALL if it is not empty *and* not
>      the same as _SOURCE (to avoid a second report about the hash).
>
> [0] Save for nodejs which invarriably wants to download stuff at build
> time. Sigh... :-( Fixing that is work for another time...

Thanks for the detailed comment!
I didn't even find any typo... :) but I have a note below.

> @@ -771,6 +775,20 @@ endif
>   endif
>   endif
>
> +# Download actual sources if they differ from the extracted sources
> +# (e.g. for external toolchains)
> +#
> +# We need to provide PKG and PKGDIR, because there's no .stamp file for
> +# the legal-info step.
> +$(1)-legal-source:	PKG=$(2)
> +$(1)-legal-source:	PKGDIR=$(pkgdir)
> +$(1)-legal-source:
> +ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),)
> +ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE))
> +	$$(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL))
> +endif # actual sources != sources
> +endif # actual sources != ""

This created a side effect that is harmless, but a bit unpleasant.

Before this patch, running 'make source' with all sources already
fetched did not produce any output. Now, when using a package that 
defines _ACTUAL_SOURCE_SITE, it does:

$ make source
arm-2014.05-29-arm-none-linux-gnueabi.src.tar.bz2: OK (sha256: [...])
$

This is caused by not having a stamp file for the actual source
tarball. I suppose we should create a .stamp_actual_downloaded.

For the rest the patch looks good and it works.

-- 
Luca

[Buildroot] [PATCH 06/16 v3] core/legal-info: ensure legal-info works in off-line mode

[Yann E. MORIN]

Luca, All,

On 2016-01-30 15:56 +0100, Luca Ceresoli spake thusly:
> Yann E. MORIN wrote:
> >Almost all packages which are saved for legal-info have their source
> >archives downloaded as part of 'make source', which makes an off-line
> >build completely possible [0].
> >
> >However, for the pre-configured external toolchains, the source tarball
> >is different, as the main tarball is a binary package. And that source
> >tarball is only downloaded during the legal-info phase, which makes it
> >inconvenient for full off-line builds.
> >
> >We fix that by adding a new rule, $(1)-legal-source which only
> >$(1)-all-source depends on, so that we only download it for a top-level
> >'make source', not as part of the standard download mechanism (i.e. only
> >what is really needed to build).
> >
> >This way, we can do a complete [0] off-line build and are still able to
> >generate legal-info, while at the same time we do not incur any download
> >overhead during a simple build.
> >
> >Also, we previously downloaded the _ACTUAL_SOURCE_TARBALL when it was
> >not empty. However, since _ACTUAL_SOURCE_TARBALL defaults to the value
> >of _SOURCE, it can not be empty when _SOURCE is not. Thus, we'd get a
> >spurious report of a missing hash for the tarball, since it was not in
> >a standard package rule (configure, build, install..) and thus would
> >miss the PKG and PKGDIR variables to find the .hash file.
> >
> >We fix that in this commit as well, by:
> >
> >   - setting PKG and PKGDIR just for the -legal-source rule;
> >
> >   - only downloading _ACTUAL_SOURCE_TARBALL if it is not empty *and* not
> >     the same as _SOURCE (to avoid a second report about the hash).
> >
> >[0] Save for nodejs which invarriably wants to download stuff at build
> >time. Sigh... :-( Fixing that is work for another time...
> 
> Thanks for the detailed comment!
> I didn't even find any typo... :) but I have a note below.

Hehe! ;-)

> >@@ -771,6 +775,20 @@ endif
> >  endif
> >  endif
> >
> >+# Download actual sources if they differ from the extracted sources
> >+# (e.g. for external toolchains)
> >+#
> >+# We need to provide PKG and PKGDIR, because there's no .stamp file for
> >+# the legal-info step.
> >+$(1)-legal-source:	PKG=$(2)
> >+$(1)-legal-source:	PKGDIR=$(pkgdir)
> >+$(1)-legal-source:
> >+ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),)
> >+ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE))
> >+	$$(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL))
> >+endif # actual sources != sources
> >+endif # actual sources != ""
> 
> This created a side effect that is harmless, but a bit unpleasant.
> 
> Before this patch, running 'make source' with all sources already
> fetched did not produce any output. Now, when using a package that defines
> _ACTUAL_SOURCE_SITE, it does:
> 
> $ make source
> arm-2014.05-29-arm-none-linux-gnueabi.src.tar.bz2: OK (sha256: [...])
> $
> 
> This is caused by not having a stamp file for the actual source
> tarball. I suppose we should create a .stamp_actual_downloaded.

Hmmm.. Indeed, that's not very nice... :-( I'll see to use a stamp file.

Side note in this respect: is _ACTUAL_SOURCE supposed to support
multiple archives in there? If yes, then the current code (before that
patch) is not correct, as it does not treat _ACTUAL_SOURCE as a list and
does not iterate over it when calling the DOWNLOAD macro.

I think it would make sense to support to support a list in there, but
that is work for a separate patch.

> For the rest the patch looks good and it works.

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 06/16 v3] core/legal-info: ensure legal-info works in off-line mode

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Luca, All,
>
> On 2016-01-30 15:56 +0100, Luca Ceresoli spake thusly:
>> Yann E. MORIN wrote:
>>> Almost all packages which are saved for legal-info have their source
>>> archives downloaded as part of 'make source', which makes an off-line
>>> build completely possible [0].
>>>
>>> However, for the pre-configured external toolchains, the source tarball
>>> is different, as the main tarball is a binary package. And that source
>>> tarball is only downloaded during the legal-info phase, which makes it
>>> inconvenient for full off-line builds.
>>>
>>> We fix that by adding a new rule, $(1)-legal-source which only
>>> $(1)-all-source depends on, so that we only download it for a top-level
>>> 'make source', not as part of the standard download mechanism (i.e. only
>>> what is really needed to build).
>>>
>>> This way, we can do a complete [0] off-line build and are still able to
>>> generate legal-info, while at the same time we do not incur any download
>>> overhead during a simple build.
>>>
>>> Also, we previously downloaded the _ACTUAL_SOURCE_TARBALL when it was
>>> not empty. However, since _ACTUAL_SOURCE_TARBALL defaults to the value
>>> of _SOURCE, it can not be empty when _SOURCE is not. Thus, we'd get a
>>> spurious report of a missing hash for the tarball, since it was not in
>>> a standard package rule (configure, build, install..) and thus would
>>> miss the PKG and PKGDIR variables to find the .hash file.
>>>
>>> We fix that in this commit as well, by:
>>>
>>>    - setting PKG and PKGDIR just for the -legal-source rule;
>>>
>>>    - only downloading _ACTUAL_SOURCE_TARBALL if it is not empty *and* not
>>>      the same as _SOURCE (to avoid a second report about the hash).
>>>
>>> [0] Save for nodejs which invarriably wants to download stuff at build
>>> time. Sigh... :-( Fixing that is work for another time...
>>
>> Thanks for the detailed comment!
>> I didn't even find any typo... :) but I have a note below.
>
> Hehe! ;-)
>
>>> @@ -771,6 +775,20 @@ endif
>>>   endif
>>>   endif
>>>
>>> +# Download actual sources if they differ from the extracted sources
>>> +# (e.g. for external toolchains)
>>> +#
>>> +# We need to provide PKG and PKGDIR, because there's no .stamp file for
>>> +# the legal-info step.
>>> +$(1)-legal-source:	PKG=$(2)
>>> +$(1)-legal-source:	PKGDIR=$(pkgdir)
>>> +$(1)-legal-source:
>>> +ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),)
>>> +ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE))
>>> +	$$(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL))
>>> +endif # actual sources != sources
>>> +endif # actual sources != ""
>>
>> This created a side effect that is harmless, but a bit unpleasant.
>>
>> Before this patch, running 'make source' with all sources already
>> fetched did not produce any output. Now, when using a package that defines
>> _ACTUAL_SOURCE_SITE, it does:
>>
>> $ make source
>> arm-2014.05-29-arm-none-linux-gnueabi.src.tar.bz2: OK (sha256: [...])
>> $
>>
>> This is caused by not having a stamp file for the actual source
>> tarball. I suppose we should create a .stamp_actual_downloaded.
>
> Hmmm.. Indeed, that's not very nice... :-( I'll see to use a stamp file.
>
> Side note in this respect: is _ACTUAL_SOURCE supposed to support
> multiple archives in there? If yes, then the current code (before that
> patch) is not correct, as it does not treat _ACTUAL_SOURCE as a list and
> does not iterate over it when calling the DOWNLOAD macro.
>
> I think it would make sense to support to support a list in there, but
> that is work for a separate patch.

It would be nice, but I don't think there is any actual use case in the
foreseeable future.

-- 
Luca

[Buildroot] [PATCH 06/16 v3] core/legal-info: ensure legal-info works in off-line mode

[Yann E. MORIN]

Luca, All,

On 2016-01-30 16:31 +0100, Luca Ceresoli spake thusly:
> Yann E. MORIN wrote:
[--SNIP--]
> >Side note in this respect: is _ACTUAL_SOURCE supposed to support
> >multiple archives in there? If yes, then the current code (before that
> >patch) is not correct, as it does not treat _ACTUAL_SOURCE as a list and
> >does not iterate over it when calling the DOWNLOAD macro.
> >
> >I think it would make sense to support to support a list in there, but
> >that is work for a separate patch.
> 
> It would be nice, but I don't think there is any actual use case in the
> foreseeable future.

Except I can very easily see separate archvies for pre-built toolchains'
actual sources, one for each of gcc, binutils, glibc and so on...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 07/16 v3] core/pkg-generic: add variable to store the package rawname-version

[Yann E. MORIN]

Introduce a new per-package variable to store the 'rawname-version'
tuple, instead of computing it every time we need it.

Currently, it's only a single location, but follow-up patches will
introduce more use of it.

Reported-by: Luca Ceresoli <luca@lucaceresoli.net>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---
 package/pkg-generic.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index f8ff6f1..b7d374a 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -387,6 +387,7 @@ else
  $(2)_DL_VERSION := $$(strip $$($(2)_VERSION))
 endif
 $(2)_VERSION := $$(call sanitize,$$($(2)_DL_VERSION))
+$(2)_RAWNAME_VERSION = $$($(2)_RAWNAME)-$$($(2)_VERSION)
 
 ifdef $(3)_OVERRIDE_SRCDIR
   $(2)_OVERRIDE_SRCDIR ?= $$($(3)_OVERRIDE_SRCDIR)
@@ -423,7 +424,7 @@ ifndef $(2)_SOURCE
  ifdef $(3)_SOURCE
   $(2)_SOURCE = $$($(3)_SOURCE)
  else
-  $(2)_SOURCE			?= $$($(2)_RAWNAME)-$$($(2)_VERSION).tar.gz
+  $(2)_SOURCE			?= $$($(2)_RAWNAME_VERSION).tar.gz
  endif
 endif
 
-- 
1.9.1

[Buildroot] [PATCH 08/16 v3] core/legal-info: install source archives in their own sub-dir

[Yann E. MORIN]

Currently, we put all source archives side-by-side in the same
directory.

Since we're about to also save individual patches that were applied
on those sources, we don't want to make that directory a complete
mess of unassorted files.

So, we install each source archive in its own sub-directory, where
we'll later store the patches too. Store that location in a variable,
so it can be re-used later on (to install patches in a future commit).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

---
Changes v1 -> v2:
  - perl no longer has a post-legal-info hook  (Thoams, Luca)
---
 package/pkg-generic.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b7d374a..27fd98d 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -490,6 +490,8 @@ endif
 
 $(2)_REDISTRIBUTE		?= YES
 
+$(2)_REDIST_SOURCES_DIR = $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))/$$($(2)_RAWNAME_VERSION)
+
 # When a target package is a toolchain dependency set this variable to
 # 'NO' so the 'toolchain' dependency is not added to prevent a circular
 # dependency
@@ -823,7 +825,7 @@ ifeq ($$($(2)_REDISTRIBUTE),YES)
 # Copy the source tarball
 	$$(Q)$$(call hardlink-copy,\
 		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
-		     $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))))
+		     $$($(2)_REDIST_SOURCES_DIR))
 endif # redistribute
 
 endif # other packages
-- 
1.9.1

[Buildroot] [PATCH 09/16 v3] core/legal-info: add package version to license directory

[Yann E. MORIN]

Now that we save the source archives in a directory named after the
package and its version, do the same for the license files, for
consistency.

It has a not-so-bad side-effect of also saving the version string in
the all-licenses list.

The only (small) side-effect, is that the warnings about undefined
_LICENSE_FILES now contains the version string, too. That's unavoidable,
since that's what is stored in the legal report.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

---
Changes v1 -> v2:
  - s/drawback/side-effect/  (Luca)
---
 package/pkg-generic.mk | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 27fd98d..7ad3bc3 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -805,10 +805,10 @@ ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 # is that the license still applies to the files distributed as part
 # of the rootfs, even if the sources are not themselves redistributed.
 ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),)
-	@$$(call legal-license-nofiles,$$($(2)_RAWNAME),$$(call UPPERCASE,$(4)))
-	@$$(call legal-warning-pkg,$$($(2)_RAWNAME),cannot save license ($(2)_LICENSE_FILES not defined))
+	@$$(call legal-license-nofiles,$$($(2)_RAWNAME_VERSION),$$(call UPPERCASE,$(4)))
+	@$$(call legal-warning-pkg,$$($(2)_RAWNAME_VERSION),cannot save license ($(2)_LICENSE_FILES not defined))
 else
-	@$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep))
+	@$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME_VERSION),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep))
 endif # license files
 
 ifeq ($$($(2)_SITE_METHOD),local)
-- 
1.9.1

[Buildroot] [PATCH 10/16 v3] core/apply-patches: store full path of applied patches

[Yann E. MORIN]

Currently, we only store the filename of the applied patches.

However, we are soon to want to install those patches in the legal-info
directory, so we'll have to know where those patches come from.

Instead of duplicating the logic to find the patches (bundled,
downloaded, from a global patch dir...), just store the full path to
each of those patches so we can retrieve them more easily later on.

Also always create the list-file, even if empty, so that we need not
test for its existence before reading it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
[Tested only with patches in the Buildroot sources]
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

---
Changes v1 -> v2:
  - do not duplicate '/' in paths  (Luca)
---
 support/scripts/apply-patches.sh | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/support/scripts/apply-patches.sh b/support/scripts/apply-patches.sh
index 201278d..20a1552 100755
--- a/support/scripts/apply-patches.sh
+++ b/support/scripts/apply-patches.sh
@@ -63,8 +63,12 @@ find ${builddir}/ '(' -name '*.rej' -o -name '.*.rej' ')' -print0 | \
     xargs -0 -r rm -f
 
 function apply_patch {
-    path=$1
-    patch=$2
+    path="${1%%/}"
+    patch="${2}"
+    case "${path}" in
+        /*) ;;
+        *)  path="$(pwd)/${path}";;
+    esac
     if [ "$3" ]; then
         type="series"; uncomp="cat"
     else
@@ -99,7 +103,7 @@ function apply_patch {
         echo "Error: missing patch file ${path}/$patch"
         exit 1
     fi
-    echo $patch >> ${builddir}/.applied_patches_list
+    echo "${path}/${patch}" >> ${builddir}/.applied_patches_list
     ${uncomp} "${path}/$patch" | patch -g0 -p1 -E -d "${builddir}" -t -N $silent
     if [ $? != 0 ] ; then
         echo "Patch failed!  Please fix ${patch}!"
@@ -141,6 +145,7 @@ function scan_patchdir {
     fi
 }
 
+touch ${builddir}/.applied_patches_list
 scan_patchdir "$patchdir" "$patchpattern"
 
 # Check for rejects...
-- 
1.9.1

[Buildroot] [PATCH 11/16 v3] core/legal-info: also save patches

[Yann E. MORIN]

Currently, the legal-info infra only saves the source archive of a
package. However, that's not enough as we may apply some patches on
packages sources.

We do suggest users to also redistribute the Buildroot sources as part
of their compliance distribution, so the patches bundled in Buildroot
would indeed be included in the compliance distribution.

However, that's still not enough, since we may download some patches, or
the user may use a global patch directory. Patches in there might not
end up in the compliance distribution, and there are risks of
non-conformity.

So, always include patches alongside the source archive.

To ensure reproducibility, we also generate a series file, so patches
can be re-applied in the correct order.

We get the list of patches to include from the list of patches that were
applied by the package infrastructure (via the apply-patches support
script). So, we need to get packages properly extracted and patched
before we can save their legal-info, not just in the case they define
_LICENSE_FILES.

Update the legal-info header accordingly.

Note: this means that, when a package is not patched and defines no
LICENSE_FILES, we will extract and patch it for nothing. There is no
easy way to know whether we have to patch a package or not. We can only
either duplicate the logic to detect patches (bad) or rely on the infra
actually patching the package. Also, a vast majority of packages are
either patched, or define _LICENSE_FILES, so it is best and easiest to
always extract and patch them prior to legal-info.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

---
Changes v2 -> v3:
  - also mention that patches have been saved  (Luca)

Changes v1 -> v2:
  - don't recompute rawname-version needlessly  (Luca)
---
 package/pkg-generic.mk           | 13 ++++++++-----
 support/legal-info/README.header |  9 +++++----
 2 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 7ad3bc3..c5b66db 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -759,12 +759,10 @@ $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
 endif
 $(2)_MANIFEST_LICENSE_FILES ?= not saved
 
-# If the package declares _LICENSE_FILES, we need to extract it,
-# for overriden, local or normal remote packages alike, whether
-# we want to redistribute it or not.
-ifneq ($$($(2)_LICENSE_FILES),)
+# We need to extract and patch a package to be able to retrieve its
+# license files (if any) and the list of patches applied to it (if
+# any).
 $(1)-legal-info: $(1)-patch
-endif
 
 # We only save the sources of packages we want to redistribute, that are
 # non-local, and non-overriden. So only store, in the manifest, the tarball
@@ -826,6 +824,11 @@ ifeq ($$($(2)_REDISTRIBUTE),YES)
 	$$(Q)$$(call hardlink-copy,\
 		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
 		     $$($(2)_REDIST_SOURCES_DIR))
+# Copy patches and generate the series file
+	$$(Q)while read f; do \
+		$$(call hardlink-copy,$$$${f},$$($(2)_REDIST_SOURCES_DIR)) || exit 1; \
+		printf "%s\n" "$$$${f##*/}" >>$$($(2)_REDIST_SOURCES_DIR)/series || exit 1; \
+	done <$$($(2)_DIR)/.applied_patches_list
 endif # redistribute
 
 endif # other packages
diff --git a/support/legal-info/README.header b/support/legal-info/README.header
index d07c45d..0b5aa66 100644
--- a/support/legal-info/README.header
+++ b/support/legal-info/README.header
@@ -14,10 +14,11 @@ This material is composed of the following items.
    compiled programs.
    Note: this may have not been saved due to technical limitations, you may
    need to collect it manually.
- * The source code for all packages; this has been saved in the sources/
-   subdirectory (except for the non-redistributable packages, which have not
-   been saved); patches applied to some packages by Buildroot are included in
-   the Buildroot sources and were not duplicated in the sources/ subdirectory.
+ * The original source code for all packages; this has been saved in theo
+   sources/ subdirectory (except for the non-redistributable packages, which
+   have not been saved). Patches that were applied are also saved, along
+   with a file named 'series' that lists the patches in the order they were
+   applied.
  * A manifest file listing the configured packages and related information.
  * The license text of the packages; they have been saved in the licenses/
    subdirectory.
-- 
1.9.1

[Buildroot] [PATCH 11/16 v3] core/legal-info: also save patches

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Currently, the legal-info infra only saves the source archive of a
> package. However, that's not enough as we may apply some patches on
> packages sources.
>
> We do suggest users to also redistribute the Buildroot sources as part
> of their compliance distribution, so the patches bundled in Buildroot
> would indeed be included in the compliance distribution.
>
> However, that's still not enough, since we may download some patches, or
> the user may use a global patch directory. Patches in there might not
> end up in the compliance distribution, and there are risks of
> non-conformity.
>
> So, always include patches alongside the source archive.
>
> To ensure reproducibility, we also generate a series file, so patches
> can be re-applied in the correct order.
>
> We get the list of patches to include from the list of patches that were
> applied by the package infrastructure (via the apply-patches support
> script). So, we need to get packages properly extracted and patched
> before we can save their legal-info, not just in the case they define
> _LICENSE_FILES.
>
> Update the legal-info header accordingly.
>
> Note: this means that, when a package is not patched and defines no
> LICENSE_FILES, we will extract and patch it for nothing. There is no
> easy way to know whether we have to patch a package or not. We can only
> either duplicate the logic to detect patches (bad) or rely on the infra
> actually patching the package. Also, a vast majority of packages are
> either patched, or define _LICENSE_FILES, so it is best and easiest to
> always extract and patch them prior to legal-info.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>
>
> ---
> Changes v2 -> v3:
>    - also mention that patches have been saved  (Luca)
>
> Changes v1 -> v2:
>    - don't recompute rawname-version needlessly  (Luca)
> ---
>   package/pkg-generic.mk           | 13 ++++++++-----
>   support/legal-info/README.header |  9 +++++----
>   2 files changed, 13 insertions(+), 9 deletions(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 7ad3bc3..c5b66db 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -759,12 +759,10 @@ $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
>   endif
>   $(2)_MANIFEST_LICENSE_FILES ?= not saved
>
> -# If the package declares _LICENSE_FILES, we need to extract it,
> -# for overriden, local or normal remote packages alike, whether
> -# we want to redistribute it or not.
> -ifneq ($$($(2)_LICENSE_FILES),)
> +# We need to extract and patch a package to be able to retrieve its
> +# license files (if any) and the list of patches applied to it (if
> +# any).
>   $(1)-legal-info: $(1)-patch
> -endif
>
>   # We only save the sources of packages we want to redistribute, that are
>   # non-local, and non-overriden. So only store, in the manifest, the tarball
> @@ -826,6 +824,11 @@ ifeq ($$($(2)_REDISTRIBUTE),YES)
>   	$$(Q)$$(call hardlink-copy,\
>   		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
>   		     $$($(2)_REDIST_SOURCES_DIR))
> +# Copy patches and generate the series file
> +	$$(Q)while read f; do \
> +		$$(call hardlink-copy,$$$${f},$$($(2)_REDIST_SOURCES_DIR)) || exit 1; \
> +		printf "%s\n" "$$$${f##*/}" >>$$($(2)_REDIST_SOURCES_DIR)/series || exit 1; \
> +	done <$$($(2)_DIR)/.applied_patches_list
>   endif # redistribute
>
>   endif # other packages
> diff --git a/support/legal-info/README.header b/support/legal-info/README.header
> index d07c45d..0b5aa66 100644
> --- a/support/legal-info/README.header
> +++ b/support/legal-info/README.header
> @@ -14,10 +14,11 @@ This material is composed of the following items.
>      compiled programs.
>      Note: this may have not been saved due to technical limitations, you may
>      need to collect it manually.
> - * The source code for all packages; this has been saved in the sources/
> -   subdirectory (except for the non-redistributable packages, which have not
> -   been saved); patches applied to some packages by Buildroot are included in
> -   the Buildroot sources and were not duplicated in the sources/ subdirectory.
> + * The original source code for all packages; this has been saved in theo

theo -> the

With that fixed:
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>

You already carried my Tested-by tag, I re-tested this and it works
fine.

-- 
Luca

[Buildroot] [PATCH 12/16 v3] core/legal-info: renumber saved patches

[Yann E. MORIN]

Patches we save can come from various locations;
  - bundled with Buildroot
  - downloaded
  - from one or more global-patch-dir

It is possible that two patches lying into different locations have the
same basename, like so (first is bundled, second is from an hypothetical
global-patch-dir):
    package/foo/0001-fix-Makefile.patch
    /path/to/my/patches/foo/0001-fix-Makefile.patch

In that case, we'd save onlt the second patch, overwriting the first.

We fix that by forcibly prefixing saved patches with a new numbering, to
guarantee the unicity of saved files.

The unfortunate side-effects are that:
  - most saved patches will be twice-numbered,
  - the series file is now redundant.

While the former is mostly not nice-looking, we shouldn't try to strip
any leading numbering, as that might not be a numbering (e.g.
42-retries.patch which is a patch add 42 retries).

The latter is not really problematic. A lot of tools (and people) can
work with, and prefer to have a series file.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>

---
Note that I've made this a separate change, not because the patch would
be too complex if squahed with the previous, but rather to have a more
detailed commit log about the reason for the renumbering; squashing the
two changes together would make for a really long commit log, at the
risk of being more difficult to follow.
---
 package/pkg-generic.mk | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index c5b66db..f6132b3 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -825,9 +825,16 @@ ifeq ($$($(2)_REDISTRIBUTE),YES)
 		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
 		     $$($(2)_REDIST_SOURCES_DIR))
 # Copy patches and generate the series file
-	$$(Q)while read f; do \
-		$$(call hardlink-copy,$$$${f},$$($(2)_REDIST_SOURCES_DIR)) || exit 1; \
-		printf "%s\n" "$$$${f##*/}" >>$$($(2)_REDIST_SOURCES_DIR)/series || exit 1; \
+# Because patches may come from various places (bundled in Buildroot,
+# from one or more global-patch-dir), there might be collisions on the
+# basenames of those files.
+# We add a new numbering to each patch to ensure unicity of the filenames.
+	$$(Q)cpt=1; while read f; do \
+		_c=$$$$(printf "%04d" $$$${cpt}); \
+		$$(call hardlink-copy,$$$${f},$$($(2)_REDIST_SOURCES_DIR),\
+			$$$${_c}-$$$${f##*/}) || exit 1; \
+		printf "%s\n" "$$$${_c}-$$$${f##*/}" >>$$($(2)_REDIST_SOURCES_DIR)/series || exit 1; \
+		: $$$$((cpt++)); \
 	done <$$($(2)_DIR)/.applied_patches_list
 endif # redistribute
 
-- 
1.9.1

[Buildroot] [PATCH 12/16 v3] core/legal-info: renumber saved patches

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Patches we save can come from various locations;
>    - bundled with Buildroot
>    - downloaded
>    - from one or more global-patch-dir
>
> It is possible that two patches lying into different locations have the
> same basename, like so (first is bundled, second is from an hypothetical
> global-patch-dir):
>      package/foo/0001-fix-Makefile.patch
>      /path/to/my/patches/foo/0001-fix-Makefile.patch
>
> In that case, we'd save onlt the second patch, overwriting the first.

onlt -> only

Indeed for legal-info it would be a serious issue if we silently
overwrite a patch, resulting in _silently_ missing a patch. Thus this
must be fixed somehow, and I would not merge the previous patch without
this one (or any variation we come up with).

As an alternative solution, Arnout proposed to error out if two patches
have the same basename. Even though if would avoid the ugly prefixing,
it would unnecessarily break builds without a need (as far as the
development is concerned).

Thus, I moderately prefer Yann's solution. It could be improved by
renaming only files that have identical names, but that might be
overkill, and can definitely be done later.

>
> We fix that by forcibly prefixing saved patches with a new numbering, to
> guarantee the unicity of saved files.
>
> The unfortunate side-effects are that:
>    - most saved patches will be twice-numbered,
>    - the series file is now redundant.
>
> While the former is mostly not nice-looking, we shouldn't try to strip
> any leading numbering, as that might not be a numbering (e.g.
> 42-retries.patch which is a patch add 42 retries).
>
> The latter is not really problematic. A lot of tools (and people) can
> work with, and prefer to have a series file.

I agree, leaving the series file may be useful to somebody, and it
doesn't hurt anyway.

>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
>
> ---
> Note that I've made this a separate change, not because the patch would
> be too complex if squahed with the previous, but rather to have a more
> detailed commit log about the reason for the renumbering; squashing the
> two changes together would make for a really long commit log, at the
> risk of being more difficult to follow.
> ---
>   package/pkg-generic.mk | 13 ++++++++++---
>   1 file changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index c5b66db..f6132b3 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -825,9 +825,16 @@ ifeq ($$($(2)_REDISTRIBUTE),YES)
>   		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
>   		     $$($(2)_REDIST_SOURCES_DIR))
>   # Copy patches and generate the series file
> -	$$(Q)while read f; do \
> -		$$(call hardlink-copy,$$$${f},$$($(2)_REDIST_SOURCES_DIR)) || exit 1; \
> -		printf "%s\n" "$$$${f##*/}" >>$$($(2)_REDIST_SOURCES_DIR)/series || exit 1; \
> +# Because patches may come from various places (bundled in Buildroot,
> +# from one or more global-patch-dir), there might be collisions on the
> +# basenames of those files.
> +# We add a new numbering to each patch to ensure unicity of the filenames.
> +	$$(Q)cpt=1; while read f; do \
> +		_c=$$$$(printf "%04d" $$$${cpt}); \
> +		$$(call hardlink-copy,$$$${f},$$($(2)_REDIST_SOURCES_DIR),\
> +			$$$${_c}-$$$${f##*/}) || exit 1; \
> +		printf "%s\n" "$$$${_c}-$$$${f##*/}" >>$$($(2)_REDIST_SOURCES_DIR)/series || exit 1; \
> +		: $$$$((cpt++)); \

I must admit this is not the most readable piece of code I have seen so
far... ;) I cannot do anything about the '$$$$'s, but at least I can
suggest to rename the variables:

  - cpt -> patch_num
  - _c -> prefix_num

But it works fine, thus:
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 12/16 v3] core/legal-info: renumber saved patches

[Yann E. MORIN]

Luca, All,

On 2016-01-31 20:42 +0100, Luca Ceresoli spake thusly:
> Yann E. MORIN wrote:
> >Patches we save can come from various locations;
> >   - bundled with Buildroot
> >   - downloaded
> >   - from one or more global-patch-dir
> >
> >It is possible that two patches lying into different locations have the
> >same basename, like so (first is bundled, second is from an hypothetical
> >global-patch-dir):
> >     package/foo/0001-fix-Makefile.patch
> >     /path/to/my/patches/foo/0001-fix-Makefile.patch
> >
> >In that case, we'd save onlt the second patch, overwriting the first.
> 
> onlt -> only

Damned... ;-)

> Indeed for legal-info it would be a serious issue if we silently
> overwrite a patch, resulting in _silently_ missing a patch.

It's not only about missing a patch (althigh that is the worst
outocome), it's also about having dependent patches, like:

    /path/to/buildroot/package/foo/0001-fix-Makefile.patch
    /path/to/global/patch/dir/foo/0001-fix-Makefile.patch

with the second patch furhter changing the *same* part of the Makefile,
so it would overwrite the previous one, but fail to apply.

So, two issues;
  - patches faling to apply (bad, because technically incorrect)
  - missign patches (the worst, because legally incorrect)

> As an alternative solution, Arnout proposed to error out if two patches
> have the same basename. Even though if would avoid the ugly prefixing,
> it would unnecessarily break builds without a need (as far as the
> development is concerned).

I'm not very much in favour of erroring out either... The resulting file
names may _look_ ugly, but I don't care: they are for legal people. ;-)

Seriously, I have the same point of view as Luca here.

[--SNIP--]
> >+# Because patches may come from various places (bundled in Buildroot,
> >+# from one or more global-patch-dir), there might be collisions on the
> >+# basenames of those files.
> >+# We add a new numbering to each patch to ensure unicity of the filenames.
> >+	$$(Q)cpt=1; while read f; do \
> >+		_c=$$$$(printf "%04d" $$$${cpt}); \
> >+		$$(call hardlink-copy,$$$${f},$$($(2)_REDIST_SOURCES_DIR),\
> >+			$$$${_c}-$$$${f##*/}) || exit 1; \
> >+		printf "%s\n" "$$$${_c}-$$$${f##*/}" >>$$($(2)_REDIST_SOURCES_DIR)/series || exit 1; \
> >+		: $$$$((cpt++)); \
> 
> I must admit this is not the most readable piece of code I have seen so
> far... ;)

Hehe! ;-]

> I cannot do anything about the '$$$$'s, but at least I can
> suggest to rename the variables:
> 
>  - cpt -> patch_num
>  - _c -> prefix_num

OK, workd for me.

> But it works fine, thus:
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

Thanks.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 13/16 v3] core/legal-info: also save extra downloads

[Yann E. MORIN]

Some packages, like perl, download extra files that end up as part of
the source that Buildroot builds. Up until now, those files were not
saved in the legal-info output.

Add those files to the legal-info output.

The unfortunate side-effect is that we will also save the secondary
archive for the external blackfin toolchains; however, we already do
save the binary release of some external toolchains when they do not
provide actual source archives.

This is inherently bad, as those are not source archives, but solving
this is a bigger concern, for another series...

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>

---
Changes v2 -> v3:
  - typo  (Luca)
  - incorporate the post-commit log message (the part about the
    side-effect) into the commit log itself, it makes sense to not
    forget about that
---
 package/pkg-generic.mk | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index f6132b3..b59af9e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -820,10 +820,12 @@ else
 # Other packages
 
 ifeq ($$($(2)_REDISTRIBUTE),YES)
-# Copy the source tarball
-	$$(Q)$$(call hardlink-copy,\
-		     $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL),\
-		     $$($(2)_REDIST_SOURCES_DIR))
+# Copy the source tarball and any extra downloads, but not
+# patches, as they are handled specially afterwards.
+	$$(foreach e,$$($(2)_ACTUAL_SOURCE_TARBALL) $$(notdir $$($(2)_EXTRA_DOWNLOADS)),\
+			$$(Q)$$(call hardlink-copy,\
+				$$(DL_DIR)/$$(e),\
+				$$($(2)_REDIST_SOURCES_DIR))$$(sep))
 # Copy patches and generate the series file
 # Because patches may come from various places (bundled in Buildroot,
 # from one or more global-patch-dir), there might be collisions on the
-- 
1.9.1

[Buildroot] [PATCH 13/16 v3] core/legal-info: also save extra downloads

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Some packages, like perl, download extra files that end up as part of
> the source that Buildroot builds. Up until now, those files were not
> saved in the legal-info output.
>
> Add those files to the legal-info output.
>
> The unfortunate side-effect is that we will also save the secondary
> archive for the external blackfin toolchains; however, we already do
> save the binary release of some external toolchains when they do not
> provide actual source archives.
>
> This is inherently bad, as those are not source archives, but solving
> this is a bigger concern, for another series...
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>

And:
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 14/16 v3] core/legal-info: generate a hash of all saved files

[Yann E. MORIN]

Having a hash of the saved files can be interesting for the recipient to
verify the integrity of the files.

We remove the warning file earlier, to exclude it from the hash list.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>

---
Changes v1 -> v2:
  - simplify getting rid of the ..../legal-info/ prefix  (Luca)
  - always sort with the C locale
---
 Makefile | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 17c181b..c3427ed 100644
--- a/Makefile
+++ b/Makefile
@@ -696,8 +696,14 @@ legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p
 		cat support/legal-info/README.warnings-header \
 			$(LEGAL_WARNINGS) >>$(LEGAL_REPORT); \
 		cat $(LEGAL_WARNINGS); fi
-	@echo "Legal info produced in $(LEGAL_INFO_DIR)"
 	@rm -f $(LEGAL_WARNINGS)
+	@(cd $(LEGAL_INFO_DIR); \
+	  find * -type f -exec sha256sum {} + \
+	  |LC_ALL=C sort -k2 \
+	  >.legal-info.sha256; \
+	  mv .legal-info.sha256 legal-info.sha256 \
+	)
+	@echo "Legal info produced in $(LEGAL_INFO_DIR)"
 
 show-targets:
 	@echo $(PACKAGES) $(TARGETS_ROOTFS)
-- 
1.9.1

[Buildroot] [PATCH 14/16 v3] core/legal-info: generate a hash of all saved files

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Having a hash of the saved files can be interesting for the recipient to
> verify the integrity of the files.
>
> We remove the warning file earlier, to exclude it from the hash list.
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>

Acked-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

-- 
Luca

[Buildroot] [PATCH 15/16 v3] core/legal-info: allow ignoring packages from the legal-info

[Yann E. MORIN]

It might be necessary to not even mention a package in the output of
legal-info:

  - virtual package have virtually nothing to save in the legal-info
    output;

  - for Buildroot itself, host-gcc-initial and host-gcc-final are
    not real packages, they are just two different steps of the same
    package, gcc;

  - for proprietary packages, it might not even be legal to even
    mention them, being under NDA or some other such restrictive
    conditions.

Introduce the new FOO_LEGAL_INGORE variable that a package can set
to 'YES' (default to 'NO') to indicate that the package should be
completely ignored from the legal-info output, in which case the
package is not metionned in the maniufest, its source archive,
patches and license files are not saved into legal-info/ .

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>

---
Changes v1 -> v2:
  - introduce a new variable, instead of making _REDISTRIBUTE a
    tri-state  (Thomas, Peter, Luca)
---
 docs/manual/adding-packages-generic.txt |  9 +++++++++
 package/pkg-generic.mk                  | 15 +++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
index 1f9c7bb..1b07a14 100644
--- a/docs/manual/adding-packages-generic.txt
+++ b/docs/manual/adding-packages-generic.txt
@@ -421,6 +421,15 @@ information is (assuming the package name is +libfoo+) :
   non-opensource packages: Buildroot will not save the source code for this
   package when collecting the +legal-info+.
 
+* +LIBFOO_LEGAL_IGNORE+ can be set to +YES+ or +NO+ (the default) to indicate
+  that this package should be completely ignored when saving the licensing
+  information. If set to +YES+, then the package is not listed in the manifest,
+  its source archive and its license files are not saved. You probably do not
+  want to set it to +YES+, unless under very specific conditions (e.g. when
+  you use the `legal-info/` output directory as-is to provide a compliance
+  delivery, and do not want your proprietary, non-redistributable packages to
+  even be mentionned in the manifest).
+
 * +LIBFOO_FLAT_STACKSIZE+ defines the stack size of an application built into
   the FLAT binary format. The application stack size on the NOMMU architecture
   processors can't be enlarged at run time. The default stack size for the
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b59af9e..9c02372 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -490,6 +490,14 @@ endif
 
 $(2)_REDISTRIBUTE		?= YES
 
+ifndef $(2)_LEGAL_IGNORE
+ ifdef $(3)_LEGAL_IGNORE
+  $(2)_LEGAL_IGNORE = $$($(3)_LEGAL_IGNORE)
+ endif
+endif
+
+$(2)_LEGAL_IGNORE		?= NO
+
 $(2)_REDIST_SOURCES_DIR = $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))/$$($(2)_RAWNAME_VERSION)
 
 # When a target package is a toolchain dependency set this variable to
@@ -762,7 +770,10 @@ $(2)_MANIFEST_LICENSE_FILES ?= not saved
 # We need to extract and patch a package to be able to retrieve its
 # license files (if any) and the list of patches applied to it (if
 # any).
+# But not if we want to ignore that package completely.
+ifneq ($$($(2)_LEGAL_IGNORE),YES)
 $(1)-legal-info: $(1)-patch
+endif
 
 # We only save the sources of packages we want to redistribute, that are
 # non-local, and non-overriden. So only store, in the manifest, the tarball
@@ -792,6 +803,8 @@ endif # actual sources != ""
 
 # legal-info: produce legally relevant info.
 $(1)-legal-info:
+ifneq ($$($(2)_LEGAL_IGNORE),YES)
+
 # Packages without a source are assumed to be part of Buildroot, skip them.
 	$$(foreach hook,$$($(2)_PRE_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
 ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
@@ -845,6 +858,8 @@ endif # other packages
 endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 	$$(foreach hook,$$($(2)_POST_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
 
+endif # $(2)_LEGAL_IGNORE != YES
+
 # add package to the general list of targets if requested by the buildroot
 # configuration
 ifeq ($$($$($(2)_KCONFIG_VAR)),y)
-- 
1.9.1

[Buildroot] [PATCH 15/16 v3] core/legal-info: allow ignoring packages from the legal-info

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> It might be necessary to not even mention a package in the output of
> legal-info:
>
>    - virtual package have virtually nothing to save in the legal-info

package -> packages

>      output;
>
>    - for Buildroot itself, host-gcc-initial and host-gcc-final are
>      not real packages, they are just two different steps of the same
>      package, gcc;
>
>    - for proprietary packages, it might not even be legal to even
>      mention them, being under NDA or some other such restrictive
>      conditions.
>
> Introduce the new FOO_LEGAL_INGORE variable that a package can set
> to 'YES' (default to 'NO') to indicate that the package should be
> completely ignored from the legal-info output, in which case the
> package is not metionned in the maniufest, its source archive,

metionned in the maniufest -> mentioned in the manifest

> patches and license files are not saved into legal-info/ .
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
>
> ---
> Changes v1 -> v2:
>    - introduce a new variable, instead of making _REDISTRIBUTE a
>      tri-state  (Thomas, Peter, Luca)
> ---
>   docs/manual/adding-packages-generic.txt |  9 +++++++++
>   package/pkg-generic.mk                  | 15 +++++++++++++++
>   2 files changed, 24 insertions(+)
>
> diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt
> index 1f9c7bb..1b07a14 100644
> --- a/docs/manual/adding-packages-generic.txt
> +++ b/docs/manual/adding-packages-generic.txt
> @@ -421,6 +421,15 @@ information is (assuming the package name is +libfoo+) :
>     non-opensource packages: Buildroot will not save the source code for this
>     package when collecting the +legal-info+.
>
> +* +LIBFOO_LEGAL_IGNORE+ can be set to +YES+ or +NO+ (the default) to indicate
> +  that this package should be completely ignored when saving the licensing
> +  information. If set to +YES+, then the package is not listed in the manifest,
> +  its source archive and its license files are not saved. You probably do not
> +  want to set it to +YES+, unless under very specific conditions (e.g. when
> +  you use the `legal-info/` output directory as-is to provide a compliance
> +  delivery, and do not want your proprietary, non-redistributable packages to
> +  even be mentionned in the manifest).

mentionned -> mentioned

> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index b59af9e..9c02372 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -490,6 +490,14 @@ endif
>
>   $(2)_REDISTRIBUTE		?= YES
>
> +ifndef $(2)_LEGAL_IGNORE
> + ifdef $(3)_LEGAL_IGNORE
> +  $(2)_LEGAL_IGNORE = $$($(3)_LEGAL_IGNORE)
> + endif
> +endif
> +
> +$(2)_LEGAL_IGNORE		?= NO
> +
>   $(2)_REDIST_SOURCES_DIR = $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))/$$($(2)_RAWNAME_VERSION)
>
>   # When a target package is a toolchain dependency set this variable to
> @@ -762,7 +770,10 @@ $(2)_MANIFEST_LICENSE_FILES ?= not saved
>   # We need to extract and patch a package to be able to retrieve its
>   # license files (if any) and the list of patches applied to it (if
>   # any).
> +# But not if we want to ignore that package completely.
> +ifneq ($$($(2)_LEGAL_IGNORE),YES)
>   $(1)-legal-info: $(1)-patch
> +endif
>
>   # We only save the sources of packages we want to redistribute, that are
>   # non-local, and non-overriden. So only store, in the manifest, the tarball
> @@ -792,6 +803,8 @@ endif # actual sources != ""
>
>   # legal-info: produce legally relevant info.
>   $(1)-legal-info:
> +ifneq ($$($(2)_LEGAL_IGNORE),YES)
> +
>   # Packages without a source are assumed to be part of Buildroot, skip them.
>   	$$(foreach hook,$$($(2)_PRE_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
>   ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
> @@ -845,6 +858,8 @@ endif # other packages
>   endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
>   	$$(foreach hook,$$($(2)_POST_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
>
> +endif # $(2)_LEGAL_IGNORE != YES
> +
>   # add package to the general list of targets if requested by the buildroot
>   # configuration
>   ifeq ($$($$($(2)_KCONFIG_VAR)),y)
>

Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
Tested-by: Luca Ceresoli <luca@lucaceresoli.net>


-- 
Luca

[Buildroot] [PATCH 15/16 v3] core/legal-info: allow ignoring packages from the legal-info

[Yann E. MORIN]

Luca, All,

On 2016-01-31 21:11 +0100, Luca Ceresoli spake thusly:
> Yann E. MORIN wrote:
> >It might be necessary to not even mention a package in the output of
> >legal-info:
[--SNIP--]

Tyopes fixed.

> Reviewed-by: Luca Ceresoli <luca@lucaceresoli.net>
> Tested-by: Luca Ceresoli <luca@lucaceresoli.net>

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 16/16 v3] core/pkg-virtual: ignore from legal-info output

[Yann E. MORIN]

Virtual packages really have nothing to save for legal-info, so we just
ignore them.

Currently, packages are only extracted if they define _LICENSE_FILES.
Follow-up patches will always extract and patch non-ignored packages, so
we would potentially see spurious "extracting" and "patching" messages
for virtual packages on the console, which would be a bit strange.
Ignoring virtual packages will get us a cleaner log.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Luca Ceresoli <luca@lucaceresoli.net>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <jacmet@uclibc.org>

---
Changes v1 -> v2:
  - use the new _LEGAL_IGNORE variable  (Thomas, Luca, Peter)
---
 docs/manual/adding-packages-virtual.txt | 3 +++
 package/pkg-virtual.mk                  | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/docs/manual/adding-packages-virtual.txt b/docs/manual/adding-packages-virtual.txt
index a5f17a2..11a497d 100644
--- a/docs/manual/adding-packages-virtual.txt
+++ b/docs/manual/adding-packages-virtual.txt
@@ -56,6 +56,9 @@ The +.mk+ for the virtual package should just evaluate the +virtual-package+ mac
 The ability to have target and host packages is also available, with the
 +host-virtual-package+ macro.
 
+.Note:
+Virtual packages are automatically ignored when generating the +legal-info+ output.
+
 ==== Provider's +Config.in+ file
 
 When adding a package as a provider, only the +Config.in+ file requires some
diff --git a/package/pkg-virtual.mk b/package/pkg-virtual.mk
index 244c1d0..445f09a 100644
--- a/package/pkg-virtual.mk
+++ b/package/pkg-virtual.mk
@@ -59,6 +59,8 @@ endif
 # Add dependency against the provider
 $(2)_DEPENDENCIES += $$(call qstrip,$$(BR2_PACKAGE_PROVIDES_$(2)))
 
+$(2)_LEGAL_IGNORE = YES
+
 # Call the generic package infrastructure to generate the necessary
 # make targets
 $(call inner-generic-package,$(1),$(2),$(3),$(4))
-- 
1.9.1

[Buildroot] [PATCH 16/16 v3] core/pkg-virtual: ignore from legal-info output

[Luca Ceresoli]

Hi Yann,

Yann E. MORIN wrote:
> Virtual packages really have nothing to save for legal-info, so we just
> ignore them.
>
> Currently, packages are only extracted if they define _LICENSE_FILES.
> Follow-up patches will always extract and patch non-ignored packages, so
> we would potentially see spurious "extracting" and "patching" messages
> for virtual packages on the console, which would be a bit strange.
> Ignoring virtual packages will get us a cleaner log.

This comment is incorrect after patch 11 ("core/legal-info: also save
patches").

>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Luca Ceresoli <luca@lucaceresoli.net>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <jacmet@uclibc.org>
>
> ---
> Changes v1 -> v2:
>    - use the new _LEGAL_IGNORE variable  (Thomas, Luca, Peter)
> ---
>   docs/manual/adding-packages-virtual.txt | 3 +++
>   package/pkg-virtual.mk                  | 2 ++
>   2 files changed, 5 insertions(+)
>
> diff --git a/docs/manual/adding-packages-virtual.txt b/docs/manual/adding-packages-virtual.txt
> index a5f17a2..11a497d 100644
> --- a/docs/manual/adding-packages-virtual.txt
> +++ b/docs/manual/adding-packages-virtual.txt
> @@ -56,6 +56,9 @@ The +.mk+ for the virtual package should just evaluate the +virtual-package+ mac
>   The ability to have target and host packages is also available, with the
>   +host-virtual-package+ macro.
>
> +.Note:
> +Virtual packages are automatically ignored when generating the +legal-info+ output.
> +

A minor nit: to make it clearer, I would append "by setting
FOO_LEGAL_IGNORE" to this sentence.

-- 
Luca