[Buildroot] [PATCH v5 00/19] Reproducible builds

[Buildroot] [PATCH v5 00/19] Reproducible builds

[Jérôme Pouiller]

This series try to continue work initiated by Gilles Chanteperdrix:
  http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
  http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

I dropped some patchs from original series because either:
  - I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
    in gcc, ...)
  - I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
  - They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)

There are many changes since previous version. While earlier versions focus on
timestamps, v5 allows to make reproducible build even using different build
path. In particular it remove rpaths from ELF generated with libtool.

This series is divided in two parts:
  - patches 1 to 8 only concern timestamps
  - patches 9 to 19 only concern build path

Mainly cosmetics changes has been made on patches imported from v4.  I
respinned v4 on top of master. Commit log has been improved for patches 5 to 7.

I added patch 8 where I now detect common patterns that break reproducibility
(`uname -n`, `uname -r`, etc...).

I temporary dropped patch that disable build-id for kernel. When build paths are
identical, build-id are also identical. When build paths are different, build-id
are sometime different, but I did not yet identified exact origin (maybe
-fdebug-prefix-map= would be sufficient to solve problem).

In add, for other packages I noticed that symbol table contains absolute path
to some objects:

    $ arm-linux-objdump -t col | grep crt
    00000000 l    df *ABS*  00000000     /[...]/host/usr/arm-buildroot-linux-gnueabi/sysroot/lib/crti.o
    00000000 l    df *ABS*  00000000     /[...]/host/usr/arm-buildroot-linux-gnueabi/sysroot/lib/crtn.o

Symbol tables are not a problem them self, but they imply unreproducible
build-id. I think I could pass "-Wl,-x" to compiler to remove these
occurrences, but I have not yet evaluated all impacts of this solution. So,
I just postpone this problem.


Patches 11 to 19 are nearly a series in the series. Until now, most of binaries
installed by libtool was configured with RPATH pointingto their build
directory. Indeed, libtool add this path during compilation in order to be able
to execute them directly from build directory. This path should normally
removed during install, but Buildroot disable this behavior (patch 15). Simply
re-enabling this behavior does not work. Indeed, during relink, libtool try to
use .la that are not yet patched and fail to find libraries. On another side,
libtool support usage of a sysroot since v1.5. To enable this support, we have
to keep original values from .la file (patch 12 and 13) and inform libtool we
are using a sysroot (patch 11).

Patch 14 fix a small incompatibility with unsafe path detection.

Since libtool is now correctly used, it is not more necessary to disable
install directory sanity check (patch 17).

From libtool point of view, sysroot is not reachable from $(TARGET_DIR). So,
during installation to $(TARGET_DIR), it add an entry in RPATH that point to
$(STAGING_DIR). To fix this problem, we just have to inform libtool that
$(STAGING_DIR) is reachable (patch 16).

Finally, I also clean up libtool infra from modification that seems useless now
(patches 18 and 19).

I tested this series using internal toolchain and Linaro toolchain with these
packages:

    BR2_INIT_SYSTEMD=y       (install public libraries in /usr/lib/systemd)
    BR2_PACKAGE_LIBCDAUDIO=y (libtool 1.5)
    BR2_PACKAGE_LIBLO=y      (libtool 2.2)
    BR2_PACKAGE_MADPLAY=y    (unpatched libtool)
    BR2_PACKAGE_ALSA_LIB=y   (optional dependency to Madplay)
    BR2_PACKAGE_PYTHON=y
    BR2_PACKAGE_PYTHON_PY_PYC=y
    BR2_PACKAGE_GNUPG2=y

Except patches 12 and 13, I think whole series is bisectable.

Known issues that break reproducibility:
  - Use of lzop (it unconditionally include timestamps in result)
  - Since we build our own toolchain and toolchain include BR2_FULL_VERSION,
    ccache is incompatible with reproducible
  - Some external toolchains (Linaro for exemple) enable build-id by default.
  - Build path appears in symbol tables. So TARGET_DIR must be striped even if
    it was not built with debug infos.
  - Linux kernel shouldn't be built with CONFIG_DEBUG_INFO, else build-id will
    differ.
  - sysconfigdata.py provided by Python contains build path
  - Some libraries (libassuan, ksba, libxml2, libxslt, ...) install
    configurations scripts (that contain build path) on target.


J?r?me Pouiller (19):
  reproducible: fix DATE/TIME macros in toolchain-wrapper
  fakedate: new package
  core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk
  reproducible: enable fakedate
  pycompile: allow to force compilation
  python2: generate reproducible .pyc
  python3: generate reproducible .pyc
  reproducible: try to detect most common errors
  python2: remove full path from .pyc
  python3: remove full path from .pyc
  infra-libtool: pass sysroot information to libtool
  infra-libtool: no longer prepend STAGING_DIR to libdir
  infra-libtool: correctly prefix $libdir with $STAGING_DIR
  infra-libtool: drop original $libdir (i.e. /usr/lib) from library
    paths
  infra-libtool: relink binaries on install
  infra-libtool: inform libtool that STAGING_DIR is reachable at runtime
  infra-libtool: no longer disable install directory sanity check
  infra-libtool: remove workaround for calls without `--tag'
  infra-libtool: no longer force sys_lib_search_path

 Makefile                                       |  6 ++
 package/Makefile.in                            |  2 +-
 package/fakedate/fakedate                      | 59 +++++++++++++++++
 package/fakedate/fakedate.mk                   | 15 +++++
 package/pkg-autotools.mk                       |  1 +
 package/pkg-generic.mk                         | 11 ----
 package/python/python.mk                       | 15 ++++-
 package/python3/python3.mk                     | 15 ++++-
 support/dependencies/dependencies.mk           |  2 -
 support/libtool/buildroot-libtool-v1.5.patch   | 85 ++++++-------------------
 support/libtool/buildroot-libtool-v2.2.patch   | 88 +++++++-------------------
 support/libtool/buildroot-libtool-v2.4.4.patch | 63 ++++--------------
 support/libtool/buildroot-libtool-v2.4.patch   | 62 ++++--------------
 support/scripts/pycompile.py                   | 11 +++-
 toolchain/toolchain-wrapper.c                  | 74 +++++++++++++++++++++-
 15 files changed, 255 insertions(+), 254 deletions(-)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

-- 
1.9.1

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Jérôme Pouiller]

The use __DATE__ and __TIME__ are one of most common sources of
non-reproducible binaries. In order to fix that, gcc begin to support
SOURCE_DATE_EPOCH variable. This patch take advantage of toolchain-wrapper
to provide support of SOURCE_DATE_EPOCH to older gcc versions.

Function get_source_date_epoch() come directly from gcc git.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Handle $SOURCE_DATE_EPOCH at runtime (Thomas)
    v2:
      - Overload __TIME__ and __DATE__ instead of patching gcc (Thomas)

 toolchain/toolchain-wrapper.c | 74 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 73 insertions(+), 1 deletion(-)

diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
index d59629b..6150574 100644
--- a/toolchain/toolchain-wrapper.c
+++ b/toolchain/toolchain-wrapper.c
@@ -22,12 +22,17 @@
 #include <unistd.h>
 #include <stdlib.h>
 #include <errno.h>
+#include <time.h>
 
 #ifdef BR_CCACHE
 static char ccache_path[PATH_MAX];
 #endif
 static char path[PATH_MAX];
 static char sysroot[PATH_MAX];
+// strlen("-D__TIME__=\"HH:MM:SS\"") + 1 = 22
+static char source_time[22];
+// strlen("-D__DATE__=\"MMM DD YYYY\"") + 1 = 25
+static char source_date[25];
 
 /**
  * GCC errors out with certain combinations of arguments (examples are
@@ -39,8 +44,11 @@ static char sysroot[PATH_MAX];
  * 	-mfloat-abi=
  * 	-march=
  * 	-mcpu=
+ * 	-D__TIME__=
+ * 	-D__DATE__=
+ * 	-Wno-builtin-macro-redefined
  */
-#define EXCLUSIVE_ARGS	3
+#define EXCLUSIVE_ARGS	6
 
 static char *predef_args[] = {
 #ifdef BR_CCACHE
@@ -139,6 +147,47 @@ static void check_unsafe_path(const char *arg,
 	}
 }
 
+/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
+ * timestamp to replace embedded current dates to get reproducible
+ * results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.
+ */
+time_t get_source_date_epoch()
+{
+	char *source_date_epoch;
+	long long epoch;
+	char *endptr;
+
+	source_date_epoch = getenv("SOURCE_DATE_EPOCH");
+	if (!source_date_epoch)
+		return (time_t) -1;
+
+	errno = 0;
+	epoch = strtoll (source_date_epoch, &endptr, 10);
+	if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
+			|| (errno != 0 && epoch == 0)) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"strtoll: %s\n", strerror(errno));
+		exit(2);
+	}
+	if (endptr == source_date_epoch) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"no digits were found: %s\n", endptr);
+		exit(2);
+	}
+	if (*endptr != '\0') {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"trailing garbage: %s\n", endptr);
+		exit(2);
+	}
+	if (epoch < 0) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"value must be nonnegative: %lld \n", epoch);
+		exit(2);
+	}
+
+	return (time_t) epoch;
+}
+
 int main(int argc, char **argv)
 {
 	char **args, **cur, **exec_args;
@@ -149,6 +198,7 @@ int main(int argc, char **argv)
 	char *paranoid_wrapper;
 	int paranoid;
 	int ret, i, count = 0, debug;
+	time_t source_date_epoch;
 
 	/* Calculate the relative paths */
 	basename = strrchr(progpath, '/');
@@ -254,6 +304,28 @@ int main(int argc, char **argv)
 	}
 #endif /* ARCH || CPU */
 
+	source_date_epoch = get_source_date_epoch();
+	if (source_date_epoch != -1) {
+		struct tm *tm = localtime(&source_date_epoch);
+		if (!tm) {
+			perror("__FILE__: localtime");
+			return 3;
+		}
+		ret = strftime(source_time, sizeof(source_time), "-D__TIME__=\"%T\"", tm);
+		if (!ret) {
+			perror("__FILE__: overflow");
+			return 3;
+		}
+		*cur++ = source_time;
+		ret = strftime(source_date, sizeof(source_date), "-D__DATE__=\"%b %e %Y\"", tm);
+		if (!ret) {
+			perror("__FILE__: overflow");
+			return 3;
+		}
+		*cur++ = source_date;
+		*cur++ = "-Wno-builtin-macro-redefined";
+	}
+
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
 		paranoid = 1;
-- 
1.9.1

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> The use __DATE__ and __TIME__ are one of most common sources of
> non-reproducible binaries. In order to fix that, gcc begin to support
> SOURCE_DATE_EPOCH variable. This patch take advantage of toolchain-wrapper
> to provide support of SOURCE_DATE_EPOCH to older gcc versions.
>
> Function get_source_date_epoch() come directly from gcc git.
>
> This work was sponsored by `BA Robotic Systems'.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

Regards,


-- 
Samuel

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Peter Korsgaard]

>>>>> "J?r?me" == J?r?me Pouiller <jezz@sysmic.org> writes:

 > The use __DATE__ and __TIME__ are one of most common sources of

The use of.

 > non-reproducible binaries. In order to fix that, gcc begin to support
 > SOURCE_DATE_EPOCH variable. This patch take advantage of toolchain-wrapper
 > to provide support of SOURCE_DATE_EPOCH to older gcc versions.

 > Function get_source_date_epoch() come directly from gcc git.

 > This work was sponsored by `BA Robotic Systems'.

 > Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
 > ---

 > Notes:
 >     v3:
 >       - Handle $SOURCE_DATE_EPOCH at runtime (Thomas)
 >     v2:
 >       - Overload __TIME__ and __DATE__ instead of patching gcc (Thomas)

 >  toolchain/toolchain-wrapper.c | 74 ++++++++++++++++++++++++++++++++++++++++++-
 >  1 file changed, 73 insertions(+), 1 deletion(-)

 > diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
 > index d59629b..6150574 100644
 > --- a/toolchain/toolchain-wrapper.c
 > +++ b/toolchain/toolchain-wrapper.c
 > @@ -22,12 +22,17 @@
 >  #include <unistd.h>
 >  #include <stdlib.h>
 >  #include <errno.h>
 > +#include <time.h>
 
 >  #ifdef BR_CCACHE
 >  static char ccache_path[PATH_MAX];
 >  #endif
 >  static char path[PATH_MAX];
 >  static char sysroot[PATH_MAX];
 > +// strlen("-D__TIME__=\"HH:MM:SS\"") + 1 = 22
 > +static char source_time[22];
 > +// strlen("-D__DATE__=\"MMM DD YYYY\"") + 1 = 25
 > +static char source_date[25];

It is nicer to simply use sizeof on that string, E.G.:

static char source_time[sizeof("-D__TIME__=\"HH:MM:SS\"")];

Then the comment can be dropped and the comment never gets out of sync
with the size.


 >  /**
 >   * GCC errors out with certain combinations of arguments (examples are
 > @@ -39,8 +44,11 @@ static char sysroot[PATH_MAX];
 >   * 	-mfloat-abi=
 >   * 	-march=
 >   * 	-mcpu=
 > + * 	-D__TIME__=
 > + * 	-D__DATE__=
 > + * 	-Wno-builtin-macro-redefined
 >   */
 > -#define EXCLUSIVE_ARGS	3
 > +#define EXCLUSIVE_ARGS	6
 
 >  static char *predef_args[] = {
 >  #ifdef BR_CCACHE
 > @@ -139,6 +147,47 @@ static void check_unsafe_path(const char *arg,
 >  	}
 >  }
 
 > +/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
 > + * timestamp to replace embedded current dates to get reproducible
 > + * results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.
 > + */
 > +time_t get_source_date_epoch()

This should be static.


> +{
 > +	char *source_date_epoch;
 > +	long long epoch;
 > +	char *endptr;
 > +
 > +	source_date_epoch = getenv("SOURCE_DATE_EPOCH");
 > +	if (!source_date_epoch)
 > +		return (time_t) -1;
 > +
 > +	errno = 0;
 > +	epoch = strtoll (source_date_epoch, &endptr, 10);

NIT: No space between strtoll and '('.

> +	if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
 > +			|| (errno != 0 && epoch == 0)) {
 > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
 > +				"strtoll: %s\n", strerror(errno));
 > +		exit(2);
 > +	}
 > +	if (endptr == source_date_epoch) {
 > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
 > +				"no digits were found: %s\n", endptr);
 > +		exit(2);
 > +	}
 > +	if (*endptr != '\0') {
 > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
 > +				"trailing garbage: %s\n", endptr);
 > +		exit(2);
 > +	}
 > +	if (epoch < 0) {
 > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
 > +				"value must be nonnegative: %lld \n", epoch);
 > +		exit(2);
 > +	}

I'm not sure this detailed error handling is really needed, but OK.

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Jérôme Pouiller]

Hello Peter,

On Tuesday 07 February 2017 21:41:41 Peter Korsgaard wrote:
[...]
> > +{
>  > +	char *source_date_epoch;
>  > +	long long epoch;
>  > +	char *endptr;
>  > +
>  > +	source_date_epoch = getenv("SOURCE_DATE_EPOCH");
>  > +	if (!source_date_epoch)
>  > +		return (time_t) -1;
>  > +
>  > +	errno = 0;
>  > +	epoch = strtoll (source_date_epoch, &endptr, 10);
> 
> NIT: No space between strtoll and '('.
> 
> > +	if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
>  > +			|| (errno != 0 && epoch == 0)) {
>  > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
>  > +				"strtoll: %s\n", strerror(errno));
>  > +		exit(2);
>  > +	}
>  > +	if (endptr == source_date_epoch) {
>  > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
>  > +				"no digits were found: %s\n", endptr);
>  > +		exit(2);
>  > +	}
>  > +	if (*endptr != '\0') {
>  > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
>  > +				"trailing garbage: %s\n", endptr);
>  > +		exit(2);
>  > +	}
>  > +	if (epoch < 0) {
>  > +		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
>  > +				"value must be nonnegative: %lld \n", epoch);
>  > +		exit(2);
>  > +	}
> 
> I'm not sure this detailed error handling is really needed, but OK.

In fact I copy-paste get_source_date_epoch() from gcc git. So toolchain
wrapper will produce exactly same error messages than upstream gcc
(and it include same coding style error :-) ).



-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Peter Korsgaard]

>>>>> "J?r?me" == J?r?me Pouiller <jezz@sysmic.org> writes:

Hi,

 >> I'm not sure this detailed error handling is really needed, but OK.

 > In fact I copy-paste get_source_date_epoch() from gcc git. So toolchain
 > wrapper will produce exactly same error messages than upstream gcc
 > (and it include same coding style error :-) ).

Yes, I noticed that later on as well (which is why I added the reference
to gcc git).

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Thomas Petazzoni]

Hello,

On Wed, 08 Feb 2017 11:07:07 +0100, J?r?me Pouiller wrote:

> > I'm not sure this detailed error handling is really needed, but OK.  
> 
> In fact I copy-paste get_source_date_epoch() from gcc git. So toolchain
> wrapper will produce exactly same error messages than upstream gcc
> (and it include same coding style error :-) ).

So now our toolchain-wrapper is under GPLv3 :-/

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

 > Hello,
 > On Wed, 08 Feb 2017 11:07:07 +0100, J?r?me Pouiller wrote:

 >> > I'm not sure this detailed error handling is really needed, but OK.  
 >> 
 >> In fact I copy-paste get_source_date_epoch() from gcc git. So toolchain
 >> wrapper will produce exactly same error messages than upstream gcc
 >> (and it include same coding style error :-) ).

 > So now our toolchain-wrapper is under GPLv3 :-/

Yes, if the function is considered large and and original enough to be
considered copyrightable - Afterall it is just a getenv + strtoll and a
bit of error checking.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Jérôme Pouiller]

On Wednesday 08 February 2017 14:46:09 Peter Korsgaard wrote:
> Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:
>  > On Wed, 08 Feb 2017 11:07:07 +0100, J?r?me Pouiller wrote:
> 
>  >> > I'm not sure this detailed error handling is really needed, but 
OK.  
>  >> 
>  >> In fact I copy-paste get_source_date_epoch() from gcc git. So 
toolchain
>  >> wrapper will produce exactly same error messages than upstream gcc
>  >> (and it include same coding style error :-) ).
> 
>  > So now our toolchain-wrapper is under GPLv3 :-/

Indeed, I didn't noticed this problem.

> Yes, if the function is considered large and and original enough to be
> considered copyrightable - Afterall it is just a getenv + strtoll and 
a
> bit of error checking.

I am not a lawyer, but my code is clearly a "derivative work" of 
get_source_date_epoch() from gcc.

What do you prefer?

  1. Revert and rewrite function: it is not a big amount of work.

  2. Postpone problem: Buildroot also contains patches under GPLv3 and
     anyway, users will use a GPLv3 compiler.


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Peter Korsgaard]

>>>>> "J?r?me" == J?r?me Pouiller <jezz@sysmic.org> writes:


 >> Yes, if the function is considered large and and original enough to be
 >> considered copyrightable - Afterall it is just a getenv + strtoll and a
 >> bit of error checking.

 > I am not a lawyer, but my code is clearly a "derivative work" of 
 > get_source_date_epoch() from gcc.

Neither am I. I guess the question is if that function contains
"substantial amount of creative work" to make it copyrightable.


 > What do you prefer?

 >   1. Revert and rewrite function: it is not a big amount of work.

But it is hard to claim that your reimplementation isn't a derrived work
of get_source_date_epoch() - I mean, there is a limit to how many
different ways you can write getenv + strtoll and a bit of error
handling.


 >   2. Postpone problem: Buildroot also contains patches under GPLv3 and
 >      anyway, users will use a GPLv3 compiler.

I think that is the most sensible thing to do. What do others say?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v5 01/19] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Thomas Petazzoni]

Hello,

On Wed, 08 Feb 2017 15:29:27 +0100, Peter Korsgaard wrote:

>  >   2. Postpone problem: Buildroot also contains patches under GPLv3 and
>  >      anyway, users will use a GPLv3 compiler.  
> 
> I think that is the most sensible thing to do. What do others say?

Indeed, gcc itself, which is called by this wrapper, is under GPLv3. So
it's not a big deal if the wrapper is GPLv3 I believe.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 02/19] fakedate: new package

[Jérôme Pouiller]

`date' is widely used by packages to include build information in their
binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.

Instead of having to identify all `date' invocations in the different
packages, this commit adds a small tool that allows to always return
the same date.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Do not force $PATH. Find true `date' binary instead (Thomas)
      - Add explanations (Thomas)
      - Fix options detection (Arnout)
      - Rename INHIBIT in FORCE_EPOCH (Thomas)
      - Break after FORCE_EPOCH=0 (Arnout)
      - Add support for '--reference'
      - Add full copyright blurb (Arnout)

 package/fakedate/fakedate    | 59 ++++++++++++++++++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk | 15 +++++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

diff --git a/package/fakedate/fakedate b/package/fakedate/fakedate
new file mode 100755
index 0000000..074c517
--- /dev/null
+++ b/package/fakedate/fakedate
@@ -0,0 +1,59 @@
+#!/bin/sh
+# vim: set sw=4 expandtab:
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+# Copyright (C) 2016 J?r?me Pouiller <jezz@sysmic.org>
+#
+LOG=/dev/null
+
+# Sanity check
+if ! readlink -f "$0" | grep -q fakedate; then
+    echo "fakedate: Please name this script \`fakedate'"
+    exit 1
+fi
+
+DATE_BIN=false
+# Do not call `date'd directly since it will produce an infinite recursion.
+# Instead, find path of true `date' binary.
+for P in `echo $PATH | tr ':' ' '`; do
+    if [ -x "$P/date" ]; then
+        if readlink -f "$P/date" | grep -qv fakedate; then
+            DATE_BIN="$P/date"
+            break;
+        fi
+    fi
+done
+
+if [ -n "$SOURCE_DATE_EPOCH" ]; then
+    FORCE_EPOCH=1
+    for i in "$@"; do
+        # Use of --date, --file and --reference (and their short option counter
+        # parts) is incompatible with SOURCE_DATE_EPOCH.
+        # -u and -R are the only short options without argument. So they could
+        # appear between '-' and option we want to match.
+        if echo "$i" | grep -qE '^-([uR]*d|-date|[uR]*f|-file|[uR]*r|--reference)'; then
+            FORCE_EPOCH=0
+            break;
+        fi
+    done
+    if [ $FORCE_EPOCH -eq 1 ]; then
+        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
+        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
+        exec $DATE_BIN -d "@$SOURCE_DATE_EPOCH" "$@"
+    fi
+fi
+
+exec $DATE_BIN "$@"
diff --git a/package/fakedate/fakedate.mk b/package/fakedate/fakedate.mk
new file mode 100644
index 0000000..61d4bd7
--- /dev/null
+++ b/package/fakedate/fakedate.mk
@@ -0,0 +1,15 @@
+################################################################################
+#
+# fakedate
+#
+################################################################################
+
+# source included in buildroot
+HOST_FAKEDATE_LICENSE = GPLv2+
+
+define HOST_FAKEDATE_INSTALL_CMDS
+	$(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/fakedate
+	ln -sfn fakedate $(HOST_DIR)/usr/bin/date
+endef
+
+$(eval $(host-generic-package))
-- 
1.9.1

[Buildroot] [PATCH v5 02/19] fakedate: new package

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> `date' is widely used by packages to include build information in their
> binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.
>
> Instead of having to identify all `date' invocations in the different
> packages, this commit adds a small tool that allows to always return
> the same date.
>
> This work was sponsored by `BA Robotic Systems'.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 02/19] fakedate: new package

[Peter Korsgaard]

>>>>> "J?r?me" == J?r?me Pouiller <jezz@sysmic.org> writes:

 > `date' is widely used by packages to include build information in their
 > binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.

 > Instead of having to identify all `date' invocations in the different
 > packages, this commit adds a small tool that allows to always return
 > the same date.

 > This work was sponsored by `BA Robotic Systems'.

 > Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
 > ---

 > Notes:
 >     v3:
 >       - Do not force $PATH. Find true `date' binary instead (Thomas)
 >       - Add explanations (Thomas)
 >       - Fix options detection (Arnout)
 >       - Rename INHIBIT in FORCE_EPOCH (Thomas)
 >       - Break after FORCE_EPOCH=0 (Arnout)
 >       - Add support for '--reference'
 >       - Add full copyright blurb (Arnout)

Would using something like libfaketime not be a more generic solution?

https://github.com/wolfcw/libfaketime

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v5 02/19] fakedate: new package

[Jérôme Pouiller]

Hello Peter,

On Tuesday 07 February 2017 21:49:30 Peter Korsgaard wrote:
> >>>>> "J?r?me" == J?r?me Pouiller <jezz@sysmic.org> writes:
> 
>  > `date' is widely used by packages to include build information in 
their
>  > binaries. Unfortunately, this is incompatible with  
BR2_REPRODUCIBLE.
> 
>  > Instead of having to identify all `date' invocations in the 
different
>  > packages, this commit adds a small tool that allows to always 
return
>  > the same date.
[...]

> Would using something like libfaketime not be a more generic solution?
> 
> https://github.com/wolfcw/libfaketime

You mean enabling something like libfaketime for whole compile process?

I think it would break compilation process. In add, I prefer to have a 
very tight control over necessary conditions to override 'date' output.

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v5 02/19] fakedate: new package

[Peter Korsgaard]

>>>>> "J?r?me" == J?r?me Pouiller <jezz@sysmic.org> writes:

 > `date' is widely used by packages to include build information in their
 > binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.

 > Instead of having to identify all `date' invocations in the different
 > packages, this commit adds a small tool that allows to always return
 > the same date.

 > This work was sponsored by `BA Robotic Systems'.

 > Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
 > ---

 > Notes:
 >     v3:
 >       - Do not force $PATH. Find true `date' binary instead (Thomas)
 >       - Add explanations (Thomas)
 >       - Fix options detection (Arnout)
 >       - Rename INHIBIT in FORCE_EPOCH (Thomas)
 >       - Break after FORCE_EPOCH=0 (Arnout)
 >       - Add support for '--reference'
 >       - Add full copyright blurb (Arnout)

 >  package/fakedate/fakedate    | 59 ++++++++++++++++++++++++++++++++++++++++++++
 >  package/fakedate/fakedate.mk | 15 +++++++++++
 >  2 files changed, 74 insertions(+)
 >  create mode 100755 package/fakedate/fakedate
 >  create mode 100644 package/fakedate/fakedate.mk

 > diff --git a/package/fakedate/fakedate b/package/fakedate/fakedate
 > new file mode 100755
 > index 0000000..074c517
 > --- /dev/null
 > +++ b/package/fakedate/fakedate
 > @@ -0,0 +1,59 @@
 > +#!/bin/sh
 > +# vim: set sw=4 expandtab:
 > +#
 > +# This program is free software; you can redistribute it and/or modify
 > +# it under the terms of the GNU General Public License as published by
 > +# the Free Software Foundation; either version 2 of the License, or
 > +# (at your option) any later version.
 > +#
 > +# This program is distributed in the hope that it will be useful,
 > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
 > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
 > +# General Public License for more details.
 > +#
 > +# You should have received a copy of the GNU General Public License
 > +# along with this program; if not, write to the Free Software
 > +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 > +#
 > +# Copyright (C) 2016 J?r?me Pouiller <jezz@sysmic.org>
 > +#
 > +LOG=/dev/null
 > +
 > +# Sanity check
 > +if ! readlink -f "$0" | grep -q fakedate; then
 > +    echo "fakedate: Please name this script \`fakedate'"
 > +    exit 1
 > +fi
 > +
 > +DATE_BIN=false
 > +# Do not call `date'd directly since it will produce an infinite recursion.
 > +# Instead, find path of true `date' binary.
 > +for P in `echo $PATH | tr ':' ' '`; do
 > +    if [ -x "$P/date" ]; then
 > +        if readlink -f "$P/date" | grep -qv fakedate; then
 > +            DATE_BIN="$P/date"
 > +            break;
 > +        fi
 > +    fi
 > +done
 > +
 > +if [ -n "$SOURCE_DATE_EPOCH" ]; then
 > +    FORCE_EPOCH=1
 > +    for i in "$@"; do
 > +        # Use of --date, --file and --reference (and their short option counter
 > +        # parts) is incompatible with SOURCE_DATE_EPOCH.
 > +        # -u and -R are the only short options without argument. So they could
 > +        # appear between '-' and option we want to match.
 > +        if echo "$i" | grep -qE '^-([uR]*d|-date|[uR]*f|-file|[uR]*r|--reference)'; then
 > +            FORCE_EPOCH=0
 > +            break;
 > +        fi
 > +    done
 > +    if [ $FORCE_EPOCH -eq 1 ]; then
 > +        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
 > +        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG

I don't think this LOG debug stuff makes much sense as you have to edit the
file anyway, so you can just add any adhoc debug statements on the fly
anyway.

Committed with that removed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v5 03/19] core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk

[Jérôme Pouiller]

Usually, Buildroot does never initialize variables with empty content.
DEPENDENCIES_HOST_PREREQ was an unjustified exception.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - New patch

 support/dependencies/dependencies.mk | 2 --
 1 file changed, 2 deletions(-)

diff --git a/support/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
index 4334dac..d4b0409 100644
--- a/support/dependencies/dependencies.mk
+++ b/support/dependencies/dependencies.mk
@@ -5,8 +5,6 @@
 #
 ################################################################################
 
-DEPENDENCIES_HOST_PREREQ :=
-
 # suitable-host-pkg: calls check-host-$(1).sh shell script. Parameter (2)
 # can be the candidate to be checked. If not present, the check-host-$(1).sh
 # script should use 'which' to find a candidate. The script should return
-- 
1.9.1

[Buildroot] [PATCH v5 03/19] core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:20 +0100, J?r?me Pouiller wrote:
> Usually, Buildroot does never initialize variables with empty content.
> DEPENDENCIES_HOST_PREREQ was an unjustified exception.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v5 04/19] reproducible: enable fakedate

[Jérôme Pouiller]

Enable fakedate for whole build process.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Build host-fakedate before toolchain (Thomas)

 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index b80b9e1..ad7fde5 100644
--- a/Makefile
+++ b/Makefile
@@ -254,6 +254,7 @@ export LC_ALL = C
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
+DEPENDENCIES_HOST_PREREQ += host-fakedate
 endif
 
 # To put more focus on warnings, be less verbose as default
-- 
1.9.1

[Buildroot] [PATCH v5 04/19] reproducible: enable fakedate

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> Enable fakedate for whole build process.
>
> This work was sponsored by `BA Robotic Systems'.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>
> Notes:
>     v3:
>       - Build host-fakedate before toolchain (Thomas)
>
>  Makefile | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/Makefile b/Makefile
> index b80b9e1..ad7fde5 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -254,6 +254,7 @@ export LC_ALL = C
>  export GZIP = -n
>  BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
>  export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
> +DEPENDENCIES_HOST_PREREQ += host-fakedate
>  endif
>
>  # To put more focus on warnings, be less verbose as default
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 04/19] reproducible: enable fakedate

[Peter Korsgaard]

>>>>> "J?r?me" == J?r?me Pouiller <jezz@sysmic.org> writes:

 > Enable fakedate for whole build process.
 > This work was sponsored by `BA Robotic Systems'.

 > Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
 > ---

 > Notes:
 >     v3:
 >       - Build host-fakedate before toolchain (Thomas)

Committed, thanks.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v5 05/19] pycompile: allow to force compilation

[Jérôme Pouiller]

By default, compile_dir() rely on modification time to know if a python file
have to be built again. However in some circumstances (when doing reproducible
builds), modification times are not reliable. Thus, this patch add a way to
force rebuild of all python sources.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v5:
      - Improve commit log
    v3:
      - new patch

 support/scripts/pycompile.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/support/scripts/pycompile.py b/support/scripts/pycompile.py
index fde711a..5598f8a 100644
--- a/support/scripts/pycompile.py
+++ b/support/scripts/pycompile.py
@@ -10,6 +10,7 @@ from __future__ import print_function
 import sys
 import py_compile
 import compileall
+import argparse
 
 class ReportProblem:
     def __nonzero__(self):
@@ -21,4 +22,12 @@ class ReportProblem:
 
 report_problem = ReportProblem()
 
-compileall.compile_dir(sys.argv[1], quiet=report_problem)
+parser = argparse.ArgumentParser(description='Compile Python source files in a directory tree.')
+parser.add_argument("target", metavar='DIRECTORY',
+        help='Directory to scan')
+parser.add_argument("--force", action='store_true',
+        help="Force compilation even if alread compiled")
+
+args = parser.parse_args()
+
+compileall.compile_dir(args.target, force=args.force, quiet=report_problem)
-- 
1.9.1

[Buildroot] [PATCH v5 05/19] pycompile: allow to force compilation

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> By default, compile_dir() rely on modification time to know if a python file
> have to be built again. However in some circumstances (when doing reproducible
> builds), modification times are not reliable. Thus, this patch add a way to
> force rebuild of all python sources.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>
> Notes:
>     v5:
>       - Improve commit log
>     v3:
>       - new patch
>
>  support/scripts/pycompile.py | 11 ++++++++++-
>  1 file changed, 10 insertions(+), 1 deletion(-)
>
> diff --git a/support/scripts/pycompile.py b/support/scripts/pycompile.py
> index fde711a..5598f8a 100644
> --- a/support/scripts/pycompile.py
> +++ b/support/scripts/pycompile.py
> @@ -10,6 +10,7 @@ from __future__ import print_function
>  import sys
>  import py_compile
>  import compileall
> +import argparse
>
>  class ReportProblem:
>      def __nonzero__(self):
> @@ -21,4 +22,12 @@ class ReportProblem:
>
>  report_problem = ReportProblem()
>
> -compileall.compile_dir(sys.argv[1], quiet=report_problem)
> +parser = argparse.ArgumentParser(description='Compile Python source files in a directory tree.')
> +parser.add_argument("target", metavar='DIRECTORY',
> +        help='Directory to scan')
> +parser.add_argument("--force", action='store_true',
> +        help="Force compilation even if alread compiled")
> +
> +args = parser.parse_args()
> +
> +compileall.compile_dir(args.target, force=args.force, quiet=report_problem)
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 05/19] pycompile: allow to force compilation

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:22 +0100, J?r?me Pouiller wrote:
> By default, compile_dir() rely on modification time to know if a python file
> have to be built again. However in some circumstances (when doing reproducible
> builds), modification times are not reliable. Thus, this patch add a way to
> force rebuild of all python sources.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v5 06/19] python2: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before compiling .pyc
files.

In add, since pycompile rely on modification times to know if a file need to be
recompiled, it is safer to force recompilation of all source files.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v5:
      - Typo in commit log
      - Explain why we pass --force to pycompile
      - Fix indent
    v3:
      - Force compilation instead of removing .pyc

 package/python/python.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index cfd5a0a..cbe7c19 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -233,10 +233,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON_FIX_TIME
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
+endif
+
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
+		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
 		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
-- 
1.9.1

[Buildroot] [PATCH v5 06/19] python2: generate reproducible .pyc

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> .pyc files contain modification time of .py source. In order to make
> build reproducible, we fix modification time of all .py before compiling .pyc
> files.
>
> In add, since pycompile rely on modification times to know if a file need to be
> recompiled, it is safer to force recompilation of all source files.
>
> This work was sponsored by `BA Robotic Systems'.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>
> Notes:
>     v5:
>       - Typo in commit log
>       - Explain why we pass --force to pycompile
>       - Fix indent
>     v3:
>       - Force compilation instead of removing .pyc
>
>  package/python/python.mk | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/package/python/python.mk b/package/python/python.mk
> index cfd5a0a..cbe7c19 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -233,10 +233,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define PYTHON_FIX_TIME
> +       find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
> +               xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
> +endef
> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
> +endif
> +
>  define PYTHON_CREATE_PYC_FILES
>         PYTHONPATH="$(PYTHON_PATH)" \
>         $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> -               support/scripts/pycompile.py \
> +               support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
>                 $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
>  endef
>
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 06/19] python2: generate reproducible .pyc

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:23 +0100, J?r?me Pouiller wrote:
> .pyc files contain modification time of .py source. In order to make
> build reproducible, we fix modification time of all .py before compiling .pyc
> files.
> 
> In add, since pycompile rely on modification times to know if a file need to be
> recompiled, it is safer to force recompilation of all source files.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

I've adjusted the commit log to fix various typoes.

> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define PYTHON_FIX_TIME
> +	find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
> +		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
> +endef
> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME

And instead of calling this through PYTHON_TARGET_FINALIZE_HOOKS, I
called it explicitly in PYTHON_CREATE_PYC_FILES, before doing the byte
compilation. The two are very closely related so I believe it made more
sense this way.

Applied with those changes. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 07/19] python3: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make build
reproducible, we fix modification time of all .py before compiling .pyc files.

In add, since pycompile rely on modification times to know if a file need to be
recompiled, it is safer to force recompilation of all source files.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v5:
      - Typo in commit log
      - Explain why we pass --force to pycompile
    v3:
      - Force compilation instead of removing .pyc

 package/python3/python3.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index 8bfae79..1817b22 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -227,10 +227,18 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON3_FIX_TIME
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME
+endif
+
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
+		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
 		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
-- 
1.9.1

[Buildroot] [PATCH v5 07/19] python3: generate reproducible .pyc

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> .pyc files contain modification time of .py source. In order to make build
> reproducible, we fix modification time of all .py before compiling .pyc files.
>
> In add, since pycompile rely on modification times to know if a file need to be
> recompiled, it is safer to force recompilation of all source files.
>
> This work was sponsored by `BA Robotic Systems'.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>
> Notes:
>     v5:
>       - Typo in commit log
>       - Explain why we pass --force to pycompile
>     v3:
>       - Force compilation instead of removing .pyc
>
>  package/python3/python3.mk | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/package/python3/python3.mk b/package/python3/python3.mk
> index 8bfae79..1817b22 100644
> --- a/package/python3/python3.mk
> +++ b/package/python3/python3.mk
> @@ -227,10 +227,18 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define PYTHON3_FIX_TIME
> +       find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
> +               xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
> +endef
> +PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME
> +endif
> +
>  define PYTHON3_CREATE_PYC_FILES
>         PYTHONPATH="$(PYTHON3_PATH)" \
>         $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
> -               support/scripts/pycompile.py \
> +               support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
>                 $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
>  endef
>
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 07/19] python3: generate reproducible .pyc

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:24 +0100, J?r?me Pouiller wrote:
> .pyc files contain modification time of .py source. In order to make build
> reproducible, we fix modification time of all .py before compiling .pyc files.
> 
> In add, since pycompile rely on modification times to know if a file need to be
> recompiled, it is safer to force recompilation of all source files.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---

Applied with the same changes as for PATCH 06/19.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v5 08/19] reproducible: try to detect most common errors

[Jérôme Pouiller]

Some package includes some information from build environment in their results.
This practice is incompatible with reproducible builds.

This patch scans final target to research most common patterns.

Since we only search fixed strings (grep is called with -F), this search is
fast (on my station, 60ms for a target of 40MB).

Note, it could be a good idea to also match current user name. However, build
path often contains username and, until now, we do not try to avoid build path
in result.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/Makefile b/Makefile
index ad7fde5..5b504c1 100644
--- a/Makefile
+++ b/Makefile
@@ -708,6 +708,11 @@ endif
 		$(call MESSAGE,"Executing post-build script $(s)"); \
 		$(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+	grep -raoF -e "`uname -r`" -e "`uname -v`" -e "`uname -n`" $(TARGET_DIR) | \
+		sed 's/\(.*\):\(.*\)/Warning: \1 may contain unreproducible information: \2/' | uniq
+endif
+
 target-post-image: $(TARGETS_ROOTFS) target-finalize
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
-- 
1.9.1

[Buildroot] [PATCH v5 08/19] reproducible: try to detect most common errors

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> Some package includes some information from build environment in their results.
> This practice is incompatible with reproducible builds.
>
> This patch scans final target to research most common patterns.
>
> Since we only search fixed strings (grep is called with -F), this search is
> fast (on my station, 60ms for a target of 40MB).
>
> Note, it could be a good idea to also match current user name. However, build
> path often contains username and, until now, we do not try to avoid build path
> in result.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>  Makefile | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index ad7fde5..5b504c1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -708,6 +708,11 @@ endif
>                 $(call MESSAGE,"Executing post-build script $(s)"); \
>                 $(EXTRA_ENV) $(s) $(TARGET_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
>
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +       grep -raoF -e "`uname -r`" -e "`uname -v`" -e "`uname -n`" $(TARGET_DIR) | \
> +               sed 's/\(.*\):\(.*\)/Warning: \1 may contain unreproducible information: \2/' | uniq
> +endif
> +
>  target-post-image: $(TARGETS_ROOTFS) target-finalize
>         @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
>                 $(call MESSAGE,"Executing post-image script $(s)"); \
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 08/19] reproducible: try to detect most common errors

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:25 +0100, J?r?me Pouiller wrote:
> Some package includes some information from build environment in their results.
> This practice is incompatible with reproducible builds.
> 
> This patch scans final target to research most common patterns.
> 
> Since we only search fixed strings (grep is called with -F), this search is
> fast (on my station, 60ms for a target of 40MB).
> 
> Note, it could be a good idea to also match current user name. However, build
> path often contains username and, until now, we do not try to avoid build path
> in result.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

I am not entirely convinced it makes sense to grep more or less
randomly in all files specifically for those strings.

Or perhaps, we should have a separate shell scripts that checks for
several classes of obviously non-reproducible behaviors?

Arnout, Peter, Yann, what do you think?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 08/19] reproducible: try to detect most common errors

[Yann E. MORIN]

Thomas, J?r?me, All,

On 2017-04-01 16:50 +0200, Thomas Petazzoni spake thusly:
> On Tue, 20 Dec 2016 14:46:25 +0100, J?r?me Pouiller wrote:
> > Some package includes some information from build environment in their results.
> > This practice is incompatible with reproducible builds.
> > 
> > This patch scans final target to research most common patterns.
> > 
> > Since we only search fixed strings (grep is called with -F), this search is
> > fast (on my station, 60ms for a target of 40MB).
> > 
> > Note, it could be a good idea to also match current user name. However, build
> > path often contains username and, until now, we do not try to avoid build path
> > in result.
> > 
> > Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> 
> I am not entirely convinced it makes sense to grep more or less
> randomly in all files specifically for those strings.

Indeed. My hostnames usually cary some computer-related keywords, which
can very well occur in binary files (e.g. 'segfault').

> Or perhaps, we should have a separate shell scripts that checks for
> several classes of obviously non-reproducible behaviors?

Indeed, but such a script can only be a helper that a user would
voluntarily run on their own.

We can't have such a script automatically run, because those strings can
be too common.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH v5 08/19] reproducible: try to detect most common errors

[Arnout Vandecappelle]

On 01-04-17 23:13, Yann E. MORIN wrote:
> Thomas, J?r?me, All,
> 
> On 2017-04-01 16:50 +0200, Thomas Petazzoni spake thusly:
>> On Tue, 20 Dec 2016 14:46:25 +0100, J?r?me Pouiller wrote:
>>> Some package includes some information from build environment in their results.
>>> This practice is incompatible with reproducible builds.
>>>
>>> This patch scans final target to research most common patterns.
>>>
>>> Since we only search fixed strings (grep is called with -F), this search is
>>> fast (on my station, 60ms for a target of 40MB).
>>>
>>> Note, it could be a good idea to also match current user name. However, build
>>> path often contains username and, until now, we do not try to avoid build path
>>> in result.
>>>
>>> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
>>
>> I am not entirely convinced it makes sense to grep more or less
>> randomly in all files specifically for those strings.
> 
> Indeed. My hostnames usually cary some computer-related keywords, which
> can very well occur in binary files (e.g. 'segfault').

 Even more so for uname -r, which could be something like 4.10...

> 
>> Or perhaps, we should have a separate shell scripts that checks for
>> several classes of obviously non-reproducible behaviors?
> 
> Indeed, but such a script can only be a helper that a user would
> voluntarily run on their own.
> 
> We can't have such a script automatically run, because those strings can
> be too common.

 Well, as proposed by Jerome, it would just print a warning at the end of the
build, which is probably acceptable.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 09/19] python2: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch changes the way
`pycompile.py' is launched in order to only keep the part relative to
$TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---

Notes:
    v3:
      - Typo in commit log (Arnout)

 package/python/python.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index cbe7c19..18ab14c 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -243,9 +243,10 @@ endif
 
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		$(if $(BR2_REPRODUCIBLE),--force) \
+		usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH v5 09/19] python2: remove full path from .pyc

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> .pyc files include path to source .py file. This patch changes the way
> `pycompile.py' is launched in order to only keep the part relative to
> $TARGET_DIR.
>
> This work was sponsored by `BA Robotic Systems'.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>
> Notes:
>     v3:
>       - Typo in commit log (Arnout)
>
>  package/python/python.mk | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/package/python/python.mk b/package/python/python.mk
> index cbe7c19..18ab14c 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -243,9 +243,10 @@ endif
>
>  define PYTHON_CREATE_PYC_FILES
>         PYTHONPATH="$(PYTHON_PATH)" \
> -       $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> -               support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
> -               $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
> +       cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> +               $(TOPDIR)/support/scripts/pycompile.py \
> +               $(if $(BR2_REPRODUCIBLE),--force) \
> +               usr/lib/python$(PYTHON_VERSION_MAJOR)
>  endef
>
>  ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 09/19] python2: remove full path from .pyc

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:26 +0100, J?r?me Pouiller wrote:
> .pyc files include path to source .py file. This patch changes the way
> `pycompile.py' is launched in order to only keep the part relative to
> $TARGET_DIR.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v5 10/19] python3: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch changes the way
`pycompile.py' is launched in order to only keep the part relative to
$TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---

Notes:
    v3:
      - Typo in commit log (Arnout)

 package/python3/python3.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index 1817b22..2562be7 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -237,9 +237,10 @@ endif
 
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		$(if $(BR2_REPRODUCIBLE),--force) \
+		usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH v5 10/19] python3: remove full path from .pyc

[Samuel Martin]

Hi J?r?me, all,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> .pyc files include path to source .py file. This patch changes the way
> `pycompile.py' is launched in order to only keep the part relative to
> $TARGET_DIR.
>
> This work was sponsored by `BA Robotic Systems'.
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Reviewed-by: Samuel Martin <s.martin49@gmail.com>

> ---
>
> Notes:
>     v3:
>       - Typo in commit log (Arnout)
>
>  package/python3/python3.mk | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/package/python3/python3.mk b/package/python3/python3.mk
> index 1817b22..2562be7 100644
> --- a/package/python3/python3.mk
> +++ b/package/python3/python3.mk
> @@ -237,9 +237,10 @@ endif
>
>  define PYTHON3_CREATE_PYC_FILES
>         PYTHONPATH="$(PYTHON3_PATH)" \
> -       $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
> -               support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
> -               $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
> +       cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
> +               $(TOPDIR)/support/scripts/pycompile.py \
> +               $(if $(BR2_REPRODUCIBLE),--force) \
> +               usr/lib/python$(PYTHON3_VERSION_MAJOR)
>  endef
>
>  ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 10/19] python3: remove full path from .pyc

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:27 +0100, J?r?me Pouiller wrote:
> .pyc files include path to source .py file. This patch changes the way
> `pycompile.py' is launched in order to only keep the part relative to
> $TARGET_DIR.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Jérôme Pouiller]

libtool include support for cross-compilation since v1.5. In order to enable it,
we have to pass `--with-sysroot' to `configure' script. With this option,
autoconf fill `$lt_sysroot' and libtool will search libraries in `$lt_sysroot'
subtree.

Note Yocto force `$lt_sysroot' even if --with-sysroot is not provided:
  http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/fix-resolve-lt-sysroot.patch

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/pkg-autotools.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index f7b8488..bb20eec 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -192,6 +192,7 @@ define $(2)_CONFIGURE_CMDS
 		--sysconfdir=/etc \
 		--localstatedir=/var \
 		--program-prefix="" \
+		--with-sysroot \
 		--disable-gtk-doc \
 		--disable-gtk-doc-html \
 		--disable-doc \
-- 
1.9.1

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Arnout Vandecappelle]

 Hi Jerome,

 Please wrap your commit messages at 72 columns (vim wrap will do that for you :-).

On 20-12-16 14:46, J?r?me Pouiller wrote:
> libtool include support for cross-compilation since v1.5. In order to enable it,
                 ^s

> we have to pass `--with-sysroot' to `configure' script. With this option,
                                     ^the

> autoconf fill `$lt_sysroot' and libtool will search libraries in `$lt_sysroot'
               ^s                                                 ^the

> subtree.
> 
> Note Yocto force `$lt_sysroot' even if --with-sysroot is not provided:
      ^:          ^s
>   http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/fix-resolve-lt-sysroot.patch

 What a horrible hack is that patch, turning 'yes' into 'no'...

> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)


 Regards,
 Arnout

> ---
>  package/pkg-autotools.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
> index f7b8488..bb20eec 100644
> --- a/package/pkg-autotools.mk
> +++ b/package/pkg-autotools.mk
> @@ -192,6 +192,7 @@ define $(2)_CONFIGURE_CMDS
>  		--sysconfdir=/etc \
>  		--localstatedir=/var \
>  		--program-prefix="" \
> +		--with-sysroot \
>  		--disable-gtk-doc \
>  		--disable-gtk-doc-html \
>  		--disable-doc \
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Thomas Petazzoni]

Hello,

On Tue, 20 Dec 2016 14:46:28 +0100, J?r?me Pouiller wrote:

> libtool include support for cross-compilation since v1.5.

Are you sure this is true?

In the release notes of libtool v2.4
(https://savannah.gnu.org/forum/forum.php?forum_id=6525), I see:

"""


New in 2.4 2010-09-22:

    New features:

- Sysroot support. This allows you to build cross-compiled packages
with the same prefix that will be used on the destination machine,
and still find dependent libraries under the compiler's "sysroot".
Without sysroot support, paths internal to the build system may leak
into the product of the build.

Sysroot support is disabled unless the --with-sysroot configure
option is passed to configure, because .la files generated with
sysroot support will not be usable in general with older Libtools. 

"""

So are you sure that this feature didn't appear in libtool 2.4 ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Arnout Vandecappelle]

On 20-03-17 22:52, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 20 Dec 2016 14:46:28 +0100, J?r?me Pouiller wrote:
> 
>> libtool include support for cross-compilation since v1.5.
> 
> Are you sure this is true?
> 
> In the release notes of libtool v2.4
> (https://savannah.gnu.org/forum/forum.php?forum_id=6525), I see:
> 
> """
> 
> 
> New in 2.4 2010-09-22:
> 
>     New features:
> 
> - Sysroot support. This allows you to build cross-compiled packages
> with the same prefix that will be used on the destination machine,
> and still find dependent libraries under the compiler's "sysroot".
> Without sysroot support, paths internal to the build system may leak
> into the product of the build.
> 
> Sysroot support is disabled unless the --with-sysroot configure
> option is passed to configure, because .la files generated with
> sysroot support will not be usable in general with older Libtools. 
> 
> """
> 
> So are you sure that this feature didn't appear in libtool 2.4 ?

 Yeah, looks like it. E.g. speex has libtool 1.5, and it doesn't have any
sysroot support (not in configure, not in ltmain.sh).

 Obviously, *this* patch is not affected by it - it certainly doesn't hurt to
pass the --with-sysroot configure option. It's only patches 12+13 that we
wouldn't be able to remove.

 However, considering:

- most packages install things in /usr/lib, which is already searched by default;

- most packages use libtool 2.4 or later,

I think we can take the risk. The autobuilders will discover any issues, which
we can fix per package (e.g. with AUTORECONF = YES).

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Thomas Petazzoni]

Hello,

On Wed, 22 Mar 2017 00:57:39 +0100, Arnout Vandecappelle wrote:

> > So are you sure that this feature didn't appear in libtool 2.4 ?  
> 
>  Yeah, looks like it. E.g. speex has libtool 1.5, and it doesn't have any
> sysroot support (not in configure, not in ltmain.sh).
> 
>  Obviously, *this* patch is not affected by it - it certainly doesn't hurt to
> pass the --with-sysroot configure option. It's only patches 12+13 that we
> wouldn't be able to remove.
> 
>  However, considering:
> 
> - most packages install things in /usr/lib, which is already searched by default;
> 
> - most packages use libtool 2.4 or later,
> 
> I think we can take the risk. The autobuilders will discover any issues, which
> we can fix per package (e.g. with AUTORECONF = YES).

Since I would prefer to avoid autoreconfiguring many packages, it would
be useful to know how many packages are still using libtool versions
earlier than 2.4.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Arnout Vandecappelle]

On 22-03-17 09:14, Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 22 Mar 2017 00:57:39 +0100, Arnout Vandecappelle wrote:
> 
>>> So are you sure that this feature didn't appear in libtool 2.4 ?  
>>
>>  Yeah, looks like it. E.g. speex has libtool 1.5, and it doesn't have any
>> sysroot support (not in configure, not in ltmain.sh).
>>
>>  Obviously, *this* patch is not affected by it - it certainly doesn't hurt to
>> pass the --with-sysroot configure option. It's only patches 12+13 that we
>> wouldn't be able to remove.
>>
>>  However, considering:
>>
>> - most packages install things in /usr/lib, which is already searched by default;
>>
>> - most packages use libtool 2.4 or later,
>>
>> I think we can take the risk. The autobuilders will discover any issues, which
>> we can fix per package (e.g. with AUTORECONF = YES).
> 
> Since I would prefer to avoid autoreconfiguring many packages, it would
> be useful to know how many packages are still using libtool versions
> earlier than 2.4.

 I'm doing an allyesconfig download and will check the ltmain.sh in all the
tarballs. It will take some time before that finishes.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Jérôme Pouiller]

Hello,

On Wednesday 22 March 2017 09:14:37 CET Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 22 Mar 2017 00:57:39 +0100, Arnout Vandecappelle wrote:
> 
> > > So are you sure that this feature didn't appear in libtool 2.4 ?  
> > 
> >  Yeah, looks like it. E.g. speex has libtool 1.5, and it doesn't have any
> > sysroot support (not in configure, not in ltmain.sh).
> > 
> >  Obviously, *this* patch is not affected by it - it certainly doesn't hurt
> >  to pass the --with-sysroot configure option. It's only patches 12+13 that
> >  we wouldn't be able to remove.
> > 
> >  However, considering:
> > 
> > - most packages install things in /usr/lib, which is already searched by
> > default;
> > 
> > - most packages use libtool 2.4 or later,
> > 
> > I think we can take the risk. The autobuilders will discover any issues,
> > which we can fix per package (e.g. with AUTORECONF = YES).
> 
> Since I would prefer to avoid autoreconfiguring many packages, it would
> be useful to know how many packages are still using libtool versions
> earlier than 2.4.

This is precisely what I wanted to known before to replying.

I extracted all (or nearly all) packages existing in Buildroot. I found 269 
packages that use libool:

$ grep -ho 'VERSION="[0-9.]*' */ltmain.sh  | sort | uniq -c | sort -n
      1 VERSION="1.5.24
      1 VERSION="2.2.4
      3 VERSION="1.5.22
      5 VERSION="1.5.26
     13 VERSION="2.2.6
     29 VERSION="2.4.6
     67 VERSION="2.4.2

There 23 packages that use libtool < 2.4:
  - bellagio
  - celt051
  - dbus-cpp
  - eventlog
  - gtk2-engines
  - lesstif
  - libargtable2
  - libesmtp
  - liblo
  - libmhash
  - libnetfilter_conntrack
  - libnetfilter_log
  - libnfnetlink
  - libpthread-stubs
  - lite
  - log4cxx
  - matchbox-lib
  - metacity
  - sdl_sound
  - sylpheed
  - xapp_xinput-calibrator
  - xdriver_xf86-input-tslib
  - xdriver_xf86-video-imx

4 of them already call autoreconf: bellagio, dbus-cpp, libnetfilter_log, 
libnfnetlink.

I am going to create a configuration with all of these packages.

++

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Arnout Vandecappelle]

On 22-03-17 10:20, J?r?me Pouiller wrote:
> Hello,
> 
> On Wednesday 22 March 2017 09:14:37 CET Thomas Petazzoni wrote:
>> Hello,
>>
>> On Wed, 22 Mar 2017 00:57:39 +0100, Arnout Vandecappelle wrote:
>>
>>>> So are you sure that this feature didn't appear in libtool 2.4 ?  
>>>
>>>  Yeah, looks like it. E.g. speex has libtool 1.5, and it doesn't have any
>>> sysroot support (not in configure, not in ltmain.sh).
>>>
>>>  Obviously, *this* patch is not affected by it - it certainly doesn't hurt
>>>  to pass the --with-sysroot configure option. It's only patches 12+13 that
>>>  we wouldn't be able to remove.
>>>
>>>  However, considering:
>>>
>>> - most packages install things in /usr/lib, which is already searched by
>>> default;
>>>
>>> - most packages use libtool 2.4 or later,
>>>
>>> I think we can take the risk. The autobuilders will discover any issues,
>>> which we can fix per package (e.g. with AUTORECONF = YES).
>>
>> Since I would prefer to avoid autoreconfiguring many packages, it would
>> be useful to know how many packages are still using libtool versions
>> earlier than 2.4.
> 
> This is precisely what I wanted to known before to replying.
> 
> I extracted all (or nearly all) packages existing in Buildroot. I found 269 
> packages that use libool:
> 
> $ grep -ho 'VERSION="[0-9.]*' */ltmain.sh  | sort | uniq -c | sort -n
>       1 VERSION="1.5.24
>       1 VERSION="2.2.4
>       3 VERSION="1.5.22
>       5 VERSION="1.5.26
>      13 VERSION="2.2.6
>      29 VERSION="2.4.6
>      67 VERSION="2.4.2

 Your command isn't good enough... It skips ltmain.sh which are not in the top
directory, and it skips the ones where VERSION isn't quoted. I got quite a few more:

$ for i in *.tar*; do tar tf $i | grep ltmain.sh | xargs -r tar -xf $i
--to-command="grep ^VERSION= | { grep -v '[^.]2\.4' && printf ' @@$i\\r'; } | tr
'\\n\\r' ' \\n'"; done

VERSION=1.5.26  @@CEGUI-0.6.2b.tar.gz
VERSION="1.5.26 Debian 1.5.26-1ubuntu1"  @@LiTE-0.8.10.tar.gz
VERSION=2.2.6  @@SDL-1.2.15.tar.gz
VERSION=2.2.6  @@SDL2_image-2.0.1.tar.gz
VERSION=2.2.6  @@SDL2_ttf-2.0.14.tar.gz
VERSION=2.2.6  @@SDL_image-1.2.12.tar.gz
VERSION=2.2.6  @@SDL_mixer-1.2.12.tar.gz
VERSION=2.2.6  @@SDL_net-1.2.8.tar.gz
VERSION="1.5.24 Debian 1.5.24-1ubuntu1"  @@SDL_sound-1.0.3.tar.gz
VERSION=2.2.6  @@SDL_ttf-2.0.11.tar.gz
VERSION="1.5.22 Debian 1.5.22-2"  @@apache-log4cxx-0.10.0.tar.gz
VERSION=2.2.6b  @@arago-2011.09-armv5te-linux-gnueabi-sdk.tar.bz2
VERSION=2.2.6b  @@arago-2011.09-armv7a-linux-gnueabi-sdk.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@argtable2-13.tar.gz
VERSION=2.2.10  @@axfsutils-64886703fdff6cfc64eb20fbe7d7295bbeec867b.tar.gz
VERSION=2.2.6  @@beecrypt-4.2.1.tar.gz
VERSION=2.2.6b  @@bind-9.11.0-P3.tar.gz
VERSION=1.3.5  @@bind-9.11.0-P3.tar.gz
VERSION=2.2.7a  @@binutils-2.26.1.tar.bz2
VERSION=2.2.7a  @@binutils-2.27.tar.bz2
VERSION=1.5.22  @@boost_1_63_0.tar.bz2
VERSION="1.5.26 Debian 1.5.26-4"  @@celt-0.5.1.3.tar.gz
VERSION=1.5.22  @@clamav-0.99.2.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@collectd-5.5.1.tar.bz2
VERSION="1.5.26 Debian 1.5.26-1ubuntu1"  @@curlftpfs-0.9.2.tar.gz
VERSION=2.2.6b  @@db-5.3.28.NC.tar.gz
VERSION=2.2.6  @@db-5.3.28.NC.tar.gz
VERSION=1.5.22  @@db-5.3.28.NC.tar.gz
VERSION="2.2.6 Debian-2.2.6a-1ubuntu1"  @@eventlog_0.2.12.tar.gz
VERSION="1.5.22 Debian 1.5.22-4"  @@faad2-2.7.tar.gz
VERSION=1.4  @@fcgi-2.4.0.tar.gz
VERSION=1.5.24  @@gamin-0.1.10.tar.gz
VERSION=2.2.7a  @@gcc-4.9.4.tar.bz2
VERSION=2.2.7a  @@gcc-4.9.4.tar.bz2
VERSION=2.2.7a  @@gcc-4.9.4.tar.bz2
VERSION=1.5.16  @@gcc-4.9.4.tar.bz2
VERSION=2.2.7a  @@gcc-5.4.0.tar.bz2
VERSION=2.2.7a  @@gcc-5.4.0.tar.bz2
VERSION=2.2.7a  @@gcc-5.4.0.tar.bz2
VERSION=1.5.16  @@gcc-5.4.0.tar.bz2
VERSION=2.2.7a  @@gcc-6.2.0.tar.bz2
VERSION=2.2.7a  @@gcc-6.2.0.tar.bz2


VERSION=2.2.7a  @@gcc-6.2.0.tar.bz2


VERSION=1.5.16  @@gcc-6.2.0.tar.bz2


VERSION=2.2.7a  @@gdb-7.10.1.tar.xz
VERSION=2.2.7a  @@gdb-7.11.1.tar.xz
VERSION="1.5.26 Debian 1.5.26-4"  @@genpart-1.0.2.tar.bz2
VERSION=1.4e  @@gettext-0.19.8.1.tar.xz
VERSION="2.2.6b Debian-2.2.6b-2ubuntu3"  @@gmpc-11.8.16.tar.gz
VERSION=2.2.6b  @@google-breakpad-7515ab13768c7edc09f0f2ec2354dc6c928239a6.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2"  @@gtk-engines-2.20.2.tar.bz2
VERSION=1.5.2  @@ijs-0.35.tar.bz2
VERSION=1.5.20  @@ipkg-0.99.163.tar.gz
VERSION=1.5.22  @@jasper-1.900.1.tar.bz2
VERSION=2.2.6b  @@json-c-0.11.tar.gz
VERSION="1.5.26 Debian 1.5.26-4"  @@kodi-16.1-Jarvis.tar.gz
VERSION=2.2.10  @@lame-3.99.5.tar.gz
VERSION=2.2.6  @@libart_lgpl-2.3.21.tar.gz
VERSION=2.2.4  @@libass-0.13.1.tar.xz
VERSION=1.5.6  @@libcdaudio-0.99.12p2.tar.gz
VERSION=2.2.6  @@libcddb-1.3.2.tar.bz2
VERSION=2.2.6  @@libdaemon-0.14.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@libdbus-c++-0.9.0.tar.gz
VERSION=2.2.10  @@libdmtx-0.7.4.tar.gz
VERSION=1.5.6  @@libdnet-1.11.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2"  @@libeXosip2-3.6.0.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2ubuntu3"  @@libee-0.4.1.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@libesmtp-1.0.6.tar.bz2
VERSION=1.5.6  @@libfakekey-0.1.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@libgeotiff-1.4.0.tar.gz
VERSION=1.5.26  @@libglade-2.6.4.tar.bz2
VERSION=1.5.22  @@libhid-0.2.16.tar.gz
VERSION=1.5.2  @@libid3tag-0.15.1b.tar.gz
VERSION="2.2.4 Debian-2.2.4-0ubuntu4"  @@liblo-0.26.tar.gz
VERSION=1.5.22  @@liblog4c-localtime-v1.0.tar.gz
VERSION=1.5.2  @@libmad-0.15.1b.tar.gz
VERSION="1.5.22 Debian 1.5.22-2"  @@libmatchbox-1.9.tar.bz2
VERSION=1.5.22  @@libmcrypt-2.5.8.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2"  @@libmnl-1.0.3.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2ubuntu3"  @@libmpd-11.8.17.tar.gz
VERSION="1.5.26 Debian 1.5.26-4"  @@libmpeg2-0.5.1.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2"  @@libnetfilter_conntrack-1.0.6.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2"  @@libnetfilter_log-1.0.1.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2"  @@libnetfilter_log-1.0.1.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2"  @@libnfnetlink-1.0.1.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2"  @@libosip2-3.6.0.tar.gz
VERSION="2.2.6 Debian-2.2.6a-4"  @@libpthread-stubs-0.3.tar.bz2
VERSION=1.5.6  @@librsync-0.9.7.tar.gz
VERSION="1.5.22 Debian 1.5.22-2"  @@libsexy-0.1.11.tar.gz
VERSION=1.5.14  @@libsvg-0.1.4.tar.gz
VERSION=1.5.14  @@libsvg-cairo-0.1.6.tar.gz
VERSION=2.2.6  @@libtheora-1.1.1.tar.xz
VERSION=1.4.3  @@libunwind-1.1.tar.gz
VERSION=2.2.10  @@lzop-1.03.tar.gz
VERSION=1.5.2  @@madplay-0.15.2b.tar.gz
VERSION=1.5.6  @@mb-applet-startup-monitor-0.1.tar.bz2
VERSION="2.2.4 Debian-2.2.4-0ubuntu4"  @@mcrypt-2.6.8.tar.gz
VERSION=2.2.6b  @@msgpack-0.5.4.tar.gz
VERSION=2.2.6  @@mysql-5.1.73.tar.gz
VERSION=1.5.8  @@ngrep-1.45.tar.bz2
VERSION=1.5.26  @@php-7.1.3.tar.xz
VERSION=1.5.22  @@pjproject-2.4.5.tar.bz2
VERSION=2.2.6  @@popt-1.16.tar.gz
VERSION=1.3.4  @@privoxy-3.0.26-stable-src.tar.gz
VERSION="2.2.6 Debian-2.2.6a-1ubuntu1"  @@proftpd-1.3.5d.tar.gz
VERSION=1.5.26  @@proftpd-1.3.5d.tar.gz
VERSION="2.2.6 Debian-2.2.6a-1ubuntu1"  @@proftpd-1.3.5d.tar.gz
VERSION=2.2.6  @@pthsem_2.0.8.tar.gz
VERSION=2.2.6  @@qt-everywhere-opensource-src-4.8.7.tar.gz
VERSION=2.2.7a  @@rustc-1.10.0-src.tar.gz
VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
VERSION=1.4  @@sidplay-libs-2.1.1.tar.gz
VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@sofia-sip-1.12.11.tar.gz
VERSION="1.5.22 Debian 1.5.22-4"  @@speex-1.2rc1.tar.gz
VERSION=2.2.6  @@sqlcipher-v3.2.0.tar.gz
VERSION=1.5.6  @@startup-notification-0.9.tar.gz
VERSION=1.5.22  @@sysfsutils-2.1.0.tar.gz
VERSION=1.5.22  @@szip-2.1.tar.gz
VERSION=1.5.24  @@tn5250-0.17.4.tar.gz
VERSION=1.5.6  @@ucl-1.03.tar.gz
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@unixODBC-2.3.2.tar.gz
VERSION="2.2.6 Debian-2.2.6a-4"  @@vorbis-tools-1.4.0.tar.gz
VERSION=1.4.3  @@xcursor-transparent-theme-0.1.1.tar.gz
VERSION="1.5.26 Debian 1.5.26-4"  @@xf86-input-tslib-0.0.6.tar.bz2
VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@xinput_calibrator-0.7.5.tar.gz

 That's 109 tarballs with some older libtool.

 Note that this just iterates over the tarballs that I have in my download
directory, there may be some old versions of packages in there. Anyway, I think
109 out of the 2150 tarballs in that directory is still extremely small.

 Note also that in many of these, the ltmain will be in some subdirectory
containing bundled source that we don't even use.

 Note, finally, that there are also quite a few host-only packages in there
(e.g. gcc, but that one is special because it also builds stuff for the target.
It doesn't seem to install .la files, however).


> There 23 packages that use libtool < 2.4:
>   - bellagio
>   - celt051
>   - dbus-cpp
>   - eventlog
>   - gtk2-engines
>   - lesstif
>   - libargtable2
>   - libesmtp
>   - liblo
>   - libmhash
>   - libnetfilter_conntrack
>   - libnetfilter_log
>   - libnfnetlink
>   - libpthread-stubs
>   - lite
>   - log4cxx
>   - matchbox-lib
>   - metacity
>   - sdl_sound
>   - sylpheed
>   - xapp_xinput-calibrator
>   - xdriver_xf86-input-tslib
>   - xdriver_xf86-video-imx
> 
> 4 of them already call autoreconf: bellagio, dbus-cpp, libnetfilter_log, 
> libnfnetlink.
> 
> I am going to create a configuration with all of these packages.

 That's not necessarily sufficient: the generated .la file could be wrong, so
you have to check if packages that use it are still working correctly.
Alternatively, just check all the .la files.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Arnout Vandecappelle]

On 22-03-17 11:05, Arnout Vandecappelle wrote:
> 
> 
> On 22-03-17 10:20, J?r?me Pouiller wrote:
>> Hello,
>>
>> On Wednesday 22 March 2017 09:14:37 CET Thomas Petazzoni wrote:
>>> Hello,
>>>
>>> On Wed, 22 Mar 2017 00:57:39 +0100, Arnout Vandecappelle wrote:
>>>
>>>>> So are you sure that this feature didn't appear in libtool 2.4 ?  
>>>>
>>>>  Yeah, looks like it. E.g. speex has libtool 1.5, and it doesn't have any
>>>> sysroot support (not in configure, not in ltmain.sh).
>>>>
>>>>  Obviously, *this* patch is not affected by it - it certainly doesn't hurt
>>>>  to pass the --with-sysroot configure option. It's only patches 12+13 that
>>>>  we wouldn't be able to remove.
>>>>
>>>>  However, considering:
>>>>
>>>> - most packages install things in /usr/lib, which is already searched by
>>>> default;
>>>>
>>>> - most packages use libtool 2.4 or later,
>>>>
>>>> I think we can take the risk. The autobuilders will discover any issues,
>>>> which we can fix per package (e.g. with AUTORECONF = YES).

 By the way, this series also *fixes* issues. E.g. I couldn't reproduce
http://autobuild.buildroot.net/results/e8c6bdc314b9a8d5de303633a3abd7b7e44beb2e
after applying this series, whereas I could reproduce it before.

 Regards,
 Arnout

>>>
>>> Since I would prefer to avoid autoreconfiguring many packages, it would
>>> be useful to know how many packages are still using libtool versions
>>> earlier than 2.4.
>>
>> This is precisely what I wanted to known before to replying.
>>
>> I extracted all (or nearly all) packages existing in Buildroot. I found 269 
>> packages that use libool:
>>
>> $ grep -ho 'VERSION="[0-9.]*' */ltmain.sh  | sort | uniq -c | sort -n
>>       1 VERSION="1.5.24
>>       1 VERSION="2.2.4
>>       3 VERSION="1.5.22
>>       5 VERSION="1.5.26
>>      13 VERSION="2.2.6
>>      29 VERSION="2.4.6
>>      67 VERSION="2.4.2
> 
>  Your command isn't good enough... It skips ltmain.sh which are not in the top
> directory, and it skips the ones where VERSION isn't quoted. I got quite a few more:
> 
> $ for i in *.tar*; do tar tf $i | grep ltmain.sh | xargs -r tar -xf $i
> --to-command="grep ^VERSION= | { grep -v '[^.]2\.4' && printf ' @@$i\\r'; } | tr
> '\\n\\r' ' \\n'"; done
> 
> VERSION=1.5.26  @@CEGUI-0.6.2b.tar.gz
> VERSION="1.5.26 Debian 1.5.26-1ubuntu1"  @@LiTE-0.8.10.tar.gz
> VERSION=2.2.6  @@SDL-1.2.15.tar.gz
> VERSION=2.2.6  @@SDL2_image-2.0.1.tar.gz
> VERSION=2.2.6  @@SDL2_ttf-2.0.14.tar.gz
> VERSION=2.2.6  @@SDL_image-1.2.12.tar.gz
> VERSION=2.2.6  @@SDL_mixer-1.2.12.tar.gz
> VERSION=2.2.6  @@SDL_net-1.2.8.tar.gz
> VERSION="1.5.24 Debian 1.5.24-1ubuntu1"  @@SDL_sound-1.0.3.tar.gz
> VERSION=2.2.6  @@SDL_ttf-2.0.11.tar.gz
> VERSION="1.5.22 Debian 1.5.22-2"  @@apache-log4cxx-0.10.0.tar.gz
> VERSION=2.2.6b  @@arago-2011.09-armv5te-linux-gnueabi-sdk.tar.bz2
> VERSION=2.2.6b  @@arago-2011.09-armv7a-linux-gnueabi-sdk.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@argtable2-13.tar.gz
> VERSION=2.2.10  @@axfsutils-64886703fdff6cfc64eb20fbe7d7295bbeec867b.tar.gz
> VERSION=2.2.6  @@beecrypt-4.2.1.tar.gz
> VERSION=2.2.6b  @@bind-9.11.0-P3.tar.gz
> VERSION=1.3.5  @@bind-9.11.0-P3.tar.gz
> VERSION=2.2.7a  @@binutils-2.26.1.tar.bz2
> VERSION=2.2.7a  @@binutils-2.27.tar.bz2
> VERSION=1.5.22  @@boost_1_63_0.tar.bz2
> VERSION="1.5.26 Debian 1.5.26-4"  @@celt-0.5.1.3.tar.gz
> VERSION=1.5.22  @@clamav-0.99.2.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@collectd-5.5.1.tar.bz2
> VERSION="1.5.26 Debian 1.5.26-1ubuntu1"  @@curlftpfs-0.9.2.tar.gz
> VERSION=2.2.6b  @@db-5.3.28.NC.tar.gz
> VERSION=2.2.6  @@db-5.3.28.NC.tar.gz
> VERSION=1.5.22  @@db-5.3.28.NC.tar.gz
> VERSION="2.2.6 Debian-2.2.6a-1ubuntu1"  @@eventlog_0.2.12.tar.gz
> VERSION="1.5.22 Debian 1.5.22-4"  @@faad2-2.7.tar.gz
> VERSION=1.4  @@fcgi-2.4.0.tar.gz
> VERSION=1.5.24  @@gamin-0.1.10.tar.gz
> VERSION=2.2.7a  @@gcc-4.9.4.tar.bz2
> VERSION=2.2.7a  @@gcc-4.9.4.tar.bz2
> VERSION=2.2.7a  @@gcc-4.9.4.tar.bz2
> VERSION=1.5.16  @@gcc-4.9.4.tar.bz2
> VERSION=2.2.7a  @@gcc-5.4.0.tar.bz2
> VERSION=2.2.7a  @@gcc-5.4.0.tar.bz2
> VERSION=2.2.7a  @@gcc-5.4.0.tar.bz2
> VERSION=1.5.16  @@gcc-5.4.0.tar.bz2
> VERSION=2.2.7a  @@gcc-6.2.0.tar.bz2
> VERSION=2.2.7a  @@gcc-6.2.0.tar.bz2
> 
> 
> VERSION=2.2.7a  @@gcc-6.2.0.tar.bz2
> 
> 
> VERSION=1.5.16  @@gcc-6.2.0.tar.bz2
> 
> 
> VERSION=2.2.7a  @@gdb-7.10.1.tar.xz
> VERSION=2.2.7a  @@gdb-7.11.1.tar.xz
> VERSION="1.5.26 Debian 1.5.26-4"  @@genpart-1.0.2.tar.bz2
> VERSION=1.4e  @@gettext-0.19.8.1.tar.xz
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu3"  @@gmpc-11.8.16.tar.gz
> VERSION=2.2.6b  @@google-breakpad-7515ab13768c7edc09f0f2ec2354dc6c928239a6.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2"  @@gtk-engines-2.20.2.tar.bz2
> VERSION=1.5.2  @@ijs-0.35.tar.bz2
> VERSION=1.5.20  @@ipkg-0.99.163.tar.gz
> VERSION=1.5.22  @@jasper-1.900.1.tar.bz2
> VERSION=2.2.6b  @@json-c-0.11.tar.gz
> VERSION="1.5.26 Debian 1.5.26-4"  @@kodi-16.1-Jarvis.tar.gz
> VERSION=2.2.10  @@lame-3.99.5.tar.gz
> VERSION=2.2.6  @@libart_lgpl-2.3.21.tar.gz
> VERSION=2.2.4  @@libass-0.13.1.tar.xz
> VERSION=1.5.6  @@libcdaudio-0.99.12p2.tar.gz
> VERSION=2.2.6  @@libcddb-1.3.2.tar.bz2
> VERSION=2.2.6  @@libdaemon-0.14.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@libdbus-c++-0.9.0.tar.gz
> VERSION=2.2.10  @@libdmtx-0.7.4.tar.gz
> VERSION=1.5.6  @@libdnet-1.11.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2"  @@libeXosip2-3.6.0.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu3"  @@libee-0.4.1.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@libesmtp-1.0.6.tar.bz2
> VERSION=1.5.6  @@libfakekey-0.1.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@libgeotiff-1.4.0.tar.gz
> VERSION=1.5.26  @@libglade-2.6.4.tar.bz2
> VERSION=1.5.22  @@libhid-0.2.16.tar.gz
> VERSION=1.5.2  @@libid3tag-0.15.1b.tar.gz
> VERSION="2.2.4 Debian-2.2.4-0ubuntu4"  @@liblo-0.26.tar.gz
> VERSION=1.5.22  @@liblog4c-localtime-v1.0.tar.gz
> VERSION=1.5.2  @@libmad-0.15.1b.tar.gz
> VERSION="1.5.22 Debian 1.5.22-2"  @@libmatchbox-1.9.tar.bz2
> VERSION=1.5.22  @@libmcrypt-2.5.8.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2"  @@libmnl-1.0.3.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu3"  @@libmpd-11.8.17.tar.gz
> VERSION="1.5.26 Debian 1.5.26-4"  @@libmpeg2-0.5.1.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2"  @@libnetfilter_conntrack-1.0.6.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2"  @@libnetfilter_log-1.0.1.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2"  @@libnetfilter_log-1.0.1.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2"  @@libnfnetlink-1.0.1.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2"  @@libosip2-3.6.0.tar.gz
> VERSION="2.2.6 Debian-2.2.6a-4"  @@libpthread-stubs-0.3.tar.bz2
> VERSION=1.5.6  @@librsync-0.9.7.tar.gz
> VERSION="1.5.22 Debian 1.5.22-2"  @@libsexy-0.1.11.tar.gz
> VERSION=1.5.14  @@libsvg-0.1.4.tar.gz
> VERSION=1.5.14  @@libsvg-cairo-0.1.6.tar.gz
> VERSION=2.2.6  @@libtheora-1.1.1.tar.xz
> VERSION=1.4.3  @@libunwind-1.1.tar.gz
> VERSION=2.2.10  @@lzop-1.03.tar.gz
> VERSION=1.5.2  @@madplay-0.15.2b.tar.gz
> VERSION=1.5.6  @@mb-applet-startup-monitor-0.1.tar.bz2
> VERSION="2.2.4 Debian-2.2.4-0ubuntu4"  @@mcrypt-2.6.8.tar.gz
> VERSION=2.2.6b  @@msgpack-0.5.4.tar.gz
> VERSION=2.2.6  @@mysql-5.1.73.tar.gz
> VERSION=1.5.8  @@ngrep-1.45.tar.bz2
> VERSION=1.5.26  @@php-7.1.3.tar.xz
> VERSION=1.5.22  @@pjproject-2.4.5.tar.bz2
> VERSION=2.2.6  @@popt-1.16.tar.gz
> VERSION=1.3.4  @@privoxy-3.0.26-stable-src.tar.gz
> VERSION="2.2.6 Debian-2.2.6a-1ubuntu1"  @@proftpd-1.3.5d.tar.gz
> VERSION=1.5.26  @@proftpd-1.3.5d.tar.gz
> VERSION="2.2.6 Debian-2.2.6a-1ubuntu1"  @@proftpd-1.3.5d.tar.gz
> VERSION=2.2.6  @@pthsem_2.0.8.tar.gz
> VERSION=2.2.6  @@qt-everywhere-opensource-src-4.8.7.tar.gz
> VERSION=2.2.7a  @@rustc-1.10.0-src.tar.gz
> VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
> VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
> VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
> VERSION=1.4  @@sidplay-libs-2.1.1.tar.gz
> VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
> VERSION=1.5.6  @@sidplay-libs-2.1.1.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@sofia-sip-1.12.11.tar.gz
> VERSION="1.5.22 Debian 1.5.22-4"  @@speex-1.2rc1.tar.gz
> VERSION=2.2.6  @@sqlcipher-v3.2.0.tar.gz
> VERSION=1.5.6  @@startup-notification-0.9.tar.gz
> VERSION=1.5.22  @@sysfsutils-2.1.0.tar.gz
> VERSION=1.5.22  @@szip-2.1.tar.gz
> VERSION=1.5.24  @@tn5250-0.17.4.tar.gz
> VERSION=1.5.6  @@ucl-1.03.tar.gz
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@unixODBC-2.3.2.tar.gz
> VERSION="2.2.6 Debian-2.2.6a-4"  @@vorbis-tools-1.4.0.tar.gz
> VERSION=1.4.3  @@xcursor-transparent-theme-0.1.1.tar.gz
> VERSION="1.5.26 Debian 1.5.26-4"  @@xf86-input-tslib-0.0.6.tar.bz2
> VERSION="2.2.6b Debian-2.2.6b-2ubuntu1"  @@xinput_calibrator-0.7.5.tar.gz
> 
>  That's 109 tarballs with some older libtool.
> 
>  Note that this just iterates over the tarballs that I have in my download
> directory, there may be some old versions of packages in there. Anyway, I think
> 109 out of the 2150 tarballs in that directory is still extremely small.
> 
>  Note also that in many of these, the ltmain will be in some subdirectory
> containing bundled source that we don't even use.
> 
>  Note, finally, that there are also quite a few host-only packages in there
> (e.g. gcc, but that one is special because it also builds stuff for the target.
> It doesn't seem to install .la files, however).
> 
> 
>> There 23 packages that use libtool < 2.4:
>>   - bellagio
>>   - celt051
>>   - dbus-cpp
>>   - eventlog
>>   - gtk2-engines
>>   - lesstif
>>   - libargtable2
>>   - libesmtp
>>   - liblo
>>   - libmhash
>>   - libnetfilter_conntrack
>>   - libnetfilter_log
>>   - libnfnetlink
>>   - libpthread-stubs
>>   - lite
>>   - log4cxx
>>   - matchbox-lib
>>   - metacity
>>   - sdl_sound
>>   - sylpheed
>>   - xapp_xinput-calibrator
>>   - xdriver_xf86-input-tslib
>>   - xdriver_xf86-video-imx
>>
>> 4 of them already call autoreconf: bellagio, dbus-cpp, libnetfilter_log, 
>> libnfnetlink.
>>
>> I am going to create a configuration with all of these packages.
> 
>  That's not necessarily sufficient: the generated .la file could be wrong, so
> you have to check if packages that use it are still working correctly.
> Alternatively, just check all the .la files.
> 
>  Regards,
>  Arnout
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Thomas Petazzoni]

Hello,

On Wed, 22 Mar 2017 22:28:27 +0100, Arnout Vandecappelle wrote:

> >>>> I think we can take the risk. The autobuilders will discover any issues,
> >>>> which we can fix per package (e.g. with AUTORECONF = YES).  
> 
>  By the way, this series also *fixes* issues. E.g. I couldn't reproduce
> http://autobuild.buildroot.net/results/e8c6bdc314b9a8d5de303633a3abd7b7e44beb2e
> after applying this series, whereas I could reproduce it before.

So it allows to have better .la files, that help with "transitive"
dependencies in static linking scenarios?

Also, you don't need to convince me that we need to use the libtool
sysroot support. Anything to cleanup/remove the libtool hacks that we
have is welcome, so this series is definitely welcome, and I'm not
trying to push back on it.

It's just that Jerome's assumption was that the sysroot support was in
libtool since 1.5, which is not the case, and the fact that it's only
there since libtool 2.4 and we have packages using earlier libtool
versions needs to be handled by the series.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 11/19] infra-libtool: pass sysroot information to libtool

[Thomas Petazzoni]

Hello,

On Mon, 20 Mar 2017 22:52:50 +0100, Thomas Petazzoni wrote:

> On Tue, 20 Dec 2016 14:46:28 +0100, J?r?me Pouiller wrote:
> 
> > libtool include support for cross-compilation since v1.5.  
> 
> Are you sure this is true?
> 
> In the release notes of libtool v2.4
> (https://savannah.gnu.org/forum/forum.php?forum_id=6525), I see:

So, because of this issue, I've marked patches 11 to 19 from your
series as "Changes Requested". Things are indeed not as simple as "use
sysroot support from libtool".

Of course, it would definitely be nice to continue working on this
topic, and benefit from libtool sysroot support when available.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v5 12/19] infra-libtool: no longer prepend STAGING_DIR to libdir

[Jérôme Pouiller]

Since `$lt_sysroot' is now correctly filled, it is no more necessary to patch
`.la' files. Indeed, if $libdir point to `/usr/lib', libtool will also search
in `$(lt_sysroot)/usr/lib'

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/pkg-generic.mk | 11 -----------
 1 file changed, 11 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 827de62..5d12ac3 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -268,17 +268,6 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 				-e "s, at BASE_DIR@,$(BASE_DIR),g" \
 				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
 	fi
-	@$(call MESSAGE,"Fixing libtool files")
-	$(Q)find $(STAGING_DIR)/usr/lib* -name "*.la" | xargs --no-run-if-empty \
-		$(SED) "s:$(BASE_DIR):@BASE_DIR@:g" \
-			-e "s:$(STAGING_DIR):@STAGING_DIR@:g" \
-			$(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
-				-e "s:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:g") \
-			-e "s:\(['= ]\)/usr:\\1 at STAGING_DIR@/usr:g" \
-			$(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
-				-e "s:@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):g") \
-			-e "s:@STAGING_DIR@:$(STAGING_DIR):g" \
-			-e "s:@BASE_DIR@:$(BASE_DIR):g"
 	@$(call step_end,install-staging)
 	$(Q)touch $@
 
-- 
1.9.1

[Buildroot] [PATCH v5 12/19] infra-libtool: no longer prepend STAGING_DIR to libdir

[Arnout Vandecappelle]

 Hi Jerome,

On 20-12-16 14:46, J?r?me Pouiller wrote:
> Since `$lt_sysroot' is now correctly filled, it is no more necessary to patch
> `.la' files. Indeed, if $libdir point to `/usr/lib', libtool will also search
> in `$(lt_sysroot)/usr/lib'

 Unfortunately, not all packages that use libtool also use automake. So some
packages may die because of this.

 That said, we should probably just fix these individual packages.

 (Not yet giving my reviewed-by, I'm first going to check the rest of the series.)

 Regards,
 Arnout

> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  package/pkg-generic.mk | 11 -----------
>  1 file changed, 11 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 827de62..5d12ac3 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -268,17 +268,6 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
>  				-e "s, at BASE_DIR@,$(BASE_DIR),g" \
>  				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
>  	fi
> -	@$(call MESSAGE,"Fixing libtool files")
> -	$(Q)find $(STAGING_DIR)/usr/lib* -name "*.la" | xargs --no-run-if-empty \
> -		$(SED) "s:$(BASE_DIR):@BASE_DIR@:g" \
> -			-e "s:$(STAGING_DIR):@STAGING_DIR@:g" \
> -			$(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
> -				-e "s:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:g") \
> -			-e "s:\(['= ]\)/usr:\\1 at STAGING_DIR@/usr:g" \
> -			$(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
> -				-e "s:@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):g") \
> -			-e "s:@STAGING_DIR@:$(STAGING_DIR):g" \
> -			-e "s:@BASE_DIR@:$(BASE_DIR):g"
>  	@$(call step_end,install-staging)
>  	$(Q)touch $@
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 12/19] infra-libtool: no longer prepend STAGING_DIR to libdir

[Jérôme Pouiller]

Hello Anrout,

On Saturday 18 March 2017 16:37:31 CET Arnout Vandecappelle wrote:
>  Hi Jerome,
> 
> On 20-12-16 14:46, J?r?me Pouiller wrote:
> > Since `$lt_sysroot' is now correctly filled, it is no more necessary to patch
> > `.la' files. Indeed, if $libdir point to `/usr/lib', libtool will also search
> > in `$(lt_sysroot)/usr/lib'
> 
>  Unfortunately, not all packages that use libtool also use automake.

You mean autoconf? Do you have an example of this?

>  So some packages may die because of this.
>
>  That said, we should probably just fix these individual packages.

yes, I think it very unusual.


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v5 12/19] infra-libtool: no longer prepend STAGING_DIR to libdir

[Arnout Vandecappelle]

On 22-03-17 11:21, J?r?me Pouiller wrote:
> Hello Anrout,
> 
> On Saturday 18 March 2017 16:37:31 CET Arnout Vandecappelle wrote:
>>  Hi Jerome,
>>
>> On 20-12-16 14:46, J?r?me Pouiller wrote:
>>> Since `$lt_sysroot' is now correctly filled, it is no more necessary to patch
>>> `.la' files. Indeed, if $libdir point to `/usr/lib', libtool will also search
>>> in `$(lt_sysroot)/usr/lib'
>>
>>  Unfortunately, not all packages that use libtool also use automake.
> 
> You mean autoconf? Do you have an example of this?

 No, I mean automake. If it's not using autoconf that it shouldn't even be an
autotools-package, and we don't apply the libtool patches.

 IIUC, the --with-sysroot option is going to be substituted in the Makefiles
with an extra option to libtool. However, if a package is not using automake, it
is possible that that extra option doesn't get propagated. I think.


 Regards,
 Arnout

> 
>>  So some packages may die because of this.
>>
>>  That said, we should probably just fix these individual packages.
> 
> yes, I think it very unusual.
> 
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 12/19] infra-libtool: no longer prepend STAGING_DIR to libdir

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> Since `$lt_sysroot' is now correctly filled, it is no more necessary to patch
> `.la' files. Indeed, if $libdir point to `/usr/lib', libtool will also search
                                       ^s
> in `$(lt_sysroot)/usr/lib'
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)


 Regards,
 Arnout

> ---
>  package/pkg-generic.mk | 11 -----------
>  1 file changed, 11 deletions(-)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 827de62..5d12ac3 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -268,17 +268,6 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
>  				-e "s, at BASE_DIR@,$(BASE_DIR),g" \
>  				$(addprefix $(STAGING_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ;\
>  	fi
> -	@$(call MESSAGE,"Fixing libtool files")
> -	$(Q)find $(STAGING_DIR)/usr/lib* -name "*.la" | xargs --no-run-if-empty \
> -		$(SED) "s:$(BASE_DIR):@BASE_DIR@:g" \
> -			-e "s:$(STAGING_DIR):@STAGING_DIR@:g" \
> -			$(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
> -				-e "s:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:g") \
> -			-e "s:\(['= ]\)/usr:\\1 at STAGING_DIR@/usr:g" \
> -			$(if $(TOOLCHAIN_EXTERNAL_INSTALL_DIR),\
> -				-e "s:@TOOLCHAIN_EXTERNAL_INSTALL_DIR@:$(TOOLCHAIN_EXTERNAL_INSTALL_DIR):g") \
> -			-e "s:@STAGING_DIR@:$(STAGING_DIR):g" \
> -			-e "s:@BASE_DIR@:$(BASE_DIR):g"
>  	@$(call step_end,install-staging)
>  	$(Q)touch $@
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 13/19] infra-libtool: correctly prefix $libdir with $STAGING_DIR

[Jérôme Pouiller]

Until now Builroot patched libtool in order to ignore `$libdir' provided by .la
file. It used the path where it found the .la instead of `$libdir'. However,
since v1.5 libtool correctly prepend sysroot to `$libdir' as we can see a few
lines below patched lines:

  func_append newdependency_libs " ${lt_sysroot:+=}$libdir/$name"

(in fact, this line just prepend '=', and '=' is substituted later)

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 support/libtool/buildroot-libtool-v1.5.patch   | 32 ++------------------------
 support/libtool/buildroot-libtool-v2.2.patch   | 30 +-----------------------
 support/libtool/buildroot-libtool-v2.4.4.patch | 11 ---------
 support/libtool/buildroot-libtool-v2.4.patch   | 11 ---------
 4 files changed, 3 insertions(+), 81 deletions(-)

diff --git a/support/libtool/buildroot-libtool-v1.5.patch b/support/libtool/buildroot-libtool-v1.5.patch
index cdd9c5a..609006a 100644
--- a/support/libtool/buildroot-libtool-v1.5.patch
+++ b/support/libtool/buildroot-libtool-v1.5.patch
@@ -34,24 +34,7 @@
  	if test -n "$link_static_flag"; then
  	  compile_command="$compile_command $link_static_flag"
  	  finalize_command="$finalize_command $link_static_flag"
-@@ -2146,8 +2153,14 @@
- 	    absdir="$abs_ladir"
- 	    libdir="$abs_ladir"
- 	  else
--	    dir="$libdir"
--	    absdir="$libdir"
-+            # Adding 'libdir' from the .la file to our library search paths
-+            # breaks crosscompilation horribly.  We cheat here and don't add
-+            # it, instead adding the path where we found the .la.  -CL
-+	    dir="$abs_ladir"
-+	    absdir="$abs_ladir"
-+	    libdir="$abs_ladir"
-+	    #dir="$libdir"
-+	    #absdir="$libdir"
- 	  fi
- 	else
- 	  dir="$ladir/$objdir"
-@@ -2272,7 +2285,7 @@
+@@ -2272,7 +2279,7 @@
  	   { test "$prefer_static_libs" = no || test -z "$old_library"; }; then
  	  if test "$installed" = no; then
  	    notinst_deplibs="$notinst_deplibs $lib"
@@ -60,18 +43,7 @@
  	  fi
  	  # This is a shared library
  
-@@ -5169,6 +5182,10 @@
- 	    # Replace all uninstalled libtool libraries with the installed ones
- 	    newdependency_libs=
- 	    for deplib in $dependency_libs; do
-+              # Replacing uninstalled with installed can easily break crosscompilation,
-+              # since the installed path is generally the wrong architecture.  -CL
-+              newdependency_libs="$newdependency_libs $deplib"
-+              continue
- 	      case $deplib in
- 	      *.la)
- 		name=`$echo "X$deplib" | $Xsed -e 's%^.*/%%'`
-@@ -5487,10 +5504,13 @@
+@@ -5487,10 +5494,13 @@
  	  # At present, this check doesn't affect windows .dll's that
  	  # are installed into $libdir/../bin (currently, that works fine)
  	  # but it's something to keep an eye on.
diff --git a/support/libtool/buildroot-libtool-v2.2.patch b/support/libtool/buildroot-libtool-v2.2.patch
index 8bb7826..3af0604 100644
--- a/support/libtool/buildroot-libtool-v2.2.patch
+++ b/support/libtool/buildroot-libtool-v2.2.patch
@@ -67,24 +67,7 @@
  	# The effects of -static are defined in a previous loop.
  	# We used to do the same as -all-static on platforms that
  	# didn't have a PIC flag, but the assumption that the effects
-@@ -5739,8 +5738,14 @@
- 	    absdir="$abs_ladir"
- 	    libdir="$abs_ladir"
- 	  else
--	    dir="$libdir"
--	    absdir="$libdir"
-+            # Adding 'libdir' from the .la file to our library search paths
-+            # breaks crosscompilation horribly.  We cheat here and don't add
-+            # it, instead adding the path where we found the .la.  -CL
-+	    dir="$abs_ladir"
-+	    absdir="$abs_ladir"
-+	    libdir="$abs_ladir"
-+	    #dir="$libdir"
-+	    #absdir="$libdir"
- 	  fi
- 	  test "X$hardcode_automatic" = Xyes && avoidtemprpath=yes
- 	else
-@@ -5891,7 +5896,7 @@
+@@ -5891,7 +5890,7 @@
  	  *)
  	    if test "$installed" = no; then
  	      notinst_deplibs="$notinst_deplibs $lib"
@@ -93,14 +76,3 @@
  	    fi
  	    ;;
  	  esac
-@@ -8373,6 +8378,10 @@
- 	    # Replace all uninstalled libtool libraries with the installed ones
- 	    newdependency_libs=
- 	    for deplib in $dependency_libs; do
-+              # Replacing uninstalled with installed can easily break crosscompilation,
-+              # since the installed path is generally the wrong architecture.  -CL
-+              newdependency_libs="$newdependency_libs $deplib"
-+              continue
- 	      case $deplib in
- 	      *.la)
- 		func_basename "$deplib"
diff --git a/support/libtool/buildroot-libtool-v2.4.4.patch b/support/libtool/buildroot-libtool-v2.4.4.patch
index bcdf30c..d71669b 100644
--- a/support/libtool/buildroot-libtool-v2.4.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.4.patch
@@ -82,14 +82,3 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  	    fi
  	    ;;
  	  esac
-@@ -10710,6 +10710,10 @@
- 	    # Replace all uninstalled libtool libraries with the installed ones
- 	    newdependency_libs=
- 	    for deplib in $dependency_libs; do
-+	      # Replacing uninstalled with installed can easily break crosscompilation,
-+	      # since the installed path is generally the wrong architecture.  -CL
-+	      newdependency_libs="$newdependency_libs $deplib"
-+	      continue
- 	      case $deplib in
- 	      *.la)
- 		func_basename "$deplib"
diff --git a/support/libtool/buildroot-libtool-v2.4.patch b/support/libtool/buildroot-libtool-v2.4.patch
index f610b1b..bf40c94 100644
--- a/support/libtool/buildroot-libtool-v2.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.patch
@@ -76,14 +76,3 @@
  	    fi
  	    ;;
  	  esac
-@@ -9275,6 +9274,10 @@
- 	    # Replace all uninstalled libtool libraries with the installed ones
- 	    newdependency_libs=
- 	    for deplib in $dependency_libs; do
-+              # Replacing uninstalled with installed can easily break crosscompilation,
-+              # since the installed path is generally the wrong architecture.  -CL
-+              newdependency_libs="$newdependency_libs $deplib"
-+              continue
- 	      case $deplib in
- 	      *.la)
- 		func_basename "$deplib"
-- 
1.9.1

[Buildroot] [PATCH v5 13/19] infra-libtool: correctly prefix $libdir with $STAGING_DIR

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> Until now Builroot patched libtool in order to ignore `$libdir' provided by .la
                                                                             ^the
> file. It used the path where it found the .la instead of `$libdir'. However,
> since v1.5 libtool correctly prepend sysroot to `$libdir' as we can see a few
                                      ^s the
> lines below patched lines:
             ^the
> 
>   func_append newdependency_libs " ${lt_sysroot:+=}$libdir/$name"
> 
> (in fact, this line just prepend '=', and '=' is substituted later)
                                  ^s
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)


 Regards,
 Arnout

> ---
>  support/libtool/buildroot-libtool-v1.5.patch   | 32 ++------------------------
>  support/libtool/buildroot-libtool-v2.2.patch   | 30 +-----------------------
>  support/libtool/buildroot-libtool-v2.4.4.patch | 11 ---------
>  support/libtool/buildroot-libtool-v2.4.patch   | 11 ---------
>  4 files changed, 3 insertions(+), 81 deletions(-)
> 
> diff --git a/support/libtool/buildroot-libtool-v1.5.patch b/support/libtool/buildroot-libtool-v1.5.patch
> index cdd9c5a..609006a 100644
> --- a/support/libtool/buildroot-libtool-v1.5.patch
> +++ b/support/libtool/buildroot-libtool-v1.5.patch
> @@ -34,24 +34,7 @@
>   	if test -n "$link_static_flag"; then
>   	  compile_command="$compile_command $link_static_flag"
>   	  finalize_command="$finalize_command $link_static_flag"
> -@@ -2146,8 +2153,14 @@
> - 	    absdir="$abs_ladir"
> - 	    libdir="$abs_ladir"
> - 	  else
> --	    dir="$libdir"
> --	    absdir="$libdir"
> -+            # Adding 'libdir' from the .la file to our library search paths
> -+            # breaks crosscompilation horribly.  We cheat here and don't add
> -+            # it, instead adding the path where we found the .la.  -CL
> -+	    dir="$abs_ladir"
> -+	    absdir="$abs_ladir"
> -+	    libdir="$abs_ladir"
> -+	    #dir="$libdir"
> -+	    #absdir="$libdir"
> - 	  fi
> - 	else
> - 	  dir="$ladir/$objdir"
> -@@ -2272,7 +2285,7 @@
> +@@ -2272,7 +2279,7 @@
>   	   { test "$prefer_static_libs" = no || test -z "$old_library"; }; then
>   	  if test "$installed" = no; then
>   	    notinst_deplibs="$notinst_deplibs $lib"
> @@ -60,18 +43,7 @@
>   	  fi
>   	  # This is a shared library
>   
> -@@ -5169,6 +5182,10 @@
> - 	    # Replace all uninstalled libtool libraries with the installed ones
> - 	    newdependency_libs=
> - 	    for deplib in $dependency_libs; do
> -+              # Replacing uninstalled with installed can easily break crosscompilation,
> -+              # since the installed path is generally the wrong architecture.  -CL
> -+              newdependency_libs="$newdependency_libs $deplib"
> -+              continue
> - 	      case $deplib in
> - 	      *.la)
> - 		name=`$echo "X$deplib" | $Xsed -e 's%^.*/%%'`
> -@@ -5487,10 +5504,13 @@
> +@@ -5487,10 +5494,13 @@
>   	  # At present, this check doesn't affect windows .dll's that
>   	  # are installed into $libdir/../bin (currently, that works fine)
>   	  # but it's something to keep an eye on.
> diff --git a/support/libtool/buildroot-libtool-v2.2.patch b/support/libtool/buildroot-libtool-v2.2.patch
> index 8bb7826..3af0604 100644
> --- a/support/libtool/buildroot-libtool-v2.2.patch
> +++ b/support/libtool/buildroot-libtool-v2.2.patch
> @@ -67,24 +67,7 @@
>   	# The effects of -static are defined in a previous loop.
>   	# We used to do the same as -all-static on platforms that
>   	# didn't have a PIC flag, but the assumption that the effects
> -@@ -5739,8 +5738,14 @@
> - 	    absdir="$abs_ladir"
> - 	    libdir="$abs_ladir"
> - 	  else
> --	    dir="$libdir"
> --	    absdir="$libdir"
> -+            # Adding 'libdir' from the .la file to our library search paths
> -+            # breaks crosscompilation horribly.  We cheat here and don't add
> -+            # it, instead adding the path where we found the .la.  -CL
> -+	    dir="$abs_ladir"
> -+	    absdir="$abs_ladir"
> -+	    libdir="$abs_ladir"
> -+	    #dir="$libdir"
> -+	    #absdir="$libdir"
> - 	  fi
> - 	  test "X$hardcode_automatic" = Xyes && avoidtemprpath=yes
> - 	else
> -@@ -5891,7 +5896,7 @@
> +@@ -5891,7 +5890,7 @@
>   	  *)
>   	    if test "$installed" = no; then
>   	      notinst_deplibs="$notinst_deplibs $lib"
> @@ -93,14 +76,3 @@
>   	    fi
>   	    ;;
>   	  esac
> -@@ -8373,6 +8378,10 @@
> - 	    # Replace all uninstalled libtool libraries with the installed ones
> - 	    newdependency_libs=
> - 	    for deplib in $dependency_libs; do
> -+              # Replacing uninstalled with installed can easily break crosscompilation,
> -+              # since the installed path is generally the wrong architecture.  -CL
> -+              newdependency_libs="$newdependency_libs $deplib"
> -+              continue
> - 	      case $deplib in
> - 	      *.la)
> - 		func_basename "$deplib"
> diff --git a/support/libtool/buildroot-libtool-v2.4.4.patch b/support/libtool/buildroot-libtool-v2.4.4.patch
> index bcdf30c..d71669b 100644
> --- a/support/libtool/buildroot-libtool-v2.4.4.patch
> +++ b/support/libtool/buildroot-libtool-v2.4.4.patch
> @@ -82,14 +82,3 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
>   	    fi
>   	    ;;
>   	  esac
> -@@ -10710,6 +10710,10 @@
> - 	    # Replace all uninstalled libtool libraries with the installed ones
> - 	    newdependency_libs=
> - 	    for deplib in $dependency_libs; do
> -+	      # Replacing uninstalled with installed can easily break crosscompilation,
> -+	      # since the installed path is generally the wrong architecture.  -CL
> -+	      newdependency_libs="$newdependency_libs $deplib"
> -+	      continue
> - 	      case $deplib in
> - 	      *.la)
> - 		func_basename "$deplib"
> diff --git a/support/libtool/buildroot-libtool-v2.4.patch b/support/libtool/buildroot-libtool-v2.4.patch
> index f610b1b..bf40c94 100644
> --- a/support/libtool/buildroot-libtool-v2.4.patch
> +++ b/support/libtool/buildroot-libtool-v2.4.patch
> @@ -76,14 +76,3 @@
>   	    fi
>   	    ;;
>   	  esac
> -@@ -9275,6 +9274,10 @@
> - 	    # Replace all uninstalled libtool libraries with the installed ones
> - 	    newdependency_libs=
> - 	    for deplib in $dependency_libs; do
> -+              # Replacing uninstalled with installed can easily break crosscompilation,
> -+              # since the installed path is generally the wrong architecture.  -CL
> -+              newdependency_libs="$newdependency_libs $deplib"
> -+              continue
> - 	      case $deplib in
> - 	      *.la)
> - 		func_basename "$deplib"
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 14/19] infra-libtool: drop original $libdir (i.e. /usr/lib) from library paths

[Jérôme Pouiller]

When it compute list of directories to include in link, libtool has a
conservative behavior. It add `$inst_prefix_dir$libdir' but also keep original
`$libdir'. Thus, it call:
   -L/usr/lib -L/buildroot/output/host/usr/arm-buildroot-linux-gnueabihf/sysroot/usr/lib

It works since the last path has priority. However, toolchan-wrapper consider
`/usr/lib' an an unsafe path.

This patch remove original `$libdir'.

Since last versions of binutils provide `-Wpoison-system-directories' option, I
think it make sense to try to upstream this patch.

Note Yocto has a similar patch:
  http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/use-sysroot-in-libpath.patch

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 support/libtool/buildroot-libtool-v1.5.patch   | 18 ++++++++++++++++++
 support/libtool/buildroot-libtool-v2.2.patch   | 18 ++++++++++++++++++
 support/libtool/buildroot-libtool-v2.4.4.patch |  9 +++++++++
 support/libtool/buildroot-libtool-v2.4.patch   |  9 +++++++++
 4 files changed, 54 insertions(+)

diff --git a/support/libtool/buildroot-libtool-v1.5.patch b/support/libtool/buildroot-libtool-v1.5.patch
index 609006a..51d049f 100644
--- a/support/libtool/buildroot-libtool-v1.5.patch
+++ b/support/libtool/buildroot-libtool-v1.5.patch
@@ -43,6 +43,24 @@
  	  fi
  	  # This is a shared library
  
+@@ -2412,7 +2419,7 @@
+ 		if test -n "$inst_prefix_dir"; then
+ 		  case "$libdir" in
+ 		    [\\/]*)
+-		      add_dir="$add_dir -L$inst_prefix_dir$libdir"
++		      add_dir="-L$inst_prefix_dir$libdir"
+ 		      ;;
+ 		  esac
+ 		fi
+@@ -2492,7 +2499,7 @@
+ 	      if test -n "$inst_prefix_dir"; then
+ 		case "$libdir" in
+ 		  [\\/]*)
+-		    add_dir="$add_dir -L$inst_prefix_dir$libdir"
++		    add_dir="-L$inst_prefix_dir$libdir"
+ 		    ;;
+ 		esac
+ 	      fi
 @@ -5487,10 +5494,13 @@
  	  # At present, this check doesn't affect windows .dll's that
  	  # are installed into $libdir/../bin (currently, that works fine)
diff --git a/support/libtool/buildroot-libtool-v2.2.patch b/support/libtool/buildroot-libtool-v2.2.patch
index 3af0604..be4e5e3 100644
--- a/support/libtool/buildroot-libtool-v2.2.patch
+++ b/support/libtool/buildroot-libtool-v2.2.patch
@@ -76,3 +76,21 @@
  	    fi
  	    ;;
  	  esac
+@@ -6094,7 +6093,7 @@
+ 		if test -n "$inst_prefix_dir"; then
+ 		  case $libdir in
+ 		    [\\/]*)
+-		      add_dir="$add_dir -L$inst_prefix_dir$libdir"
++		      add_dir="-L$inst_prefix_dir$libdir"
+ 		      ;;
+ 		  esac
+ 		fi
+@@ -6167,7 +6166,7 @@
+ 	      if test -n "$inst_prefix_dir"; then
+ 		case $libdir in
+ 		  [\\/]*)
+-		    add_dir="$add_dir -L$inst_prefix_dir$libdir"
++		    add_dir="-L$inst_prefix_dir$libdir"
+ 		    ;;
+ 		esac
+ 	      fi
diff --git a/support/libtool/buildroot-libtool-v2.4.4.patch b/support/libtool/buildroot-libtool-v2.4.4.patch
index d71669b..1458b5e 100644
--- a/support/libtool/buildroot-libtool-v2.4.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.4.patch
@@ -82,3 +82,12 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  	    fi
  	    ;;
  	  esac
+@@ -8854,7 +8854,7 @@
+ 	      if test -n "$inst_prefix_dir"; then
+ 		case $libdir in
+ 		  [\\/]*)
+-		    func_append add_dir " -L$inst_prefix_dir$libdir"
++		    add_dir="-L$inst_prefix_dir$libdir"
+ 		    ;;
+ 		esac
+ 	      fi
diff --git a/support/libtool/buildroot-libtool-v2.4.patch b/support/libtool/buildroot-libtool-v2.4.patch
index bf40c94..c2edd99 100644
--- a/support/libtool/buildroot-libtool-v2.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.patch
@@ -76,3 +76,12 @@
  	    fi
  	    ;;
  	  esac
+@@ -8854,7 +8854,7 @@
+ 	      if test -n "$inst_prefix_dir"; then
+ 		case $libdir in
+ 		  [\\/]*)
+-		    func_append add_dir " -L$inst_prefix_dir$libdir"
++		    add_dir="-L$inst_prefix_dir$libdir"
+ 		    ;;
+ 		esac
+ 	      fi
-- 
1.9.1

[Buildroot] [PATCH v5 14/19] infra-libtool: drop original $libdir (i.e. /usr/lib) from library paths

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> When it compute list of directories to include in link, libtool has a
                 ^s the                            ^the
> conservative behavior. It add `$inst_prefix_dir$libdir' but also keep original
                               ^s                                      ^s the
> `$libdir'. Thus, it call:
                          ^s
>    -L/usr/lib -L/buildroot/output/host/usr/arm-buildroot-linux-gnueabihf/sysroot/usr/lib
> 
> It works since the last path has priority. However, toolchan-wrapper consider
                                                                              ^s

 However, I think the first path has priority, no? According to man gcc: "the
directories are searched in the order specified". So I don't think this works.
In other words, I think patch 11/19 without this patch might break the build if
paranoid path is enabled. In that case, this patch should come first in the series.

> `/usr/lib' an an unsafe path.
             ^^as
> 
> This patch remove original `$libdir'.
                   ^s the
> 
> Since last versions of binutils provide `-Wpoison-system-directories' option, I
> think it make sense to try to upstream this patch.

 So, did you try to upstream it?

> 
> Note Yocto has a similar patch:
>   http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/use-sysroot-in-libpath.patch

 Well, yocto has a non-upstreamable patch :-)

> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)

 Regards,
 Arnout

> ---
>  support/libtool/buildroot-libtool-v1.5.patch   | 18 ++++++++++++++++++
>  support/libtool/buildroot-libtool-v2.2.patch   | 18 ++++++++++++++++++
>  support/libtool/buildroot-libtool-v2.4.4.patch |  9 +++++++++
>  support/libtool/buildroot-libtool-v2.4.patch   |  9 +++++++++
>  4 files changed, 54 insertions(+)
> 
> diff --git a/support/libtool/buildroot-libtool-v1.5.patch b/support/libtool/buildroot-libtool-v1.5.patch
> index 609006a..51d049f 100644
> --- a/support/libtool/buildroot-libtool-v1.5.patch
> +++ b/support/libtool/buildroot-libtool-v1.5.patch
> @@ -43,6 +43,24 @@
>   	  fi
>   	  # This is a shared library
>   
> +@@ -2412,7 +2419,7 @@
> + 		if test -n "$inst_prefix_dir"; then
> + 		  case "$libdir" in
> + 		    [\\/]*)
> +-		      add_dir="$add_dir -L$inst_prefix_dir$libdir"
> ++		      add_dir="-L$inst_prefix_dir$libdir"
> + 		      ;;
> + 		  esac
> + 		fi
> +@@ -2492,7 +2499,7 @@
> + 	      if test -n "$inst_prefix_dir"; then
> + 		case "$libdir" in
> + 		  [\\/]*)
> +-		    add_dir="$add_dir -L$inst_prefix_dir$libdir"
> ++		    add_dir="-L$inst_prefix_dir$libdir"
> + 		    ;;
> + 		esac
> + 	      fi
>  @@ -5487,10 +5494,13 @@
>   	  # At present, this check doesn't affect windows .dll's that
>   	  # are installed into $libdir/../bin (currently, that works fine)
> diff --git a/support/libtool/buildroot-libtool-v2.2.patch b/support/libtool/buildroot-libtool-v2.2.patch
> index 3af0604..be4e5e3 100644
> --- a/support/libtool/buildroot-libtool-v2.2.patch
> +++ b/support/libtool/buildroot-libtool-v2.2.patch
> @@ -76,3 +76,21 @@
>   	    fi
>   	    ;;
>   	  esac
> +@@ -6094,7 +6093,7 @@
> + 		if test -n "$inst_prefix_dir"; then
> + 		  case $libdir in
> + 		    [\\/]*)
> +-		      add_dir="$add_dir -L$inst_prefix_dir$libdir"
> ++		      add_dir="-L$inst_prefix_dir$libdir"
> + 		      ;;
> + 		  esac
> + 		fi
> +@@ -6167,7 +6166,7 @@
> + 	      if test -n "$inst_prefix_dir"; then
> + 		case $libdir in
> + 		  [\\/]*)
> +-		    add_dir="$add_dir -L$inst_prefix_dir$libdir"
> ++		    add_dir="-L$inst_prefix_dir$libdir"
> + 		    ;;
> + 		esac
> + 	      fi
> diff --git a/support/libtool/buildroot-libtool-v2.4.4.patch b/support/libtool/buildroot-libtool-v2.4.4.patch
> index d71669b..1458b5e 100644
> --- a/support/libtool/buildroot-libtool-v2.4.4.patch
> +++ b/support/libtool/buildroot-libtool-v2.4.4.patch
> @@ -82,3 +82,12 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
>   	    fi
>   	    ;;
>   	  esac
> +@@ -8854,7 +8854,7 @@
> + 	      if test -n "$inst_prefix_dir"; then
> + 		case $libdir in
> + 		  [\\/]*)
> +-		    func_append add_dir " -L$inst_prefix_dir$libdir"
> ++		    add_dir="-L$inst_prefix_dir$libdir"
> + 		    ;;
> + 		esac
> + 	      fi
> diff --git a/support/libtool/buildroot-libtool-v2.4.patch b/support/libtool/buildroot-libtool-v2.4.patch
> index bf40c94..c2edd99 100644
> --- a/support/libtool/buildroot-libtool-v2.4.patch
> +++ b/support/libtool/buildroot-libtool-v2.4.patch
> @@ -76,3 +76,12 @@
>   	    fi
>   	    ;;
>   	  esac
> +@@ -8854,7 +8854,7 @@
> + 	      if test -n "$inst_prefix_dir"; then
> + 		case $libdir in
> + 		  [\\/]*)
> +-		    func_append add_dir " -L$inst_prefix_dir$libdir"
> ++		    add_dir="-L$inst_prefix_dir$libdir"
> + 		    ;;
> + 		esac
> + 	      fi
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 15/19] infra-libtool: relink binaries on install

[Jérôme Pouiller]

Until now, libtool was not configured for cross-compilation. It did works,
because Buildroot patched $libdir in .la files to include directories from
sysroot. However, this was done after package installation. Thus, a package was
not able to use its own (not yet patched) .la files. Because of that, libtool
was not able to relink binaries during installation. So, Buildroot disabled
relink on installation. Consequently, binaries that depends on internal
libraries contains their build path in their RPATH:

  $ readelf -d target/bin/mount | grep RPATH
  0x0000000f (RPATH)  Library rpath: [/home/buildbot/output/build/util-linux-2.27.1/.libs]

This particularity was incompatible with creation of reproducible binaries.

However, with previous changes, libtool is now correctly configured and
binaries can be relinked on install. So we can re-enable this feature.

Note Yocto have a patch that remove extra paths during compilation and make
relink unnecessary:
   http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/fixinstall.patch

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 support/libtool/buildroot-libtool-v1.5.patch   | 9 ---------
 support/libtool/buildroot-libtool-v2.2.patch   | 9 ---------
 support/libtool/buildroot-libtool-v2.4.4.patch | 9 ---------
 support/libtool/buildroot-libtool-v2.4.patch   | 9 ---------
 4 files changed, 36 deletions(-)

diff --git a/support/libtool/buildroot-libtool-v1.5.patch b/support/libtool/buildroot-libtool-v1.5.patch
index 51d049f..0429c58 100644
--- a/support/libtool/buildroot-libtool-v1.5.patch
+++ b/support/libtool/buildroot-libtool-v1.5.patch
@@ -34,15 +34,6 @@
  	if test -n "$link_static_flag"; then
  	  compile_command="$compile_command $link_static_flag"
  	  finalize_command="$finalize_command $link_static_flag"
-@@ -2272,7 +2279,7 @@
- 	   { test "$prefer_static_libs" = no || test -z "$old_library"; }; then
- 	  if test "$installed" = no; then
- 	    notinst_deplibs="$notinst_deplibs $lib"
--	    need_relink=yes
-+	    need_relink=no
- 	  fi
- 	  # This is a shared library
- 
 @@ -2412,7 +2419,7 @@
  		if test -n "$inst_prefix_dir"; then
  		  case "$libdir" in
diff --git a/support/libtool/buildroot-libtool-v2.2.patch b/support/libtool/buildroot-libtool-v2.2.patch
index be4e5e3..b2058b4 100644
--- a/support/libtool/buildroot-libtool-v2.2.patch
+++ b/support/libtool/buildroot-libtool-v2.2.patch
@@ -67,15 +67,6 @@
  	# The effects of -static are defined in a previous loop.
  	# We used to do the same as -all-static on platforms that
  	# didn't have a PIC flag, but the assumption that the effects
-@@ -5891,7 +5890,7 @@
- 	  *)
- 	    if test "$installed" = no; then
- 	      notinst_deplibs="$notinst_deplibs $lib"
--	      need_relink=yes
-+	      need_relink=no
- 	    fi
- 	    ;;
- 	  esac
 @@ -6094,7 +6093,7 @@
  		if test -n "$inst_prefix_dir"; then
  		  case $libdir in
diff --git a/support/libtool/buildroot-libtool-v2.4.4.patch b/support/libtool/buildroot-libtool-v2.4.4.patch
index 1458b5e..8e25d3d 100644
--- a/support/libtool/buildroot-libtool-v2.4.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.4.patch
@@ -73,15 +73,6 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  	# The effects of -static are defined in a previous loop.
  	# We used to do the same as -all-static on platforms that
  	# didn't have a PIC flag, but the assumption that the effects
-@@ -8112,7 +8112,7 @@
- 	  *)
- 	    if test no = "$installed"; then
- 	      func_append notinst_deplibs " $lib"
--	      need_relink=yes
-+	      need_relink=no
- 	    fi
- 	    ;;
- 	  esac
 @@ -8854,7 +8854,7 @@
  	      if test -n "$inst_prefix_dir"; then
  		case $libdir in
diff --git a/support/libtool/buildroot-libtool-v2.4.patch b/support/libtool/buildroot-libtool-v2.4.patch
index c2edd99..c475ec9 100644
--- a/support/libtool/buildroot-libtool-v2.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.patch
@@ -67,15 +67,6 @@
  	# The effects of -static are defined in a previous loop.
  	# We used to do the same as -all-static on platforms that
  	# didn't have a PIC flag, but the assumption that the effects
-@@ -6697,7 +6696,7 @@
- 	  *)
- 	    if test "$installed" = no; then
- 	      func_append notinst_deplibs " $lib"
--	      need_relink=yes
-+	      need_relink=no
- 	    fi
- 	    ;;
- 	  esac
 @@ -8854,7 +8854,7 @@
  	      if test -n "$inst_prefix_dir"; then
  		case $libdir in
-- 
1.9.1

[Buildroot] [PATCH v5 15/19] infra-libtool: relink binaries on install

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> Until now, libtool was not configured for cross-compilation. It did works,
                                                                      work
> because Buildroot patched $libdir in .la files to include directories from
> sysroot. However, this was done after package installation. Thus, a package was
> not able to use its own (not yet patched) .la files. Because of that, libtool
> was not able to relink binaries during installation. So, Buildroot disabled
> relink on installation. Consequently, binaries that depends on internal
                                                      depend
> libraries contains their build path in their RPATH:
            contain
> 
>   $ readelf -d target/bin/mount | grep RPATH
>   0x0000000f (RPATH)  Library rpath: [/home/buildbot/output/build/util-linux-2.27.1/.libs]
> 
> This particularity was incompatible with creation of reproducible binaries.

 Note that Wolfgang's RPATH mangling patches will also fix this. But of course,
not having the problem to begin with is better :-)

> 
> However, with previous changes, libtool is now correctly configured and
> binaries can be relinked on install. So we can re-enable this feature.
> 
> Note Yocto have a patch that remove extra paths during compilation and make
> relink unnecessary:
>    http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/fixinstall.patch
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)


 Regards,
 Arnout

> ---
>  support/libtool/buildroot-libtool-v1.5.patch   | 9 ---------
>  support/libtool/buildroot-libtool-v2.2.patch   | 9 ---------
>  support/libtool/buildroot-libtool-v2.4.4.patch | 9 ---------
>  support/libtool/buildroot-libtool-v2.4.patch   | 9 ---------
>  4 files changed, 36 deletions(-)
> 
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 16/19] infra-libtool: inform libtool that STAGING_DIR is reachable at runtime

[Jérôme Pouiller]

When installing binaries to $TARGET_DIR, libtool add `$STAGING_DIR{,/usr}/lib'
to RPATH. This is annoying to create reproducible binaries. This patch just
inform libtool that these paths are reachable on runtime and it is necessary to
add then to RPATH.

Notice `$lt_cv_sys_lib_dlsearch_path_spec' was introduced in libtool v2.0.
Packages with libtool v1.5 continue to include `$STAGING_DIR{,/usr}/lib' in
their RPATH. The few packages that use libtool v1.5 have to be patched
individually in order to make them reproducible (adding AUTORECONF=YES should
be sufficient in most cases).

Note, instead of providing $lt_cv_sys_lib_dlsearch_path_spec, Yocto patch
libtool:
   http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/fix-final-rpath.patch

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/Makefile.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/Makefile.in b/package/Makefile.in
index 688d5b1..6a341f2 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -355,6 +355,7 @@ TARGET_CONFIGURE_ARGS = \
 	ac_cv_func_calloc_0_nonnull=yes \
 	ac_cv_func_realloc_0_nonnull=yes \
 	lt_cv_sys_lib_search_path_spec="" \
+	lt_cv_sys_lib_dlsearch_path_spec="/lib /usr/lib $(STAGING_DIR)/lib $(STAGING_DIR)/usr/lib" \
 	$(BR2_AC_CV_C_BIGENDIAN) \
 	$(BR2_GT_CV_FUNC_GNUGETTEXT_LIBC)
 
-- 
1.9.1

[Buildroot] [PATCH v5 16/19] infra-libtool: inform libtool that STAGING_DIR is reachable at runtime

[Samuel Martin]

Hi J?r?me,

On Tue, Dec 20, 2016 at 2:46 PM, J?r?me Pouiller <jezz@sysmic.org> wrote:
> When installing binaries to $TARGET_DIR, libtool add `$STAGING_DIR{,/usr}/lib'
> to RPATH. This is annoying to create reproducible binaries. This patch just
> inform libtool that these paths are reachable on runtime and it is necessary to

s/it is necessary/it is not necessary/

> add then to RPATH.
>
> Notice `$lt_cv_sys_lib_dlsearch_path_spec' was introduced in libtool v2.0.
> Packages with libtool v1.5 continue to include `$STAGING_DIR{,/usr}/lib' in
> their RPATH. The few packages that use libtool v1.5 have to be patched
> individually in order to make them reproducible (adding AUTORECONF=YES should
> be sufficient in most cases).
>
> Note, instead of providing $lt_cv_sys_lib_dlsearch_path_spec, Yocto patch
> libtool:
>    http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/fix-final-rpath.patch
>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
>  package/Makefile.in | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 688d5b1..6a341f2 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -355,6 +355,7 @@ TARGET_CONFIGURE_ARGS = \
>         ac_cv_func_calloc_0_nonnull=yes \
>         ac_cv_func_realloc_0_nonnull=yes \
>         lt_cv_sys_lib_search_path_spec="" \
> +       lt_cv_sys_lib_dlsearch_path_spec="/lib /usr/lib $(STAGING_DIR)/lib $(STAGING_DIR)/usr/lib" \
>         $(BR2_AC_CV_C_BIGENDIAN) \
>         $(BR2_GT_CV_FUNC_GNUGETTEXT_LIBC)
>
> --
> 1.9.1
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

Regards,

-- 
Samuel

[Buildroot] [PATCH v5 16/19] infra-libtool: inform libtool that STAGING_DIR is reachable at runtime

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> When installing binaries to $TARGET_DIR, libtool add `$STAGING_DIR{,/usr}/lib'
                                                      ^s
> to RPATH. This is annoying to create reproducible binaries. This patch just
> inform libtool that these paths are reachable on runtime and it is necessary to
        ^s                                      at                  ^not
> add then to RPATH.
      them
> 
> Notice `$lt_cv_sys_lib_dlsearch_path_spec' was introduced in libtool v2.0.
> Packages with libtool v1.5 continue to include `$STAGING_DIR{,/usr}/lib' in
> their RPATH. The few packages that use libtool v1.5 have to be patched
> individually in order to make them reproducible (adding AUTORECONF=YES should
> be sufficient in most cases).

 Again, this is also solved by Wolfgang's rpath series.

> 
> Note, instead of providing $lt_cv_sys_lib_dlsearch_path_spec, Yocto patch
> libtool:
>    http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-devtools/libtool/libtool/fix-final-rpath.patch
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)

 Since this is already handled by Wolgang's rpath series, this patch could be
dropped. However, it is not invasive and it solves the problem earlier, so I vot
to keep it.

 Regards,
 Arnout

> ---
>  package/Makefile.in | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 688d5b1..6a341f2 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -355,6 +355,7 @@ TARGET_CONFIGURE_ARGS = \
>  	ac_cv_func_calloc_0_nonnull=yes \
>  	ac_cv_func_realloc_0_nonnull=yes \
>  	lt_cv_sys_lib_search_path_spec="" \
> +	lt_cv_sys_lib_dlsearch_path_spec="/lib /usr/lib $(STAGING_DIR)/lib $(STAGING_DIR)/usr/lib" \
>  	$(BR2_AC_CV_C_BIGENDIAN) \
>  	$(BR2_GT_CV_FUNC_GNUGETTEXT_LIBC)
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 17/19] infra-libtool: no longer disable install directory sanity check

[Jérôme Pouiller]

During install, libtool check if $libdir is a subdirectory of $DESTDIR. Until
now $DESTDIR was $(TARGET_DIR) and $libdir was $(STAGING_DIR)/usr/lib.
Consequently, this test failed. Since $libdir is now /usr/lib, there is not more
reason to skipped that test.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 support/libtool/buildroot-libtool-v1.5.patch   | 18 ------------------
 support/libtool/buildroot-libtool-v2.2.patch   | 26 ++++++--------------------
 support/libtool/buildroot-libtool-v2.4.4.patch | 25 +++++--------------------
 support/libtool/buildroot-libtool-v2.4.patch   | 24 +++++-------------------
 4 files changed, 16 insertions(+), 77 deletions(-)

diff --git a/support/libtool/buildroot-libtool-v1.5.patch b/support/libtool/buildroot-libtool-v1.5.patch
index 0429c58..3d78831 100644
--- a/support/libtool/buildroot-libtool-v1.5.patch
+++ b/support/libtool/buildroot-libtool-v1.5.patch
@@ -52,21 +52,3 @@
  		    ;;
  		esac
  	      fi
-@@ -5487,10 +5494,13 @@
- 	  # At present, this check doesn't affect windows .dll's that
- 	  # are installed into $libdir/../bin (currently, that works fine)
- 	  # but it's something to keep an eye on.
--	  if test "$inst_prefix_dir" = "$destdir"; then
--	    $echo "$modename: error: cannot install \`$file' to a directory not ending in $libdir" 1>&2
--	    exit $EXIT_FAILURE
--	  fi
-+	  #
-+	  # This breaks install into our staging area.  -PB
-+	  # 
-+	  # if test "$inst_prefix_dir" = "$destdir"; then
-+	  #   $echo "$modename: error: cannot install \`$file' to a directory not ending in $libdir" 1>&2
-+	  #   exit $EXIT_FAILURE
-+	  # fi
- 
- 	  if test -n "$inst_prefix_dir"; then
- 	    # Stick the inst_prefix_dir data into the link command.
diff --git a/support/libtool/buildroot-libtool-v2.2.patch b/support/libtool/buildroot-libtool-v2.2.patch
index b2058b4..6d9bac7 100644
--- a/support/libtool/buildroot-libtool-v2.2.patch
+++ b/support/libtool/buildroot-libtool-v2.2.patch
@@ -11,21 +11,7 @@
  #	else
  #	  func_verbose "using $tagname tagged configuration"
  	fi
-@@ -2239,8 +2239,11 @@
- 	  # At present, this check doesn't affect windows .dll's that
- 	  # are installed into $libdir/../bin (currently, that works fine)
- 	  # but it's something to keep an eye on.
--	  test "$inst_prefix_dir" = "$destdir" && \
--	    func_fatal_error "error: cannot install \`$file' to a directory not ending in $libdir"
-+	  #
-+	  # This breaks install into our staging area.  -PB
-+	  #
-+	  # test "$inst_prefix_dir" = "$destdir" && \
-+	  #   func_fatal_error "error: cannot install \`$file' to a directory not ending in $libdir"
- 
- 	  if test -n "$inst_prefix_dir"; then
- 	    # Stick the inst_prefix_dir data into the link command.
-@@ -4429,7 +4432,8 @@
+@@ -4429,7 +4429,8 @@
  	;;
        -all-static | -static | -static-libtool-libs)
  	case $arg in
@@ -35,7 +21,7 @@
  	  if test "$build_libtool_libs" = yes && test -z "$link_static_flag"; then
  	    func_warning "complete static linking is impossible in this configuration"
  	  fi
-@@ -4438,12 +4442,6 @@
+@@ -4438,12 +4439,6 @@
  	  fi
  	  prefer_static_libs=yes
  	  ;;
@@ -48,7 +34,7 @@
  	-static-libtool-libs)
  	  if test -z "$pic_flag" && test -n "$link_static_flag"; then
  	    dlopen_self=$dlopen_self_static
-@@ -4726,7 +4724,8 @@
+@@ -4726,7 +4721,8 @@
        prevarg="$arg"
  
        case $arg in
@@ -58,7 +44,7 @@
  	if test -n "$link_static_flag"; then
  	  # See comment for -static flag below, for more details.
  	  func_append compile_command " $link_static_flag"
-@@ -4995,7 +4994,7 @@
+@@ -4995,7 +4991,7 @@
  	continue
  	;;
  
@@ -67,7 +53,7 @@
  	# The effects of -static are defined in a previous loop.
  	# We used to do the same as -all-static on platforms that
  	# didn't have a PIC flag, but the assumption that the effects
-@@ -6094,7 +6093,7 @@
+@@ -6094,7 +6090,7 @@
  		if test -n "$inst_prefix_dir"; then
  		  case $libdir in
  		    [\\/]*)
@@ -76,7 +62,7 @@
  		      ;;
  		  esac
  		fi
-@@ -6167,7 +6166,7 @@
+@@ -6167,7 +6163,7 @@
  	      if test -n "$inst_prefix_dir"; then
  		case $libdir in
  		  [\\/]*)
diff --git a/support/libtool/buildroot-libtool-v2.4.4.patch b/support/libtool/buildroot-libtool-v2.4.4.patch
index 8e25d3d..2400aa3 100644
--- a/support/libtool/buildroot-libtool-v2.4.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.4.patch
@@ -16,22 +16,7 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  #	else
  #	  func_verbose "using $tagname tagged configuration"
  	fi
-@@ -4277,8 +4277,12 @@
- 	  # At present, this check doesn't affect windows .dll's that
- 	  # are installed into $libdir/../bin (currently, that works fine)
- 	  # but it's something to keep an eye on.
--	  test "$inst_prefix_dir" = "$destdir" && \
--	    func_fatal_error "error: cannot install '$file' to a directory not ending in $libdir"
-+
-+	  #
-+	  # This breaks install into our staging area.  -PB
-+	  #
-+	  #test "$inst_prefix_dir" = "$destdir" && \
-+	  #  func_fatal_error "error: cannot install '$file' to a directory not ending in $libdir"
- 
- 	  if test -n "$inst_prefix_dir"; then
- 	    # Stick the inst_prefix_dir data into the link command.
-@@ -6550,7 +6554,8 @@
+@@ -6550,7 +6550,8 @@
  	;;
        -all-static | -static | -static-libtool-libs)
  	case $arg in
@@ -41,7 +26,7 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  	  if test yes = "$build_libtool_libs" && test -z "$link_static_flag"; then
  	    func_warning "complete static linking is impossible in this configuration"
  	  fi
-@@ -6559,12 +6564,6 @@
+@@ -6559,12 +6560,6 @@
  	  fi
  	  prefer_static_libs=yes
  	  ;;
@@ -54,7 +39,7 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  	-static-libtool-libs)
  	  if test -z "$pic_flag" && test -n "$link_static_flag"; then
  	    dlopen_self=$dlopen_self_static
-@@ -6858,7 +6857,8 @@
+@@ -6858,7 +6853,8 @@
        prevarg=$arg
  
        case $arg in
@@ -64,7 +49,7 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  	if test -n "$link_static_flag"; then
  	  # See comment for -static flag below, for more details.
  	  func_append compile_command " $link_static_flag"
-@@ -7149,7 +7149,7 @@
+@@ -7149,7 +7145,7 @@
  	continue
  	;;
  
@@ -73,7 +58,7 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
  	# The effects of -static are defined in a previous loop.
  	# We used to do the same as -all-static on platforms that
  	# didn't have a PIC flag, but the assumption that the effects
-@@ -8854,7 +8854,7 @@
+@@ -8854,7 +8850,7 @@
  	      if test -n "$inst_prefix_dir"; then
  		case $libdir in
  		  [\\/]*)
diff --git a/support/libtool/buildroot-libtool-v2.4.patch b/support/libtool/buildroot-libtool-v2.4.patch
index c475ec9..7165bea 100644
--- a/support/libtool/buildroot-libtool-v2.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.patch
@@ -11,21 +11,7 @@
  #	else
  #	  func_verbose "using $tagname tagged configuration"
  	fi
-@@ -2963,8 +2963,11 @@
- 	  # At present, this check doesn't affect windows .dll's that
- 	  # are installed into $libdir/../bin (currently, that works fine)
- 	  # but it's something to keep an eye on.
--	  test "$inst_prefix_dir" = "$destdir" && \
--	    func_fatal_error "error: cannot install \`$file' to a directory not ending in $libdir"
-+	  #
-+	  # This breaks install into our staging area.  -PB
-+	  #
-+	  # test "$inst_prefix_dir" = "$destdir" && \
-+	  #   func_fatal_error "error: cannot install \`$file' to a directory not ending in $libdir"
- 
- 	  if test -n "$inst_prefix_dir"; then
- 	    # Stick the inst_prefix_dir data into the link command.
-@@ -5176,7 +5179,8 @@
+@@ -5176,7 +5176,8 @@
  	;;
        -all-static | -static | -static-libtool-libs)
  	case $arg in
@@ -35,7 +21,7 @@
  	  if test "$build_libtool_libs" = yes && test -z "$link_static_flag"; then
  	    func_warning "complete static linking is impossible in this configuration"
  	  fi
-@@ -5185,12 +5189,6 @@
+@@ -5185,12 +5186,6 @@
  	  fi
  	  prefer_static_libs=yes
  	  ;;
@@ -48,7 +34,7 @@
  	-static-libtool-libs)
  	  if test -z "$pic_flag" && test -n "$link_static_flag"; then
  	    dlopen_self=$dlopen_self_static
-@@ -5473,7 +5471,8 @@
+@@ -5473,7 +5468,8 @@
        prevarg="$arg"
  
        case $arg in
@@ -58,7 +44,7 @@
  	if test -n "$link_static_flag"; then
  	  # See comment for -static flag below, for more details.
  	  func_append compile_command " $link_static_flag"
-@@ -5754,7 +5753,7 @@
+@@ -5754,7 +5750,7 @@
  	continue
  	;;
  
@@ -67,7 +53,7 @@
  	# The effects of -static are defined in a previous loop.
  	# We used to do the same as -all-static on platforms that
  	# didn't have a PIC flag, but the assumption that the effects
-@@ -8854,7 +8854,7 @@
+@@ -8854,7 +8851,7 @@
  	      if test -n "$inst_prefix_dir"; then
  		case $libdir in
  		  [\\/]*)
-- 
1.9.1

[Buildroot] [PATCH v5 17/19] infra-libtool: no longer disable install directory sanity check

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> During install, libtool check if $libdir is a subdirectory of $DESTDIR. Until
                               ^s
> now $DESTDIR was $(TARGET_DIR) and $libdir was $(STAGING_DIR)/usr/lib.
> Consequently, this test failed. Since $libdir is now /usr/lib, there is not more
                                 ^Therefore, the Buildroot libtool patch
disabled that test.
                                                                          no
> reason to skipped that test.
            skip
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)


 Regards,
 Arnout


> ---
>  support/libtool/buildroot-libtool-v1.5.patch   | 18 ------------------
>  support/libtool/buildroot-libtool-v2.2.patch   | 26 ++++++--------------------
>  support/libtool/buildroot-libtool-v2.4.4.patch | 25 +++++--------------------
>  support/libtool/buildroot-libtool-v2.4.patch   | 24 +++++-------------------
>  4 files changed, 16 insertions(+), 77 deletions(-)
[snip]
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 18/19] infra-libtool: remove workaround for calls without `--tag'

[Jérôme Pouiller]

Buildroot included a work around for the case libtool was called with a
non-existant `--tag'. We was not able create reproduce this problem anymore. So,
we drop this workaround.

Please send a bug report if you encounter this problem again.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 support/libtool/buildroot-libtool-v1.5.patch   | 12 ------------
 support/libtool/buildroot-libtool-v2.2.patch   | 11 -----------
 support/libtool/buildroot-libtool-v2.4.4.patch | 11 -----------
 support/libtool/buildroot-libtool-v2.4.patch   | 11 -----------
 4 files changed, 45 deletions(-)

diff --git a/support/libtool/buildroot-libtool-v1.5.patch b/support/libtool/buildroot-libtool-v1.5.patch
index 3d78831..3664221 100644
--- a/support/libtool/buildroot-libtool-v1.5.patch
+++ b/support/libtool/buildroot-libtool-v1.5.patch
@@ -1,17 +1,5 @@
 --- a/ltmain.sh	2014-07-31 10:43:01.634568799 -0300
 +++ b/ltmain.sh	2014-07-31 10:44:14.032018999 -0300
-@@ -226,8 +226,9 @@
- 	# line option must be used.
- 	if test -z "$tagname"; then
- 	  $echo "$modename: unable to infer tagged configuration"
--	  $echo "$modename: specify a tag with \`--tag'" 1>&2
--	  exit $EXIT_FAILURE
-+	  $echo "$modename: defaulting to \`CC'"
-+	  $echo "$modename: if this is not correct, specify a tag with \`--tag'"
-+#	  exit $EXIT_FAILURE
- #        else
- #          $echo "$modename: using $tagname tagged configuration"
- 	fi
 @@ -247,6 +248,11 @@
    arg="$1"
    shift
diff --git a/support/libtool/buildroot-libtool-v2.2.patch b/support/libtool/buildroot-libtool-v2.2.patch
index 6d9bac7..c7dc33c 100644
--- a/support/libtool/buildroot-libtool-v2.2.patch
+++ b/support/libtool/buildroot-libtool-v2.2.patch
@@ -1,16 +1,5 @@
 --- a/ltmain.sh	2010-06-09 15:08:53.000000000 +0200
 +++ b/ltmain.sh	2014-07-30 22:33:41.176710372 +0200
-@@ -1214,8 +1214,8 @@
- 	# was found and let the user know that the "--tag" command
- 	# line option must be used.
- 	if test -z "$tagname"; then
--	  func_echo "unable to infer tagged configuration"
--	  func_fatal_error "specify a tag with \`--tag'"
-+	  func_echo "defaulting to \`CC'"
-+	  func_echo "if this is not correct, specify a tag with \`--tag'"
- #	else
- #	  func_verbose "using $tagname tagged configuration"
- 	fi
 @@ -4429,7 +4429,8 @@
  	;;
        -all-static | -static | -static-libtool-libs)
diff --git a/support/libtool/buildroot-libtool-v2.4.4.patch b/support/libtool/buildroot-libtool-v2.4.4.patch
index 2400aa3..ddd5675 100644
--- a/support/libtool/buildroot-libtool-v2.4.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.4.patch
@@ -5,17 +5,6 @@ Signed-off-by: Gustavo Zacarias <gustavo@zacarias.com.ar>
 
 --- a/ltmain.sh	2014-12-16 09:51:23.068441045 -0300
 +++ b/ltmain.sh	2014-12-16 09:57:10.509430339 -0300
-@@ -2687,8 +2687,8 @@
- 	# was found and let the user know that the "--tag" command
- 	# line option must be used.
- 	if test -z "$tagname"; then
--	  func_echo "unable to infer tagged configuration"
--	  func_fatal_error "specify a tag with '--tag'"
-+	  func_echo "defaulting to \`CC'"
-+	  func_echo "if this is not correct, specify a tag with \`--tag'"
- #	else
- #	  func_verbose "using $tagname tagged configuration"
- 	fi
 @@ -6550,7 +6550,8 @@
  	;;
        -all-static | -static | -static-libtool-libs)
diff --git a/support/libtool/buildroot-libtool-v2.4.patch b/support/libtool/buildroot-libtool-v2.4.patch
index 7165bea..0008980 100644
--- a/support/libtool/buildroot-libtool-v2.4.patch
+++ b/support/libtool/buildroot-libtool-v2.4.patch
@@ -1,16 +1,5 @@
 --- a/ltmain.sh	2014-07-30 22:21:24.664684143 +0200
 +++ b/ltmain.sh	2014-07-30 22:23:02.440687625 +0200
-@@ -1417,8 +1417,8 @@
- 	# was found and let the user know that the "--tag" command
- 	# line option must be used.
- 	if test -z "$tagname"; then
--	  func_echo "unable to infer tagged configuration"
--	  func_fatal_error "specify a tag with \`--tag'"
-+	  func_echo "defaulting to \`CC'"
-+	  func_echo "if this is not correct, specify a tag with \`--tag'"
- #	else
- #	  func_verbose "using $tagname tagged configuration"
- 	fi
 @@ -5176,7 +5176,8 @@
  	;;
        -all-static | -static | -static-libtool-libs)
-- 
1.9.1

[Buildroot] [PATCH v5 18/19] infra-libtool: remove workaround for calls without `--tag'

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> Buildroot included a work around for the case libtool was called with a
> non-existant `--tag'. We was not able create reproduce this problem anymore. So,
> we drop this workaround.

 It's better to refer to the commit adding it. Something like:

Buildroot included a work around for the case libtool was called with a
non-existent '--tag'. This was introduced as a series of hacks to make
cross-compilation of fontconfig work in commit ee8f2133, ten years ago. The hack
was carried over in various incarnations of the libtool patch, but is no longer
relevant.

> 
> Please send a bug report if you encounter this problem again.

 I don't think this sentence is necessary. The hack is just too far out - if the
package build system calls libtool with an unknown --tag option, something is
truly rotten.

> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)


 Regards,
 Arnout

> ---
>  support/libtool/buildroot-libtool-v1.5.patch   | 12 ------------
>  support/libtool/buildroot-libtool-v2.2.patch   | 11 -----------
>  support/libtool/buildroot-libtool-v2.4.4.patch | 11 -----------
>  support/libtool/buildroot-libtool-v2.4.patch   | 11 -----------
>  4 files changed, 45 deletions(-)
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 19/19] infra-libtool: no longer force sys_lib_search_path

[Jérôme Pouiller]

Commit 6246704008 forced $sys_lib_search_path_spec to be empty. Commit log said
that it was filled with native paths. However, recent tests show that
$sys_lib_search_path_spec is correctly filled with result of
`$CC -print-search-dirs' as it should.

So this patch remove this workaround.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
 package/Makefile.in | 1 -
 1 file changed, 1 deletion(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 6a341f2..ee43985 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -354,7 +354,6 @@ TARGET_CONFIGURE_ARGS = \
 	ac_cv_func_malloc_0_nonnull=yes \
 	ac_cv_func_calloc_0_nonnull=yes \
 	ac_cv_func_realloc_0_nonnull=yes \
-	lt_cv_sys_lib_search_path_spec="" \
 	lt_cv_sys_lib_dlsearch_path_spec="/lib /usr/lib $(STAGING_DIR)/lib $(STAGING_DIR)/usr/lib" \
 	$(BR2_AC_CV_C_BIGENDIAN) \
 	$(BR2_GT_CV_FUNC_GNUGETTEXT_LIBC)
-- 
1.9.1

[Buildroot] [PATCH v5 19/19] infra-libtool: no longer force sys_lib_search_path

[Arnout Vandecappelle]

On 20-12-16 14:46, J?r?me Pouiller wrote:
> Commit 6246704008 forced $sys_lib_search_path_spec to be empty. Commit log said
> that it was filled with native paths. However, recent tests show that
> $sys_lib_search_path_spec is correctly filled with result of
> `$CC -print-search-dirs' as it should.

 I guess that that's due to passing --with-sysroot, no?

> 
> So this patch remove this workaround.
                      ^s

> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>


Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
(tested on a static arm musl build with about 20 packages)


 Regards,
 Arnout

> ---
>  package/Makefile.in | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 6a341f2..ee43985 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -354,7 +354,6 @@ TARGET_CONFIGURE_ARGS = \
>  	ac_cv_func_malloc_0_nonnull=yes \
>  	ac_cv_func_calloc_0_nonnull=yes \
>  	ac_cv_func_realloc_0_nonnull=yes \
> -	lt_cv_sys_lib_search_path_spec="" \
>  	lt_cv_sys_lib_dlsearch_path_spec="/lib /usr/lib $(STAGING_DIR)/lib $(STAGING_DIR)/usr/lib" \
>  	$(BR2_AC_CV_C_BIGENDIAN) \
>  	$(BR2_GT_CV_FUNC_GNUGETTEXT_LIBC)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v5 00/19] Reproducible builds

[Jérôme Pouiller]

Hello, 

On Tuesday 20 December 2016 14:46:17 CET J?r?me Pouiller wrote:
[...]
> Patches 11 to 19 are nearly a series in the series. Until now, most of binaries
> installed by libtool was configured with RPATH pointingto their build
> directory. Indeed, libtool add this path during compilation in order to be able
> to execute them directly from build directory. This path should normally
> removed during install, but Buildroot disable this behavior (patch 15). Simply
> re-enabling this behavior does not work. Indeed, during relink, libtool try to
> use .la that are not yet patched and fail to find libraries. On another side,
> libtool support usage of a sysroot since v1.5. To enable this support, we have
> to keep original values from .la file (patch 12 and 13) and inform libtool we
> are using a sysroot (patch 11).
> 
> Patch 14 fix a small incompatibility with unsafe path detection.
> 
> Since libtool is now correctly used, it is not more necessary to disable
> install directory sanity check (patch 17).
> 
> From libtool point of view, sysroot is not reachable from $(TARGET_DIR). So,
> during installation to $(TARGET_DIR), it add an entry in RPATH that point to
> $(STAGING_DIR). To fix this problem, we just have to inform libtool that
> $(STAGING_DIR) is reachable (patch 16).
> 
> Finally, I also clean up libtool infra from modification that seems useless now
> (patches 18 and 19).
> 
> I tested this series using internal toolchain and Linaro toolchain with these
> packages:
> 
>     BR2_INIT_SYSTEMD=y       (install public libraries in /usr/lib/systemd)
>     BR2_PACKAGE_LIBCDAUDIO=y (libtool 1.5)
>     BR2_PACKAGE_LIBLO=y      (libtool 2.2)
>     BR2_PACKAGE_MADPLAY=y    (unpatched libtool)
>     BR2_PACKAGE_ALSA_LIB=y   (optional dependency to Madplay)
>     BR2_PACKAGE_PYTHON=y
>     BR2_PACKAGE_PYTHON_PY_PYC=y
>     BR2_PACKAGE_GNUPG2=y
> 
> Except patches 12 and 13, I think whole series is bisectable.
[...]

As I said above, patches 11 to 19 of this series do a strong clean up
of libtool infrastructure. I think these patches are important for two
reasons:

   1. It is the best way to support reproducible build from different
      build paths
   2. Currently BR usage of libtool is not standard. These patches use
      libtool in the right way. It drop unnecessary RPATH from binaries.
      It drop most of patches applied to libtool.

However, this series do very intrusive changes and it is difficult to
review. Consequently, I think it is necessary to apply them very early
in a development cycle. So, I suggest to apply them to 'next' now.

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v5 00/19] Reproducible builds

[Arnout Vandecappelle]

On 21-02-17 09:17, J?r?me Pouiller wrote:
> Hello, 
> 
> On Tuesday 20 December 2016 14:46:17 CET J?r?me Pouiller wrote:
> [...]
>> Patches 11 to 19 are nearly a series in the series. Until now, most of binaries
>> installed by libtool was configured with RPATH pointingto their build
>> directory. Indeed, libtool add this path during compilation in order to be able
>> to execute them directly from build directory. This path should normally
>> removed during install, but Buildroot disable this behavior (patch 15). Simply
>> re-enabling this behavior does not work. Indeed, during relink, libtool try to
>> use .la that are not yet patched and fail to find libraries. On another side,
>> libtool support usage of a sysroot since v1.5. To enable this support, we have
>> to keep original values from .la file (patch 12 and 13) and inform libtool we
>> are using a sysroot (patch 11).
>>
>> Patch 14 fix a small incompatibility with unsafe path detection.
>>
>> Since libtool is now correctly used, it is not more necessary to disable
>> install directory sanity check (patch 17).
>>
>> From libtool point of view, sysroot is not reachable from $(TARGET_DIR). So,
>> during installation to $(TARGET_DIR), it add an entry in RPATH that point to
>> $(STAGING_DIR). To fix this problem, we just have to inform libtool that
>> $(STAGING_DIR) is reachable (patch 16).
>>
>> Finally, I also clean up libtool infra from modification that seems useless now
>> (patches 18 and 19).
>>
>> I tested this series using internal toolchain and Linaro toolchain with these
>> packages:
>>
>>     BR2_INIT_SYSTEMD=y       (install public libraries in /usr/lib/systemd)
>>     BR2_PACKAGE_LIBCDAUDIO=y (libtool 1.5)
>>     BR2_PACKAGE_LIBLO=y      (libtool 2.2)
>>     BR2_PACKAGE_MADPLAY=y    (unpatched libtool)
>>     BR2_PACKAGE_ALSA_LIB=y   (optional dependency to Madplay)
>>     BR2_PACKAGE_PYTHON=y
>>     BR2_PACKAGE_PYTHON_PY_PYC=y
>>     BR2_PACKAGE_GNUPG2=y
>>
>> Except patches 12 and 13, I think whole series is bisectable.
> [...]
> 
> As I said above, patches 11 to 19 of this series do a strong clean up
> of libtool infrastructure. I think these patches are important for two
> reasons:
> 
>    1. It is the best way to support reproducible build from different
>       build paths
>    2. Currently BR usage of libtool is not standard. These patches use
>       libtool in the right way. It drop unnecessary RPATH from binaries.
>       It drop most of patches applied to libtool.
> 
> However, this series do very intrusive changes and it is difficult to
> review. Consequently, I think it is necessary to apply them very early
> in a development cycle. So, I suggest to apply them to 'next' now.

 Yes, it's time to apply the libtool patches. I have a few spelling corrections
for the commit messages lined up but nothing more, and I'm waiting for a test
build to finish to give my Ack.

 However, I wonder, wouldn't it be a good idea to change our approach? Instead
of patching the ltmain files, why don't we just overwrite them with our own
host-libtool?

 Hm, host-libtool depends on host-m4, so that adds (on my machine) 50 seconds to
the build whenever an autotools package is included... Possibly not acceptable.
We could still take that approach for the case when AUTORECONF=YES, however. But
then again, that would break the symmetry we have now...

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF