[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

Hi all,
this patch set aims to improve build reproducibility by using a
per-package staging directory.
Also this feature aims to enable as default the top-level parallel make.

Every package instead of installing the files in the shared staging
directory (STAGING_DIR) install all files in a subdirectory of the
STAGINGPKG_DIR.

At the begginning of the configure stage, all the dependencies staging
directory are copied in the package staging directory, e.g.:

zlib-install-staging: insall files in /output/stagingpkg/zlib/

cramsfs-configure: copy all files from /output/stagingpkg/zlib/* to
  /output/stagingpkg/cramfs/

The STAGING_DIR variable is redefined for every package in order to
point to the per-package directory, so a package cannot access
automatically to staging files of packages that aren't in the
(PKG)_DENDENDENCIES variable.

The most significant changes are in the pkg-generic.mk files.

The other modifications are just to be able to access to per-package
staging directory, like adding -L$(STAGING)/usr/lib or -I$(STAGING)/usr/include

This patch set is a working in progess, but it seems to builds fine
the configuration that i mangaged to test.

TODO:
 - clean up
 - rebase over master, right now is based on the 2015.02 release (I've
   not rebased it because there are some regressions on master after
   the merge of the next branch)
 - test more packages (I've not done it, because there are already
   build problems without using this patch set, at least on Fedora 21)

For convenience this patch set is avaiable in the branch "pps-2015.02"
of the repository https://github.com/fabio-porcedda/buildroot.git

This patch set is based on the release 2015.02, it builds fine the
following configuration on a Fedora 21:
  make distclean defconfig allyespackageconfig -s && sed -e "s/BR2_PACKAGE_LUAJIT=y//" -i .config && yes '' | make oldconfig
  make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Best regards

Fabio Porcedda (30):
  rtmpdump: use TARGET_LDFLAGS instead of TARGET_CFLAGS for XLDFLAGS
  xinetd: use TARGET_LDFLAGS in order to support per-package staging
  iproute2: use the TARGET_LDFLAGS variable
  opentyrian: use TARGET_LDFLAGS
  pppd: use TARGET_LDFLAGS
  openswan: set LDFLAGS
  exim: use TARGET_LDFLAGS
  fbv: use TARGET_LDFLAGS
  cups: use TARGET_LDFLAGS
  faifa: use TARGET_LDFLAGS
  iw: use TARGET_CONFIGURE_OPTS
  dhcpdump: use TARGET_CONFIGURE_OPTS in order to support PPS
  dtc: add add support for per-package staging directory
  openssh: add support to the per-package staging directory.
  mjpg-streamer: add support for the per-package staging directory
  tcpreplay: delay the execution of pcap-config
  erlang: add support for the per-package staging directory
  perl: don't loose the -shared flag when TARGET_LDFLAGS isn't empty
  erlang-p1-iconv: bump to a version that use TARGET_CFLAGS
  erlang-p1-zlib: bump to a version that use TARGET_CFLAGS
  lmbench: to be checked
  Makefile: add the STAGINGNOPKG_DIR variable
  gpsd: add support for per-package staging directory
  triggerhappy: add support for per-package staging directory
  ipsec-tools: add support for per-package staging directory
  pkg-cmake: add support for per-package staging directory
  pkg-luarocks: add support for per-package staging directory
  pkgconf: Move PKG_CONFIG_HOST_BINARY to Makefile.in
  pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages
  pkg-generic: add support for per-package staging directory

 Makefile                                           |  2 +-
 package/Makefile.in                                |  4 ++
 package/cups/cups.mk                               |  1 +
 package/dhcpdump/dhcpdump.mk                       |  2 +-
 package/dtc/dtc.mk                                 |  7 +--
 package/erlang-p1-iconv/erlang-p1-iconv.mk         |  2 +-
 package/erlang-p1-zlib/erlang-p1-zlib.mk           |  2 +-
 package/erlang/erlang.mk                           |  2 +-
 package/exim/exim.mk                               |  1 +
 ...-handle-LDFLAGS-passed-to-the-configure-s.patch | 40 ++++++++++++++
 package/fbv/fbv.mk                                 |  2 +-
 package/gpsd/gpsd.mk                               |  4 +-
 package/iproute2/iproute2.mk                       |  2 +-
 package/ipsec-tools/ipsec-tools.mk                 |  2 +-
 package/iw/iw.mk                                   |  5 +-
 package/lmbench/lmbench.mk                         |  5 +-
 package/mjpg-streamer/mjpg-streamer.mk             |  3 +-
 package/openssh/openssh.mk                         |  2 +-
 package/openswan/openswan.mk                       |  3 +-
 package/opentyrian/opentyrian.mk                   |  2 +-
 package/perl/perl.mk                               |  4 +-
 package/pkg-cmake.mk                               |  7 +++
 package/pkg-generic.mk                             | 63 ++++++++++++++++++++--
 package/pkg-luarocks.mk                            |  5 +-
 package/pkgconf/pkgconf.mk                         |  2 -
 package/pppd/pppd.mk                               |  2 +-
 package/rtmpdump/rtmpdump.mk                       |  2 +-
 package/tcpreplay/tcpreplay.mk                     |  4 +-
 package/triggerhappy/triggerhappy.mk               |  2 +-
 package/xinetd/xinetd.mk                           |  1 +
 support/misc/toolchainfile.cmake.in                |  6 ++-
 31 files changed, 151 insertions(+), 40 deletions(-)
 create mode 100644 package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch

-- 
2.3.1

[Buildroot] [RFC v3 01/30] rtmpdump: use TARGET_LDFLAGS instead of TARGET_CFLAGS for XLDFLAGS

[Fabio Porcedda]

The XLDFLAGS is for ld related flags so use TARGET_LDFLAGS instead of
TARGET_CFLAGS, also the TARGET_CFLAGS is already used for XCFLAGS.

This fix is also needed for supporting the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/rtmpdump/rtmpdump.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/rtmpdump/rtmpdump.mk b/package/rtmpdump/rtmpdump.mk
index 14037fb..613f1cd 100644
--- a/package/rtmpdump/rtmpdump.mk
+++ b/package/rtmpdump/rtmpdump.mk
@@ -39,7 +39,7 @@ define RTMPDUMP_BUILD_CMDS
 	$(MAKE) CRYPTO=$(RTMPDUMP_CRYPTO) \
 		prefix=/usr \
 		XCFLAGS="$(RTMPDUMP_CFLAGS)" \
-		XLDFLAGS="$(TARGET_CFLAGS)" \
+		XLDFLAGS="$(TARGET_LDFLAGS)" \
 		CROSS_COMPILE="$(TARGET_CROSS)" \
 		$(RTMPDUMP_SHARED) \
 		-C $(@D)/librtmp
-- 
2.3.1

[Buildroot] [RFC v3 02/30] xinetd: use TARGET_LDFLAGS in order to support per-package staging

[Fabio Porcedda]

---
 package/xinetd/xinetd.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/xinetd/xinetd.mk b/package/xinetd/xinetd.mk
index 92d0d53..a0d41f7 100644
--- a/package/xinetd/xinetd.mk
+++ b/package/xinetd/xinetd.mk
@@ -10,6 +10,7 @@ XINETD_LICENSE = xinetd license
 XINETD_LICENSE_FILES = COPYRIGHT
 
 XINETD_CFLAGS = $(TARGET_CFLAGS)
+XINETD_LIBS = $(TARGET_LDFLAGS)
 
 # Three cases here:
 #  1. We have libtirpc, use it by passing special flags
-- 
2.3.1

[Buildroot] [RFC v3 03/30] iproute2: use the TARGET_LDFLAGS variable

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/iproute2/iproute2.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/iproute2/iproute2.mk b/package/iproute2/iproute2.mk
index 0838463..5f144cd 100644
--- a/package/iproute2/iproute2.mk
+++ b/package/iproute2/iproute2.mk
@@ -59,7 +59,7 @@ endef
 
 define IPROUTE2_BUILD_CMDS
 	$(SED) 's/$$(CCOPTS)//' $(@D)/netem/Makefile
-	$(TARGET_MAKE_ENV) $(MAKE) \
+	$(TARGET_MAKE_ENV) LDFLAGS="$(TARGET_LDFLAGS)" $(MAKE) \
 		DBM_INCLUDE="$(STAGING_DIR)/usr/include" \
 		CCOPTS="$(TARGET_CFLAGS) -D_GNU_SOURCE" -C $(@D)
 endef
-- 
2.3.1

[Buildroot] [RFC v3 04/30] opentyrian: use TARGET_LDFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/opentyrian/opentyrian.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/opentyrian/opentyrian.mk b/package/opentyrian/opentyrian.mk
index f00a52b..5a4f278 100644
--- a/package/opentyrian/opentyrian.mk
+++ b/package/opentyrian/opentyrian.mk
@@ -24,7 +24,7 @@ define OPENTYRIAN_BUILD_CMDS
 		CC="$(TARGET_CC)" \
 		STRIP="/bin/true" \
 		SDL_CONFIG="$(STAGING_DIR)/usr/bin/sdl-config" \
-		LDFLAGS="-lm" \
+		LDFLAGS="-lm $(TARGET_LDFLAGS)" \
 		WITH_NETWORK="$(OPENTYRIAN_NETWORK)" \
 		-C $(@D) release
 endef
-- 
2.3.1

[Buildroot] [RFC v3 05/30] pppd: use TARGET_LDFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pppd/pppd.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pppd/pppd.mk b/package/pppd/pppd.mk
index 453445b..a5095fe 100644
--- a/package/pppd/pppd.mk
+++ b/package/pppd/pppd.mk
@@ -53,7 +53,7 @@ define PPPD_CONFIGURE_CMDS
 endef
 
 define PPPD_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" COPTS="$(TARGET_CFLAGS)" \
+	$(MAKE) CC="$(TARGET_CC)" COPTS="$(TARGET_CFLAGS) $(TARGET_LDFLAGS)" \
 		-C $(@D) $(PPPD_MAKE_OPTS)
 endef
 
-- 
2.3.1

[Buildroot] [RFC v3 06/30] openswan: set LDFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/openswan/openswan.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/openswan/openswan.mk b/package/openswan/openswan.mk
index ba36f7c..79f1409 100644
--- a/package/openswan/openswan.mk
+++ b/package/openswan/openswan.mk
@@ -12,7 +12,8 @@ OPENSWAN_LICENSE_FILES = COPYING LICENSE
 OPENSWAN_DEPENDENCIES = host-bison host-flex gmp iproute2
 OPENSWAN_MAKE_OPTS = ARCH=$(BR2_ARCH) CC="$(TARGET_CC)" \
 	USERCOMPILE="$(TARGET_CFLAGS) -fPIE" INC_USRLOCAL=/usr \
-	USE_KLIPS=false USE_MAST=false USE_NM=false
+	USE_KLIPS=false USE_MAST=false USE_NM=false \
+	LDFLAGS="$(TARGET_LDFLAGS)"
 
 ifeq ($(BR2_PACKAGE_LIBCURL),y)
 	OPENSWAN_DEPENDENCIES += libcurl
-- 
2.3.1

[Buildroot] [RFC v3 07/30] exim: use TARGET_LDFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/exim/exim.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/exim/exim.mk b/package/exim/exim.mk
index 34a6893..43867d5 100644
--- a/package/exim/exim.mk
+++ b/package/exim/exim.mk
@@ -65,6 +65,7 @@ endif
 define EXIM_CONFIGURE_TOOLCHAIN
 	$(call exim-config-add,CC,$(TARGET_CC))
 	$(call exim-config-add,CFLAGS,$(TARGET_CFLAGS))
+	$(call exim-config-add,LFLAGS,$(TARGET_LDFLAGS))
 	$(call exim-config-add,AR,$(TARGET_AR) cq)
 	$(call exim-config-add,RANLIB,$(TARGET_RANLIB))
 	$(call exim-config-add,HOSTCC,$(HOSTCC))
-- 
2.3.1

[Buildroot] [RFC v3 08/30] fbv: use TARGET_LDFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/fbv/fbv.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/fbv/fbv.mk b/package/fbv/fbv.mk
index 8788b0e..d46bcac 100644
--- a/package/fbv/fbv.mk
+++ b/package/fbv/fbv.mk
@@ -12,7 +12,7 @@ FBV_LICENSE_FILES = COPYING
 
 ### image format dependencies and configure options
 FBV_DEPENDENCIES = # empty
-FBV_CONFIGURE_OPTS = # empty
+FBV_CONFIGURE_OPTS = --libs="$(TARGET_LDFLAGS)"
 ifeq ($(BR2_PACKAGE_FBV_PNG),y)
 FBV_DEPENDENCIES += libpng
 
-- 
2.3.1

[Buildroot] [RFC v3 09/30] cups: use TARGET_LDFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/cups/cups.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/cups/cups.mk b/package/cups/cups.mk
index cc87574..502f7a3 100644
--- a/package/cups/cups.mk
+++ b/package/cups/cups.mk
@@ -12,6 +12,7 @@ CUPS_LICENSE_FILES = LICENSE.txt
 CUPS_INSTALL_STAGING = YES
 CUPS_INSTALL_STAGING_OPTS = DESTDIR=$(STAGING_DIR) DSTROOT=$(STAGING_DIR) install
 CUPS_INSTALL_TARGET_OPTS = DESTDIR=$(TARGET_DIR) DSTROOT=$(TARGET_DIR) install
+CUPS_CONF_ENV = DSOFLAGS="$(TARGET_LDFLAGS)"
 CUPS_CONF_OPTS = \
 	--without-perl \
 	--without-java \
-- 
2.3.1

[Buildroot] [RFC v3 10/30] faifa: use TARGET_LDFLAGS

[Fabio Porcedda]

To do that add a patch already sent to upstream:
https://github.com/ffainelli/faifa/pull/9

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 ...-handle-LDFLAGS-passed-to-the-configure-s.patch | 40 ++++++++++++++++++++++
 1 file changed, 40 insertions(+)
 create mode 100644 package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch

diff --git a/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch b/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
new file mode 100644
index 0000000..5d54374
--- /dev/null
+++ b/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
@@ -0,0 +1,40 @@
+From 62cafe305f4b9f5fb4305dcfb68ae3144efc3353 Mon Sep 17 00:00:00 2001
+From: Fabio Porcedda <Fabio Porcedda fabio.porcedda@gmail>
+Date: Wed, 18 Feb 2015 17:58:23 +0100
+Subject: [PATCH] Makefile.in: handle LDFLAGS passed to the configure script
+
+Taking in account CFLAGS and LDFLAGS passed to the configure script is a
+common behaviour.
+
+This fix is also useful for supporting the per-package staging directory
+within Buildroot.
+
+Signed-off-by: Fabio Porcedda <Fabio Porcedda fabio.porcedda@gmail>
+---
+ Makefile.in | 3 ++-
+ 1 file changed, 2 insertions(+), 1 deletion(-)
+
+diff --git a/Makefile.in b/Makefile.in
+index b1b7c58..792ac82 100644
+--- a/Makefile.in
++++ b/Makefile.in
+@@ -14,6 +14,7 @@
+ CC	= @CC@
+ STRIP	?= $(CROSS)strip
+ CFLAGS	= @CFLAGS@
++LDFLAGS	= @LDFLAGS@
+ INSTALL	= @INSTALL@
+ LIBS	= @LIBS@
+ 
+@@ -50,7 +51,7 @@ SIM_LIBS:=-levent
+ SIM_CFLAGS:=-Wno-unused
+ 
+ ifeq ($(OS),DARWIN)
+-LDFLAGS:=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
++LDFLAGS+=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
+ endif
+ 
+ LIBS:=$(LDFLAGS) $(LIBS)
+-- 
+2.3.0
+
-- 
2.3.1

[Buildroot] [RFC v3 11/30] iw: use TARGET_CONFIGURE_OPTS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/iw/iw.mk | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/package/iw/iw.mk b/package/iw/iw.mk
index df6e257..460c83b 100644
--- a/package/iw/iw.mk
+++ b/package/iw/iw.mk
@@ -10,9 +10,8 @@ IW_SITE = $(BR2_KERNEL_MIRROR)/software/network/iw
 IW_LICENSE = iw license
 IW_LICENSE_FILES = COPYING
 IW_DEPENDENCIES = host-pkgconf libnl
-IW_MAKE_OPTS = CC="$(TARGET_CC)" LD="$(TARGET_LD)" LDFLAGS="$(TARGET_LDFLAGS)"
-IW_MAKE_ENV = PKG_CONFIG="$(HOST_DIR)/usr/bin/pkg-config" \
-	GIT_DIR=$(IW_DIR)
+IW_MAKE_OPTS =
+IW_MAKE_ENV = $(TARGET_CONFIGURE_OPTS) GIT_DIR=$(IW_DIR)
 
 ifeq ($(BR2_STATIC_LIBS),y)
 # libnl needs pthread/m, so we need to explicitly with them when static
-- 
2.3.1

[Buildroot] [RFC v3 12/30] dhcpdump: use TARGET_CONFIGURE_OPTS in order to support PPS

[Fabio Porcedda]

In order to support the per-package staging directory we need to pass
the TARGET_LDFLAGS, so instead of adding directly to the make command
line use the TARGET_CONFIGURE_OPTS variable that already contain the
TARGET_LDFALGS contents.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/dhcpdump/dhcpdump.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/dhcpdump/dhcpdump.mk b/package/dhcpdump/dhcpdump.mk
index 0756319..d9fb98f 100644
--- a/package/dhcpdump/dhcpdump.mk
+++ b/package/dhcpdump/dhcpdump.mk
@@ -16,7 +16,7 @@ DHCPDUMP_LIBS += $(shell $(STAGING_DIR)/usr/bin/pcap-config --static --additiona
 endif
 
 define DHCPDUMP_BUILD_CMDS
-	$(MAKE) -C $(@D) CC="$(TARGET_CC) $(TARGET_CFLAGS)" LIBS="$(DHCPDUMP_LIBS)"
+	$(MAKE) -C $(@D) $(TARGET_CONFIGURE_OPTS) LIBS="$(DHCPDUMP_LIBS)"
 endef
 
 define DHCPDUMP_INSTALL_TARGET_CMDS
-- 
2.3.1

[Buildroot] [RFC v3 13/30] dtc: add add support for per-package staging directory

[Fabio Porcedda]

Use $(TARGET_CONFIGURE_OPTS) in the install commands because it builds
something so it needs it to suppoer the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/dtc/dtc.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/package/dtc/dtc.mk b/package/dtc/dtc.mk
index e094f7b..3da3369 100644
--- a/package/dtc/dtc.mk
+++ b/package/dtc/dtc.mk
@@ -36,12 +36,13 @@ endef
 
 # For staging, only the library is needed
 define DTC_INSTALL_STAGING_CMDS
-	$(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR) PREFIX=/usr install-lib \
-		install-includes
+	$(TARGET_CONFIGURE_OPTS) $(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR) \
+		PREFIX=/usr install-lib install-includes
 endef
 
 define DTC_INSTALL_TARGET_CMDS
-	$(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) PREFIX=/usr $(DTC_INSTALL_GOAL)
+	$(TARGET_CONFIGURE_OPTS) $(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) \
+		PREFIX=/usr $(DTC_INSTALL_GOAL)
 endef
 
 $(eval $(generic-package))
-- 
2.3.1

[Buildroot] [RFC v3 14/30] openssh: add support to the per-package staging directory.

[Fabio Porcedda]

Add "-L$(STAGING_DIR)/usr/lib" to the LDFLAGS environment variable to be
used by the configure script.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/openssh/openssh.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/openssh/openssh.mk b/package/openssh/openssh.mk
index be471b6..22cdb12 100644
--- a/package/openssh/openssh.mk
+++ b/package/openssh/openssh.mk
@@ -8,7 +8,7 @@ OPENSSH_VERSION = 6.7p1
 OPENSSH_SITE = http://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable
 OPENSSH_LICENSE = BSD-3c BSD-2c Public Domain
 OPENSSH_LICENSE_FILES = LICENCE
-OPENSSH_CONF_ENV = LD="$(TARGET_CC)" LDFLAGS="$(TARGET_CFLAGS)"
+OPENSSH_CONF_ENV = LD="$(TARGET_CC)" LDFLAGS="$(TARGET_CFLAGS) -L$(STAGING_DIR)/usr/lib"
 OPENSSH_CONF_OPTS = \
 	--sysconfdir=/etc/ssh \
 	--disable-lastlog \
-- 
2.3.1

[Buildroot] [RFC v3 15/30] mjpg-streamer: add support for the per-package staging directory

[Fabio Porcedda]

Add -I$(STAGING_DIR)/usr/include to the CFLAGS environment variable and
$(TARGET_LDFLAGS) to the LFAGS environment variable of the make command.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/mjpg-streamer/mjpg-streamer.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/mjpg-streamer/mjpg-streamer.mk b/package/mjpg-streamer/mjpg-streamer.mk
index 2385429..7f3c355 100644
--- a/package/mjpg-streamer/mjpg-streamer.mk
+++ b/package/mjpg-streamer/mjpg-streamer.mk
@@ -15,7 +15,8 @@ MJPG_STREAMER_LICENSE_FILES = LICENSE
 MJPG_STREAMER_DEPENDENCIES = jpeg
 
 define MJPG_STREAMER_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE) CC=$(TARGET_CC) -C $(@D)
+	$(TARGET_MAKE_ENV) CFLAGS="-I$(STAGING_DIR)/usr/include" \
+	LFLAGS="$(TARGET_LDFLAGS)" $(MAKE) CC=$(TARGET_CC) -C $(@D)
 endef
 
 define MJPG_STREAMER_INSTALL_TARGET_CMDS
-- 
2.3.1

[Buildroot] [RFC v3 16/30] tcpreplay: delay the execution of pcap-config

[Fabio Porcedda]

Delay the execution of the pcap-config tool after the creation of the
package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/tcpreplay/tcpreplay.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/tcpreplay/tcpreplay.mk b/package/tcpreplay/tcpreplay.mk
index 1dbe453..4378c2c 100644
--- a/package/tcpreplay/tcpreplay.mk
+++ b/package/tcpreplay/tcpreplay.mk
@@ -16,8 +16,8 @@ TCPREPLAY_CONF_OPTS = --with-libpcap=$(STAGING_DIR)/usr
 TCPREPLAY_DEPENDENCIES = libpcap
 
 # libpcap may depend on symbols in other libs
-TCPREPLAY_LIBS = $(shell $(STAGING_DIR)/usr/bin/pcap-config --static --libs)
-TCPREPLAY_CONF_ENV += ac_cv_search_pcap_close='$(TCPREPLAY_LIBS)' \
+TCPREPLAY_LIBS = `$(STAGING_DIR)/usr/bin/pcap-config --static --libs`
+TCPREPLAY_CONF_ENV += ac_cv_search_pcap_close="$(TCPREPLAY_LIBS)" \
 	LIBS="$(TCPREPLAY_LIBS)"
 
 ifeq ($(BR2_PACKAGE_TCPDUMP),y)
-- 
2.3.1

[Buildroot] [RFC v3 17/30] erlang: add support for the per-package staging directory

[Fabio Porcedda]

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/erlang/erlang.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/erlang/erlang.mk b/package/erlang/erlang.mk
index 0c76531..85063b1 100644
--- a/package/erlang/erlang.mk
+++ b/package/erlang/erlang.mk
@@ -31,7 +31,7 @@ ERLANG_CONF_ENV += erl_xcomp_sysroot=$(STAGING_DIR)
 ERLANG_CONF_OPTS = --without-javac
 
 ERLANG_DEPENDENCIES += libatomic_ops
-ERLANG_CONF_OPTS += --with-libatomic_ops=$(STAGING_DIR)/usr LIBS=-latomic_ops
+ERLANG_CONF_OPTS += --with-libatomic_ops=$(STAGING_DIR)/usr LIBS="-latomic_ops -L$(STAGING_DIR)/usr/lib"
 
 # erlang uses openssl for all things crypto. Since the host tools (such as
 # rebar) uses crypto, we need to build host-erlang with support for openssl.
-- 
2.3.1

[Buildroot] [RFC v3 18/30] perl: don't loose the -shared flag when TARGET_LDFLAGS isn't empty

[Fabio Porcedda]

This is in order to support the per-package staging directory.
---
 package/perl/perl.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/perl/perl.mk b/package/perl/perl.mk
index b1f2101..2b9cdff 100644
--- a/package/perl/perl.mk
+++ b/package/perl/perl.mk
@@ -33,6 +33,8 @@ PERL_POST_DOWNLOAD_HOOKS += PERL_CROSS_DOWNLOAD
 define PERL_CROSS_EXTRACT
 	$(call suitable-extractor,$(PERL_CROSS_SOURCE)) $(DL_DIR)/$(PERL_CROSS_SOURCE) | \
 	$(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(@D) $(TAR_OPTIONS) -
+	$(SED) "s/-L\*|-R\*|-Wl,-R\*)/-L\*|-R\*|-Wl,-R\*|-shared)/" \
+		$(@D)/cnf/configure_misc.sh
 endef
 PERL_POST_EXTRACT_HOOKS += PERL_CROSS_EXTRACT
 
@@ -57,7 +59,7 @@ PERL_CONF_OPTS = \
 	--prefix=/usr \
 	-Dld="$(TARGET_CC_NOCCACHE)" \
 	-Dccflags="$(TARGET_CFLAGS)" \
-	-Dldflags="$(TARGET_LDFLAGS) -lm" \
+	-Dldflags="$(TARGET_LDFLAGS) -lm -shared" \
 	-Dmydomain="" \
 	-Dmyhostname="$(BR2_TARGET_GENERIC_HOSTNAME)" \
 	-Dmyuname="Buildroot $(BR2_VERSION_FULL)" \
-- 
2.3.1

[Buildroot] [RFC v3 19/30] erlang-p1-iconv: bump to a version that use TARGET_CFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Cc: Johan Oudinet <johan.oudinet@gmail.com>
---
 package/erlang-p1-iconv/erlang-p1-iconv.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/erlang-p1-iconv/erlang-p1-iconv.mk b/package/erlang-p1-iconv/erlang-p1-iconv.mk
index 2189f01..6d1d992 100644
--- a/package/erlang-p1-iconv/erlang-p1-iconv.mk
+++ b/package/erlang-p1-iconv/erlang-p1-iconv.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-ERLANG_P1_ICONV_VERSION = dff5ee5
+ERLANG_P1_ICONV_VERSION = a16ac9b918626665e066534878eab7a878eea275
 ERLANG_P1_ICONV_SITE = $(call github,processone,eiconv,$(ERLANG_P1_ICONV_VERSION))
 ERLANG_P1_ICONV_LICENSE = GPLv2+
 ERLANG_P1_ICONV_LICENSE_FILES = COPYING
-- 
2.3.1

[Buildroot] [RFC v3 20/30] erlang-p1-zlib: bump to a version that use TARGET_CFLAGS

[Fabio Porcedda]

This is in order to support the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
Cc: Johan Oudinet <johan.oudinet@gmail.com>
---
 package/erlang-p1-zlib/erlang-p1-zlib.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/erlang-p1-zlib/erlang-p1-zlib.mk b/package/erlang-p1-zlib/erlang-p1-zlib.mk
index 4ba3332..7744f5a 100644
--- a/package/erlang-p1-zlib/erlang-p1-zlib.mk
+++ b/package/erlang-p1-zlib/erlang-p1-zlib.mk
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-ERLANG_P1_ZLIB_VERSION = 52e82bb
+ERLANG_P1_ZLIB_VERSION = 513c7e52a01616e7af74c67ac246dc09a3c071e4
 ERLANG_P1_ZLIB_SITE = $(call github,processone,zlib,$(ERLANG_P1_ZLIB_VERSION))
 ERLANG_P1_ZLIB_LICENSE = GPLv2+
 ERLANG_P1_ZLIB_LICENSE_FILES = COPYING
-- 
2.3.1

[Buildroot] [RFC v3 21/30] lmbench: to be checked

[Fabio Porcedda]

---
 package/lmbench/lmbench.mk | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/package/lmbench/lmbench.mk b/package/lmbench/lmbench.mk
index 63a83fd..954f7eb 100644
--- a/package/lmbench/lmbench.mk
+++ b/package/lmbench/lmbench.mk
@@ -10,9 +10,6 @@ LMBENCH_SITE = http://downloads.sourceforge.net/project/lmbench/development/lmbe
 LMBENCH_LICENSE = lmbench license (based on GPLv2)
 LMBENCH_LICENSE_FILES = COPYING COPYING-2
 
-LMBENCH_CFLAGS = $(TARGET_CFLAGS)
-LMBENCH_LDLIBS = $(TARGET_LDFLAGS)
-
 ifeq ($(BR2_PACKAGE_LIBTIRPC),y)
 LMBENCH_DEPENDENCIES += libtirpc
 LMBENCH_CFLAGS += -I$(STAGING_DIR)/usr/include/tirpc/
@@ -28,7 +25,7 @@ define LMBENCH_CONFIGURE_CMDS
 endef
 
 define LMBENCH_BUILD_CMDS
-	$(MAKE) CFLAGS="$(LMBENCH_CFLAGS)" LDFLAGS="$(LMBENCH_LDFLAGS)" OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
+	$(MAKE) CFLAGS="$(TARGET_CFLAGS) $(LMBENCH_CFLAGS)" LDFLAGS="$(TARGET_LDFLAGS) $(LMBENCH_LDFLAGS)" OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
 endef
 
 define LMBENCH_INSTALL_TARGET_CMDS
-- 
2.3.1

[Buildroot] [RFC v3 22/30] Makefile: add the STAGINGNOPKG_DIR variable

[Fabio Porcedda]

This is to be able to access the common staging directory even when in a
successive commit the STAGING_DIR will point to the per-package staging
directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/Makefile.in | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/Makefile.in b/package/Makefile.in
index 803b162..88ace05 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -108,6 +108,7 @@ endif
 
 STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
 STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
+STAGINGNOPKG_DIR := $(STAGING_DIR)
 
 TARGET_OPTIMIZATION := $(call qstrip,$(BR2_TARGET_OPTIMIZATION))
 
-- 
2.3.1

[Buildroot] [RFC v3 23/30] gpsd: add support for per-package staging directory

[Fabio Porcedda]

Pass the TARGET_LDFLAGS variable.

Also use $(STAGINGNOPKG_DIR) instead of $(STAGING_DIR) because now the
STAGING_DIR variable point to the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/gpsd/gpsd.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/gpsd/gpsd.mk b/package/gpsd/gpsd.mk
index 3e6411b..62fd207 100644
--- a/package/gpsd/gpsd.mk
+++ b/package/gpsd/gpsd.mk
@@ -14,12 +14,12 @@ GPSD_DEPENDENCIES = host-scons host-pkgconf
 
 GPSD_LDFLAGS = $(TARGET_LDFLAGS)
 
-GPSD_SCONS_ENV = $(TARGET_CONFIGURE_OPTS)
+GPSD_SCONS_ENV = $(TARGET_CONFIGURE_OPTS) LINKFLAGS="$(TARGET_LDFLAGS)"
 
 GPSD_SCONS_OPTS = \
 	arch=$(ARCH)\
 	prefix=/usr\
-	sysroot=$(STAGING_DIR)\
+	sysroot=$(STAGINGNOPKG_DIR) \
 	strip=no\
 	python=no
 
-- 
2.3.1

[Buildroot] [RFC v3 24/30] triggerhappy: add support for per-package staging directory

[Fabio Porcedda]

Use $(STAGINGNOPKG_DIR) instead of $(STAGING_DIR) because now the
STAGING_DIR point to the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/triggerhappy/triggerhappy.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/triggerhappy/triggerhappy.mk b/package/triggerhappy/triggerhappy.mk
index d3dfc6a..ff1b379 100644
--- a/package/triggerhappy/triggerhappy.mk
+++ b/package/triggerhappy/triggerhappy.mk
@@ -11,7 +11,7 @@ TRIGGERHAPPY_LICENSE_FILES = COPYING
 
 define TRIGGERHAPPY_BUILD_CMDS
 	$(MAKE) $(TARGET_CONFIGURE_OPTS) \
-		LINUX_INPUT_H=$(STAGING_DIR)/usr/include/linux/input.h \
+		LINUX_INPUT_H=$(STAGINGNOPKG_DIR)/usr/include/linux/input.h \
 		-C $(@D) thd th-cmd
 endef
 
-- 
2.3.1

[Buildroot] [RFC v3 25/30] ipsec-tools: add support for per-package staging directory

[Fabio Porcedda]

Use $(STAGINGNOPKG_DIR) instead of $(STAGING_DIR) because now the
STAGING_DIR point to the per-package staging directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/ipsec-tools/ipsec-tools.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/ipsec-tools/ipsec-tools.mk b/package/ipsec-tools/ipsec-tools.mk
index 9454c4e..738b616 100644
--- a/package/ipsec-tools/ipsec-tools.mk
+++ b/package/ipsec-tools/ipsec-tools.mk
@@ -23,7 +23,7 @@ endif
 IPSEC_TOOLS_CONF_OPTS = \
 	  --without-libpam \
 	  --disable-gssapi \
-	  --with-kernel-headers=$(STAGING_DIR)/usr/include
+	  --with-kernel-headers=$(STAGINGNOPKG_DIR)/usr/include
 
 ifeq ($(BR2_PACKAGE_IPSEC_TOOLS_ADMINPORT),y)
 IPSEC_TOOLS_CONF_OPTS += --enable-adminport
-- 
2.3.1

[Buildroot] [RFC v3 26/30] pkg-cmake: add support for per-package staging directory

[Fabio Porcedda]

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-cmake.mk                | 7 +++++++
 support/misc/toolchainfile.cmake.in | 6 ++++--
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
index eacd641..8cda3b3 100644
--- a/package/pkg-cmake.mk
+++ b/package/pkg-cmake.mk
@@ -72,10 +72,16 @@ define $(2)_CONFIGURE_CMDS
 	(cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
+	PKG_CONFIG_SYSROOT_DIR="$$($(2)_STAGING_DIR)" \
+	PKG_CONFIG_PATH="$$($(2)_STAGING_DIR)/usr/lib/pkgconfig" \
 	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \
 		-DCMAKE_INSTALL_PREFIX="/usr" \
+		-DCMAKE_C_FLAGS="$$(TARGET_CFLAGS)" \
+		-DCMAKE_SHARED_LINKER_FLAGS="$$(TARGET_LDFLAGS)" \
+		-DCMAKE_MODULE_LINKER_FLAGS="$$(TARGET_LDFLAGS)" \
+		-DCMAKE_EXE_LINKER_FLAGS="$$(TARGET_LDFLAGS)" \
 		-DCMAKE_COLOR_MAKEFILE=OFF \
 		-DBUILD_DOC=OFF \
 		-DBUILD_DOCS=OFF \
@@ -86,6 +92,7 @@ define $(2)_CONFIGURE_CMDS
 		-DBUILD_TESTING=OFF \
 		-DBUILD_SHARED_LIBS=$$(if $$(BR2_STATIC_LIBS),OFF,ON) \
 		-DUSE_CCACHE=$$(if $$(BR2_CCACHE),ON,OFF) \
+		-DSTAGING_DIR=$$(STAGING_DIR) \
 		$$($$(PKG)_CONF_OPTS) \
 	)
 endef
diff --git a/support/misc/toolchainfile.cmake.in b/support/misc/toolchainfile.cmake.in
index 816af13..99e2d51 100644
--- a/support/misc/toolchainfile.cmake.in
+++ b/support/misc/toolchainfile.cmake.in
@@ -19,11 +19,13 @@ set(CMAKE_EXE_LINKER_FLAGS "@@TARGET_LDFLAGS@@ ${CMAKE_EXE_LINKER_FLAGS}" CACHE
 set(CMAKE_INSTALL_SO_NO_EXE 0)
 
 set(CMAKE_PROGRAM_PATH "${RELOCATED_HOST_DIR}/usr/bin")
-set(CMAKE_FIND_ROOT_PATH "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+set(CMAKE_FIND_ROOT_PATH "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@;${STAGING_DIR}/usr")
 set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
 set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
 set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
-set(ENV{PKG_CONFIG_SYSROOT_DIR} "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+if(NOT DEFINED ENV{PKG_CONFIG_SYSROOT_DIR})
+       set(ENV{PKG_CONFIG_SYSROOT_DIR} "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+endif()
 
 # This toolchain file can be used both inside and outside Buildroot.
 # * When used inside Buildroot, ccache support is explicitly driven using the
-- 
2.3.1

[Buildroot] [RFC v3 27/30] pkg-luarocks: add support for per-package staging directory

[Fabio Porcedda]

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-luarocks.mk | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/package/pkg-luarocks.mk b/package/pkg-luarocks.mk
index 83f338e..4bdaff3 100644
--- a/package/pkg-luarocks.mk
+++ b/package/pkg-luarocks.mk
@@ -59,7 +59,10 @@ endif
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
 	cd $$($(2)_SRCDIR) && \
-		$$(LUAROCKS_RUN) make --keep $$($(2)_ROCKSPEC) $$($(2)_BUILD_OPTS)
+		$$(LUAROCKS_RUN) make --keep $$($(2)_ROCKSPEC) \
+		$$($(2)_BUILD_OPTS) OPENSSL_DIR=$$(STAGING_DIR)/usr \
+		EXPAT_DIR=$$(STAGING_DIR)/usr ZLIB_DIR=$$(STAGING_DIR)/usr \
+		SQLITE_DIR=$$(STAGING_DIR)/usr
 endef
 endif
 
-- 
2.3.1

[Buildroot] [RFC v3 28/30] pkgconf: Move PKG_CONFIG_HOST_BINARY to Makefile.in

[Fabio Porcedda]

This is in order to support the per-package staging directory.
Because the PKG_CONFIG_HOST_BINARY will be used inside the
pkg-generic.mk, move the definition in the Makefile.in so it will be
loaded before the pkg-generic.mk.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/Makefile.in        | 2 ++
 package/pkgconf/pkgconf.mk | 2 --
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 88ace05..6cfe3ed 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -247,6 +247,8 @@ export PERL=$(shell which perl)
 # finds this perl module by exporting the proper value for PERL5LIB.
 export PERL5LIB=$(HOST_DIR)/usr/lib/perl
 
+PKG_CONFIG_HOST_BINARY = $(HOST_DIR)/usr/bin/pkg-config
+
 TARGET_CONFIGURE_OPTS = \
 	PATH=$(BR_PATH) \
 	AR="$(TARGET_AR)" \
diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index a27c084..7feeb4f 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -10,8 +10,6 @@ PKGCONF_SOURCE = pkgconf-$(PKGCONF_VERSION).tar.bz2
 PKGCONF_LICENSE = pkgconf license
 PKGCONF_LICENSE_FILES = COPYING
 
-PKG_CONFIG_HOST_BINARY = $(HOST_DIR)/usr/bin/pkg-config
-
 define PKGCONF_LINK_PKGCONFIG
 	ln -sf pkgconf $(TARGET_DIR)/usr/bin/pkg-config
 endef
-- 
2.3.1

[Buildroot] [RFC v3 29/30] pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages

[Fabio Porcedda]

This will simplify the commit that add the per-package staging
directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-generic.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1b09955..43289d0 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -413,17 +413,20 @@ $(2)_REDISTRIBUTE		?= YES
 # When a target package is a toolchain dependency set this variable to
 # 'NO' so the 'toolchain' dependency is not added to prevent a circular
 # dependency
-$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+ifeq ($(4),target)
+ $(2)_ADD_TOOLCHAIN_DEPENDENCY ?= YES
+else
+ $(2)_ADD_TOOLCHAIN_DEPENDENCY ?= NO
+endif
 
 ifeq ($(4),host)
 $(2)_DEPENDENCIES ?= $$(filter-out  host-toolchain $(1),\
 	$$(patsubst host-host-%,host-%,$$(addprefix host-,$$($(3)_DEPENDENCIES))))
 endif
-ifeq ($(4),target)
+
 ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
 $(2)_DEPENDENCIES += toolchain
 endif
-endif
 
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
-- 
2.3.1

[Buildroot] [RFC v3 30/30] pkg-generic: add support for per-package staging directory

[Fabio Porcedda]

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile               |  2 +-
 package/Makefile.in    |  1 +
 package/pkg-generic.mk | 54 ++++++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 3723abc..6619bad 100644
--- a/Makefile
+++ b/Makefile
@@ -832,7 +832,7 @@ printvars:
 
 clean:
 	rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
-		$(BUILD_DIR) $(BASE_DIR)/staging \
+		$(BUILD_DIR) $(BASE_DIR)/staging $(STAGINGPKG_DIR) \
 		$(LEGAL_INFO_DIR)
 
 distclean: clean
diff --git a/package/Makefile.in b/package/Makefile.in
index 6cfe3ed..2a21c9d 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -109,6 +109,7 @@ endif
 STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
 STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
 STAGINGNOPKG_DIR := $(STAGING_DIR)
+STAGINGPKG_DIR = $(BASE_DIR)/stagingpkg
 
 TARGET_OPTIMIZATION := $(call qstrip,$(BR2_TARGET_OPTIMIZATION))
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 43289d0..c35f801 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -208,7 +208,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 		$(call MESSAGE,"Fixing package configuration files") ;\
 			$(SED)  "s,$(BASE_DIR), at BASE_DIR@,g" \
 				-e "s,$(STAGING_DIR), at STAGING_DIR@,g" \
-				-e "s,^\(exec_\)\?prefix=.*,\1prefix=@STAGING_DIR@/usr,g" \
+				-e 's,^\(exec_\)\?prefix=.*,\1prefix=`echo $$0 | sed '"'"'s_/[^/]*/[^/]*$$__'"'"'`,g' \
 				-e "s,-I/usr/,-I at STAGING_DIR@/usr/,g" \
 				-e "s,-L/usr/,-L at STAGING_DIR@/usr/,g" \
 				-e "s, at STAGING_DIR@,$(STAGING_DIR),g" \
@@ -247,7 +247,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
-	rm -Rf $(@D)
+	rm -Rf $(@D) $(STAGINGPKG_DIR)/$(call LOWERCASE,$(PKG))
 
 ################################################################################
 # virt-provides-single -- check that provider-pkg is the declared provider for
@@ -431,6 +431,38 @@ endif
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 
+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+ $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
+ $(2)_TARGET_CPPFLAGS = $$(strip $$(TARGET_CPPFLAGS) -I$$($(2)_STAGING_DIR)/usr/include)
+ $(2)_TARGET_LDFLAGS = $$(TARGET_LDFLAGS) -L$$($(2)_STAGING_DIR)/usr/lib -Wl,-rpath,$$($(2)_STAGING_DIR)/usr/lib
+ $(2)_TARGET_CONFIGURE_OPTS = \
+	$$(TARGET_CONFIGURE_OPTS) \
+	PKG_CONFIG_SYSROOT_DIR="$$($(2)_STAGING_DIR)" \
+	PKG_CONFIG_PATH="$$($(2)_STAGING_DIR)/usr/lib/pkgconfig"
+ $(2)_TARGET_MAKE_ENV = \
+	$$(TARGET_MAKE_ENV) \
+	PKG_CONFIG_SYSROOT_DIR="$$($(2)_STAGING_DIR)" \
+	PKG_CONFIG_PATH="$$($(2)_STAGING_DIR)/usr/lib/pkgconfig"
+else
+ $(2)_STAGING_DIR = $$(STAGING_DIR)
+ $(2)_TARGET_CONFIGURE_OPTS = $$(TARGET_CONFIGURE_OPTS)
+ $(2)_TARGET_MAKE_ENV = $$(TARGET_MAKE_ENV)
+endif
+
+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+ $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
+	$$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
+	$$($$(call UPPERCASE,$$(dep))_STAGING_DIR)))
+
+ define $(2)_PREPARE_STAGING_DIR
+	mkdir -p $$(STAGING_DIR)/usr/include $$(STAGING_DIR)/usr/lib
+	$$(if $$($(2)_STAGING_DIRS),
+		cp -rdpf $$(addsuffix /*,$$($(2)_STAGING_DIRS)) \
+			$$($(2)_STAGING_DIR))
+ endef
+ $(2)_PRE_CONFIGURE_HOOKS := $(2)_PREPARE_STAGING_DIR $$($(2)_PRE_CONFIGURE_HOOKS)
+endif
+
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
 $(2)_INSTALL_TARGET		?= YES
@@ -604,11 +636,29 @@ $(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
+$$($(2)_TARGET_INSTALL_TARGET):		STAGING_DIR:=$$($(2)_STAGING_DIR)
+$$($(2)_TARGET_INSTALL_TARGET):		TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
+$$($(2)_TARGET_INSTALL_TARGET):		TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
+$$($(2)_TARGET_INSTALL_TARGET):		TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
 $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
+$$($(2)_TARGET_INSTALL_STAGING):	STAGING_DIR:=$$($(2)_STAGING_DIR)
+$$($(2)_TARGET_INSTALL_STAGING):	TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
+$$($(2)_TARGET_INSTALL_STAGING):	TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
+$$($(2)_TARGET_INSTALL_STAGING):	TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
 $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
+$$($(2)_TARGET_INSTALL_HOST):		STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
+$$($(2)_TARGET_BUILD):			STAGING_DIR:=$$($(2)_STAGING_DIR)
+$$($(2)_TARGET_BUILD):			TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
+$$($(2)_TARGET_BUILD):			TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
+$$($(2)_TARGET_BUILD):			TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
+$$($(2)_TARGET_BUILD):			TARGET_MAKE_ENV:=$$($(2)_TARGET_MAKE_ENV)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
+$$($(2)_TARGET_CONFIGURE):		STAGING_DIR:=$$($(2)_STAGING_DIR)
+$$($(2)_TARGET_CONFIGURE):		TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
+$$($(2)_TARGET_CONFIGURE):		TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
+$$($(2)_TARGET_CONFIGURE):		TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
 $$($(2)_TARGET_RSYNC):                  SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
 $$($(2)_TARGET_RSYNC):                  PKG=$(2)
 $$($(2)_TARGET_RSYNC_SOURCE):		SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
-- 
2.3.1

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Tue,  3 Mar 2015 10:17:05 +0100, Fabio Porcedda wrote:

> this patch set aims to improve build reproducibility by using a
> per-package staging directory.
> Also this feature aims to enable as default the top-level parallel make.

Interesting! I must say I'm a bit worried about the implications of
having to pass custom -I and -L flags all the time: we did arrange the
compiler do not require those (by configuration for the internal
toolchain case, and using the wrapper for the external toolchain case).

Could you give some hints about the performance impact of those copies
of the staging directory? Does it continue to make the top-level
parallel build thing useful?

I.e can you compare in terms of build time the following cases:

 * Normal non-top-level parallel build.
 * A top-level parallel build without this PPS feature
 * A top-level parallel build with this PPS feature

Preferably something with a fairly deep dependency tree, i.e lots of
small packages that depend on each other. A X.org stack + gtk2 would be
a good example.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Tue, Mar 3, 2015 at 2:29 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Fabio Porcedda,
>
> On Tue,  3 Mar 2015 10:17:05 +0100, Fabio Porcedda wrote:
>
>> this patch set aims to improve build reproducibility by using a
>> per-package staging directory.
>> Also this feature aims to enable as default the top-level parallel make.
>
> Interesting! I must say I'm a bit worried about the implications of
> having to pass custom -I and -L flags all the time: we did arrange the
> compiler do not require those (by configuration for the internal
> toolchain case, and using the wrapper for the external toolchain case).
>
> Could you give some hints about the performance impact of those copies
> of the staging directory? Does it continue to make the top-level
> parallel build thing useful?

IMHO the per-package staging feature is useful even without taking in
consideration the top-level parallel build, the unspecified optional
dependency issue are not common, they can generate build faulres or
just binary with unexpected behavior.

> I.e can you compare in terms of build time the following cases:
>
>  * Normal non-top-level parallel build.
>  * A top-level parallel build without this PPS feature
>  * A top-level parallel build with this PPS feature

Sure

> Preferably something with a fairly deep dependency tree, i.e lots of
> small packages that depend on each other. A X.org stack + gtk2 would be
> a good example.

The most complete selection of package that i've tested is the
defconfig + allyespacakgeconfig (693 targets), i've tried to enable a
wider range of packages, defconfig + glibc + allyespackageconfig but
even without using the pps they fail to build.
So i will provide build times of the configuration defconfig +
allyespackageconfig.

If you provide a defconfig on the release 2015.02, i can try to build that.

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Tue, 3 Mar 2015 15:03:46 +0100, Fabio Porcedda wrote:

> > Could you give some hints about the performance impact of those copies
> > of the staging directory? Does it continue to make the top-level
> > parallel build thing useful?
> 
> IMHO the per-package staging feature is useful even without taking in
> consideration the top-level parallel build, the unspecified optional
> dependency issue are not common, they can generate build faulres or
> just binary with unexpected behavior.

Yes, correct, this is indeed another benefit of this approach.

> > Preferably something with a fairly deep dependency tree, i.e lots of
> > small packages that depend on each other. A X.org stack + gtk2 would be
> > a good example.
> 
> The most complete selection of package that i've tested is the
> defconfig + allyespacakgeconfig (693 targets), i've tried to enable a
> wider range of packages, defconfig + glibc + allyespackageconfig but
> even without using the pps they fail to build.
> So i will provide build times of the configuration defconfig +
> allyespackageconfig.

Ok.

> If you provide a defconfig on the release 2015.02, i can try to build that.

Something like:

BR2_arm=y
BR2_TOOLCHAIN_BUILDROOT_LARGEFILE=y
BR2_TOOLCHAIN_BUILDROOT_INET_IPV6=y
BR2_TOOLCHAIN_BUILDROOT_INET_RPC=y
BR2_TOOLCHAIN_BUILDROOT_LOCALE=y
BR2_TOOLCHAIN_BUILDROOT_CXX=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_LIBGTK3=y

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC v3 21/30] lmbench: to be checked

[Baruch Siach]

On Tue, Mar 03, 2015 at 10:17:26AM +0100, Fabio Porcedda wrote:
> ---
>  package/lmbench/lmbench.mk | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/package/lmbench/lmbench.mk b/package/lmbench/lmbench.mk
> index 63a83fd..954f7eb 100644
> --- a/package/lmbench/lmbench.mk
> +++ b/package/lmbench/lmbench.mk
> @@ -10,9 +10,6 @@ LMBENCH_SITE = http://downloads.sourceforge.net/project/lmbench/development/lmbe
>  LMBENCH_LICENSE = lmbench license (based on GPLv2)
>  LMBENCH_LICENSE_FILES = COPYING COPYING-2
>  
> -LMBENCH_CFLAGS = $(TARGET_CFLAGS)
> -LMBENCH_LDLIBS = $(TARGET_LDFLAGS)
> -

Please see http://patchwork.ozlabs.org/patch/442832/.

baruch

>  ifeq ($(BR2_PACKAGE_LIBTIRPC),y)
>  LMBENCH_DEPENDENCIES += libtirpc
>  LMBENCH_CFLAGS += -I$(STAGING_DIR)/usr/include/tirpc/
> @@ -28,7 +25,7 @@ define LMBENCH_CONFIGURE_CMDS
>  endef
>  
>  define LMBENCH_BUILD_CMDS
> -	$(MAKE) CFLAGS="$(LMBENCH_CFLAGS)" LDFLAGS="$(LMBENCH_LDFLAGS)" OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
> +	$(MAKE) CFLAGS="$(TARGET_CFLAGS) $(LMBENCH_CFLAGS)" LDFLAGS="$(TARGET_LDFLAGS) $(LMBENCH_LDFLAGS)" OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
>  endef
>  
>  define LMBENCH_INSTALL_TARGET_CMDS

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [RFC v3 10/30] faifa: use TARGET_LDFLAGS

[Baruch Siach]

Hi Fabio,

On Tue, Mar 03, 2015 at 10:17:15AM +0100, Fabio Porcedda wrote:
> To do that add a patch already sent to upstream:
> https://github.com/ffainelli/faifa/pull/9
> 
> This is in order to support the per-package staging directory.
> 
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
> ---
>  ...-handle-LDFLAGS-passed-to-the-configure-s.patch | 40 ++++++++++++++++++++++
>  1 file changed, 40 insertions(+)
>  create mode 100644 package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
> 
> diff --git a/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch b/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
> new file mode 100644
> index 0000000..5d54374
> --- /dev/null
> +++ b/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
> @@ -0,0 +1,40 @@
> +From 62cafe305f4b9f5fb4305dcfb68ae3144efc3353 Mon Sep 17 00:00:00 2001
> +From: Fabio Porcedda <Fabio Porcedda fabio.porcedda@gmail>
> +Date: Wed, 18 Feb 2015 17:58:23 +0100
> +Subject: [PATCH] Makefile.in: handle LDFLAGS passed to the configure script
> +
> +Taking in account CFLAGS and LDFLAGS passed to the configure script is a
> +common behaviour.
> +
> +This fix is also useful for supporting the per-package staging directory
> +within Buildroot.
> +
> +Signed-off-by: Fabio Porcedda <Fabio Porcedda fabio.porcedda@gmail>
> +---
> + Makefile.in | 3 ++-
> + 1 file changed, 2 insertions(+), 1 deletion(-)
> +
> +diff --git a/Makefile.in b/Makefile.in
> +index b1b7c58..792ac82 100644
> +--- a/Makefile.in
> ++++ b/Makefile.in
> +@@ -14,6 +14,7 @@
> + CC	= @CC@
> + STRIP	?= $(CROSS)strip
> + CFLAGS	= @CFLAGS@
> ++LDFLAGS	= @LDFLAGS@
> + INSTALL	= @INSTALL@
> + LIBS	= @LIBS@
> + 
> +@@ -50,7 +51,7 @@ SIM_LIBS:=-levent
> + SIM_CFLAGS:=-Wno-unused
> + 
> + ifeq ($(OS),DARWIN)
> +-LDFLAGS:=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
> ++LDFLAGS+=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)

Why do you need this hunk?

baruch

> + endif
> + 
> + LIBS:=$(LDFLAGS) $(LIBS)

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Jérôme Oufella]

Thomas Petazzoni wrote:
> Could you give some hints about the performance impact of those copies
> of the staging directory? Does it continue to make the top-level
> parallel build thing useful?

In this case, it may make sense to use file linking (cp -al)
instead of file copy, assuming the staging spaces are on the same
filesystem ?

J?r?me

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Thu, Mar 5, 2015 at 4:54 AM, J?r?me Oufella
<jerome.oufella@savoirfairelinux.com> wrote:
> Thomas Petazzoni wrote:
>> Could you give some hints about the performance impact of those copies
>> of the staging directory? Does it continue to make the top-level
>> parallel build thing useful?
>
> In this case, it may make sense to use file linking (cp -al)
> instead of file copy, assuming the staging spaces are on the same
> filesystem ?

My only fear if a copy will be overwritten (without using unlink),
also all the copies will be changed, so defeating the purpose of pps,
nevertheless I'm not sure if is a real problem or not.

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Arnout Vandecappelle]

On 03/03/15 14:29, Thomas Petazzoni wrote:
> Dear Fabio Porcedda,
> 
> On Tue,  3 Mar 2015 10:17:05 +0100, Fabio Porcedda wrote:
> 
>> > this patch set aims to improve build reproducibility by using a
>> > per-package staging directory.
>> > Also this feature aims to enable as default the top-level parallel make.
> Interesting! I must say I'm a bit worried about the implications of
> having to pass custom -I and -L flags all the time: we did arrange the
> compiler do not require those (by configuration for the internal
> toolchain case, and using the wrapper for the external toolchain case).

 I also like this feature very much!

 However, I don't like at all the approach of passing the custom -I and -L
flags. I think it's much better to start with something that was suggested
before in the BR developer meeting: use the toolchain wrapper also for the
internal toolchain. Then we can pass the per-package staging dir through the
environment, and use it in the --sysroot argument of the compiler.

 We'll also have to do something about ccache then - that could be a problem.
Perhaps ccache should be called from within the wrapper as well, that may solve
a bunch of other ccache problems.


 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v3 30/30] pkg-generic: add support for per-package staging directory

[Arnout Vandecappelle]

On 03/03/15 10:17, Fabio Porcedda wrote:
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
> ---
>  Makefile               |  2 +-
>  package/Makefile.in    |  1 +
>  package/pkg-generic.mk | 54 ++++++++++++++++++++++++++++++++++++++++++++++++--
>  3 files changed, 54 insertions(+), 3 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 3723abc..6619bad 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -832,7 +832,7 @@ printvars:
>  
>  clean:
>  	rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
> -		$(BUILD_DIR) $(BASE_DIR)/staging \
> +		$(BUILD_DIR) $(BASE_DIR)/staging $(STAGINGPKG_DIR) \
>  		$(LEGAL_INFO_DIR)
>  
>  distclean: clean
> diff --git a/package/Makefile.in b/package/Makefile.in
> index 6cfe3ed..2a21c9d 100644
> --- a/package/Makefile.in
> +++ b/package/Makefile.in
> @@ -109,6 +109,7 @@ endif
>  STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
>  STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
>  STAGINGNOPKG_DIR := $(STAGING_DIR)
> +STAGINGPKG_DIR = $(BASE_DIR)/stagingpkg

 Since $(STAGING_DIR) no longer contains everything, the staging symlink has
really become pointless. For me, it's only use was that you could set the
debugger's sysroot to it to get access to unstripped .so files. And then, we can
reuse $(BASE_DIR)/staging for $(STAGINGPKG_DIR).

>  
>  TARGET_OPTIMIZATION := $(call qstrip,$(BR2_TARGET_OPTIMIZATION))
>  
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 43289d0..c35f801 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -208,7 +208,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
>  		$(call MESSAGE,"Fixing package configuration files") ;\
>  			$(SED)  "s,$(BASE_DIR), at BASE_DIR@,g" \
>  				-e "s,$(STAGING_DIR), at STAGING_DIR@,g" \
> -				-e "s,^\(exec_\)\?prefix=.*,\1prefix=@STAGING_DIR@/usr,g" \
> +				-e 's,^\(exec_\)\?prefix=.*,\1prefix=`echo $$0 | sed '"'"'s_/[^/]*/[^/]*$$__'"'"'`,g' \

 Wow, ridiculously complicated quoting...
 More importantly, however: shouldn't this be for all the replacements, i.e. a
couple of lines below...

>  				-e "s,-I/usr/,-I at STAGING_DIR@/usr/,g" \
>  				-e "s,-L/usr/,-L at STAGING_DIR@/usr/,g" \
>  				-e "s, at STAGING_DIR@,$(STAGING_DIR),g" \

 ... here?

> @@ -247,7 +247,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
>  
>  # Remove package sources
>  $(BUILD_DIR)/%/.stamp_dircleaned:
> -	rm -Rf $(@D)
> +	rm -Rf $(@D) $(STAGINGPKG_DIR)/$(call LOWERCASE,$(PKG))
>  
>  ################################################################################
>  # virt-provides-single -- check that provider-pkg is the declared provider for
> @@ -431,6 +431,38 @@ endif
>  # Eliminate duplicates in dependencies
>  $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  
> +ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)

 I guess the intention here is to still install the toolchain stuff in the
original staging directory. But is that really needed? Since all packages depend
on toolchain, they'd just copy in the toolchain staging dir into their private
staging dir. Right?

> + $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
> + $(2)_TARGET_CPPFLAGS = $$(strip $$(TARGET_CPPFLAGS) -I$$($(2)_STAGING_DIR)/usr/include)

 Actually, it would be sufficient to add -I$(STAGING_DIR)/usr/include to
TARGET_CPPFLAGS. Since STAGING_DIR is set to the per-package staging dir in the
build rules below, it will get carried over into the TARGET_CPPFLAGS variable,
which only gets expanded when the rules are run. Same for all the other
variables you set here.

> + $(2)_TARGET_LDFLAGS = $$(TARGET_LDFLAGS) -L$$($(2)_STAGING_DIR)/usr/lib -Wl,-rpath,$$($(2)_STAGING_DIR)/usr/lib
> + $(2)_TARGET_CONFIGURE_OPTS = \
> +	$$(TARGET_CONFIGURE_OPTS) \
> +	PKG_CONFIG_SYSROOT_DIR="$$($(2)_STAGING_DIR)" \
> +	PKG_CONFIG_PATH="$$($(2)_STAGING_DIR)/usr/lib/pkgconfig"
> + $(2)_TARGET_MAKE_ENV = \
> +	$$(TARGET_MAKE_ENV) \
> +	PKG_CONFIG_SYSROOT_DIR="$$($(2)_STAGING_DIR)" \
> +	PKG_CONFIG_PATH="$$($(2)_STAGING_DIR)/usr/lib/pkgconfig"
> +else
> + $(2)_STAGING_DIR = $$(STAGING_DIR)
> + $(2)_TARGET_CONFIGURE_OPTS = $$(TARGET_CONFIGURE_OPTS)
> + $(2)_TARGET_MAKE_ENV = $$(TARGET_MAKE_ENV)
> +endif
> +
> +ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
> + $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
> +	$$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
> +	$$($$(call UPPERCASE,$$(dep))_STAGING_DIR)))
> +
> + define $(2)_PREPARE_STAGING_DIR
> +	mkdir -p $$(STAGING_DIR)/usr/include $$(STAGING_DIR)/usr/lib
> +	$$(if $$($(2)_STAGING_DIRS),
> +		cp -rdpf $$(addsuffix /*,$$($(2)_STAGING_DIRS)) \
> +			$$($(2)_STAGING_DIR))

 Why not just
		cp -rdpf -t $$($(2)_STAGING_DIR) $$($(2)_STAGING_DIRS)
?


 Regards,
 Arnout


> + endef
> + $(2)_PRE_CONFIGURE_HOOKS := $(2)_PREPARE_STAGING_DIR $$($(2)_PRE_CONFIGURE_HOOKS)
> +endif
> +
>  $(2)_INSTALL_STAGING		?= NO
>  $(2)_INSTALL_IMAGES		?= NO
>  $(2)_INSTALL_TARGET		?= YES
> @@ -604,11 +636,29 @@ $(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
>  # define the PKG variable for all targets, containing the
>  # uppercase package variable prefix
>  $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
> +$$($(2)_TARGET_INSTALL_TARGET):		STAGING_DIR:=$$($(2)_STAGING_DIR)
> +$$($(2)_TARGET_INSTALL_TARGET):		TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
> +$$($(2)_TARGET_INSTALL_TARGET):		TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
> +$$($(2)_TARGET_INSTALL_TARGET):		TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
>  $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
> +$$($(2)_TARGET_INSTALL_STAGING):	STAGING_DIR:=$$($(2)_STAGING_DIR)
> +$$($(2)_TARGET_INSTALL_STAGING):	TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
> +$$($(2)_TARGET_INSTALL_STAGING):	TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
> +$$($(2)_TARGET_INSTALL_STAGING):	TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
>  $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
>  $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
> +$$($(2)_TARGET_INSTALL_HOST):		STAGING_DIR:=$$($(2)_STAGING_DIR)
>  $$($(2)_TARGET_BUILD):			PKG=$(2)
> +$$($(2)_TARGET_BUILD):			STAGING_DIR:=$$($(2)_STAGING_DIR)
> +$$($(2)_TARGET_BUILD):			TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
> +$$($(2)_TARGET_BUILD):			TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
> +$$($(2)_TARGET_BUILD):			TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
> +$$($(2)_TARGET_BUILD):			TARGET_MAKE_ENV:=$$($(2)_TARGET_MAKE_ENV)
>  $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
> +$$($(2)_TARGET_CONFIGURE):		STAGING_DIR:=$$($(2)_STAGING_DIR)
> +$$($(2)_TARGET_CONFIGURE):		TARGET_CPPFLAGS:=$$($(2)_TARGET_CPPFLAGS)
> +$$($(2)_TARGET_CONFIGURE):		TARGET_LDFLAGS:=$$($(2)_TARGET_LDFLAGS)
> +$$($(2)_TARGET_CONFIGURE):		TARGET_CONFIGURE_OPTS:=$$($(2)_TARGET_CONFIGURE_OPTS)
>  $$($(2)_TARGET_RSYNC):                  SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
>  $$($(2)_TARGET_RSYNC):                  PKG=$(2)
>  $$($(2)_TARGET_RSYNC_SOURCE):		SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v3 21/30] lmbench: to be checked

[Fabio Porcedda]

On Tue, Mar 3, 2015 at 5:56 PM, Baruch Siach <baruch@tkos.co.il> wrote:
> On Tue, Mar 03, 2015 at 10:17:26AM +0100, Fabio Porcedda wrote:
>> ---
>>  package/lmbench/lmbench.mk | 5 +----
>>  1 file changed, 1 insertion(+), 4 deletions(-)
>>
>> diff --git a/package/lmbench/lmbench.mk b/package/lmbench/lmbench.mk
>> index 63a83fd..954f7eb 100644
>> --- a/package/lmbench/lmbench.mk
>> +++ b/package/lmbench/lmbench.mk
>> @@ -10,9 +10,6 @@ LMBENCH_SITE = http://downloads.sourceforge.net/project/lmbench/development/lmbe
>>  LMBENCH_LICENSE = lmbench license (based on GPLv2)
>>  LMBENCH_LICENSE_FILES = COPYING COPYING-2
>>
>> -LMBENCH_CFLAGS = $(TARGET_CFLAGS)
>> -LMBENCH_LDLIBS = $(TARGET_LDFLAGS)
>> -
>
> Please see http://patchwork.ozlabs.org/patch/442832/.

I will rebase this patch over your patch.

Thanks

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 10/30] faifa: use TARGET_LDFLAGS

[Fabio Porcedda]

On Tue, Mar 3, 2015 at 5:57 PM, Baruch Siach <baruch@tkos.co.il> wrote:
> Hi Fabio,
>
> On Tue, Mar 03, 2015 at 10:17:15AM +0100, Fabio Porcedda wrote:
>> To do that add a patch already sent to upstream:
>> https://github.com/ffainelli/faifa/pull/9
>>
>> This is in order to support the per-package staging directory.
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>> ---
>>  ...-handle-LDFLAGS-passed-to-the-configure-s.patch | 40 ++++++++++++++++++++++
>>  1 file changed, 40 insertions(+)
>>  create mode 100644 package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
>>
>> diff --git a/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch b/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
>> new file mode 100644
>> index 0000000..5d54374
>> --- /dev/null
>> +++ b/package/faifa/0001-Makefile.in-handle-LDFLAGS-passed-to-the-configure-s.patch
>> @@ -0,0 +1,40 @@
>> +From 62cafe305f4b9f5fb4305dcfb68ae3144efc3353 Mon Sep 17 00:00:00 2001
>> +From: Fabio Porcedda <Fabio Porcedda fabio.porcedda@gmail>
>> +Date: Wed, 18 Feb 2015 17:58:23 +0100
>> +Subject: [PATCH] Makefile.in: handle LDFLAGS passed to the configure script
>> +
>> +Taking in account CFLAGS and LDFLAGS passed to the configure script is a
>> +common behaviour.
>> +
>> +This fix is also useful for supporting the per-package staging directory
>> +within Buildroot.
>> +
>> +Signed-off-by: Fabio Porcedda <Fabio Porcedda fabio.porcedda@gmail>
>> +---
>> + Makefile.in | 3 ++-
>> + 1 file changed, 2 insertions(+), 1 deletion(-)
>> +
>> +diff --git a/Makefile.in b/Makefile.in
>> +index b1b7c58..792ac82 100644
>> +--- a/Makefile.in
>> ++++ b/Makefile.in
>> +@@ -14,6 +14,7 @@
>> + CC  = @CC@
>> + STRIP       ?= $(CROSS)strip
>> + CFLAGS      = @CFLAGS@
>> ++LDFLAGS     = @LDFLAGS@
>> + INSTALL     = @INSTALL@
>> + LIBS        = @LIBS@
>> +
>> +@@ -50,7 +51,7 @@ SIM_LIBS:=-levent
>> + SIM_CFLAGS:=-Wno-unused
>> +
>> + ifeq ($(OS),DARWIN)
>> +-LDFLAGS:=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
>> ++LDFLAGS+=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
>
> Why do you need this hunk?

Because without that the Makefile doesn't use the TARGET_LDFLAGS
passed by buildroot to the configure script, so it doesn't find the
libraries in the per-package staging directory, and i get this build
failure:

>>> faifa v0.1 Building
sha2.c: In function 'SHA256_Last':
sha2.c:492:2: warning: dereferencing type-punned pointer will break
strict-aliasing rules [-Wstrict-aliasing]
/home/tetsuya/buildroot/br2/output/host/opt/ext-toolchain/bin/../lib/gcc/i686-pc-linux-gnu/4.7.2/../../../../i686-pc-linux-gnu/bin/ld:
cannot find -lpcap
collect2: error: ld returned 1 exit status
Makefile:78: recipe for target 'libfaifa.so.0' failed
make[1]: *** [libfaifa.so.0] Error 1
package/pkg-generic.mk:182: recipe for target
'/home/tetsuya/buildroot/br2/output/build/faifa-v0.1/.stamp_built'
failed

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 10/30] faifa: use TARGET_LDFLAGS

[Baruch Siach]

Hi Fabio,

On Sun, Mar 08, 2015 at 04:14:51PM +0100, Fabio Porcedda wrote:
> >> +diff --git a/Makefile.in b/Makefile.in
> >> +index b1b7c58..792ac82 100644
> >> +--- a/Makefile.in
> >> ++++ b/Makefile.in
> >> +@@ -14,6 +14,7 @@
> >> + CC  = @CC@
> >> + STRIP       ?= $(CROSS)strip
> >> + CFLAGS      = @CFLAGS@
> >> ++LDFLAGS     = @LDFLAGS@
> >> + INSTALL     = @INSTALL@
> >> + LIBS        = @LIBS@
> >> +
> >> +@@ -50,7 +51,7 @@ SIM_LIBS:=-levent
> >> + SIM_CFLAGS:=-Wno-unused
> >> +
> >> + ifeq ($(OS),DARWIN)
> >> +-LDFLAGS:=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
> >> ++LDFLAGS+=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
> >
> > Why do you need this hunk?
> 
> Because without that the Makefile doesn't use the TARGET_LDFLAGS
> passed by buildroot to the configure script, so it doesn't find the
> libraries in the per-package staging directory, and i get this build
> failure:
> 
> >>> faifa v0.1 Building
> sha2.c: In function 'SHA256_Last':
> sha2.c:492:2: warning: dereferencing type-punned pointer will break
> strict-aliasing rules [-Wstrict-aliasing]
> /home/tetsuya/buildroot/br2/output/host/opt/ext-toolchain/bin/../lib/gcc/i686-pc-linux-gnu/4.7.2/../../../../i686-pc-linux-gnu/bin/ld:
> cannot find -lpcap
> collect2: error: ld returned 1 exit status
> Makefile:78: recipe for target 'libfaifa.so.0' failed
> make[1]: *** [libfaifa.so.0] Error 1
> package/pkg-generic.mk:182: recipe for target
> '/home/tetsuya/buildroot/br2/output/build/faifa-v0.1/.stamp_built'
> failed

Does $(OS) really evaluates to "DARWIN" on your system? This looks quite 
weird.

baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [RFC v3 10/30] faifa: use TARGET_LDFLAGS

[Fabio Porcedda]

On Sun, Mar 8, 2015 at 4:18 PM, Baruch Siach <baruch@tkos.co.il> wrote:
> Hi Fabio,
>
> On Sun, Mar 08, 2015 at 04:14:51PM +0100, Fabio Porcedda wrote:
>> >> +diff --git a/Makefile.in b/Makefile.in
>> >> +index b1b7c58..792ac82 100644
>> >> +--- a/Makefile.in
>> >> ++++ b/Makefile.in
>> >> +@@ -14,6 +14,7 @@
>> >> + CC  = @CC@
>> >> + STRIP       ?= $(CROSS)strip
>> >> + CFLAGS      = @CFLAGS@
>> >> ++LDFLAGS     = @LDFLAGS@
>> >> + INSTALL     = @INSTALL@
>> >> + LIBS        = @LIBS@
>> >> +
>> >> +@@ -50,7 +51,7 @@ SIM_LIBS:=-levent
>> >> + SIM_CFLAGS:=-Wno-unused
>> >> +
>> >> + ifeq ($(OS),DARWIN)
>> >> +-LDFLAGS:=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
>> >> ++LDFLAGS+=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
>> >
>> > Why do you need this hunk?
>>
>> Because without that the Makefile doesn't use the TARGET_LDFLAGS
>> passed by buildroot to the configure script, so it doesn't find the
>> libraries in the per-package staging directory, and i get this build
>> failure:
>>
>> >>> faifa v0.1 Building
>> sha2.c: In function 'SHA256_Last':
>> sha2.c:492:2: warning: dereferencing type-punned pointer will break
>> strict-aliasing rules [-Wstrict-aliasing]
>> /home/tetsuya/buildroot/br2/output/host/opt/ext-toolchain/bin/../lib/gcc/i686-pc-linux-gnu/4.7.2/../../../../i686-pc-linux-gnu/bin/ld:
>> cannot find -lpcap
>> collect2: error: ld returned 1 exit status
>> Makefile:78: recipe for target 'libfaifa.so.0' failed
>> make[1]: *** [libfaifa.so.0] Error 1
>> package/pkg-generic.mk:182: recipe for target
>> '/home/tetsuya/buildroot/br2/output/build/faifa-v0.1/.stamp_built'
>> failed
>
> Does $(OS) really evaluates to "DARWIN" on your system? This looks quite
> weird.

So are you just speaking about the last line, not the whole patch?
That line it's not useful on my system, but because the fafia project
support the darwin os, I've added the change anyway, the change
prevents that the LDFLAGS passed to the configure script will be
overwritten.
I've done that because I've sent the patch upstream.

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 10/30] faifa: use TARGET_LDFLAGS

[Baruch Siach]

Hi Fabio,

On Sun, Mar 08, 2015 at 04:30:59PM +0100, Fabio Porcedda wrote:
> On Sun, Mar 8, 2015 at 4:18 PM, Baruch Siach <baruch@tkos.co.il> wrote:
> > On Sun, Mar 08, 2015 at 04:14:51PM +0100, Fabio Porcedda wrote:
> >> >> +diff --git a/Makefile.in b/Makefile.in
> >> >> +index b1b7c58..792ac82 100644
> >> >> +--- a/Makefile.in
> >> >> ++++ b/Makefile.in
> >> >> +@@ -14,6 +14,7 @@
> >> >> + CC  = @CC@
> >> >> + STRIP       ?= $(CROSS)strip
> >> >> + CFLAGS      = @CFLAGS@
> >> >> ++LDFLAGS     = @LDFLAGS@
> >> >> + INSTALL     = @INSTALL@
> >> >> + LIBS        = @LIBS@
> >> >> +
> >> >> +@@ -50,7 +51,7 @@ SIM_LIBS:=-levent
> >> >> + SIM_CFLAGS:=-Wno-unused
> >> >> +
> >> >> + ifeq ($(OS),DARWIN)
> >> >> +-LDFLAGS:=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
> >> >> ++LDFLAGS+=-dynamiclib -Wl,-dylib_install_name -Wl,$(LIB_SONAME)
> >> >
> >> > Why do you need this hunk?
> >>
> >> Because without that the Makefile doesn't use the TARGET_LDFLAGS
> >> passed by buildroot to the configure script, so it doesn't find the
> >> libraries in the per-package staging directory, and i get this build
> >> failure:
> >>
> >> >>> faifa v0.1 Building
> >> sha2.c: In function 'SHA256_Last':
> >> sha2.c:492:2: warning: dereferencing type-punned pointer will break
> >> strict-aliasing rules [-Wstrict-aliasing]
> >> /home/tetsuya/buildroot/br2/output/host/opt/ext-toolchain/bin/../lib/gcc/i686-pc-linux-gnu/4.7.2/../../../../i686-pc-linux-gnu/bin/ld:
> >> cannot find -lpcap
> >> collect2: error: ld returned 1 exit status
> >> Makefile:78: recipe for target 'libfaifa.so.0' failed
> >> make[1]: *** [libfaifa.so.0] Error 1
> >> package/pkg-generic.mk:182: recipe for target
> >> '/home/tetsuya/buildroot/br2/output/build/faifa-v0.1/.stamp_built'
> >> failed
> >
> > Does $(OS) really evaluates to "DARWIN" on your system? This looks quite
> > weird.
> 
> So are you just speaking about the last line, not the whole patch?

Yes, I referred to that hunk (patch segment) specifically.

> That line it's not useful on my system, but because the fafia project
> support the darwin os, I've added the change anyway, the change
> prevents that the LDFLAGS passed to the configure script will be
> overwritten.
> I've done that because I've sent the patch upstream.

In this case, in my opinion, the patch description should also reference 
upstream patch or pull request, in addition to the Buildroot commit log.

Thanks,
baruch

-- 
     http://baruch.siach.name/blog/                  ~. .~   Tk Open Systems
=}------------------------------------------------ooO--U--Ooo------------{=
   - baruch at tkos.co.il - tel: +972.2.679.5364, http://www.tkos.co.il -

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Tue, Mar 3, 2015 at 3:21 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Fabio Porcedda,
>
> On Tue, 3 Mar 2015 15:03:46 +0100, Fabio Porcedda wrote:
>
>> > Could you give some hints about the performance impact of those copies
>> > of the staging directory? Does it continue to make the top-level
>> > parallel build thing useful?
>>
>> IMHO the per-package staging feature is useful even without taking in
>> consideration the top-level parallel build, the unspecified optional
>> dependency issue are not common, they can generate build faulres or
>> just binary with unexpected behavior.
>
> Yes, correct, this is indeed another benefit of this approach.
>
>> > Preferably something with a fairly deep dependency tree, i.e lots of
>> > small packages that depend on each other. A X.org stack + gtk2 would be
>> > a good example.
>>
>> The most complete selection of package that i've tested is the
>> defconfig + allyespacakgeconfig (693 targets), i've tried to enable a
>> wider range of packages, defconfig + glibc + allyespackageconfig but
>> even without using the pps they fail to build.
>> So i will provide build times of the configuration defconfig +
>> allyespackageconfig.
>
> Ok.
>
>> If you provide a defconfig on the release 2015.02, i can try to build that.
>
> Something like:
>
> BR2_arm=y
> BR2_TOOLCHAIN_BUILDROOT_LARGEFILE=y
> BR2_TOOLCHAIN_BUILDROOT_INET_IPV6=y
> BR2_TOOLCHAIN_BUILDROOT_INET_RPC=y
> BR2_TOOLCHAIN_BUILDROOT_LOCALE=y
> BR2_TOOLCHAIN_BUILDROOT_CXX=y
> BR2_PACKAGE_XORG7=y
> BR2_PACKAGE_XSERVER_XORG_SERVER=y
> BR2_PACKAGE_LIBGTK3=y

I've used a system with: i5-3470 (4 core @ 3.2GHz); RAM: 8GB; SSD.
Using you defconfig:

normal:                 20m
with parallel:          13m
with parallel & pps: 15m

Using a defconfig + allyespacakgeconfig:
normal:   85m
parallel:  40m
pps:       42m

I've repeated the test few times and taken the minimum value, but i
was still using the system so maybe the real values can change a bit.

I was surprised that the pps doesn't affect too much the build times,
but probably the ssd helps a lot.

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Thu, Mar 5, 2015 at 11:48 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 03/03/15 14:29, Thomas Petazzoni wrote:
>> Dear Fabio Porcedda,
>>
>> On Tue,  3 Mar 2015 10:17:05 +0100, Fabio Porcedda wrote:
>>
>>> > this patch set aims to improve build reproducibility by using a
>>> > per-package staging directory.
>>> > Also this feature aims to enable as default the top-level parallel make.
>> Interesting! I must say I'm a bit worried about the implications of
>> having to pass custom -I and -L flags all the time: we did arrange the
>> compiler do not require those (by configuration for the internal
>> toolchain case, and using the wrapper for the external toolchain case).
>
>  I also like this feature very much!
>
>  However, I don't like at all the approach of passing the custom -I and -L
> flags. I think it's much better to start with something that was suggested
> before in the BR developer meeting: use the toolchain wrapper also for the
> internal toolchain. Then we can pass the per-package staging dir through the
> environment, and use it in the --sysroot argument of the compiler.
>
>  We'll also have to do something about ccache then - that could be a problem.
> Perhaps ccache should be called from within the wrapper as well, that may solve
> a bunch of other ccache problems.

Using an internal wrapper we can also pass pass the -I and -L
automatically like the --sysroot option?

If we use a per-package sysroot we need to copy the sysroot directory
too, because the size of a uclibc sysroot is at least 11MB (if i
remember correctly the size of a glibc sysroot is something like
40MB), so if we copy without using hard links, it will take at least
10MB up to 40MB for every package.

The size of the output directory of a defconfig + allyespackageconfig is 11GB.
IMHO we have the following choices:
a) common sysroot & per-package staging & without hard links
  - the -I -L options are passed trough the toolchain wrapper
  - output size is 16GB (11GB+5GB)
  - no side effects to fear, because hard links are not used
b) per-package sysroot & without hard links
  - sysroot are passed trough the toolchain wrapper
  - output size will be 16GB plus 11MB * 694 = 16GB + 7GB = 23 GB
  - no side effects to fear, because hard links are not used
c) per-package sysroot & with hard links for toolchain
 - sysroot are passed trough the toolchain wrapper
  - output size will be 16GB
  - side effects to fear for the toolchain files because hard links
are used for them
d) per-package sysroot & with hard links for all files
 - sysroot are passed trough the toolchain wrapper
  - output size will be 11GB, so no increased size
  - side effects to fear because hard links are used

The cheaper one with almost no size and speed penalization is d) but
if some file is overwritten other package sysroot could be affected.
Maybe we can prevent the overwriting by removing the write permission
on staging files?

Which solution is most suited for  buildroot?

Thanks and my BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 30/30] pkg-generic: add support for per-package staging directory

[Fabio Porcedda]

On Fri, Mar 6, 2015 at 1:28 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 03/03/15 10:17, Fabio Porcedda wrote:
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>

<snip>

>> + $(2)_TARGET_LDFLAGS = $$(TARGET_LDFLAGS) -L$$($(2)_STAGING_DIR)/usr/lib -Wl,-rpath,$$($(2)_STAGING_DIR)/usr/lib
>> + $(2)_TARGET_CONFIGURE_OPTS = \
>> +     $$(TARGET_CONFIGURE_OPTS) \
>> +     PKG_CONFIG_SYSROOT_DIR="$$($(2)_STAGING_DIR)" \
>> +     PKG_CONFIG_PATH="$$($(2)_STAGING_DIR)/usr/lib/pkgconfig"
>> + $(2)_TARGET_MAKE_ENV = \
>> +     $$(TARGET_MAKE_ENV) \
>> +     PKG_CONFIG_SYSROOT_DIR="$$($(2)_STAGING_DIR)" \
>> +     PKG_CONFIG_PATH="$$($(2)_STAGING_DIR)/usr/lib/pkgconfig"
>> +else
>> + $(2)_STAGING_DIR = $$(STAGING_DIR)
>> + $(2)_TARGET_CONFIGURE_OPTS = $$(TARGET_CONFIGURE_OPTS)
>> + $(2)_TARGET_MAKE_ENV = $$(TARGET_MAKE_ENV)
>> +endif
>> +
>> +ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
>> + $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
>> +     $$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
>> +     $$($$(call UPPERCASE,$$(dep))_STAGING_DIR)))
>> +
>> + define $(2)_PREPARE_STAGING_DIR
>> +     mkdir -p $$(STAGING_DIR)/usr/include $$(STAGING_DIR)/usr/lib
>> +     $$(if $$($(2)_STAGING_DIRS),
>> +             cp -rdpf $$(addsuffix /*,$$($(2)_STAGING_DIRS)) \
>> +                     $$($(2)_STAGING_DIR))
>
>  Why not just
>                 cp -rdpf -t $$($(2)_STAGING_DIR) $$($(2)_STAGING_DIRS)
> ?

Because that command copy each source directory as a new sub directory
instead of copy the contents, e.g.:

directories:
dir_a/
  file_1
  file_2

dir_b/
  file_3

dest_dir/

command:
cp -rdpf -t dest_dir dir_a dir_b

result:
dest_dir/
  dir_a/
     file_1
     file_2
  dir_b/
     file_3

instead of:

cp -rdpf -t dest_dir dir_a/* dir_b/*

dest_dir/
  file_1
  file_2
  file_3

Thanks and BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Tue,  3 Mar 2015 10:17:05 +0100, Fabio Porcedda wrote:

> this patch set aims to improve build reproducibility by using a
> per-package staging directory.
> Also this feature aims to enable as default the top-level parallel make.

Here are some comments I wrote down a long time ago. It isn't a full
review, but it should give some initial thoughts:

If I understood correctly, the strategy used by your patches consists
in storing only the bare toolchain sysroot in output/staging/, and
using a per-package output/stagingpkg/<pkg>/ directory to store as an
input to the build the staging files needed to build the package, and
also as output the result of the build of the package.

I am not entirely happy with this solution, for the following reasons:

 * We no longer have a single staging directory that has all the
   libraries installed. This is needed for the toolchain to be a
   proper SDK usable by application developers outside of Buildroot.

   To solve this, either we install everything to both the real
   STAGING_DIR and a per-package staging directory, or we create a
   final build step that consolidates all the per-package staging
   directories into a global one. The advantage of this second option
   is that we don't have any parallel installation going on in the
   global staging directory.

 * I am not super happy with the idea of having the toolchain sysroot
   left in the global staging directory and referenced by the compiler
   --sysroot option on one side, and all other libraries found by
   using -L, -I and pkg-config tricks.

   I would actually prefer if a real complete sysroot was used when
   building each package, and the compiler --sysroot option used to
   adjust the compiler sysroot. This has however one significant
   drawback: the toolchain sysroot must be copied for each and every
   package, which can become quite time and space consuming. So on
   this aspect, I'd like to have some input from other Buildroot
   developers.

 * Your patch uses $($(2)_ADD_TOOLCHAIN_DEPENDENCY) to decide whether
   the per-package sysroot mechanism must be used or not. Which means
   it will only be used for target packages, and not for host
   packages. However, I'm wondering if we should not also apply the
   principle to host packages: to get reproducible builds, I believe
   we should also have separate sysroots when building host
   packages. Opinions from other Buildroot developers?

 * Your approach only takes care of make the sysroot handled in a
   per-package fashion. But what about HOST_DIR ? We could have the
   same inconsistencies as the ones we discussed about STAGING_DIR, but
   this time caused by the presence/absence of host utilities. One
   build may give a given result because host tool "foo" is present (it
   happened to be built before), and the next build may give a
   different result because host tool "foo" is absent (it's going to be
   build after).

Now some more specific comments:

 - The "enable parallel" patch should come last in the series.

 - The patches "popt: add to the "popt.pc" file the libintl library"
   and "logrotate: use pkg-config for the opt library" have been
   merged in master, so you could get rid of them from your series.

 - In patch "xinetd: use TARGET_LDFLAGS in order to support
   per-package staging", why don't we simply use the LDFLAGS
   environment variable, which seems to be understood by xinetd
   Makefile? Also, missing SoB line.

 - Patch "iproute2: use the TARGET_LDFLAGS variable" seems good to go,
   if the commit log doesn't mention per-package sysroot, so that we
   can merge it independently.

 - Patch "opentyrian: use TARGET_LDFLAGS", having -lm before
   $(TARGET_LDFLAGS) is probably more logical.

I didn't finished reviewing all of the package specific patches.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Fri, Jun 12, 2015 at 10:14 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Fabio Porcedda,
>
> On Tue,  3 Mar 2015 10:17:05 +0100, Fabio Porcedda wrote:
>
>> this patch set aims to improve build reproducibility by using a
>> per-package staging directory.
>> Also this feature aims to enable as default the top-level parallel make.
>
> Here are some comments I wrote down a long time ago. It isn't a full
> review, but it should give some initial thoughts:

Thanks for reviewing it.

> If I understood correctly, the strategy used by your patches consists
> in storing only the bare toolchain sysroot in output/staging/, and
> using a per-package output/stagingpkg/<pkg>/ directory to store as an
> input to the build the staging files needed to build the package, and
> also as output the result of the build of the package.

Yes

> I am not entirely happy with this solution, for the following reasons:
>
>  * We no longer have a single staging directory that has all the
>    libraries installed. This is needed for the toolchain to be a
>    proper SDK usable by application developers outside of Buildroot.
>
>    To solve this, either we install everything to both the real
>    STAGING_DIR and a per-package staging directory, or we create a
>    final build step that consolidates all the per-package staging
>    directories into a global one. The advantage of this second option
>    is that we don't have any parallel installation going on in the
>    global staging directory.

I personally prefer too the second solution.

>  * I am not super happy with the idea of having the toolchain sysroot
>    left in the global staging directory and referenced by the compiler
>    --sysroot option on one side, and all other libraries found by
>    using -L, -I and pkg-config tricks.
>
>    I would actually prefer if a real complete sysroot was used when
>    building each package, and the compiler --sysroot option used to
>    adjust the compiler sysroot. This has however one significant
>    drawback: the toolchain sysroot must be copied for each and every
>    package, which can become quite time and space consuming. So on
>    this aspect, I'd like to have some input from other Buildroot
>    developers.

I've already rewritten the patch set using the --sysroot option in the
toolchain wrapper. I just need to clean it up and send it, i hope to
be able to send it tomorrow so we can choose the best solution for
buildroot.

>  * Your patch uses $($(2)_ADD_TOOLCHAIN_DEPENDENCY) to decide whether
>    the per-package sysroot mechanism must be used or not. Which means
>    it will only be used for target packages, and not for host
>    packages. However, I'm wondering if we should not also apply the
>    principle to host packages: to get reproducible builds, I believe
>    we should also have separate sysroots when building host
>    packages. Opinions from other Buildroot developers?
>
>  * Your approach only takes care of make the sysroot handled in a
>    per-package fashion. But what about HOST_DIR ? We could have the
>    same inconsistencies as the ones we discussed about STAGING_DIR, but
>    this time caused by the presence/absence of host utilities. One
>    build may give a given result because host tool "foo" is present (it
>    happened to be built before), and the next build may give a
>    different result because host tool "foo" is absent (it's going to be
>    build after).

You are right, but if possible i just want to handle the host
utilities matter as a next step after this step is done.

> Now some more specific comments:
>
>  - The "enable parallel" patch should come last in the series.

Ok.

>  - The patches "popt: add to the "popt.pc" file the libintl library"
>    and "logrotate: use pkg-config for the opt library" have been
>    merged in master, so you could get rid of them from your series.

Great.

>  - In patch "xinetd: use TARGET_LDFLAGS in order to support
>    per-package staging", why don't we simply use the LDFLAGS
>    environment variable, which seems to be understood by xinetd
>    Makefile? Also, missing SoB line.

I need to test it.

>  - Patch "iproute2: use the TARGET_LDFLAGS variable" seems good to go,
>    if the commit log doesn't mention per-package sysroot, so that we
>    can merge it independently.

Ok.

>  - Patch "opentyrian: use TARGET_LDFLAGS", having -lm before
>    $(TARGET_LDFLAGS) is probably more logical.

Ok.

> I didn't finished reviewing all of the package specific patches.

Maybe you can review directly the rewritten patch set the use the
sysroot option.

Best regards
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Mon, 15 Jun 2015 11:06:40 +0200, Fabio Porcedda wrote:

> > I am not entirely happy with this solution, for the following reasons:
> >
> >  * We no longer have a single staging directory that has all the
> >    libraries installed. This is needed for the toolchain to be a
> >    proper SDK usable by application developers outside of Buildroot.
> >
> >    To solve this, either we install everything to both the real
> >    STAGING_DIR and a per-package staging directory, or we create a
> >    final build step that consolidates all the per-package staging
> >    directories into a global one. The advantage of this second option
> >    is that we don't have any parallel installation going on in the
> >    global staging directory.
> 
> I personally prefer too the second solution.

Ok.

> >  * I am not super happy with the idea of having the toolchain sysroot
> >    left in the global staging directory and referenced by the compiler
> >    --sysroot option on one side, and all other libraries found by
> >    using -L, -I and pkg-config tricks.
> >
> >    I would actually prefer if a real complete sysroot was used when
> >    building each package, and the compiler --sysroot option used to
> >    adjust the compiler sysroot. This has however one significant
> >    drawback: the toolchain sysroot must be copied for each and every
> >    package, which can become quite time and space consuming. So on
> >    this aspect, I'd like to have some input from other Buildroot
> >    developers.
> 
> I've already rewritten the patch set using the --sysroot option in the
> toolchain wrapper. I just need to clean it up and send it, i hope to
> be able to send it tomorrow so we can choose the best solution for
> buildroot.

Good. We'll see how it looks. So now the per-package sysroot also
contains the C library and kernel headers as well? It's really a
complete sysroot?

> >  * Your patch uses $($(2)_ADD_TOOLCHAIN_DEPENDENCY) to decide whether
> >    the per-package sysroot mechanism must be used or not. Which means
> >    it will only be used for target packages, and not for host
> >    packages. However, I'm wondering if we should not also apply the
> >    principle to host packages: to get reproducible builds, I believe
> >    we should also have separate sysroots when building host
> >    packages. Opinions from other Buildroot developers?
> >
> >  * Your approach only takes care of make the sysroot handled in a
> >    per-package fashion. But what about HOST_DIR ? We could have the
> >    same inconsistencies as the ones we discussed about STAGING_DIR, but
> >    this time caused by the presence/absence of host utilities. One
> >    build may give a given result because host tool "foo" is present (it
> >    happened to be built before), and the next build may give a
> >    different result because host tool "foo" is absent (it's going to be
> >    build after).
> 
> You are right, but if possible i just want to handle the host
> utilities matter as a next step after this step is done.

I'm indeed fine with solving this part as a next step, however, I'd
like to at least have some basic ideas of how it will be solved, to see
the big picture and see if solving the big picture is doable.

Also, and while this is definitely reaching much further than doing
top-level parallel build, I'm wondering if we shouldn't do the
Buildroot builds within a well-controlled chroot. This would allow us
to control the host environment, and ultimately cache the build results
and re-use them across builds. This is clearly not simple, as we need
to capture all the dependencies that have been built, their version,
their configuration, etc. I have no idea if it is reasonable to do that
in Buildroot, or if it is really too complicated to be compatible with
the "keep it simple" principle of Buildroot.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Arnout Vandecappelle]

On 06/15/15 11:17, Thomas Petazzoni wrote:
[snip]
> Also, and while this is definitely reaching much further than doing
> top-level parallel build, I'm wondering if we shouldn't do the
> Buildroot builds within a well-controlled chroot. This would allow us
> to control the host environment, and ultimately cache the build results
> and re-use them across builds. This is clearly not simple, as we need
> to capture all the dependencies that have been built, their version,
> their configuration, etc. I have no idea if it is reasonable to do that
> in Buildroot, or if it is really too complicated to be compatible with
> the "keep it simple" principle of Buildroot.

 That starts to smell more and more like NixOS... I.e., too complicated for
buildroot. But if someone proposes a patch and its diffstat looks acceptable,
the option is of course still open.

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Arnout Vandecappelle]

On 03/14/15 17:15, Fabio Porcedda wrote:
> On Thu, Mar 5, 2015 at 11:48 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
[snip]
>>  However, I don't like at all the approach of passing the custom -I and -L
>> flags. I think it's much better to start with something that was suggested
>> before in the BR developer meeting: use the toolchain wrapper also for the
>> internal toolchain. Then we can pass the per-package staging dir through the
>> environment, and use it in the --sysroot argument of the compiler.
>>
>>  We'll also have to do something about ccache then - that could be a problem.
>> Perhaps ccache should be called from within the wrapper as well, that may solve
>> a bunch of other ccache problems.
> 
> Using an internal wrapper we can also pass pass the -I and -L
> automatically like the --sysroot option?

 Yes we can. Of course, the path to the package sysroot has to be passed through
the environment.

> 
> If we use a per-package sysroot we need to copy the sysroot directory
> too, because the size of a uclibc sysroot is at least 11MB (if i
> remember correctly the size of a glibc sysroot is something like
> 40MB), so if we copy without using hard links, it will take at least
> 10MB up to 40MB for every package.
> 
> The size of the output directory of a defconfig + allyespackageconfig is 11GB.
> IMHO we have the following choices:
> a) common sysroot & per-package staging & without hard links
>   - the -I -L options are passed trough the toolchain wrapper
>   - output size is 16GB (11GB+5GB)

 Where does the 5GB come from? Experimental evidence from running this series?

>   - no side effects to fear, because hard links are not used
> b) per-package sysroot & without hard links
>   - sysroot are passed trough the toolchain wrapper
>   - output size will be 16GB plus 11MB * 694 = 16GB + 7GB = 23 GB
>   - no side effects to fear, because hard links are not used
> c) per-package sysroot & with hard links for toolchain
>  - sysroot are passed trough the toolchain wrapper
>   - output size will be 16GB
>   - side effects to fear for the toolchain files because hard links
> are used for them
> d) per-package sysroot & with hard links for all files
>  - sysroot are passed trough the toolchain wrapper
>   - output size will be 11GB, so no increased size
>   - side effects to fear because hard links are used

 Actually I'm not really concerned about the size, even if it's more than
doubled by this feature. We use factors less space than yocto so we can afford
to double it :-)

 I'm more concerned about the time penalty of copying things around all the
time. So in that respect, option a) or d) are more attractive. But even creating
all the hardlinks takes a lot of time (probably not that much faster than copying).

 A completely wild alternative is to build and use unionfs-fuse, but that
requires fusermount to be properly installed on the host. Can we assume that?

> 
> The cheaper one with almost no size and speed penalization is d) but
> if some file is overwritten other package sysroot could be affected.
> Maybe we can prevent the overwriting by removing the write permission
> on staging files?

 Yep, that's an option. It can actually be done while rsyncing.


 Regards,
 Arnout

> 
> Which solution is most suited for  buildroot?
> 
> Thanks and my BR
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Mon, Jun 15, 2015 at 11:17 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Fabio Porcedda,
>
> On Mon, 15 Jun 2015 11:06:40 +0200, Fabio Porcedda wrote:
>

[snip]

>> >  * I am not super happy with the idea of having the toolchain sysroot
>> >    left in the global staging directory and referenced by the compiler
>> >    --sysroot option on one side, and all other libraries found by
>> >    using -L, -I and pkg-config tricks.
>> >
>> >    I would actually prefer if a real complete sysroot was used when
>> >    building each package, and the compiler --sysroot option used to
>> >    adjust the compiler sysroot. This has however one significant
>> >    drawback: the toolchain sysroot must be copied for each and every
>> >    package, which can become quite time and space consuming. So on
>> >    this aspect, I'd like to have some input from other Buildroot
>> >    developers.
>>
>> I've already rewritten the patch set using the --sysroot option in the
>> toolchain wrapper. I just need to clean it up and send it, i hope to
>> be able to send it tomorrow so we can choose the best solution for
>> buildroot.
>
> Good. We'll see how it looks. So now the per-package sysroot also
> contains the C library and kernel headers as well? It's really a
> complete sysroot?

Yes, i copy the whole directory ${STAGING_DIR}, e.g.
output/host/usr/arm-buildroot-linux-gnueabi/sysroot

>> >  * Your patch uses $($(2)_ADD_TOOLCHAIN_DEPENDENCY) to decide whether
>> >    the per-package sysroot mechanism must be used or not. Which means
>> >    it will only be used for target packages, and not for host
>> >    packages. However, I'm wondering if we should not also apply the
>> >    principle to host packages: to get reproducible builds, I believe
>> >    we should also have separate sysroots when building host
>> >    packages. Opinions from other Buildroot developers?
>> >
>> >  * Your approach only takes care of make the sysroot handled in a
>> >    per-package fashion. But what about HOST_DIR ? We could have the
>> >    same inconsistencies as the ones we discussed about STAGING_DIR, but
>> >    this time caused by the presence/absence of host utilities. One
>> >    build may give a given result because host tool "foo" is present (it
>> >    happened to be built before), and the next build may give a
>> >    different result because host tool "foo" is absent (it's going to be
>> >    build after).
>>
>> You are right, but if possible i just want to handle the host
>> utilities matter as a next step after this step is done.
>
> I'm indeed fine with solving this part as a next step, however, I'd
> like to at least have some basic ideas of how it will be solved, to see
> the big picture and see if solving the big picture is doable.

My idea is to solve it in the same way as the staging directory,
copying the whole "output/host" directory expect for the sysroot part
if it is a host package.
Or maybe there is better way to do it?

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Thu, Jun 18, 2015 at 1:09 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 06/15/15 11:17, Thomas Petazzoni wrote:
> [snip]
>> Also, and while this is definitely reaching much further than doing
>> top-level parallel build, I'm wondering if we shouldn't do the
>> Buildroot builds within a well-controlled chroot. This would allow us
>> to control the host environment, and ultimately cache the build results
>> and re-use them across builds. This is clearly not simple, as we need
>> to capture all the dependencies that have been built, their version,
>> their configuration, etc. I have no idea if it is reasonable to do that
>> in Buildroot, or if it is really too complicated to be compatible with
>> the "keep it simple" principle of Buildroot.
>
>  That starts to smell more and more like NixOS... I.e., too complicated for
> buildroot. But if someone proposes a patch and its diffstat looks acceptable,
> the option is of course still open.

Maybe at least initially can be solved using an external script that
prepare the system and launch buildroot inside the chroot.

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Thu, Jun 18, 2015 at 1:32 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 03/14/15 17:15, Fabio Porcedda wrote:
>> On Thu, Mar 5, 2015 at 11:48 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> [snip]
>>>  However, I don't like at all the approach of passing the custom -I and -L
>>> flags. I think it's much better to start with something that was suggested
>>> before in the BR developer meeting: use the toolchain wrapper also for the
>>> internal toolchain. Then we can pass the per-package staging dir through the
>>> environment, and use it in the --sysroot argument of the compiler.
>>>
>>>  We'll also have to do something about ccache then - that could be a problem.
>>> Perhaps ccache should be called from within the wrapper as well, that may solve
>>> a bunch of other ccache problems.
>>
>> Using an internal wrapper we can also pass pass the -I and -L
>> automatically like the --sysroot option?
>
>  Yes we can. Of course, the path to the package sysroot has to be passed through
> the environment.
>
>>
>> If we use a per-package sysroot we need to copy the sysroot directory
>> too, because the size of a uclibc sysroot is at least 11MB (if i
>> remember correctly the size of a glibc sysroot is something like
>> 40MB), so if we copy without using hard links, it will take at least
>> 10MB up to 40MB for every package.
>>
>> The size of the output directory of a defconfig + allyespackageconfig is 11GB.
>> IMHO we have the following choices:
>> a) common sysroot & per-package staging & without hard links
>>   - the -I -L options are passed trough the toolchain wrapper
>>   - output size is 16GB (11GB+5GB)
>
>  Where does the 5GB come from? Experimental evidence from running this series?

If i remember correctly this was calculated from experimental evidence.

>>   - no side effects to fear, because hard links are not used
>> b) per-package sysroot & without hard links
>>   - sysroot are passed trough the toolchain wrapper
>>   - output size will be 16GB plus 11MB * 694 = 16GB + 7GB = 23 GB
>>   - no side effects to fear, because hard links are not used
>> c) per-package sysroot & with hard links for toolchain
>>  - sysroot are passed trough the toolchain wrapper
>>   - output size will be 16GB
>>   - side effects to fear for the toolchain files because hard links
>> are used for them
>> d) per-package sysroot & with hard links for all files
>>  - sysroot are passed trough the toolchain wrapper
>>   - output size will be 11GB, so no increased size
>>   - side effects to fear because hard links are used
>
>  Actually I'm not really concerned about the size, even if it's more than
> doubled by this feature. We use factors less space than yocto so we can afford
> to double it :-)
>
>  I'm more concerned about the time penalty of copying things around all the
> time. So in that respect, option a) or d) are more attractive. But even creating
> all the hardlinks takes a lot of time (probably not that much faster than copying).

I've made some tests about this, i will send those numbers with the
updated patch set.

>  A completely wild alternative is to build and use unionfs-fuse, but that
> requires fusermount to be properly installed on the host. Can we assume that?

Good question, maybe requiring fusermount is fine, but  I don't know
unionfs-fuse so i don't how much it is a viable solution.

>>
>> The cheaper one with almost no size and speed penalization is d) but
>> if some file is overwritten other package sysroot could be affected.
>> Maybe we can prevent the overwriting by removing the write permission
>> on staging files?
>
>  Yep, that's an option. It can actually be done while rsyncing.

Right, I've used rsync for creating hard links in my updated patch set.

BR
-- 
Fabio Porcedda

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Thomas Petazzoni]

Hello,

On Sun, 28 Jun 2015 17:33:21 +0200, Fabio Porcedda wrote:

> > Good. We'll see how it looks. So now the per-package sysroot also
> > contains the C library and kernel headers as well? It's really a
> > complete sysroot?
> 
> Yes, i copy the whole directory ${STAGING_DIR}, e.g.
> output/host/usr/arm-buildroot-linux-gnueabi/sysroot

Ok. When you say "whole", you say that you also "package" what the
toolchain installs, and create a complete sysroot from scratch for each
package being built, including for the toolchain libs/headers ?

> > I'm indeed fine with solving this part as a next step, however, I'd
> > like to at least have some basic ideas of how it will be solved, to see
> > the big picture and see if solving the big picture is doable.
> 
> My idea is to solve it in the same way as the staging directory,
> copying the whole "output/host" directory expect for the sysroot part
> if it is a host package.
> Or maybe there is better way to do it?

Again, when you say "whole", you also separate in multiple host dirs
whatever is installed by each package, and then create a custom
host dir when each package is being built, containing only the
dependencies that it needs?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Thomas Petazzoni]

Dear Fabio Porcedda,

On Sun, 28 Jun 2015 17:36:14 +0200, Fabio Porcedda wrote:

> >  That starts to smell more and more like NixOS... I.e., too complicated for
> > buildroot. But if someone proposes a patch and its diffstat looks acceptable,
> > the option is of course still open.
> 
> Maybe at least initially can be solved using an external script that
> prepare the system and launch buildroot inside the chroot.

The point is not to launch Buildroot instead the chroot, but to have
each package being built independently in the chroot.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC v3 00/30] Add per-package staging feature

[Fabio Porcedda]

On Sun, Jun 28, 2015 at 8:12 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Sun, 28 Jun 2015 17:33:21 +0200, Fabio Porcedda wrote:
>
>> > Good. We'll see how it looks. So now the per-package sysroot also
>> > contains the C library and kernel headers as well? It's really a
>> > complete sysroot?
>>
>> Yes, i copy the whole directory ${STAGING_DIR}, e.g.
>> output/host/usr/arm-buildroot-linux-gnueabi/sysroot
>
> Ok. When you say "whole", you say that you also "package" what the
> toolchain installs, and create a complete sysroot from scratch for each
> package being built, including for the toolchain libs/headers ?

Yes.

>> > I'm indeed fine with solving this part as a next step, however, I'd
>> > like to at least have some basic ideas of how it will be solved, to see
>> > the big picture and see if solving the big picture is doable.
>>
>> My idea is to solve it in the same way as the staging directory,
>> copying the whole "output/host" directory expect for the sysroot part
>> if it is a host package.
>> Or maybe there is better way to do it?
>
> Again, when you say "whole", you also separate in multiple host dirs
> whatever is installed by each package, and then create a custom
> host dir when each package is being built, containing only the
> dependencies that it needs?

Yes

Best regards
-- 
Fabio Porcedda