All of lore.kernel.org
 help / color / mirror / Atom feed
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
@ 2015-06-28 19:42 Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 01/16] packages: use TARGET_MAKE_ENV to add pps support Fabio Porcedda
                   ` (20 more replies)
  0 siblings, 21 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

Hi all,
this patch set aims to improve build reproducibility by using a
per-package staging directory.
Also this feature aims to enable the top-level parallel make.

This patch set is available at:
https://github.com/fabio-porcedda/buildroot/tree/pps-v4


Every package instead of installing the files in the shared staging
directory (STAGING_DIR) install all files in a subdirectory of the
STAGINGPKG_DIR.

At the beginning of the configure stage, all the dependencies staging
directory are copied in the package staging directory, e.g.:

zlib-install-staging: install files in /output/stagingpkg/zlib/

cramsfs-configure: copy all files from /output/stagingpkg/zlib/* to
  /output/stagingpkg/cramfs/

The STAGING_DIR variable is redefined for every package in order to
point to the per-package directory, so a package cannot access
automatically to staging files of packages that aren't in the
(PKG)_DENDENDENCIES variable.

To improve performance and reduce space utilization hard links are used.

To evaluate performance and space utilization I've done some tests.

HW-MED:
CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
RAM: 16GB
SSD: 512GB

HW-HIGH:
CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
RAM: 32GB
SATA: RAID0 2x2TB 3.5" 7200rpm


defconfig-small, long chain dependency, unfavorable  scenario for parallelization:

BR2_arm=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_PACKAGE_XORG7=y
BR2_PACKAGE_XSERVER_XORG_SERVER=y
BR2_PACKAGE_LIBGTK3=y

defconfig-full:
  BR2_arm=y
  BR2_TOOLCHAIN_EXTERNAL=y
make allyespackageconfig
Also i removed few packages that do not built on my system.
make show-targets | wc -w
1366

Test about output using hard links or note using the "defconfig-full"
on HW-MED:

|  35GB |  99m | master branch
|  37GB | 100m | patch set             
| 153GB | 105m | patch set using hard links only for the toolchain sysroot
| 225GB | 106m | patch set not using hardlinks at all

Test about performances of this patch set vs master branch:
| 199m | HW-MED | defconfig-full | master branch | no top-level make |
|  99m | HW-MED | defconfig-full | master branch | top-level make    | 
| 100m | HW-MED | defconfig-full | patch set     | top-level make    |

| 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
|  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
|  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |

| 10m | HW-MED | defconfig-small | master branch | no top-level make |
|  5m | HW-MED | defconfig-small | master branch | top-level make    | 
|  5m | HW-MED | defconfig-samll | patch set     | top-level make    |

| 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
|  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
|  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |


TODO:
 - use the toolchaing wrapper even for the internal toolchain
 - rebase over master (this patch set is based on 2015.05-161-gca5f11e).
 - more testing (tested only for arm target).

Best regads

Fabio Porcedda (16):
  packages: use TARGET_MAKE_ENV to add pps support
  packages: for staging stuff use backtick instead of the shell function
  packages/scons: make avaiable the GCC_SYSROOT environment variable
  packages/qt: read spec files from the per-package staging directory
  pkg-perl: use TARGET_MAKE_ENV to add pps support
  tooclahin-external: add support for GCC_SYSROOT
  toolchain-external: move some code in a new function
  toolchain-external: use the wrapper even for the linker "ld"
  infra: add GCC_SYSROOT environment variable to support pps feature
  pkg-cmake: add GCC_SYSROOT environment variable to support pps feature
  pkg-python: add GCC_SYSROOT variable to add pps support
  pacakge/luarocks: add GCC_SYSROOT environment variable to support pps
  package/pkgconf: use GCC_SYSROOT to support pps feature
  Makefile: add STAGINGNOPKG_DIR variable
  pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages
  infra: add per-package staging feature

 Makefile                                           | 16 ++--
 package/Makefile.in                                |  5 +-
 package/aiccu/aiccu.mk                             |  2 +-
 package/blktrace/blktrace.mk                       |  2 +-
 package/boost/boost.mk                             |  2 +-
 package/botan/botan.mk                             |  2 +-
 package/bsdiff/bsdiff.mk                           |  4 +-
 package/dbus-python/dbus-python.mk                 |  8 +-
 package/dhcpdump/dhcpdump.mk                       |  3 +-
 package/dtc/dtc.mk                                 |  3 +-
 package/exim/exim.mk                               |  5 +-
 package/fbv/fbv.mk                                 |  2 +-
 package/feh/feh.mk                                 |  4 +-
 package/gadgetfs-test/gadgetfs-test.mk             |  2 +-
 package/gdk-pixbuf/gdk-pixbuf.mk                   |  2 +-
 package/iodine/iodine.mk                           |  2 +-
 package/iw/iw.mk                                   |  2 +-
 package/jack2/jack2.mk                             |  3 +-
 package/libmicrohttpd/libmicrohttpd.mk             |  2 +-
 package/libnss/libnss.mk                           |  4 +-
 package/libupnpp/libupnpp.mk                       |  2 +-
 package/lmbench/lmbench.mk                         |  8 +-
 package/logrotate/logrotate.mk                     |  2 +-
 package/lua-msgpack-native/lua-msgpack-native.mk   |  3 +-
 ...ctions-to-the-global-environment-of-the-c.patch | 54 ++++++++++++
 package/luarocks/0002-fixed-typo.patch             | 27 ++++++
 ...e-environment-table-for-the-system-config.patch | 76 +++++++++++++++++
 ...is-not-being-integrated-bug-introduced-in.patch | 62 ++++++++++++++
 package/luarocks/luarocks.mk                       | 11 ++-
 package/madplay/madplay.mk                         |  2 +-
 package/moarvm/moarvm.mk                           |  2 +-
 package/mongoose/mongoose.mk                       |  8 +-
 package/mpg123/mpg123.mk                           |  2 +-
 package/mplayer/mplayer.mk                         |  2 +-
 package/ncurses/ncurses.mk                         |  4 +-
 package/neardal/neardal.mk                         |  2 +-
 package/nginx/nginx.mk                             |  1 +
 package/nmap/nmap.mk                               |  2 +-
 package/open2300/open2300.mk                       |  2 +-
 package/openldap/openldap.mk                       |  2 +-
 package/openssl/openssl.mk                         |  2 +-
 package/opentyrian/opentyrian.mk                   |  2 +-
 package/pax-utils/pax-utils.mk                     |  3 +-
 package/php/php.mk                                 |  4 +-
 package/pifmrds/pifmrds.mk                         |  3 +-
 package/pkg-cmake.mk                               |  1 +
 package/pkg-generic.mk                             | 43 +++++++++-
 package/pkg-perl.mk                                |  2 +-
 package/pkg-python.mk                              |  2 +
 package/pkgconf/pkg-config.in                      |  4 +-
 package/pkgconf/pkgconf.mk                         |  5 +-
 package/poco/poco.mk                               |  3 +-
 package/poppler/poppler.mk                         |  2 +-
 package/pppd/pppd.mk                               |  2 +-
 package/python-gobject/python-gobject.mk           |  4 +-
 package/python-pygame/python-pygame.mk             |  6 +-
 package/python-pyqt/python-pyqt.mk                 |  1 +
 package/python-sip/python-sip.mk                   |  2 +-
 package/qt/qt.mk                                   | 18 ++--
 package/qtuio/qtuio.mk                             |  3 +-
 package/rtmpdump/rtmpdump.mk                       |  2 +-
 package/scons/scons.mk                             |  3 +-
 package/scons/site_init.py                         | 19 +++++
 package/speex/speex.mk                             |  3 +-
 package/statserial/statserial.mk                   |  3 +-
 package/tcpreplay/tcpreplay.mk                     |  6 +-
 package/ti-utils/ti-utils.mk                       |  6 +-
 package/tvheadend/tvheadend.mk                     |  4 +-
 package/upmpdcli/upmpdcli.mk                       |  2 +-
 package/ushare/ushare.mk                           |  3 +-
 package/vsftpd/vsftpd.mk                           |  4 +-
 support/misc/toolchainfile.cmake.in                |  9 +-
 .../toolchain-external/ext-toolchain-wrapper.c     | 96 ++++++++++++++--------
 toolchain/toolchain-external/toolchain-external.mk |  2 +-
 74 files changed, 476 insertions(+), 147 deletions(-)
 create mode 100644 package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
 create mode 100644 package/luarocks/0002-fixed-typo.patch
 create mode 100644 package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
 create mode 100644 package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch
 create mode 100644 package/scons/site_init.py

-- 
2.4.3

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 01/16] packages: use TARGET_MAKE_ENV to add pps support
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 02/16] packages: for staging stuff use backtick instead of the shell function Fabio Porcedda
                   ` (19 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In the TARGET_MAKE_ENV variable will be added the GCC_SYSROOT variable
that is needed for per-package staging feature.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/blktrace/blktrace.mk                     | 2 +-
 package/boost/boost.mk                           | 2 +-
 package/botan/botan.mk                           | 2 +-
 package/bsdiff/bsdiff.mk                         | 4 ++--
 package/dhcpdump/dhcpdump.mk                     | 3 ++-
 package/dtc/dtc.mk                               | 3 ++-
 package/exim/exim.mk                             | 5 +++--
 package/feh/feh.mk                               | 4 ++--
 package/gadgetfs-test/gadgetfs-test.mk           | 2 +-
 package/iodine/iodine.mk                         | 2 +-
 package/iw/iw.mk                                 | 2 +-
 package/jack2/jack2.mk                           | 3 ++-
 package/libnss/libnss.mk                         | 4 ++--
 package/lmbench/lmbench.mk                       | 4 +++-
 package/lua-msgpack-native/lua-msgpack-native.mk | 3 ++-
 package/moarvm/moarvm.mk                         | 2 +-
 package/mongoose/mongoose.mk                     | 8 +++++---
 package/mplayer/mplayer.mk                       | 2 +-
 package/ncurses/ncurses.mk                       | 4 ++--
 package/nginx/nginx.mk                           | 1 +
 package/open2300/open2300.mk                     | 2 +-
 package/openssl/openssl.mk                       | 2 +-
 package/opentyrian/opentyrian.mk                 | 2 +-
 package/pax-utils/pax-utils.mk                   | 3 ++-
 package/pifmrds/pifmrds.mk                       | 3 ++-
 package/poco/poco.mk                             | 3 ++-
 package/pppd/pppd.mk                             | 2 +-
 package/qt/qt.mk                                 | 2 ++
 package/qtuio/qtuio.mk                           | 3 ++-
 package/rtmpdump/rtmpdump.mk                     | 2 +-
 package/speex/speex.mk                           | 3 ++-
 package/statserial/statserial.mk                 | 3 ++-
 package/ti-utils/ti-utils.mk                     | 2 +-
 package/tvheadend/tvheadend.mk                   | 4 ++--
 package/ushare/ushare.mk                         | 3 ++-
 package/vsftpd/vsftpd.mk                         | 2 +-
 36 files changed, 61 insertions(+), 42 deletions(-)

diff --git a/package/blktrace/blktrace.mk b/package/blktrace/blktrace.mk
index c193375..8d5e52b 100644
--- a/package/blktrace/blktrace.mk
+++ b/package/blktrace/blktrace.mk
@@ -11,7 +11,7 @@ BLKTRACE_LICENSE = GPLv2+
 BLKTRACE_LICENSE_FILES = COPYING
 
 define BLKTRACE_BUILD_CMDS
-	$(MAKE1) -C $(@D) $(TARGET_CONFIGURE_OPTS)
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) $(TARGET_CONFIGURE_OPTS)
 endef
 
 define BLKTRACE_INSTALL_TARGET_CMDS
diff --git a/package/boost/boost.mk b/package/boost/boost.mk
index dbb5de5..24f46e2 100644
--- a/package/boost/boost.mk
+++ b/package/boost/boost.mk
@@ -148,7 +148,7 @@ define HOST_BOOST_INSTALL_CMDS
 endef
 
 define BOOST_INSTALL_STAGING_CMDS
-	(cd $(@D) && ./bjam -j$(PARALLEL_JOBS) -q -d+1 \
+	(cd $(@D) && $(TARGET_MAKE_ENV) ./bjam -j$(PARALLEL_JOBS) -q -d+1 \
 	--user-config=$(@D)/user-config.jam \
 	$(BOOST_OPTS) \
 	--prefix=$(STAGING_DIR)/usr \
diff --git a/package/botan/botan.mk b/package/botan/botan.mk
index 6ac3ce7..58456fb 100644
--- a/package/botan/botan.mk
+++ b/package/botan/botan.mk
@@ -48,7 +48,7 @@ define BOTAN_CONFIGURE_CMDS
 endef
 
 define BOTAN_BUILD_CMDS
-	$(MAKE) -C $(@D) AR="$(TARGET_AR) crs"
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) AR="$(TARGET_AR) crs"
 endef
 
 define BOTAN_INSTALL_STAGING_CMDS
diff --git a/package/bsdiff/bsdiff.mk b/package/bsdiff/bsdiff.mk
index 7d2b296..eb903e0 100644
--- a/package/bsdiff/bsdiff.mk
+++ b/package/bsdiff/bsdiff.mk
@@ -11,9 +11,9 @@ BSDIFF_LICENSE = BSD-2c
 BSDIFF_LICENSE_FILES = bsdiff.c
 
 define BSDIFF_BUILD_CMDS
-	$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
+	$(TARGET_MAKE_ENV) $(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
 		$(@D)/bsdiff.c -lbz2 -o $(@D)/bsdiff
-	$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
+	$(TARGET_MAKE_ENV) $(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) \
 		$(@D)/bspatch.c -lbz2 -o $(@D)/bspatch
 endef
 
diff --git a/package/dhcpdump/dhcpdump.mk b/package/dhcpdump/dhcpdump.mk
index 0756319..8df5135 100644
--- a/package/dhcpdump/dhcpdump.mk
+++ b/package/dhcpdump/dhcpdump.mk
@@ -16,7 +16,8 @@ DHCPDUMP_LIBS += $(shell $(STAGING_DIR)/usr/bin/pcap-config --static --additiona
 endif
 
 define DHCPDUMP_BUILD_CMDS
-	$(MAKE) -C $(@D) CC="$(TARGET_CC) $(TARGET_CFLAGS)" LIBS="$(DHCPDUMP_LIBS)"
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) \
+		CC="$(TARGET_CC) $(TARGET_CFLAGS)" LIBS="$(DHCPDUMP_LIBS)"
 endef
 
 define DHCPDUMP_INSTALL_TARGET_CMDS
diff --git a/package/dtc/dtc.mk b/package/dtc/dtc.mk
index e094f7b..0800fb0 100644
--- a/package/dtc/dtc.mk
+++ b/package/dtc/dtc.mk
@@ -41,7 +41,8 @@ define DTC_INSTALL_STAGING_CMDS
 endef
 
 define DTC_INSTALL_TARGET_CMDS
-	$(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) PREFIX=/usr $(DTC_INSTALL_GOAL)
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		-C $(@D) DESTDIR=$(TARGET_DIR) PREFIX=/usr $(DTC_INSTALL_GOAL)
 endef
 
 $(eval $(generic-package))
diff --git a/package/exim/exim.mk b/package/exim/exim.mk
index b852793..4f75e9a 100644
--- a/package/exim/exim.mk
+++ b/package/exim/exim.mk
@@ -111,8 +111,9 @@ endef
 # Need to replicate the LFLAGS in install, as exim still wants to build
 # something when installing...
 define EXIM_INSTALL_TARGET_CMDS
-	DESTDIR=$(TARGET_DIR) INSTALL_ARG="-no_chown -no_symlink" build=br \
-	  $(MAKE1) -C $(@D) $(EXIM_STATIC_FLAGS) install
+	$(TARGET_MAKE_ENV) DESTDIR=$(TARGET_DIR) \
+		INSTALL_ARG="-no_chown -no_symlink" build=br \
+		$(MAKE1) -C $(@D) $(EXIM_STATIC_FLAGS) install
 	chmod u+s $(TARGET_DIR)/usr/sbin/exim
 endef
 
diff --git a/package/feh/feh.mk b/package/feh/feh.mk
index 0c4c164..27ac53b 100644
--- a/package/feh/feh.mk
+++ b/package/feh/feh.mk
@@ -12,8 +12,8 @@ FEH_LICENSE = MIT
 FEH_LICENSE_FILES = COPYING
 
 define FEH_BUILD_CMDS
-	$(MAKE1) CC="$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS)" \
-		-C $(@D) all
+	$(TARGET_MAKE_ENV) $(MAKE1) CC="$(TARGET_CC) $(TARGET_CFLAGS) \
+		$(TARGET_LDFLAGS)" -C $(@D) all
 endef
 
 define FEH_INSTALL_TARGET_CMDS
diff --git a/package/gadgetfs-test/gadgetfs-test.mk b/package/gadgetfs-test/gadgetfs-test.mk
index 655f50a..bdeaede 100644
--- a/package/gadgetfs-test/gadgetfs-test.mk
+++ b/package/gadgetfs-test/gadgetfs-test.mk
@@ -15,7 +15,7 @@ GADGETFS_TEST_MAKEOPTS += USE_AIO=y
 endif
 
 define GADGETFS_TEST_BUILD_CMDS
-	$(MAKE) -C $(@D) $(GADGETFS_TEST_MAKEOPTS)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) $(GADGETFS_TEST_MAKEOPTS)
 endef
 
 define GADGETFS_TEST_INSTALL_TARGET_CMDS
diff --git a/package/iodine/iodine.mk b/package/iodine/iodine.mk
index 57deb9e..c3df831 100644
--- a/package/iodine/iodine.mk
+++ b/package/iodine/iodine.mk
@@ -11,7 +11,7 @@ IODINE_LICENSE = MIT
 IODINE_LICENSE_FILES = README
 
 define IODINE_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" ARCH=$(BR2_ARCH) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" ARCH=$(BR2_ARCH) -C $(@D)
 endef
 
 define IODINE_INSTALL_TARGET_CMDS
diff --git a/package/iw/iw.mk b/package/iw/iw.mk
index 2766850..3aa285b 100644
--- a/package/iw/iw.mk
+++ b/package/iw/iw.mk
@@ -11,7 +11,7 @@ IW_LICENSE = iw license
 IW_LICENSE_FILES = COPYING
 IW_DEPENDENCIES = host-pkgconf libnl
 IW_MAKE_OPTS = CC="$(TARGET_CC)" LD="$(TARGET_LD)" LDFLAGS="$(TARGET_LDFLAGS)"
-IW_MAKE_ENV = PKG_CONFIG="$(HOST_DIR)/usr/bin/pkg-config" \
+IW_MAKE_ENV = $(TARGET_MAKE_ENV) PKG_CONFIG="$(HOST_DIR)/usr/bin/pkg-config" \
 	GIT_DIR=$(IW_DIR)
 
 define IW_BUILD_CMDS
diff --git a/package/jack2/jack2.mk b/package/jack2/jack2.mk
index 73328d3..3db7f3d 100644
--- a/package/jack2/jack2.mk
+++ b/package/jack2/jack2.mk
@@ -20,7 +20,8 @@ define JACK2_CONFIGURE_CMDS
 endef
 
 define JACK2_BUILD_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python2 ./waf build -j $(PARALLEL_JOBS))
+	(cd $(@D); $(TARGET_MAKE_ENV) \
+		$(HOST_DIR)/usr/bin/python2 ./waf build -j $(PARALLEL_JOBS))
 endef
 
 define JACK2_INSTALL_TARGET_CMDS
diff --git a/package/libnss/libnss.mk b/package/libnss/libnss.mk
index db272b3..529b773 100644
--- a/package/libnss/libnss.mk
+++ b/package/libnss/libnss.mk
@@ -38,12 +38,12 @@ endif
 endif
 
 define LIBNSS_BUILD_CMDS
-	$(MAKE1) -C $(@D)/nss coreconf \
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)/nss coreconf \
 		SOURCE_MD_DIR=$(@D)/$(LIBNSS_DISTDIR) \
 		DIST=$(@D)/$(LIBNSS_DISTDIR) \
 		CHECKLOC= \
 		$(LIBNSS_BUILD_VARS)
-	$(MAKE1) -C $(@D)/nss lib/dbm all \
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)/nss lib/dbm all \
 		SOURCE_MD_DIR=$(@D)/$(LIBNSS_DISTDIR) \
 		DIST=$(@D)/$(LIBNSS_DISTDIR) \
 		CHECKLOC= \
diff --git a/package/lmbench/lmbench.mk b/package/lmbench/lmbench.mk
index f593428..d4be8d4 100644
--- a/package/lmbench/lmbench.mk
+++ b/package/lmbench/lmbench.mk
@@ -31,7 +31,9 @@ endef
 # script. So the variables override below don't take direct effect in
 # src/Makefile.
 define LMBENCH_BUILD_CMDS
-	$(MAKE) CFLAGS="$(LMBENCH_CFLAGS)" LDLIBS="$(LMBENCH_LDLIBS)" OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		CFLAGS="$(LMBENCH_CFLAGS)" LDLIBS="$(LMBENCH_LDLIBS)" \
+		OS=$(ARCH) CC="$(TARGET_CC)" -C $(@D)/src
 endef
 
 define LMBENCH_INSTALL_TARGET_CMDS
diff --git a/package/lua-msgpack-native/lua-msgpack-native.mk b/package/lua-msgpack-native/lua-msgpack-native.mk
index acd3a0e..cc68b3b 100644
--- a/package/lua-msgpack-native/lua-msgpack-native.mk
+++ b/package/lua-msgpack-native/lua-msgpack-native.mk
@@ -11,7 +11,8 @@ LUA_MSGPACK_NATIVE_LICENSE = Apache-2.0
 LUA_MSGPACK_NATIVE_LICENSE_FILES = LICENSE.txt
 
 define LUA_MSGPACK_NATIVE_BUILD_CMDS
-	$(TARGET_CC) $(TARGET_CFLAGS) -fPIC -shared -o $(@D)/msgpack.so $(@D)/mp.c
+	$(TARGET_MAKE_ENV) $(TARGET_CC) \
+		$(TARGET_CFLAGS) -fPIC -shared -o $(@D)/msgpack.so $(@D)/mp.c
 endef
 
 define LUA_MSGPACK_NATIVE_INSTALL_TARGET_CMDS
diff --git a/package/moarvm/moarvm.mk b/package/moarvm/moarvm.mk
index 5889436..e00058b 100644
--- a/package/moarvm/moarvm.mk
+++ b/package/moarvm/moarvm.mk
@@ -32,7 +32,7 @@ define MOARVM_CONFIGURE_CMDS
 endef
 
 define MOARVM_BUILD_CMDS
-	$(MAKE) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define MOARVM_INSTALL_STAGING_CMDS
diff --git a/package/mongoose/mongoose.mk b/package/mongoose/mongoose.mk
index 6797e58..0961946 100644
--- a/package/mongoose/mongoose.mk
+++ b/package/mongoose/mongoose.mk
@@ -19,11 +19,13 @@ MONGOOSE_CFLAGS += -DNS_ENABLE_SSL -lssl -lcrypto -lz
 endif
 
 define MONGOOSE_BUILD_CMDS
-	$(TARGET_CC) $(@D)/examples/web_server/web_server.c $(@D)/mongoose.c \
+	$(TARGET_MAKE_ENV) $(TARGET_CC) \
+		$(@D)/examples/web_server/web_server.c $(@D)/mongoose.c \
 		-I$(@D) -o $(@D)/examples/web_server/web_server \
 		$(MONGOOSE_CFLAGS) -pthread
-	$(TARGET_CC) -c $(@D)/mongoose.c $(MONGOOSE_CFLAGS) -o $(@D)/mongoose.o
-	$(TARGET_AR) rcs $(@D)/libmongoose.a $(@D)/mongoose.o
+	$(TARGET_MAKE_ENV) $(TARGET_CC) \
+		-c $(@D)/mongoose.c $(MONGOOSE_CFLAGS) -o $(@D)/mongoose.o
+	$(TARGET_MAKE_ENV) $(TARGET_AR) rcs $(@D)/libmongoose.a $(@D)/mongoose.o
 endef
 
 define MONGOOSE_INSTALL_TARGET_CMDS
diff --git a/package/mplayer/mplayer.mk b/package/mplayer/mplayer.mk
index 3a61b8f..6ac91ab 100644
--- a/package/mplayer/mplayer.mk
+++ b/package/mplayer/mplayer.mk
@@ -269,7 +269,7 @@ define MPLAYER_CONFIGURE_CMDS
 endef
 
 define MPLAYER_BUILD_CMDS
-	$(MAKE) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define MPLAYER_INSTALL_TARGET_CMDS
diff --git a/package/ncurses/ncurses.mk b/package/ncurses/ncurses.mk
index 8368c69..6dcf496 100644
--- a/package/ncurses/ncurses.mk
+++ b/package/ncurses/ncurses.mk
@@ -115,9 +115,9 @@ endif
 # ncurses breaks with parallel build, but takes quite a while to
 # build single threaded. Work around it similar to how Gentoo does
 define NCURSES_BUILD_CMDS
-	$(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) sources
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) sources
 	rm -rf $(@D)/misc/pc-files
-	$(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR)
 endef
 
 ifneq ($(BR2_STATIC_LIBS),y)
diff --git a/package/nginx/nginx.mk b/package/nginx/nginx.mk
index 90ebbcc..3a4599b 100644
--- a/package/nginx/nginx.mk
+++ b/package/nginx/nginx.mk
@@ -31,6 +31,7 @@ endef
 
 # disable external libatomic_ops because its detection fails.
 NGINX_CONF_ENV += \
+	$(TARGET_MAKE_ENV) \
 	ngx_force_c_compiler=yes \
 	ngx_force_c99_have_variadic_macros=yes \
 	ngx_force_gcc_have_variadic_macros=yes \
diff --git a/package/open2300/open2300.mk b/package/open2300/open2300.mk
index 73db2a0..253a8cd 100644
--- a/package/open2300/open2300.mk
+++ b/package/open2300/open2300.mk
@@ -24,7 +24,7 @@ OPEN2300_LDFLAGS += -L$(STAGING_DIR)/usr/lib/mysql -lmysqlclient
 endif
 
 define OPEN2300_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_LD)" \
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_LD)" \
 		CFLAGS="$(OPEN2300_CFLAGS)" CC_LDFLAGS="$(OPEN2300_LDFLAGS)" \
 		-C $(@D) $(OPEN2300_BINS)
 endef
diff --git a/package/openssl/openssl.mk b/package/openssl/openssl.mk
index f90c811..0dc3750 100644
--- a/package/openssl/openssl.mk
+++ b/package/openssl/openssl.mk
@@ -99,7 +99,7 @@ define HOST_OPENSSL_BUILD_CMDS
 endef
 
 define OPENSSL_BUILD_CMDS
-	$(MAKE1) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)
 endef
 
 define OPENSSL_INSTALL_STAGING_CMDS
diff --git a/package/opentyrian/opentyrian.mk b/package/opentyrian/opentyrian.mk
index b74e4cf..6efbe4e 100644
--- a/package/opentyrian/opentyrian.mk
+++ b/package/opentyrian/opentyrian.mk
@@ -20,7 +20,7 @@ OPENTYRIAN_NETWORK = false
 endif
 
 define OPENTYRIAN_BUILD_CMDS
-	$(MAKE) PLATFORM=UNIX \
+	$(TARGET_MAKE_ENV) $(MAKE) PLATFORM=UNIX \
 		CC="$(TARGET_CC)" \
 		STRIP="/bin/true" \
 		SDL_CONFIG="$(STAGING_DIR)/usr/bin/sdl-config" \
diff --git a/package/pax-utils/pax-utils.mk b/package/pax-utils/pax-utils.mk
index fc0589f..462413d 100644
--- a/package/pax-utils/pax-utils.mk
+++ b/package/pax-utils/pax-utils.mk
@@ -31,7 +31,8 @@ define HOST_PAX_UTILS_BUILD_CMDS
 endef
 
 define PAX_UTILS_BUILD_CMDS
-	$(MAKE) $(TARGET_CONFIGURE_OPTS) $(PAX_UTILS_USE_CAP) -C $(@D)
+	$(TARGET_MAKE_ENV) \
+		$(MAKE) $(TARGET_CONFIGURE_OPTS) $(PAX_UTILS_USE_CAP) -C $(@D)
 endef
 
 define HOST_PAX_UTILS_INSTALL_CMDS
diff --git a/package/pifmrds/pifmrds.mk b/package/pifmrds/pifmrds.mk
index 5179715..9e71916 100644
--- a/package/pifmrds/pifmrds.mk
+++ b/package/pifmrds/pifmrds.mk
@@ -11,7 +11,8 @@ PIFMRDS_LICENSE = GPLv3+
 PIFMRDS_LICENSE_FILES = LICENSE
 
 define PIFMRDS_BUILD_CMDS
-	$(MAKE) -C $(@D)/src CC="$(TARGET_CC)" LDFLAGS="$(TARGET_LDFLAGS)" \
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		-C $(@D)/src CC="$(TARGET_CC)" LDFLAGS="$(TARGET_LDFLAGS)" \
 		CFLAGS="$(TARGET_CFLAGS) -std=gnu99 -ffast-math -c" \
 		app rds_wav
 endef
diff --git a/package/poco/poco.mk b/package/poco/poco.mk
index 7233d43..dff71e4 100644
--- a/package/poco/poco.mk
+++ b/package/poco/poco.mk
@@ -51,7 +51,8 @@ define POCO_CONFIGURE_CMDS
 endef
 
 define POCO_BUILD_CMDS
-	$(MAKE1) POCO_TARGET_OSARCH=$(ARCH) CROSSENV=$(TARGET_CROSS) \
+	$(TARGET_MAKE_ENV) $(MAKE1) \
+		POCO_TARGET_OSARCH=$(ARCH) CROSSENV=$(TARGET_CROSS) \
 		MYSQL_LIBDIR=$(STAGING_DIR)/usr/lib/mysql \
 		MYSQL_INCDIR=$(STAGING_DIR)/usr/include/mysql -C $(@D)
 endef
diff --git a/package/pppd/pppd.mk b/package/pppd/pppd.mk
index 0f9945f..e655457 100644
--- a/package/pppd/pppd.mk
+++ b/package/pppd/pppd.mk
@@ -50,7 +50,7 @@ define PPPD_CONFIGURE_CMDS
 endef
 
 define PPPD_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" COPTS="$(TARGET_CFLAGS)" \
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" COPTS="$(TARGET_CFLAGS)" \
 		-C $(@D) $(PPPD_MAKE_OPTS)
 endef
 
diff --git a/package/qt/qt.mk b/package/qt/qt.mk
index a7bbe5f..b2b9bbe 100644
--- a/package/qt/qt.mk
+++ b/package/qt/qt.mk
@@ -482,6 +482,8 @@ define QT_CONFIGURE_CONFIG_FILE
 endef
 endif
 
+QT_CONFIGURE_ENV += $(TARGET_MAKE_ENV)
+
 define QT_CONFIGURE_CMDS
 	-[ -f $(@D)/Makefile ] && $(MAKE) -C $(@D) confclean
 	$(QT_CONFIGURE_IPV6)
diff --git a/package/qtuio/qtuio.mk b/package/qtuio/qtuio.mk
index 5fa10a7..58008bb 100644
--- a/package/qtuio/qtuio.mk
+++ b/package/qtuio/qtuio.mk
@@ -32,7 +32,8 @@ endef
 ifeq ($(BR2_QTUIO_EXAMPLES),y)
 define QTUIO_BUILD_EXAMPLES
 	for example in $(QTUIO_EXAMPLES) ; do \
-		$(MAKE) -C $(@D)/examples/$${example} || exit 1; \
+		$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)/examples/$${example} || \
+			exit 1; \
 	done
 endef
 endif
diff --git a/package/rtmpdump/rtmpdump.mk b/package/rtmpdump/rtmpdump.mk
index 613f1cd..aaa0071 100644
--- a/package/rtmpdump/rtmpdump.mk
+++ b/package/rtmpdump/rtmpdump.mk
@@ -36,7 +36,7 @@ RTMPDUMP_SHARED = "SHARED="
 endif
 
 define RTMPDUMP_BUILD_CMDS
-	$(MAKE) CRYPTO=$(RTMPDUMP_CRYPTO) \
+	$(TARGET_MAKE_ENV) $(MAKE) CRYPTO=$(RTMPDUMP_CRYPTO) \
 		prefix=/usr \
 		XCFLAGS="$(RTMPDUMP_CFLAGS)" \
 		XLDFLAGS="$(TARGET_LDFLAGS)" \
diff --git a/package/speex/speex.mk b/package/speex/speex.mk
index a34bfac..c7d77a0 100644
--- a/package/speex/speex.mk
+++ b/package/speex/speex.mk
@@ -32,7 +32,8 @@ endef
 SPEEX_POST_CONFIGURE_HOOKS += SPEEX_LIBTOOL_FIXUP
 
 define SPEEX_BUILD_CMDS
-	$($(PKG)_MAKE_ENV) $(MAKE) $($(PKG)_MAKE_OPTS) -C $(@D)/$($(PKG)_SUBDIR)
+	$(TARGET_MAKE_ENV) $($(PKG)_MAKE_ENV) $(MAKE) \
+		$($(PKG)_MAKE_OPTS) -C $(@D)/$($(PKG)_SUBDIR)
 endef
 
 $(eval $(autotools-package))
diff --git a/package/statserial/statserial.mk b/package/statserial/statserial.mk
index f0e15bc..1d646d9 100644
--- a/package/statserial/statserial.mk
+++ b/package/statserial/statserial.mk
@@ -11,7 +11,8 @@ STATSERIAL_LICENSE = GPLv2+
 STATSERIAL_LICENSE_FILES = COPYING
 
 define STATSERIAL_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" LDFLAGS="$(TARGET_LDFLAGS)" -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" LD="$(TARGET_CC)" \
+		CFLAGS="$(TARGET_CFLAGS)" LDFLAGS="$(TARGET_LDFLAGS)" -C $(@D)
 endef
 
 define STATSERIAL_INSTALL_TARGET_CMDS
diff --git a/package/ti-utils/ti-utils.mk b/package/ti-utils/ti-utils.mk
index 3141219..35ecc19 100644
--- a/package/ti-utils/ti-utils.mk
+++ b/package/ti-utils/ti-utils.mk
@@ -14,7 +14,7 @@ TI_UTILS_CFLAGS = $(shell $(PKG_CONFIG_HOST_BINARY) --cflags libnl-genl-3.0)
 TI_UTILS_LIBS = $(shell $(PKG_CONFIG_HOST_BINARY) --libs libnl-genl-3.0)
 
 define TI_UTILS_BUILD_CMDS
-	$(MAKE1) NFSROOT="$(STAGING_DIR)" \
+	$(TARGET_MAKE_ENV) $(MAKE1) NFSROOT="$(STAGING_DIR)" \
 		CC="$(TARGET_CC) $(TARGET_CFLAGS) $(TARGET_LDFLAGS) $(TI_UTILS_CFLAGS)" \
 		LIBS="$(TI_UTILS_LIBS)" -C $(@D) all
 endef
diff --git a/package/tvheadend/tvheadend.mk b/package/tvheadend/tvheadend.mk
index 2a0019e..38a4ebe 100644
--- a/package/tvheadend/tvheadend.mk
+++ b/package/tvheadend/tvheadend.mk
@@ -68,11 +68,11 @@ define TVHEADEND_CONFIGURE_CMDS
 endef
 
 define TVHEADEND_BUILD_CMDS
-	$(MAKE) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define TVHEADEND_INSTALL_TARGET_CMDS
-	$(MAKE) -C $(@D) DESTDIR="$(TARGET_DIR)" install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR="$(TARGET_DIR)" install
 endef
 
 # Remove documentation and source files that are not needed because we
diff --git a/package/ushare/ushare.mk b/package/ushare/ushare.mk
index 25f880c..c9c6ae8 100644
--- a/package/ushare/ushare.mk
+++ b/package/ushare/ushare.mk
@@ -26,7 +26,8 @@ define USHARE_CONFIGURE_CMDS
 endef
 
 define USHARE_BUILD_CMDS
-	$(MAKE) LDFLAGS="$(TARGET_LDFLAGS) $(USHARE_LDFLAGS)" -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) \
+		LDFLAGS="$(TARGET_LDFLAGS) $(USHARE_LDFLAGS)" -C $(@D)
 endef
 
 define USHARE_INSTALL_TARGET_CMDS
diff --git a/package/vsftpd/vsftpd.mk b/package/vsftpd/vsftpd.mk
index 3091f06..db85015 100644
--- a/package/vsftpd/vsftpd.mk
+++ b/package/vsftpd/vsftpd.mk
@@ -39,7 +39,7 @@ VSFTPD_LIBS += -lpam
 endif
 
 define VSFTPD_BUILD_CMDS
-	$(MAKE) CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
+	$(TARGET_MAKE_ENV) $(MAKE) CC="$(TARGET_CC)" CFLAGS="$(TARGET_CFLAGS)" \
 		LDFLAGS="$(TARGET_LDFLAGS)" LIBS="$(VSFTPD_LIBS)" -C $(@D)
 endef
 
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 02/16] packages: for staging stuff use backtick instead of the shell function
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 01/16] packages: use TARGET_MAKE_ENV to add pps support Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-07-12 14:39   ` Arnout Vandecappelle
  2015-06-28 19:42 ` [Buildroot] [RFC v4 03/16] packages/scons: make avaiable the GCC_SYSROOT environment variable Fabio Porcedda
                   ` (18 subsequent siblings)
  20 siblings, 1 reply; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

When the shell function execute the command before the per-package
staging directory is created, use the backtick that is executed after
the per-package staging directory is created.

This commit prevents build failures when the per-package staging feature
is added.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/dbus-python/dbus-python.mk       |  8 ++++----
 package/python-gobject/python-gobject.mk |  4 ++--
 package/python-pygame/python-pygame.mk   |  6 +++---
 package/qt/qt.mk                         | 14 +++++++-------
 package/tcpreplay/tcpreplay.mk           |  6 +++---
 5 files changed, 19 insertions(+), 19 deletions(-)

diff --git a/package/dbus-python/dbus-python.mk b/package/dbus-python/dbus-python.mk
index 9bda018..4079967 100644
--- a/package/dbus-python/dbus-python.mk
+++ b/package/dbus-python/dbus-python.mk
@@ -17,15 +17,15 @@ DBUS_PYTHON_DEPENDENCIES += python host-python
 
 DBUS_PYTHON_CONF_ENV += \
 	PYTHON=$(HOST_DIR)/usr/bin/python2 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python2-config --includes)" \
-	PYTHON_LIBS="$(shell $(STAGING_DIR)/usr/bin/python2-config --ldflags)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python2-config --includes`" \
+	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python2-config --ldflags`"
 else
 DBUS_PYTHON_DEPENDENCIES += python3 host-python3
 
 DBUS_PYTHON_CONF_ENV += \
 	PYTHON=$(HOST_DIR)/usr/bin/python3 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python3-config --includes)" \
-	PYTHON_LIBS="$(shell $(STAGING_DIR)/usr/bin/python3-config --ldflags)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python3-config --includes`" \
+	PYTHON_LIBS="`$(STAGING_DIR)/usr/bin/python3-config --ldflags`"
 endif
 
 $(eval $(autotools-package))
diff --git a/package/python-gobject/python-gobject.mk b/package/python-gobject/python-gobject.mk
index c47954c..74cb054 100644
--- a/package/python-gobject/python-gobject.mk
+++ b/package/python-gobject/python-gobject.mk
@@ -20,13 +20,13 @@ PYTHON_GOBJECT_DEPENDENCIES += python host-python
 
 PYTHON_GOBJECT_CONF_ENV = \
 	PYTHON=$(HOST_DIR)/usr/bin/python2 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python2-config --includes)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python2-config --includes`"
 else
 PYTHON_GOBJECT_DEPENDENCIES += python3 host-python3
 
 PYTHON_GOBJECT_CONF_ENV = \
 	PYTHON=$(HOST_DIR)/usr/bin/python3 \
-	PYTHON_INCLUDES="$(shell $(STAGING_DIR)/usr/bin/python3-config --includes)"
+	PYTHON_INCLUDES="`$(STAGING_DIR)/usr/bin/python3-config --includes`"
 endif
 
 ifeq ($(BR2_PACKAGE_LIBFFI),y)
diff --git a/package/python-pygame/python-pygame.mk b/package/python-pygame/python-pygame.mk
index 51f1c20..995f80a 100644
--- a/package/python-pygame/python-pygame.mk
+++ b/package/python-pygame/python-pygame.mk
@@ -66,14 +66,14 @@ define PYTHON_PYGAME_UNCONFIGURE_SCRAP
 endef
 endif
 
-PYTHON_PYGAME_SDL_FLAGS = $(shell $(STAGING_DIR)/usr/bin/sdl-config --cflags)
-PYTHON_PYGAME_SDL_FLAGS += $(shell $(STAGING_DIR)/usr/bin/sdl-config --libs)
+PYTHON_PYGAME_SDL_FLAGS = `$(STAGING_DIR)/usr/bin/sdl-config --cflags`
+PYTHON_PYGAME_SDL_FLAGS += `$(STAGING_DIR)/usr/bin/sdl-config --libs`
 
 # Pygame needs a Setup file where options should be commented out if
 # dependencies are not available
 define PYTHON_PYGAME_CONFIGURE_CMDS
 	cp -f $(@D)/Setup.in $(@D)/Setup
-	$(SED) 's~^SDL = ~SDL = $(PYTHON_PYGAME_SDL_FLAGS) \n#~' $(@D)/Setup
+	$(SED) "s~^SDL = ~SDL = $(PYTHON_PYGAME_SDL_FLAGS) \n#~" $(@D)/Setup
 	$(SED) 's/^pypm/#pypm/' $(@D)/Setup
 	$(PYTHON_PYGAME_UNCONFIGURE_IMAGE)
 	$(PYTHON_PYGAME_UNCONFIGURE_FONT)
diff --git a/package/qt/qt.mk b/package/qt/qt.mk
index b2b9bbe..a4a55ec 100644
--- a/package/qt/qt.mk
+++ b/package/qt/qt.mk
@@ -230,9 +230,9 @@ QT_DEPENDENCIES += fontconfig xlib_libXi xlib_libX11 xlib_libXrender \
                 xlib_libXcursor xlib_libXrandr xlib_libXext xlib_libXv
 # Using pkg-config avoids us some logic to redefine and sed again mkspecs files
 # to add X11 include path and link options
-QT_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags x11)
-QT_CXXFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags x11)
-QT_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs x11 xext)
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs x11 xext`
 QT_CONFIGURE_OPTS += -arch $(QT_EMB_PLATFORM) \
 		-xplatform qws/linux-$(QT_EMB_PLATFORM)-g++ -x11 -no-gtkstyle -no-sm \
 		-no-openvg
@@ -329,9 +329,9 @@ endif
 ifeq ($(BR2_PACKAGE_QT_OPENGL_ES),y)
 QT_CONFIGURE_OPTS += -opengl es2 -egl
 QT_DEPENDENCIES += libgles libegl
-QT_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)
-QT_CXXFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags egl)
-QT_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs egl)
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs egl`
 else
 QT_CONFIGURE_OPTS += -no-opengl
 endif
@@ -473,7 +473,7 @@ QT_QMAKE = $(HOST_DIR)/usr/bin/qmake -spec qws/linux-$(QT_EMB_PLATFORM)-g++
 ################################################################################
 define QT_QMAKE_SET
 	$(SED) '/$(1)/d' $(3)/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++/qmake.conf
-	$(SED) '/include.*qws.conf/a$(1) = $(2)' $(3)/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++/qmake.conf
+	$(SED) "/include.*qws.conf/a$(1) = $(2)" $(3)/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++/qmake.conf
 endef
 
 ifneq ($(QT_CONFIG_FILE),)
diff --git a/package/tcpreplay/tcpreplay.mk b/package/tcpreplay/tcpreplay.mk
index 1dbe453..f6bd08f 100644
--- a/package/tcpreplay/tcpreplay.mk
+++ b/package/tcpreplay/tcpreplay.mk
@@ -16,9 +16,9 @@ TCPREPLAY_CONF_OPTS = --with-libpcap=$(STAGING_DIR)/usr
 TCPREPLAY_DEPENDENCIES = libpcap
 
 # libpcap may depend on symbols in other libs
-TCPREPLAY_LIBS = $(shell $(STAGING_DIR)/usr/bin/pcap-config --static --libs)
-TCPREPLAY_CONF_ENV += ac_cv_search_pcap_close='$(TCPREPLAY_LIBS)' \
-	LIBS="$(TCPREPLAY_LIBS)"
+TCPREPLAY_LIBS = $(STAGING_DIR)/usr/bin/pcap-config --static --libs
+TCPREPLAY_CONF_ENV += ac_cv_search_pcap_close="`$(TCPREPLAY_LIBS)`" \
+	LIBS="`$(TCPREPLAY_LIBS)`"
 
 ifeq ($(BR2_PACKAGE_TCPDUMP),y)
 TCPREPLAY_CONF_ENV += ac_cv_path_tcpdump_path=/usr/sbin/tcpdump
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 03/16] packages/scons: make avaiable the GCC_SYSROOT environment variable
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 01/16] packages: use TARGET_MAKE_ENV to add pps support Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 02/16] packages: for staging stuff use backtick instead of the shell function Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 04/16] packages/qt: read spec files from the per-package staging directory Fabio Porcedda
                   ` (17 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to support the per-package staging feature, make avaiable the
GCC_SYSROOT environment variable to scons packages so the toolchain
wrapper can use it.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/scons/scons.mk     |  3 ++-
 package/scons/site_init.py | 19 +++++++++++++++++++
 2 files changed, 21 insertions(+), 1 deletion(-)
 create mode 100644 package/scons/site_init.py

diff --git a/package/scons/scons.mk b/package/scons/scons.mk
index 39fb258..e8ae086 100644
--- a/package/scons/scons.mk
+++ b/package/scons/scons.mk
@@ -18,4 +18,5 @@ HOST_SCONS_INSTALL_OPTS = \
 $(eval $(host-python-package))
 
 # variables used by other packages
-SCONS = $(HOST_DIR)/usr/bin/python2 $(HOST_DIR)/usr/bin/scons $(if $(QUIET),-s)
+SCONS = $(HOST_DIR)/usr/bin/python2 $(HOST_DIR)/usr/bin/scons \
+	$(if $(QUIET),-s) --site-dir=$(TOPDIR)/package/scons
diff --git a/package/scons/site_init.py b/package/scons/site_init.py
new file mode 100644
index 0000000..f0d8b53
--- /dev/null
+++ b/package/scons/site_init.py
@@ -0,0 +1,19 @@
+"""
+Decorate SCons Environment constructor so that the "GCC_SYSROOT"
+variable is always included.
+"""
+
+from functools import wraps
+import SCons.Environment
+
+def add_gcc_sysroot(environment_init):
+    @wraps(environment_init)
+    def wrapper(self, *args, **kwargs):
+        environment_init(self, *args, **kwargs)
+
+        self.PrependENVPath('GCC_SYSROOT', os.environ['GCC_SYSROOT'])
+
+    return wrapper
+
+SCons.Environment.Environment.__init__ = \
+    add_gcc_sysroot(SCons.Environment.Environment.__init__)
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 04/16] packages/qt: read spec files from the per-package staging directory
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (2 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 03/16] packages/scons: make avaiable the GCC_SYSROOT environment variable Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 05/16] pkg-perl: use TARGET_MAKE_ENV to add pps support Fabio Porcedda
                   ` (16 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to support the per-package staging feature, read spec files
from the subdirectory "mkspecs" that is in the per-package staging
directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/python-pyqt/python-pyqt.mk | 1 +
 package/python-sip/python-sip.mk   | 2 +-
 package/qt/qt.mk                   | 2 +-
 3 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/package/python-pyqt/python-pyqt.mk b/package/python-pyqt/python-pyqt.mk
index fa298c4..0ee9352 100644
--- a/package/python-pyqt/python-pyqt.mk
+++ b/package/python-pyqt/python-pyqt.mk
@@ -73,6 +73,7 @@ PYTHON_PYQT_CONF_OPTS = \
 	--bindir $(TARGET_DIR)/usr/bin \
 	--destdir $(TARGET_DIR)/usr/lib/$(PYTHON_PYQT_PYTHON_DIR)/site-packages \
 	--qmake $(HOST_DIR)/usr/bin/qmake \
+	--spec $(STAGING_DIR)/usr/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++ \
 	--sysroot $(STAGING_DIR)/usr \
 	-w --confirm-license \
 	--no-designer-plugin \
diff --git a/package/python-sip/python-sip.mk b/package/python-sip/python-sip.mk
index 4651186..4ac6413 100644
--- a/package/python-sip/python-sip.mk
+++ b/package/python-sip/python-sip.mk
@@ -46,7 +46,7 @@ define PYTHON_SIP_CONFIGURE_CMDS
 			--sipdir $(TARGET_DIR)/usr/share/sip \
 			--sysroot $(STAGING_DIR)/usr \
 			--use-qmake && \
-		$(HOST_DIR)/usr/bin/qmake)
+		$(QT_QMAKE))
 endef
 
 define PYTHON_SIP_BUILD_CMDS
diff --git a/package/qt/qt.mk b/package/qt/qt.mk
index a4a55ec..0644fbe 100644
--- a/package/qt/qt.mk
+++ b/package/qt/qt.mk
@@ -457,7 +457,7 @@ endif
 # End of workaround.
 
 # Variable for other Qt applications to use
-QT_QMAKE = $(HOST_DIR)/usr/bin/qmake -spec qws/linux-$(QT_EMB_PLATFORM)-g++
+QT_QMAKE = $(HOST_DIR)/usr/bin/qmake -spec $(STAGING_DIR)/usr/mkspecs/qws/linux-$(QT_EMB_PLATFORM)-g++
 
 ################################################################################
 # QT_QMAKE_SET -- helper macro to set <variable> = <value> in
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 05/16] pkg-perl: use TARGET_MAKE_ENV to add pps support
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (3 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 04/16] packages/qt: read spec files from the per-package staging directory Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 06/16] tooclahin-external: add support for GCC_SYSROOT Fabio Porcedda
                   ` (15 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In the TARGET_MAKE_ENV variable will be added the GCC_SYSROOT variable
that is needed for per-package staging feature.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-perl.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-perl.mk b/package/pkg-perl.mk
index 6577588..d879771 100644
--- a/package/pkg-perl.mk
+++ b/package/pkg-perl.mk
@@ -130,7 +130,7 @@ define $(2)_BUILD_CMDS
 	cd $$($$(PKG)_SRCDIR) && if [ -f Build.PL ] ; then \
 		$$(PERL_RUN) Build $$($(2)_BUILD_OPTS) build; \
 	else \
-		$$(MAKE1) \
+		$$(TARGET_MAKE_ENV) $$(MAKE1) \
 			PERL_INC=$$(STAGING_DIR)/usr/lib/perl5/$$(PERL_VERSION)/$$(PERL_ARCHNAME)/CORE \
 			$$($(2)_BUILD_OPTS) pure_all; \
 	fi
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 06/16] tooclahin-external: add support for GCC_SYSROOT
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (4 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 05/16] pkg-perl: use TARGET_MAKE_ENV to add pps support Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 07/16] toolchain-external: move some code in a new function Fabio Porcedda
                   ` (14 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

To support the per-package staging feature, when the GCC_SYSROOT
environament variable is defined pass it to gcc in the "--sysroot"
option.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 toolchain/toolchain-external/ext-toolchain-wrapper.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/toolchain/toolchain-external/ext-toolchain-wrapper.c b/toolchain/toolchain-external/ext-toolchain-wrapper.c
index e2d01dc..f262b3d 100644
--- a/toolchain/toolchain-external/ext-toolchain-wrapper.c
+++ b/toolchain/toolchain-external/ext-toolchain-wrapper.c
@@ -98,6 +98,7 @@ int main(int argc, char **argv)
 	char *basename;
 	char *env_debug;
 	char *paranoid_wrapper;
+	char *gcc_sysroot;
 	int paranoid;
 	int ret, i, count = 0, debug;
 
@@ -145,7 +146,13 @@ int main(int argc, char **argv)
 		perror(__FILE__ ": overflow");
 		return 3;
 	}
-	ret = snprintf(sysroot, sizeof(sysroot), "%s/" BR_SYSROOT, absbasedir);
+	gcc_sysroot = getenv("GCC_SYSROOT");
+	if (gcc_sysroot) {
+		ret = snprintf(sysroot, sizeof(sysroot), "%s", gcc_sysroot);
+	} else {
+		ret = snprintf(sysroot, sizeof(sysroot), "%s/" BR_SYSROOT, \
+			       absbasedir);
+	}
 	if (ret >= sizeof(sysroot)) {
 		perror(__FILE__ ": overflow");
 		return 3;
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 07/16] toolchain-external: move some code in a new function
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (5 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 06/16] tooclahin-external: add support for GCC_SYSROOT Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 08/16] toolchain-external: use the wrapper even for the linker "ld" Fabio Porcedda
                   ` (13 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

To simplify the following commit, refactor the main() function moving
some code in a new add_gcc_args() function.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 .../toolchain-external/ext-toolchain-wrapper.c     | 72 ++++++++++++----------
 1 file changed, 40 insertions(+), 32 deletions(-)

diff --git a/toolchain/toolchain-external/ext-toolchain-wrapper.c b/toolchain/toolchain-external/ext-toolchain-wrapper.c
index f262b3d..f8695a4 100644
--- a/toolchain/toolchain-external/ext-toolchain-wrapper.c
+++ b/toolchain/toolchain-external/ext-toolchain-wrapper.c
@@ -71,6 +71,45 @@ static char *predef_args[] = {
 #endif
 };
 
+static char **add_gcc_args(int argc, char **argv, char **cur)
+{
+	int i;
+
+#ifdef BR_FLOAT_ABI
+	/* add float abi if not overridden in args */
+	for (i = 1; i < argc; i++) {
+		if (!strncmp(argv[i], "-mfloat-abi=", strlen("-mfloat-abi=")) ||
+		    !strcmp(argv[i], "-msoft-float") ||
+		    !strcmp(argv[i], "-mhard-float"))
+			break;
+	}
+
+	if (i == argc)
+		*cur++ = "-mfloat-abi=" BR_FLOAT_ABI;
+#endif
+
+#if defined(BR_ARCH) || \
+    defined(BR_CPU)
+	/* Add our -march/cpu/abi flags, but only if none are
+	 * already specified on the commandline
+	 */
+	for (i = 1; i < argc; i++) {
+		if (!strncmp(argv[i], "-march=", strlen("-march=")) ||
+		    !strncmp(argv[i], "-mcpu=",  strlen("-mcpu=" )))
+			break;
+	}
+	if (i == argc) {
+#ifdef BR_ARCH
+		*cur++ = "-march=" BR_ARCH;
+#endif
+#ifdef BR_CPU
+		*cur++ = "-mcpu=" BR_CPU;
+#endif
+	}
+#endif /* ARCH || CPU */
+	return cur;
+}
+
 static void check_unsafe_path(const char *path, int paranoid)
 {
 	char **c;
@@ -169,38 +208,7 @@ int main(int argc, char **argv)
 	memcpy(cur, predef_args, sizeof(predef_args));
 	cur += sizeof(predef_args) / sizeof(predef_args[0]);
 
-#ifdef BR_FLOAT_ABI
-	/* add float abi if not overridden in args */
-	for (i = 1; i < argc; i++) {
-		if (!strncmp(argv[i], "-mfloat-abi=", strlen("-mfloat-abi=")) ||
-		    !strcmp(argv[i], "-msoft-float") ||
-		    !strcmp(argv[i], "-mhard-float"))
-			break;
-	}
-
-	if (i == argc)
-		*cur++ = "-mfloat-abi=" BR_FLOAT_ABI;
-#endif
-
-#if defined(BR_ARCH) || \
-    defined(BR_CPU)
-	/* Add our -march/cpu/abi flags, but only if none are
-	 * already specified on the commandline
-	 */
-	for (i = 1; i < argc; i++) {
-		if (!strncmp(argv[i], "-march=", strlen("-march=")) ||
-		    !strncmp(argv[i], "-mcpu=",  strlen("-mcpu=" )))
-			break;
-	}
-	if (i == argc) {
-#ifdef BR_ARCH
-		*cur++ = "-march=" BR_ARCH;
-#endif
-#ifdef BR_CPU
-		*cur++ = "-mcpu=" BR_CPU;
-#endif
-	}
-#endif /* ARCH || CPU */
+	cur = add_gcc_args(argc, argv, cur);
 
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 08/16] toolchain-external: use the wrapper even for the linker "ld"
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (6 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 07/16] toolchain-external: move some code in a new function Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 09/16] infra: add GCC_SYSROOT environment variable to support pps feature Fabio Porcedda
                   ` (12 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to add the per-package staging feature we need to use the
wrappper even for the linker "ld".

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 .../toolchain-external/ext-toolchain-wrapper.c      | 21 +++++++++++++++------
 toolchain/toolchain-external/toolchain-external.mk  |  2 +-
 2 files changed, 16 insertions(+), 7 deletions(-)

diff --git a/toolchain/toolchain-external/ext-toolchain-wrapper.c b/toolchain/toolchain-external/ext-toolchain-wrapper.c
index f8695a4..c839377 100644
--- a/toolchain/toolchain-external/ext-toolchain-wrapper.c
+++ b/toolchain/toolchain-external/ext-toolchain-wrapper.c
@@ -39,9 +39,12 @@ static char sysroot[PATH_MAX];
  */
 #define EXCLUSIVE_ARGS	3
 
-static char *predef_args[] = {
+static char *predef_common_args[] = {
 	path,
-	"--sysroot", sysroot,
+	"--sysroot", sysroot
+};
+
+static char *predef_gcc_args[] = {
 #ifdef BR_ABI
 	"-mabi=" BR_ABI,
 #endif
@@ -197,7 +200,8 @@ int main(int argc, char **argv)
 		return 3;
 	}
 
-	cur = args = malloc(sizeof(predef_args) +
+	cur = args = malloc(sizeof(predef_common_args) +
+			    sizeof(predef_gcc_args) +
 			    (sizeof(char *) * (argc + EXCLUSIVE_ARGS)));
 	if (args == NULL) {
 		perror(__FILE__ ": malloc");
@@ -205,10 +209,15 @@ int main(int argc, char **argv)
 	}
 
 	/* start with predefined args */
-	memcpy(cur, predef_args, sizeof(predef_args));
-	cur += sizeof(predef_args) / sizeof(predef_args[0]);
+	memcpy(cur, predef_common_args, sizeof(predef_common_args));
+	cur += sizeof(predef_common_args) / sizeof(predef_common_args[0]);
 
-	cur = add_gcc_args(argc, argv, cur);
+	if (strcmp(path + strlen(path) - 2, "ld") !=  0) {
+		memcpy(cur, predef_gcc_args, sizeof(predef_gcc_args));
+		cur += sizeof(predef_gcc_args) / sizeof(predef_gcc_args[0]);
+
+		cur = add_gcc_args(argc, argv, cur);
+	}
 
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
index 22f8118..75d61a9 100644
--- a/toolchain/toolchain-external/toolchain-external.mk
+++ b/toolchain/toolchain-external/toolchain-external.mk
@@ -673,7 +673,7 @@ define TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER
 		*-ar|*-ranlib|*-nm) \
 			ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%../..%') .; \
 			;; \
-		*cc|*cc-*|*++|*++-*|*cpp) \
+		*cc|*cc-*|*++|*++-*|*cpp|*ld) \
 			ln -sf ext-toolchain-wrapper $$base; \
 			;; \
 		*gdb|*gdbtui) \
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 09/16] infra: add GCC_SYSROOT environment variable to support pps feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (7 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 08/16] toolchain-external: use the wrapper even for the linker "ld" Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 10/16] pkg-cmake: " Fabio Porcedda
                   ` (11 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to support the per-package staging feature, add the GCC_SYSROOT
environment variable to the TARGET_CONFIGURE_OPTS and TARGET_MAKE_ENV
variables.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/Makefile.in | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/Makefile.in b/package/Makefile.in
index 2ed7cf7..924c808 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -280,9 +280,10 @@ TARGET_CONFIGURE_OPTS = \
 	FCFLAGS="$(TARGET_FCFLAGS)" \
 	PKG_CONFIG="$(PKG_CONFIG_HOST_BINARY)" \
 	STAGING_DIR="$(STAGING_DIR)" \
+	GCC_SYSROOT="$(STAGING_DIR)" \
 	INTLTOOL_PERL=$(PERL)
 
-TARGET_MAKE_ENV = PATH=$(BR_PATH)
+TARGET_MAKE_ENV = PATH=$(BR_PATH) GCC_SYSROOT="$(STAGING_DIR)"
 
 
 HOST_CONFIGURE_OPTS = \
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 10/16] pkg-cmake: add GCC_SYSROOT environment variable to support pps feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (8 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 09/16] infra: add GCC_SYSROOT environment variable to support pps feature Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 11/16] pkg-python: add GCC_SYSROOT variable to add pps support Fabio Porcedda
                   ` (10 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to support the per-package staging feature, add the GCC_SYSROOT
environment variable and use it for building packages with cmake.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-cmake.mk                | 1 +
 support/misc/toolchainfile.cmake.in | 9 +++++++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
index f17a883..8af1e48 100644
--- a/package/pkg-cmake.mk
+++ b/package/pkg-cmake.mk
@@ -85,6 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
+	GCC_SYSROOT=$$(STAGING_DIR) \
 	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \
diff --git a/support/misc/toolchainfile.cmake.in b/support/misc/toolchainfile.cmake.in
index cd41254..fa82af4 100644
--- a/support/misc/toolchainfile.cmake.in
+++ b/support/misc/toolchainfile.cmake.in
@@ -19,12 +19,17 @@ set(CMAKE_EXE_LINKER_FLAGS "@@TARGET_LDFLAGS@@ ${CMAKE_EXE_LINKER_FLAGS}" CACHE
 set(CMAKE_INSTALL_SO_NO_EXE 0)
 
 set(CMAKE_PROGRAM_PATH "${RELOCATED_HOST_DIR}/usr/bin")
-set(CMAKE_FIND_ROOT_PATH "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+if(DEFINED ENV{GCC_SYSROOT})
+	set(CMAKE_FIND_ROOT_PATH "$ENV{GCC_SYSROOT}")
+	set(ENV{PKG_CONFIG_SYSROOT_DIR} "$ENV{GCC_SYSROOT}")
+else()
+	set(CMAKE_FIND_ROOT_PATH "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+	set(ENV{PKG_CONFIG_SYSROOT_DIR} "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
+endif()
 set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
 set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
 set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
 set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
-set(ENV{PKG_CONFIG_SYSROOT_DIR} "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@")
 
 # This toolchain file can be used both inside and outside Buildroot.
 # * When used inside Buildroot, ccache support is explicitly driven using the
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 11/16] pkg-python: add GCC_SYSROOT variable to add pps support
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (9 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 10/16] pkg-cmake: " Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 12/16] pacakge/luarocks: add GCC_SYSROOT environment variable to support pps Fabio Porcedda
                   ` (9 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to support the per-package staging feature, add the GCC_SYSROOT
environment variable to the python packages.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-python.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 426301a..126846e 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -27,6 +27,7 @@ PKG_PYTHON_DISTUTILS_ENV = \
 	CFLAGS="$(TARGET_CFLAGS)" \
 	LDFLAGS="$(TARGET_LDFLAGS)" \
 	LDSHARED="$(TARGET_CROSS)gcc -shared" \
+	GCC_SYSROOT="$(STAGING_DIR)" \
 	PYTHONPATH="$(if $(BR2_PACKAGE_PYTHON3),$(PYTHON3_PATH),$(PYTHON_PATH))" \
 	_python_sysroot=$(STAGING_DIR) \
 	_python_prefix=/usr \
@@ -51,6 +52,7 @@ HOST_PKG_PYTHON_DISTUTILS_INSTALL_OPTS = \
 # Target setuptools-based packages
 PKG_PYTHON_SETUPTOOLS_ENV = \
 	PATH=$(BR_PATH) \
+	GCC_SYSROOT="$(STAGING_DIR)" \
 	PYTHONPATH="$(if $(BR2_PACKAGE_PYTHON3),$(PYTHON3_PATH),$(PYTHON_PATH))" \
 	_python_sysroot=$(STAGING_DIR) \
 	_python_prefix=/usr \
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 12/16] pacakge/luarocks: add GCC_SYSROOT environment variable to support pps
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (10 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 11/16] pkg-python: add GCC_SYSROOT variable to add pps support Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 13/16] package/pkgconf: use GCC_SYSROOT to support pps feature Fabio Porcedda
                   ` (8 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to support the per-package staging feature, add the GCC_SYSROOT
environment variable and use it for building luarocks pacakges.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 ...ctions-to-the-global-environment-of-the-c.patch | 54 +++++++++++++++
 package/luarocks/0002-fixed-typo.patch             | 27 ++++++++
 ...e-environment-table-for-the-system-config.patch | 76 ++++++++++++++++++++++
 ...is-not-being-integrated-bug-introduced-in.patch | 62 ++++++++++++++++++
 package/luarocks/luarocks.mk                       | 11 ++--
 5 files changed, 226 insertions(+), 4 deletions(-)
 create mode 100644 package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
 create mode 100644 package/luarocks/0002-fixed-typo.patch
 create mode 100644 package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
 create mode 100644 package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch

diff --git a/package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch b/package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
new file mode 100644
index 0000000..f83c139
--- /dev/null
+++ b/package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
@@ -0,0 +1,54 @@
+From b3ea8783a4d73d437a27befc1d8ab47ef69d4ba1 Mon Sep 17 00:00:00 2001
+From: Thijs Schreijer <thijs@thijsschreijer.nl>
+Date: Tue, 16 Jun 2015 19:31:01 +0200
+Subject: [PATCH 1/4] added 2 functions to the global environment of the
+ config-file loader sandbox; `os_getenv()` and `__dump_env()`
+
+(Upstream patch taken from master branch)
+(cherry picked from commit e1dd6375822d800984e1d3d821899a315bc1b222)
+---
+ src/luarocks/cfg.lua | 17 ++++++++++++++++-
+ 1 file changed, 16 insertions(+), 1 deletion(-)
+
+diff --git a/src/luarocks/cfg.lua b/src/luarocks/cfg.lua
+index 99b4077..cf0c0d3 100644
+--- a/src/luarocks/cfg.lua
++++ b/src/luarocks/cfg.lua
+@@ -145,6 +145,14 @@ end
+ cfg.variables = {}
+ cfg.rocks_trees = {}
+ 
++-- some extras for the global enviornment in the config files;
++cfg.os_getenv = os.getenv
++cfg.__dump_env = function()
++  -- debug function, calling it from a config file will show all 
++  -- available globals to that config file
++  print(util.show_table(cfg, "global environment"))
++end
++
+ sys_config_file = site_config.LUAROCKS_SYSCONFIG or sys_config_dir.."/config-"..cfg.lua_version..".lua"
+ local err, errcode
+ sys_config_ok, err, errcode = persist.load_into_table(sys_config_file, cfg)
+@@ -158,11 +166,18 @@ if (not sys_config_ok) and errcode ~= "open" then -- either "load" or "run"; bad
+    os.exit(cfg.errorcodes.CONFIGFILE)
+ end
+ 
+-local env_for_config_file = {
++local env_for_config_file
++env_for_config_file = {
+    home = cfg.home,
+    lua_version = cfg.lua_version,
+    platform = util.make_shallow_copy(detected),
+    processor = proc,
++   os_getenv = os.getenv, 
++   __dump_env = function()
++     -- debug function, calling it from a config file will show all 
++     -- available globals to that config file
++     print(util.show_table(env_for_config_file, "global environment"))
++   end,
+ }
+ 
+ if not site_config.LUAROCKS_FORCE_CONFIG then
+-- 
+2.4.3
+
diff --git a/package/luarocks/0002-fixed-typo.patch b/package/luarocks/0002-fixed-typo.patch
new file mode 100644
index 0000000..f4defef
--- /dev/null
+++ b/package/luarocks/0002-fixed-typo.patch
@@ -0,0 +1,27 @@
+From 53c9599371b3e0815d62d3d34b1f1aef2b2877db Mon Sep 17 00:00:00 2001
+From: Thijs Schreijer <thijs@thijsschreijer.nl>
+Date: Wed, 17 Jun 2015 11:01:48 +0200
+Subject: [PATCH 2/4] fixed typo
+
+(Upstream patch taken from master branch)
+(cherry picked from commit 4792618d8882eb36c9e9d05efac8b1ae336f00cb)
+---
+ src/luarocks/cfg.lua | 2 +-
+ 1 file changed, 1 insertion(+), 1 deletion(-)
+
+diff --git a/src/luarocks/cfg.lua b/src/luarocks/cfg.lua
+index cf0c0d3..278be4a 100644
+--- a/src/luarocks/cfg.lua
++++ b/src/luarocks/cfg.lua
+@@ -145,7 +145,7 @@ end
+ cfg.variables = {}
+ cfg.rocks_trees = {}
+ 
+--- some extras for the global enviornment in the config files;
++-- some extras for the global environment in the config files;
+ cfg.os_getenv = os.getenv
+ cfg.__dump_env = function()
+   -- debug function, calling it from a config file will show all 
+-- 
+2.4.3
+
diff --git a/package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch b/package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
new file mode 100644
index 0000000..05e6a5e
--- /dev/null
+++ b/package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
@@ -0,0 +1,76 @@
+From a51259a3308aef9ce61f611144324b998ad67710 Mon Sep 17 00:00:00 2001
+From: Thijs Schreijer <thijs@thijsschreijer.nl>
+Date: Thu, 18 Jun 2015 18:01:19 +0200
+Subject: [PATCH 3/4] fix: use same environment table for the system config
+ file as well.
+
+(Upstream patch taken from master branch)
+(cherry picked from commit 998d012a7228099990a88454150e843559074e0c)
+---
+ src/luarocks/cfg.lua | 37 +++++++++++++++----------------------
+ 1 file changed, 15 insertions(+), 22 deletions(-)
+
+diff --git a/src/luarocks/cfg.lua b/src/luarocks/cfg.lua
+index 278be4a..5dcd59c 100644
+--- a/src/luarocks/cfg.lua
++++ b/src/luarocks/cfg.lua
+@@ -145,27 +145,7 @@ end
+ cfg.variables = {}
+ cfg.rocks_trees = {}
+ 
+--- some extras for the global environment in the config files;
+-cfg.os_getenv = os.getenv
+-cfg.__dump_env = function()
+-  -- debug function, calling it from a config file will show all 
+-  -- available globals to that config file
+-  print(util.show_table(cfg, "global environment"))
+-end
+-
+-sys_config_file = site_config.LUAROCKS_SYSCONFIG or sys_config_dir.."/config-"..cfg.lua_version..".lua"
+-local err, errcode
+-sys_config_ok, err, errcode = persist.load_into_table(sys_config_file, cfg)
+-
+-if (not sys_config_ok) and errcode == "open" then -- file not found, so try alternate file
+-   sys_config_file = sys_config_dir.."/config.lua"
+-   sys_config_ok, err, errcode = persist.load_into_table(sys_config_file, cfg)
+-end
+-if (not sys_config_ok) and errcode ~= "open" then -- either "load" or "run"; bad config file, bail out with error
+-   io.stderr:write(err.."\n")
+-   os.exit(cfg.errorcodes.CONFIGFILE)
+-end
+-
++-- The global environment in the config files;
+ local env_for_config_file
+ env_for_config_file = {
+    home = cfg.home,
+@@ -173,13 +153,26 @@ env_for_config_file = {
+    platform = util.make_shallow_copy(detected),
+    processor = proc,
+    os_getenv = os.getenv, 
+-   __dump_env = function()
++   dump_env = function()
+      -- debug function, calling it from a config file will show all 
+      -- available globals to that config file
+      print(util.show_table(env_for_config_file, "global environment"))
+    end,
+ }
+ 
++sys_config_file = site_config.LUAROCKS_SYSCONFIG or sys_config_dir.."/config-"..cfg.lua_version..".lua"
++local err, errcode
++   sys_config_ok, err, errcode = persist.load_into_table(sys_config_file, env_for_config_file)
++
++if (not sys_config_ok) and errcode == "open" then -- file not found, so try alternate file
++   sys_config_file = sys_config_dir.."/config.lua"
++      sys_config_ok, err, errcode = persist.load_into_table(sys_config_file, env_for_config_file)
++end
++if (not sys_config_ok) and errcode ~= "open" then -- either "load" or "run"; bad config file, bail out with error
++   io.stderr:write(err.."\n")
++   os.exit(cfg.errorcodes.CONFIGFILE)
++end
++
+ if not site_config.LUAROCKS_FORCE_CONFIG then
+ 
+    local home_overrides, err, errcode
+-- 
+2.4.3
+
diff --git a/package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch b/package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch
new file mode 100644
index 0000000..af702eb
--- /dev/null
+++ b/package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch
@@ -0,0 +1,62 @@
+From 3cd403fd0979328f9c6f711a3983c64c250ce7df Mon Sep 17 00:00:00 2001
+From: Thijs Schreijer <thijs@thijsschreijer.nl>
+Date: Thu, 25 Jun 2015 00:14:50 +0200
+Subject: [PATCH 4/4] config.lua is not being integrated, bug introduced in
+ #385
+
+(Upstream patch taken from master branch)
+(cherry picked from commit f6974b40735b9300f7b529da6e602459d0db9c49)
+---
+ src/luarocks/cfg.lua | 19 ++++++++++++-------
+ 1 file changed, 12 insertions(+), 7 deletions(-)
+ mode change 100644 => 100755 src/luarocks/cfg.lua
+
+diff --git a/src/luarocks/cfg.lua b/src/luarocks/cfg.lua
+old mode 100644
+new mode 100755
+index 5dcd59c..3cf0565
+--- a/src/luarocks/cfg.lua
++++ b/src/luarocks/cfg.lua
+@@ -159,6 +159,16 @@ env_for_config_file = {
+      print(util.show_table(env_for_config_file, "global environment"))
+    end,
+ }
++-- Merge values from config files read into the `cfg` table
++local merge_overrides = function(overrides)
++   if overrides.rocks_trees then
++      cfg.rocks_trees = nil
++   end
++   if overrides.rocks_servers then
++      cfg.rocks_servers = nil
++   end
++   util.deep_merge(cfg, overrides)
++end
+ 
+ sys_config_file = site_config.LUAROCKS_SYSCONFIG or sys_config_dir.."/config-"..cfg.lua_version..".lua"
+ local err, errcode
+@@ -172,6 +182,7 @@ if (not sys_config_ok) and errcode ~= "open" then -- either "load" or "run"; bad
+    io.stderr:write(err.."\n")
+    os.exit(cfg.errorcodes.CONFIGFILE)
+ end
++   merge_overrides(sys_config_ok)
+ 
+ if not site_config.LUAROCKS_FORCE_CONFIG then
+ 
+@@ -189,13 +200,7 @@ if not site_config.LUAROCKS_FORCE_CONFIG then
+    end
+    if home_overrides then
+       home_config_ok = true
+-      if home_overrides.rocks_trees then
+-         cfg.rocks_trees = nil
+-      end
+-      if home_overrides.rocks_servers then
+-         cfg.rocks_servers = nil
+-      end
+-      util.deep_merge(cfg, home_overrides)
++      merge_overrides(home_overrides)
+    else
+       home_config_ok = home_overrides
+       if errcode ~= "open" then
+-- 
+2.4.4
+
diff --git a/package/luarocks/luarocks.mk b/package/luarocks/luarocks.mk
index 413e23d..042019f 100644
--- a/package/luarocks/luarocks.mk
+++ b/package/luarocks/luarocks.mk
@@ -35,16 +35,17 @@ define HOST_LUAROCKS_INSTALL_CMDS
 	rm -f $(LUAROCKS_CONFIG_FILE)
 	$(MAKE1) -C $(@D) install \
 		PREFIX=$(HOST_DIR)/usr
+	echo "gcc_sysroot = os_getenv([[GCC_SYSROOT]])"         >> $(LUAROCKS_CONFIG_FILE)
 	echo "-- BR cross-compilation"                          >> $(LUAROCKS_CONFIG_FILE)
 	echo "variables = {"                                    >> $(LUAROCKS_CONFIG_FILE)
-	echo "   LUA_INCDIR = [[$(STAGING_DIR)/usr/include]],"  >> $(LUAROCKS_CONFIG_FILE)
-	echo "   LUA_LIBDIR = [[$(STAGING_DIR)/usr/lib]],"      >> $(LUAROCKS_CONFIG_FILE)
+	echo "   LUA_INCDIR = gcc_sysroot..[[/usr/include]],"   >> $(LUAROCKS_CONFIG_FILE)
+	echo "   LUA_LIBDIR = gcc_sysroot..[[/usr/lib]],"       >> $(LUAROCKS_CONFIG_FILE)
 	echo "   CC = [[$(TARGET_CC)]],"                        >> $(LUAROCKS_CONFIG_FILE)
 	echo "   LD = [[$(TARGET_CC)]],"                        >> $(LUAROCKS_CONFIG_FILE)
 	echo "   CFLAGS = [[$(LUAROCKS_CFLAGS)]],"              >> $(LUAROCKS_CONFIG_FILE)
 	echo "   LIBFLAG = [[-shared $(TARGET_LDFLAGS)]],"      >> $(LUAROCKS_CONFIG_FILE)
 	echo "}"                                                >> $(LUAROCKS_CONFIG_FILE)
-	echo "external_deps_dirs = { [[$(STAGING_DIR)/usr]] }"  >> $(LUAROCKS_CONFIG_FILE)
+	echo 'external_deps_dirs = { gcc_sysroot..[[/usr]] }'   >> $(LUAROCKS_CONFIG_FILE)
 	echo "gcc_rpath = false"                                >> $(LUAROCKS_CONFIG_FILE)
 	echo "rocks_trees = { [[$(TARGET_DIR)/usr]] }"          >> $(LUAROCKS_CONFIG_FILE)
 	echo "wrap_bin_scripts = false"                         >> $(LUAROCKS_CONFIG_FILE)
@@ -53,7 +54,9 @@ endef
 
 $(eval $(host-generic-package))
 
-LUAROCKS_RUN_ENV = LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua"
+LUAROCKS_RUN_ENV = \
+	GCC_SYSROOT="$(STAGING_DIR)" \
+	LUA_PATH="$(HOST_DIR)/usr/share/lua/$(LUAINTERPRETER_ABIVER)/?.lua"
 LUAROCKS_RUN_CMD = $(LUA_RUN) $(HOST_DIR)/usr/bin/luarocks
 
 LUAROCKS_RUN = $(LUAROCKS_RUN_ENV) $(LUAROCKS_RUN_CMD)
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 13/16] package/pkgconf: use GCC_SYSROOT to support pps feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (11 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 12/16] pacakge/luarocks: add GCC_SYSROOT environment variable to support pps Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 14/16] Makefile: add STAGINGNOPKG_DIR variable Fabio Porcedda
                   ` (7 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

In order to support the per-package staging feature, use the GCC_SYSROOT
environment variable to find *.pc files in the per-package staging
directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/aiccu/aiccu.mk                 |  2 +-
 package/fbv/fbv.mk                     |  2 +-
 package/gdk-pixbuf/gdk-pixbuf.mk       |  2 +-
 package/libmicrohttpd/libmicrohttpd.mk |  2 +-
 package/libupnpp/libupnpp.mk           |  2 +-
 package/lmbench/lmbench.mk             |  4 ++--
 package/logrotate/logrotate.mk         |  2 +-
 package/madplay/madplay.mk             |  2 +-
 package/mpg123/mpg123.mk               |  2 +-
 package/neardal/neardal.mk             |  2 +-
 package/nmap/nmap.mk                   |  2 +-
 package/openldap/openldap.mk           |  2 +-
 package/php/php.mk                     |  4 ++--
 package/pkgconf/pkg-config.in          |  4 +++-
 package/pkgconf/pkgconf.mk             |  5 ++---
 package/poppler/poppler.mk             |  2 +-
 package/qt/qt.mk                       | 12 ++++++------
 package/ti-utils/ti-utils.mk           |  4 ++--
 package/upmpdcli/upmpdcli.mk           |  2 +-
 package/vsftpd/vsftpd.mk               |  2 +-
 20 files changed, 31 insertions(+), 30 deletions(-)

diff --git a/package/aiccu/aiccu.mk b/package/aiccu/aiccu.mk
index f2cd3de..b906b63 100644
--- a/package/aiccu/aiccu.mk
+++ b/package/aiccu/aiccu.mk
@@ -16,7 +16,7 @@ AICCU_LFDLAGS = $(TARGET_LDFLAGS)
 # aiccu forgets to link with gnutls' dependencies breaking the build when
 # linking statically
 ifeq ($(BR2_STATIC_LIBS),y)
-AICCU_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --static --libs gnutls)
+AICCU_LDFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --static --libs gnutls)
 endif
 
 # dummy RPM_OPT_FLAGS to disable stripping
diff --git a/package/fbv/fbv.mk b/package/fbv/fbv.mk
index 8788b0e..993125d 100644
--- a/package/fbv/fbv.mk
+++ b/package/fbv/fbv.mk
@@ -18,7 +18,7 @@ FBV_DEPENDENCIES += libpng
 
 # libpng in turn depends on other libraries
 ifeq ($(BR2_STATIC_LIBS),y)
-FBV_CONFIGURE_OPTS += "--libs=$(shell $(PKG_CONFIG_HOST_BINARY) --libs libpng --static)"
+FBV_CONFIGURE_OPTS += "--libs=$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libpng --static)"
 endif
 
 else
diff --git a/package/gdk-pixbuf/gdk-pixbuf.mk b/package/gdk-pixbuf/gdk-pixbuf.mk
index eb30215..c408170 100644
--- a/package/gdk-pixbuf/gdk-pixbuf.mk
+++ b/package/gdk-pixbuf/gdk-pixbuf.mk
@@ -35,7 +35,7 @@ GDK_PIXBUF_CONF_OPTS += --without-libtiff
 else
 GDK_PIXBUF_DEPENDENCIES += tiff host-pkgconf
 GDK_PIXBUF_CONF_ENV += \
-	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs libtiff-4)'
+	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libtiff-4)'
 endif
 
 ifeq ($(BR2_PACKAGE_XLIB_LIBX11),y)
diff --git a/package/libmicrohttpd/libmicrohttpd.mk b/package/libmicrohttpd/libmicrohttpd.mk
index a6662a7..ebaf248 100644
--- a/package/libmicrohttpd/libmicrohttpd.mk
+++ b/package/libmicrohttpd/libmicrohttpd.mk
@@ -13,7 +13,7 @@ LIBMICROHTTPD_CONF_OPTS = --disable-curl --disable-spdy --disable-examples
 ifeq ($(BR2_PACKAGE_LIBMICROHTTPD_SSL),y)
 LIBMICROHTTPD_LICENSE = LGPLv2.1+
 LIBMICROHTTPD_DEPENDENCIES += host-pkgconf gnutls libgcrypt
-LIBMICROHTTPD_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs gnutls)"
+LIBMICROHTTPD_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs gnutls)"
 LIBMICROHTTPD_CONF_OPTS += --enable-https --with-gnutls=$(STAGING_DIR)/usr \
 	--with-libgcrypt-prefix=$(STAGING_DIR)/usr
 else
diff --git a/package/libupnpp/libupnpp.mk b/package/libupnpp/libupnpp.mk
index defc16a..3960c2b 100644
--- a/package/libupnpp/libupnpp.mk
+++ b/package/libupnpp/libupnpp.mk
@@ -16,7 +16,7 @@ LIBUPNPP_DEPENDENCIES = expat libcurl libupnp
 ifeq ($(BR2_STATIC_LIBS),y)
 LIBUPNPP_DEPENDENCIES += host-pkgconf
 LIBUPNPP_CONF_ENV += \
-	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs libupnp libcurl)'
+	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libupnp libcurl)'
 endif
 
 $(eval $(autotools-package))
diff --git a/package/lmbench/lmbench.mk b/package/lmbench/lmbench.mk
index d4be8d4..92ea257 100644
--- a/package/lmbench/lmbench.mk
+++ b/package/lmbench/lmbench.mk
@@ -14,8 +14,8 @@ LMBENCH_CFLAGS = $(TARGET_CFLAGS)
 
 ifeq ($(BR2_PACKAGE_LIBTIRPC),y)
 LMBENCH_DEPENDENCIES += host-pkgconf libtirpc
-LMBENCH_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY) --cflags libtirpc)
-LMBENCH_LDLIBS = $(shell $(PKG_CONFIG_HOST_BINARY) --libs libtirpc)
+LMBENCH_CFLAGS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags libtirpc)
+LMBENCH_LDLIBS = $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libtirpc)
 endif
 
 define LMBENCH_CONFIGURE_CMDS
diff --git a/package/logrotate/logrotate.mk b/package/logrotate/logrotate.mk
index 0af7f4a..7cfb7a1 100644
--- a/package/logrotate/logrotate.mk
+++ b/package/logrotate/logrotate.mk
@@ -11,7 +11,7 @@ LOGROTATE_LICENSE_FILES = COPYING
 LOGROTATE_DEPENDENCIES = popt host-pkgconf
 # tarball does not have a generated configure script
 LOGROTATE_AUTORECONF = YES
-LOGROTATE_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs popt)"
+LOGROTATE_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs popt)"
 LOGROTATE_CONF_OPTS = --without-selinux
 
 ifeq ($(BR2_PACKAGE_ACL),y)
diff --git a/package/madplay/madplay.mk b/package/madplay/madplay.mk
index ff12aa1..c39d0e8 100644
--- a/package/madplay/madplay.mk
+++ b/package/madplay/madplay.mk
@@ -16,7 +16,7 @@ MADPLAY_DEPENDENCIES = libmad libid3tag $(if $(BR2_PACKAGE_GETTEXT),gettext)
 ifeq ($(BR2_PACKAGE_MADPLAY_ALSA),y)
 MADPLAY_CONF_OPTS += --with-alsa
 MADPLAY_DEPENDENCIES += host-pkgconf alsa-lib
-MADPLAY_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs alsa)"
+MADPLAY_CONF_ENV += LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs alsa)"
 endif
 
 $(eval $(autotools-package))
diff --git a/package/mpg123/mpg123.mk b/package/mpg123/mpg123.mk
index a16e400..f02ba9c 100644
--- a/package/mpg123/mpg123.mk
+++ b/package/mpg123/mpg123.mk
@@ -54,7 +54,7 @@ MPG123_CONF_OPTS += --with-default-audio=portaudio
 MPG123_DEPENDENCIES += portaudio
 # configure script does NOT use pkg-config to figure out how to link
 # with portaudio, breaking static linking as portaudio uses pthreads
-MPG123_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs portaudio-2.0)'
+MPG123_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs portaudio-2.0)'
 endif
 
 ifeq ($(BR2_PACKAGE_SDL),y)
diff --git a/package/neardal/neardal.mk b/package/neardal/neardal.mk
index ddaecbe..1a7ec7f 100644
--- a/package/neardal/neardal.mk
+++ b/package/neardal/neardal.mk
@@ -25,7 +25,7 @@ endif
 # forgets to take that into account, causing the detection to fail
 # when linking statically
 ifeq ($(BR2_STATIC_LIBS),y)
-NEARDAL_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs ncurses)'
+NEARDAL_CONF_ENV += LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs ncurses)'
 endif
 
 define NEARDAL_INSTALL_NCL
diff --git a/package/nmap/nmap.mk b/package/nmap/nmap.mk
index d46d58b..35c6a82 100644
--- a/package/nmap/nmap.mk
+++ b/package/nmap/nmap.mk
@@ -30,7 +30,7 @@ NMAP_POST_PATCH_HOOKS += NMAP_WRAPPER_EXEC
 ifeq ($(BR2_PACKAGE_OPENSSL),y)
 NMAP_CONF_OPTS += --with-openssl="$(STAGING_DIR)/usr"
 NMAP_DEPENDENCIES += host-pkgconf openssl
-NMAP_LIBS_FOR_STATIC_LINK += $(shell $(PKG_CONFIG_HOST_BINARY) --libs --static openssl)
+NMAP_LIBS_FOR_STATIC_LINK += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs --static openssl)
 else
 NMAP_CONF_OPTS += --without-openssl
 endif
diff --git a/package/openldap/openldap.mk b/package/openldap/openldap.mk
index 68464ab..93facb4 100644
--- a/package/openldap/openldap.mk
+++ b/package/openldap/openldap.mk
@@ -31,7 +31,7 @@ endif
 ifeq ($(BR2_PACKAGE_OPENSSL),y)
 OPENLDAP_MP = bignum
 OPENLDAP_DEPENDENCIES += openssl
-OPENLDAP_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs libssl libcrypto)"
+OPENLDAP_CONF_ENV = LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libssl libcrypto)"
 else ifeq ($(BR2_PACKAGE_GMP),y)
 OPENLDAP_MP = gmp
 OPENLDAP_DEPENDENCIES += gmp
diff --git a/package/php/php.mk b/package/php/php.mk
index c9bcfe3..f7c064c 100644
--- a/package/php/php.mk
+++ b/package/php/php.mk
@@ -116,7 +116,7 @@ PHP_CONF_OPTS += --with-openssl=$(STAGING_DIR)/usr
 PHP_DEPENDENCIES += openssl
 # openssl needs zlib, but the configure script forgets to link against
 # it causing detection failures with static linking
-PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs --static openssl)
+PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs --static openssl)
 endif
 
 ifeq ($(BR2_PACKAGE_PHP_EXT_LIBXML2),y)
@@ -187,7 +187,7 @@ endif
 ifeq ($(BR2_PACKAGE_PHP_EXT_SQLITE),y)
 PHP_CONF_OPTS += --with-sqlite3=$(STAGING_DIR)/usr
 PHP_DEPENDENCIES += sqlite
-PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs --static sqlite3)
+PHP_STATIC_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs --static sqlite3)
 endif
 
 ### PDO
diff --git a/package/pkgconf/pkg-config.in b/package/pkgconf/pkg-config.in
index 4dec487..9e5408f 100644
--- a/package/pkgconf/pkg-config.in
+++ b/package/pkgconf/pkg-config.in
@@ -1,2 +1,4 @@
 #!/bin/sh
-PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:- at PKG_CONFIG_LIBDIR@} PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:- at STAGING_DIR@} $(dirname $0)/pkgconf @STATIC@ $@
+PKG_CONFIG_LIBDIR=${PKG_CONFIG_LIBDIR:-$GCC_SYSROOT/usr/lib/pkgconfig:$GCC_SYSROOT/usr/share/pkgconfig} \
+    PKG_CONFIG_SYSROOT_DIR=${PKG_CONFIG_SYSROOT_DIR:-$GCC_SYSROOT} \
+    $(dirname $0)/pkgconf @STATIC@ $@
diff --git a/package/pkgconf/pkgconf.mk b/package/pkgconf/pkgconf.mk
index 3c93fd5..0c496be 100644
--- a/package/pkgconf/pkgconf.mk
+++ b/package/pkgconf/pkgconf.mk
@@ -11,6 +11,8 @@ PKGCONF_LICENSE = pkgconf license
 PKGCONF_LICENSE_FILES = COPYING
 
 PKG_CONFIG_HOST_BINARY = $(HOST_DIR)/usr/bin/pkg-config
+PKG_CONFIG_HOST_BINARY_GCC_SYSROOT = \
+	GCC_SYSROOT=$(STAGING_DIR) $(HOST_DIR)/usr/bin/pkg-config
 
 define PKGCONF_LINK_PKGCONFIG
 	ln -sf pkgconf $(TARGET_DIR)/usr/bin/pkg-config
@@ -19,9 +21,6 @@ endef
 define HOST_PKGCONF_INSTALL_WRAPPER
 	$(INSTALL) -m 0755 -D package/pkgconf/pkg-config.in \
 		$(HOST_DIR)/usr/bin/pkg-config
-	$(SED) 's, at PKG_CONFIG_LIBDIR@,$(STAGING_DIR)/usr/lib/pkgconfig:$(STAGING_DIR)/usr/share/pkgconfig,' \
-		-e 's, at STAGING_DIR@,$(STAGING_DIR),' \
-		$(HOST_DIR)/usr/bin/pkg-config
 endef
 
 define HOST_PKGCONF_STATIC
diff --git a/package/poppler/poppler.mk b/package/poppler/poppler.mk
index 234b7cc..c5ff161 100644
--- a/package/poppler/poppler.mk
+++ b/package/poppler/poppler.mk
@@ -24,7 +24,7 @@ ifeq ($(BR2_PACKAGE_TIFF),y)
 POPPLER_CONF_OPTS += --enable-libtiff
 # Help poppler to find libtiff in static linking scenarios
 POPPLER_CONF_ENV += \
-	LIBTIFF_LIBS="$(shell $(PKG_CONFIG_HOST_BINARY) --libs libtiff-4)"
+	LIBTIFF_LIBS="$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libtiff-4)"
 POPPLER_DEPENDENCIES += tiff
 else
 POPPLER_CONF_OPTS += --disable-libtiff
diff --git a/package/qt/qt.mk b/package/qt/qt.mk
index 0644fbe..3a9eda6 100644
--- a/package/qt/qt.mk
+++ b/package/qt/qt.mk
@@ -230,9 +230,9 @@ QT_DEPENDENCIES += fontconfig xlib_libXi xlib_libX11 xlib_libXrender \
                 xlib_libXcursor xlib_libXrandr xlib_libXext xlib_libXv
 # Using pkg-config avoids us some logic to redefine and sed again mkspecs files
 # to add X11 include path and link options
-QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
-QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags x11`
-QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs x11 xext`
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags x11`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags x11`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs x11 xext`
 QT_CONFIGURE_OPTS += -arch $(QT_EMB_PLATFORM) \
 		-xplatform qws/linux-$(QT_EMB_PLATFORM)-g++ -x11 -no-gtkstyle -no-sm \
 		-no-openvg
@@ -329,9 +329,9 @@ endif
 ifeq ($(BR2_PACKAGE_QT_OPENGL_ES),y)
 QT_CONFIGURE_OPTS += -opengl es2 -egl
 QT_DEPENDENCIES += libgles libegl
-QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
-QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY) --cflags egl`
-QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY) --libs egl`
+QT_CFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags egl`
+QT_CXXFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags egl`
+QT_LDFLAGS += `$(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs egl`
 else
 QT_CONFIGURE_OPTS += -no-opengl
 endif
diff --git a/package/ti-utils/ti-utils.mk b/package/ti-utils/ti-utils.mk
index 35ecc19..19d53fd 100644
--- a/package/ti-utils/ti-utils.mk
+++ b/package/ti-utils/ti-utils.mk
@@ -10,8 +10,8 @@ TI_UTILS_DEPENDENCIES = libnl host-pkgconf
 TI_UTILS_LICENSE = BSD-3c
 TI_UTILS_LICENSE_FILES = COPYING
 
-TI_UTILS_CFLAGS = $(shell $(PKG_CONFIG_HOST_BINARY) --cflags libnl-genl-3.0)
-TI_UTILS_LIBS = $(shell $(PKG_CONFIG_HOST_BINARY) --libs libnl-genl-3.0)
+TI_UTILS_CFLAGS = $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --cflags libnl-genl-3.0)
+TI_UTILS_LIBS = $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libnl-genl-3.0)
 
 define TI_UTILS_BUILD_CMDS
 	$(TARGET_MAKE_ENV) $(MAKE1) NFSROOT="$(STAGING_DIR)" \
diff --git a/package/upmpdcli/upmpdcli.mk b/package/upmpdcli/upmpdcli.mk
index a5c6370..1587a85 100644
--- a/package/upmpdcli/upmpdcli.mk
+++ b/package/upmpdcli/upmpdcli.mk
@@ -19,7 +19,7 @@ ifeq ($(BR2_STATIC_LIBS),y)
 # manually pass it to make.
 UPMPDCLI_DEPENDENCIES += host-pkgconf
 UPMPDCLI_MAKE_OPTS = \
-	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY) --libs expat libcurl libupnp)'
+	LIBS='$(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs expat libcurl libupnp)'
 endif
 
 # Upmpdcli only runs if user upmpdcli exists
diff --git a/package/vsftpd/vsftpd.mk b/package/vsftpd/vsftpd.mk
index db85015..b6517d4 100644
--- a/package/vsftpd/vsftpd.mk
+++ b/package/vsftpd/vsftpd.mk
@@ -24,7 +24,7 @@ endif
 
 ifeq ($(BR2_PACKAGE_OPENSSL),y)
 VSFTPD_DEPENDENCIES += openssl host-pkgconf
-VSFTPD_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY) --libs libssl libcrypto)
+VSFTPD_LIBS += $(shell $(PKG_CONFIG_HOST_BINARY_GCC_SYSROOT) --libs libssl libcrypto)
 VSFTPD_POST_CONFIGURE_HOOKS += VSFTPD_ENABLE_SSL
 endif
 
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 14/16] Makefile: add STAGINGNOPKG_DIR variable
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (12 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 13/16] package/pkgconf: use GCC_SYSROOT to support pps feature Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 15/16] pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages Fabio Porcedda
                   ` (6 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

When the per-package staging feature will be added, the STAGING_DIR
variable will be used to refer to the per-package staging directory so
to be able to refer to the original STAGING_DIR a STAGINGNOPKG_DIR is
added.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile            | 14 +++++---------
 package/Makefile.in |  1 +
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/Makefile b/Makefile
index a0d4abb..b4e5a10 100644
--- a/Makefile
+++ b/Makefile
@@ -437,15 +437,11 @@ else
 LIB_SYMLINK = lib32
 endif
 
-$(STAGING_DIR):
-	@mkdir -p $(STAGING_DIR)/bin
-	@mkdir -p $(STAGING_DIR)/lib
-	@ln -snf lib $(STAGING_DIR)/$(LIB_SYMLINK)
-	@mkdir -p $(STAGING_DIR)/usr/lib
-	@ln -snf lib $(STAGING_DIR)/usr/$(LIB_SYMLINK)
-	@mkdir -p $(STAGING_DIR)/usr/include
-	@mkdir -p $(STAGING_DIR)/usr/bin
-	@ln -snf $(STAGING_DIR) $(BASE_DIR)/staging
+$(STAGINGNOPKG_DIR):
+	@mkdir -p $@/bin $@/lib $@/usr/lib $@/usr/include $@/usr/bin
+	@ln -snf lib $@/$(LIB_SYMLINK)
+	@ln -snf lib $@/usr/$(LIB_SYMLINK)
+	@ln -snf $@ $(BASE_DIR)/staging
 
 ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 TARGET_SKELETON = $(BR2_ROOTFS_SKELETON_CUSTOM_PATH)
diff --git a/package/Makefile.in b/package/Makefile.in
index 924c808..169c6b3 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -109,6 +109,7 @@ endif
 
 STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
 STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
+STAGINGNOPKG_DIR := $(STAGING_DIR)
 
 TARGET_OPTIMIZATION := $(call qstrip,$(BR2_TARGET_OPTIMIZATION))
 
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 15/16] pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (13 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 14/16] Makefile: add STAGINGNOPKG_DIR variable Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-06-28 19:42 ` [Buildroot] [RFC v4 16/16] infra: add per-package staging feature Fabio Porcedda
                   ` (5 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

This will simplify the commit that add the per-package staging
directory.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-generic.mk | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 3da5a87..376b025 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -394,17 +394,20 @@ $(2)_REDISTRIBUTE		?= YES
 # When a target package is a toolchain dependency set this variable to
 # 'NO' so the 'toolchain' dependency is not added to prevent a circular
 # dependency
-$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+ifeq ($(4),target)
+ $(2)_ADD_TOOLCHAIN_DEPENDENCY ?= YES
+else
+ $(2)_ADD_TOOLCHAIN_DEPENDENCY ?= NO
+endif
 
 ifeq ($(4),host)
 $(2)_DEPENDENCIES ?= $$(filter-out  host-toolchain $(1),\
 	$$(patsubst host-host-%,host-%,$$(addprefix host-,$$($(3)_DEPENDENCIES))))
 endif
-ifeq ($(4),target)
+
 ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
 $(2)_DEPENDENCIES += toolchain
 endif
-endif
 
 # Eliminate duplicates in dependencies
 $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 16/16] infra: add per-package staging feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (14 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 15/16] pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages Fabio Porcedda
@ 2015-06-28 19:42 ` Fabio Porcedda
  2015-07-11 22:56   ` Romain Naour
  2015-06-28 19:53 ` [Buildroot] [RFC v4 00/16] Add " Fabio Porcedda
                   ` (4 subsequent siblings)
  20 siblings, 1 reply; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:42 UTC (permalink / raw)
  To: buildroot

To improve the build reproducibility, instead of sharing a common
staging are, every package has a private staging directory where only
the declared dependency are present.

Having a per-package staging directory ensure that every package use
only the declared dependency, because even if a undeclared dependency is
build before, it will be not included in the private package directory.

This feature is useful for both top-level non-parallel make and
top-level parallel make.

This feature is useful for target dependencies but not for host
dependencies, to solve that a similar feature for host packages must be
developed.

Implementation details:
To minimize change the STAGING_DIR variable will be redefined for each
package to point to the per-package staging directory.

At the beginning of each configuration stage the per-package staging
directory will be created copying using hard links all the declared
target dependencies.
Also each path of every copied configuration files will be changed to
point to the per-package

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile               |  2 +-
 package/Makefile.in    |  1 +
 package/pkg-cmake.mk   |  2 +-
 package/pkg-generic.mk | 34 +++++++++++++++++++++++++++++++++-
 4 files changed, 36 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index b4e5a10..01a7198 100644
--- a/Makefile
+++ b/Makefile
@@ -800,7 +800,7 @@ printvars:
 
 clean:
 	rm -rf $(TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) \
-		$(BUILD_DIR) $(BASE_DIR)/staging \
+		$(BUILD_DIR) $(BASE_DIR)/staging $(STAGINGPKG_DIR) \
 		$(LEGAL_INFO_DIR) $(GRAPHS_DIR)
 
 distclean: clean
diff --git a/package/Makefile.in b/package/Makefile.in
index 169c6b3..ea26772 100644
--- a/package/Makefile.in
+++ b/package/Makefile.in
@@ -110,6 +110,7 @@ endif
 STAGING_SUBDIR = usr/$(GNU_TARGET_NAME)/sysroot
 STAGING_DIR    = $(HOST_DIR)/$(STAGING_SUBDIR)
 STAGINGNOPKG_DIR := $(STAGING_DIR)
+STAGINGPKG_DIR = $(BASE_DIR)/stagingpkg
 
 TARGET_OPTIMIZATION := $(call qstrip,$(BR2_TARGET_OPTIMIZATION))
 
diff --git a/package/pkg-cmake.mk b/package/pkg-cmake.mk
index 8af1e48..eb80466 100644
--- a/package/pkg-cmake.mk
+++ b/package/pkg-cmake.mk
@@ -85,7 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	GCC_SYSROOT=$$(STAGING_DIR) \
+	GCC_SYSROOT=$$($(2)_STAGING_DIR) \
 	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 376b025..edac936 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -221,7 +221,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 
 # Remove package sources
 $(BUILD_DIR)/%/.stamp_dircleaned:
-	rm -Rf $(@D)
+	rm -Rf $(@D) $(STAGING_DIR)
 
 ################################################################################
 # virt-provides-single -- check that provider-pkg is the declared provider for
@@ -414,6 +414,33 @@ $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
 $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
 $(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))
 
+ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
+  $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
+	$$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
+	$$($$(call UPPERCASE,$$(dep))_STAGING_DIR)/))
+
+  $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
+
+  define $(2)_PREPARE_STAGING_DIR
+	mkdir -p $$($(2)_STAGING_DIR)/usr/mkspecs/qws
+	cp -rl $(STAGING_DIR)/* $$($(2)_STAGING_DIR)
+	$$(if $$($(2)_STAGING_DIRS), $$(foreach dir,$$($(2)_STAGING_DIRS),\
+		rsync -au --link-dest=$$(dir) $$(dir) $$($(2)_STAGING_DIR); ))
+	find $$($(2)_STAGING_DIR)/usr/{bin,lib,mkspecs/qws} \
+		-ignore_readdir_race \
+		-name "*[-_]config" -or -name "*.la" -or -name "*.pc" -or \
+		-name "*.prl" -or -name "qmake.conf" | xargs -r sed -i -r \
+		-e "s|$$(STAGINGPKG_DIR)/[^/]*/+usr|$$($(2)_STAGING_DIR)/usr|g" \
+		-e "s|$$(BUILD_DIR)/[^ ]+/([^/ ]+).la|$$($(2)_STAGING_DIR)/usr/lib/\1.la|g" \
+		-e "s|/lib/libpulsecommon|/lib/pulseaudio/libpulsecommon|g"
+  endef
+
+  $(2)_PRE_CONFIGURE_HOOKS := \
+	$(2)_PREPARE_STAGING_DIR $$($(2)_PRE_CONFIGURE_HOOKS)
+else
+  $(2)_STAGING_DIR = $$(STAGING_DIR)
+endif
+
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
 $(2)_INSTALL_TARGET		?= YES
@@ -617,11 +644,15 @@ $(1)-reconfigure:	$(1)-clean-for-reconfigure $(1)
 # define the PKG variable for all targets, containing the
 # uppercase package variable prefix
 $$($(2)_TARGET_INSTALL_TARGET):		PKG=$(2)
+$$($(2)_TARGET_INSTALL_TARGET):		STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_INSTALL_STAGING):	PKG=$(2)
+$$($(2)_TARGET_INSTALL_STAGING):	STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_INSTALL_IMAGES):		PKG=$(2)
 $$($(2)_TARGET_INSTALL_HOST):           PKG=$(2)
 $$($(2)_TARGET_BUILD):			PKG=$(2)
+$$($(2)_TARGET_BUILD):			STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_CONFIGURE):		PKG=$(2)
+$$($(2)_TARGET_CONFIGURE):		STAGING_DIR:=$$($(2)_STAGING_DIR)
 $$($(2)_TARGET_RSYNC):                  SRCDIR=$$($(2)_OVERRIDE_SRCDIR)
 $$($(2)_TARGET_RSYNC):                  PKG=$(2)
 $$($(2)_TARGET_PATCH):			PKG=$(2)
@@ -631,6 +662,7 @@ $$($(2)_TARGET_EXTRACT):		PKG=$(2)
 $$($(2)_TARGET_SOURCE):			PKG=$(2)
 $$($(2)_TARGET_SOURCE):			PKGDIR=$(pkgdir)
 $$($(2)_TARGET_DIRCLEAN):		PKG=$(2)
+$$($(2)_TARGET_DIRCLEAN):		STAGING_DIR:=$$($(2)_STAGING_DIR)
 
 # Compute the name of the Kconfig option that correspond to the
 # package being enabled. We handle three cases: the special Linux
-- 
2.4.3

^ permalink raw reply related	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (15 preceding siblings ...)
  2015-06-28 19:42 ` [Buildroot] [RFC v4 16/16] infra: add per-package staging feature Fabio Porcedda
@ 2015-06-28 19:53 ` Fabio Porcedda
  2015-06-28 20:34 ` Thomas Petazzoni
                   ` (3 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-28 19:53 UTC (permalink / raw)
  To: buildroot

On Sun, Jun 28, 2015 at 9:42 PM, Fabio Porcedda
<fabio.porcedda@gmail.com> wrote:
> Hi all,
> this patch set aims to improve build reproducibility by using a
> per-package staging directory.
> Also this feature aims to enable the top-level parallel make.
>
> This patch set is available at:
> https://github.com/fabio-porcedda/buildroot/tree/pps-v4
>
>
> Every package instead of installing the files in the shared staging
> directory (STAGING_DIR) install all files in a subdirectory of the
> STAGINGPKG_DIR.
>
> At the beginning of the configure stage, all the dependencies staging
> directory are copied in the package staging directory, e.g.:
>
> zlib-install-staging: install files in /output/stagingpkg/zlib/
>
> cramsfs-configure: copy all files from /output/stagingpkg/zlib/* to
>   /output/stagingpkg/cramfs/
>
> The STAGING_DIR variable is redefined for every package in order to
> point to the per-package directory, so a package cannot access
> automatically to staging files of packages that aren't in the
> (PKG)_DENDENDENCIES variable.
>
> To improve performance and reduce space utilization hard links are used.
>
> To evaluate performance and space utilization I've done some tests.
>
> HW-MED:
> CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
> RAM: 16GB
> SSD: 512GB
>
> HW-HIGH:
> CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
> RAM: 32GB
> SATA: RAID0 2x2TB 3.5" 7200rpm
>
>
> defconfig-small, long chain dependency, unfavorable  scenario for parallelization:
>
> BR2_arm=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_PACKAGE_XORG7=y
> BR2_PACKAGE_XSERVER_XORG_SERVER=y
> BR2_PACKAGE_LIBGTK3=y
>
> defconfig-full:
>   BR2_arm=y
>   BR2_TOOLCHAIN_EXTERNAL=y
> make allyespackageconfig
> Also i removed few packages that do not built on my system.
> make show-targets | wc -w
> 1366
>
> Test about output using hard links or note using the "defconfig-full"
> on HW-MED:
>
> |  35GB |  99m | master branch
> |  37GB | 100m | patch set
> | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> | 225GB | 106m | patch set not using hardlinks at all
>
> Test about performances of this patch set vs master branch:
> | 199m | HW-MED | defconfig-full | master branch | no top-level make |
> |  99m | HW-MED | defconfig-full | master branch | top-level make    |
> | 100m | HW-MED | defconfig-full | patch set     | top-level make    |
>
> | 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
> |  73m | HW-HIGH | defconfig-full | master branch | top-level make    |
> |  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |
>
> | 10m | HW-MED | defconfig-small | master branch | no top-level make |
> |  5m | HW-MED | defconfig-small | master branch | top-level make    |
> |  5m | HW-MED | defconfig-samll | patch set     | top-level make    |
>
> | 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
> |  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    |
> |  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |
>
>
> TODO:
>  - use the toolchaing wrapper even for the internal toolchain
>  - rebase over master (this patch set is based on 2015.05-161-gca5f11e).
>  - more testing (tested only for arm target).


v4:
 - Rewritten the path set to use the gcc --sysroot option trough the
toolchain wrapper.

> Best regads
>
> Fabio Porcedda (16):
>   packages: use TARGET_MAKE_ENV to add pps support
>   packages: for staging stuff use backtick instead of the shell function
>   packages/scons: make avaiable the GCC_SYSROOT environment variable
>   packages/qt: read spec files from the per-package staging directory
>   pkg-perl: use TARGET_MAKE_ENV to add pps support
>   tooclahin-external: add support for GCC_SYSROOT
>   toolchain-external: move some code in a new function
>   toolchain-external: use the wrapper even for the linker "ld"
>   infra: add GCC_SYSROOT environment variable to support pps feature
>   pkg-cmake: add GCC_SYSROOT environment variable to support pps feature
>   pkg-python: add GCC_SYSROOT variable to add pps support
>   pacakge/luarocks: add GCC_SYSROOT environment variable to support pps
>   package/pkgconf: use GCC_SYSROOT to support pps feature
>   Makefile: add STAGINGNOPKG_DIR variable
>   pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages
>   infra: add per-package staging feature
>
>  Makefile                                           | 16 ++--
>  package/Makefile.in                                |  5 +-
>  package/aiccu/aiccu.mk                             |  2 +-
>  package/blktrace/blktrace.mk                       |  2 +-
>  package/boost/boost.mk                             |  2 +-
>  package/botan/botan.mk                             |  2 +-
>  package/bsdiff/bsdiff.mk                           |  4 +-
>  package/dbus-python/dbus-python.mk                 |  8 +-
>  package/dhcpdump/dhcpdump.mk                       |  3 +-
>  package/dtc/dtc.mk                                 |  3 +-
>  package/exim/exim.mk                               |  5 +-
>  package/fbv/fbv.mk                                 |  2 +-
>  package/feh/feh.mk                                 |  4 +-
>  package/gadgetfs-test/gadgetfs-test.mk             |  2 +-
>  package/gdk-pixbuf/gdk-pixbuf.mk                   |  2 +-
>  package/iodine/iodine.mk                           |  2 +-
>  package/iw/iw.mk                                   |  2 +-
>  package/jack2/jack2.mk                             |  3 +-
>  package/libmicrohttpd/libmicrohttpd.mk             |  2 +-
>  package/libnss/libnss.mk                           |  4 +-
>  package/libupnpp/libupnpp.mk                       |  2 +-
>  package/lmbench/lmbench.mk                         |  8 +-
>  package/logrotate/logrotate.mk                     |  2 +-
>  package/lua-msgpack-native/lua-msgpack-native.mk   |  3 +-
>  ...ctions-to-the-global-environment-of-the-c.patch | 54 ++++++++++++
>  package/luarocks/0002-fixed-typo.patch             | 27 ++++++
>  ...e-environment-table-for-the-system-config.patch | 76 +++++++++++++++++
>  ...is-not-being-integrated-bug-introduced-in.patch | 62 ++++++++++++++
>  package/luarocks/luarocks.mk                       | 11 ++-
>  package/madplay/madplay.mk                         |  2 +-
>  package/moarvm/moarvm.mk                           |  2 +-
>  package/mongoose/mongoose.mk                       |  8 +-
>  package/mpg123/mpg123.mk                           |  2 +-
>  package/mplayer/mplayer.mk                         |  2 +-
>  package/ncurses/ncurses.mk                         |  4 +-
>  package/neardal/neardal.mk                         |  2 +-
>  package/nginx/nginx.mk                             |  1 +
>  package/nmap/nmap.mk                               |  2 +-
>  package/open2300/open2300.mk                       |  2 +-
>  package/openldap/openldap.mk                       |  2 +-
>  package/openssl/openssl.mk                         |  2 +-
>  package/opentyrian/opentyrian.mk                   |  2 +-
>  package/pax-utils/pax-utils.mk                     |  3 +-
>  package/php/php.mk                                 |  4 +-
>  package/pifmrds/pifmrds.mk                         |  3 +-
>  package/pkg-cmake.mk                               |  1 +
>  package/pkg-generic.mk                             | 43 +++++++++-
>  package/pkg-perl.mk                                |  2 +-
>  package/pkg-python.mk                              |  2 +
>  package/pkgconf/pkg-config.in                      |  4 +-
>  package/pkgconf/pkgconf.mk                         |  5 +-
>  package/poco/poco.mk                               |  3 +-
>  package/poppler/poppler.mk                         |  2 +-
>  package/pppd/pppd.mk                               |  2 +-
>  package/python-gobject/python-gobject.mk           |  4 +-
>  package/python-pygame/python-pygame.mk             |  6 +-
>  package/python-pyqt/python-pyqt.mk                 |  1 +
>  package/python-sip/python-sip.mk                   |  2 +-
>  package/qt/qt.mk                                   | 18 ++--
>  package/qtuio/qtuio.mk                             |  3 +-
>  package/rtmpdump/rtmpdump.mk                       |  2 +-
>  package/scons/scons.mk                             |  3 +-
>  package/scons/site_init.py                         | 19 +++++
>  package/speex/speex.mk                             |  3 +-
>  package/statserial/statserial.mk                   |  3 +-
>  package/tcpreplay/tcpreplay.mk                     |  6 +-
>  package/ti-utils/ti-utils.mk                       |  6 +-
>  package/tvheadend/tvheadend.mk                     |  4 +-
>  package/upmpdcli/upmpdcli.mk                       |  2 +-
>  package/ushare/ushare.mk                           |  3 +-
>  package/vsftpd/vsftpd.mk                           |  4 +-
>  support/misc/toolchainfile.cmake.in                |  9 +-
>  .../toolchain-external/ext-toolchain-wrapper.c     | 96 ++++++++++++++--------
>  toolchain/toolchain-external/toolchain-external.mk |  2 +-
>  74 files changed, 476 insertions(+), 147 deletions(-)
>  create mode 100644 package/luarocks/0001-added-2-functions-to-the-global-environment-of-the-c.patch
>  create mode 100644 package/luarocks/0002-fixed-typo.patch
>  create mode 100644 package/luarocks/0003-fix-use-same-environment-table-for-the-system-config.patch
>  create mode 100644 package/luarocks/0004-config.lua-is-not-being-integrated-bug-introduced-in.patch
>  create mode 100644 package/scons/site_init.py
>
> --
> 2.4.3
>

Best regards
-- 
Fabio Porcedda

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (16 preceding siblings ...)
  2015-06-28 19:53 ` [Buildroot] [RFC v4 00/16] Add " Fabio Porcedda
@ 2015-06-28 20:34 ` Thomas Petazzoni
  2015-06-28 20:46 ` Yann E. MORIN
                   ` (2 subsequent siblings)
  20 siblings, 0 replies; 37+ messages in thread
From: Thomas Petazzoni @ 2015-06-28 20:34 UTC (permalink / raw)
  To: buildroot

Dear Fabio Porcedda,

I haven't looked at the implementation yet, so I'll just comment on the
results.

On Sun, 28 Jun 2015 21:42:34 +0200, Fabio Porcedda wrote:

> To improve performance and reduce space utilization hard links are used.
> 
> To evaluate performance and space utilization I've done some tests.
> 
> HW-MED:
> CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
> RAM: 16GB
> SSD: 512GB
> 
> HW-HIGH:
> CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
> RAM: 32GB
> SATA: RAID0 2x2TB 3.5" 7200rpm
> 
> 
> defconfig-small, long chain dependency, unfavorable  scenario for parallelization:
> 
> BR2_arm=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_PACKAGE_XORG7=y
> BR2_PACKAGE_XSERVER_XORG_SERVER=y
> BR2_PACKAGE_LIBGTK3=y
> 
> defconfig-full:
>   BR2_arm=y
>   BR2_TOOLCHAIN_EXTERNAL=y
> make allyespackageconfig
> Also i removed few packages that do not built on my system.
> make show-targets | wc -w
> 1366
> 
> Test about output using hard links or note using the "defconfig-full"
> on HW-MED:
> 
> |  35GB |  99m | master branch
> |  37GB | 100m | patch set             
> | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> | 225GB | 106m | patch set not using hardlinks at all

So clearly hardlinks are great, as they limit a lot the increase of
disk usage. Going from 35 GB to 37 GB for a full defconfig is quite
reasonable IMO, especially considering the additional safety wrt
optional dependencies that per-package staging gives (even without
considering the top-level parallel build feature).

Also, I guess you are keeping all the temporary per-package staging
directories, right? We could probably save even more disk-space by
getting rid of the per-package staging areas after each package is
built (but that would be this per-package staging area would have to be
re-created every time you do "make <pkg>-rebuild", which isn't nice).

> Test about performances of this patch set vs master branch:
> | 199m | HW-MED | defconfig-full | master branch | no top-level make |
> |  99m | HW-MED | defconfig-full | master branch | top-level make    | 
> | 100m | HW-MED | defconfig-full | patch set     | top-level make    |
> 
> | 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
> |  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
> |  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |

So your high-end hardware in fact needs more time than the medium-end
hardware to build with top-level parallel, probably because the I/Os
come from HDDs instead of SSD. However, with top-level parallel build,
you high-end hardware performs better.

> | 10m | HW-MED | defconfig-small | master branch | no top-level make |
> |  5m | HW-MED | defconfig-small | master branch | top-level make    | 
> |  5m | HW-MED | defconfig-samll | patch set     | top-level make    |
> 
> | 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
> |  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
> |  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

So all in all, top-level parallel build makes the build from twice
faster to three times faster (or even more for defconfig-full on
HW-HIGH). And the time impact of the per-package staging feature is
minimal.

So to me, this is very encouraging, and clearly shows that it is worth
diving into your implementation and see how we can progressively merge
that.

Can I ask a last test to be done: what happens if you apply your patch
set, but do not use top-level parallel make ? Just to see if there
would be a significant impact to people not enabling top-level parallel
make.

Thanks a lot!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (17 preceding siblings ...)
  2015-06-28 20:34 ` Thomas Petazzoni
@ 2015-06-28 20:46 ` Yann E. MORIN
  2015-06-28 20:49   ` Yann E. MORIN
  2015-06-28 21:00   ` Thomas Petazzoni
  2015-07-05  8:51 ` Fabio Porcedda
  2015-10-04 17:21 ` Arnout Vandecappelle
  20 siblings, 2 replies; 37+ messages in thread
From: Yann E. MORIN @ 2015-06-28 20:46 UTC (permalink / raw)
  To: buildroot

Fabio, All,

On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:
> this patch set aims to improve build reproducibility by using a
> per-package staging directory.
> Also this feature aims to enable the top-level parallel make.

I haven't yet had time to look in the different patches, but I have a
preliminary comment about your benchmarks:

[--SNIP--]
> HW-MED:
> CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
> RAM: 16GB
> SSD: 512GB
> 
> HW-HIGH:
> CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
> RAM: 32GB
> SATA: RAID0 2x2TB 3.5" 7200rpm
> 1366
> 
> Test about output using hard links or note using the "defconfig-full"
> on HW-MED:
> 
> |  35GB |  99m | master branch
> |  37GB | 100m | patch set             
> | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> | 225GB | 106m | patch set not using hardlinks at all
> 
> Test about performances of this patch set vs master branch:
> | 199m | HW-MED | defconfig-full | master branch | no top-level make |
> |  99m | HW-MED | defconfig-full | master branch | top-level make    | 
> | 100m | HW-MED | defconfig-full | patch set     | top-level make    |
> 
> | 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
> |  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
> |  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |
> 
> | 10m | HW-MED | defconfig-small | master branch | no top-level make |
> |  5m | HW-MED | defconfig-small | master branch | top-level make    | 
> |  5m | HW-MED | defconfig-samll | patch set     | top-level make    |
> 
> | 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
> |  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
> |  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

OK, so those benchmarks show that:

  - HDD are terribly slow when compared to SSDs

  - the build-time sppedup of parallel builds is huge, up to 4.5 on
    HW-HIGH, but mostly around 2 when HDDs do not play in the equation
    (i.e. on SSDs, or with all cached in RAM (small config)).

  - the build-time overhead is low, 5% in the worst case (73min -> 77min)

  - the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without
    hardlinks

  - we're missing the benchmarks for this patchset without top-level
    parallel make (especially for the size overhead). Unless it no
    longer makes sense?

So, I'm really a bit skeptical. About five time the size for only about
twice the speedup, is it worth it? Sure some people will easily favour
speed over anything else, still the size overhead is really huge.

But I still think the idea is worth investigating.

Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which
might be better as a HW_HIGH system. I can spin a test-build with this
patchset on this machine to see what we get...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
  2015-06-28 20:46 ` Yann E. MORIN
@ 2015-06-28 20:49   ` Yann E. MORIN
  2015-06-28 20:57     ` Thomas Petazzoni
  2015-06-28 21:00   ` Thomas Petazzoni
  1 sibling, 1 reply; 37+ messages in thread
From: Yann E. MORIN @ 2015-06-28 20:49 UTC (permalink / raw)
  To: buildroot

Fabio, All,

On 2015-06-28 22:46 +0200, Yann E. MORIN spake thusly:
> On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:
[--SNIP--]
> > |  35GB |  99m | master branch
> > |  37GB | 100m | patch set             
> > | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> > | 225GB | 106m | patch set not using hardlinks at all
[--SNIP--]
>   - the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without
>     hardlinks

Oh wait, I read the wrong line...

OK, disregard my comments about the size overhead. It's only 2GiB, i.e.
roughly 5%.

Sorry for the confusion...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
  2015-06-28 20:49   ` Yann E. MORIN
@ 2015-06-28 20:57     ` Thomas Petazzoni
  2015-06-28 21:04       ` Yann E. MORIN
  0 siblings, 1 reply; 37+ messages in thread
From: Thomas Petazzoni @ 2015-06-28 20:57 UTC (permalink / raw)
  To: buildroot

Hello,

On Sun, 28 Jun 2015 22:49:12 +0200, Yann E. MORIN wrote:

> On 2015-06-28 22:46 +0200, Yann E. MORIN spake thusly:
> > On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:
> [--SNIP--]
> > > |  35GB |  99m | master branch
> > > |  37GB | 100m | patch set             
> > > | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> > > | 225GB | 106m | patch set not using hardlinks at all
> [--SNIP--]
> >   - the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without
> >     hardlinks
> 
> Oh wait, I read the wrong line...
> 
> OK, disregard my comments about the size overhead. It's only 2GiB, i.e.
> roughly 5%.

Yes, the size overhead is IMO very reasonable. So do you reconsider the
general opinion you gave in your previous e-mail?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
  2015-06-28 20:46 ` Yann E. MORIN
  2015-06-28 20:49   ` Yann E. MORIN
@ 2015-06-28 21:00   ` Thomas Petazzoni
  2015-06-28 21:25     ` Yann E. MORIN
  2015-06-29  8:38     ` Fabio Porcedda
  1 sibling, 2 replies; 37+ messages in thread
From: Thomas Petazzoni @ 2015-06-28 21:00 UTC (permalink / raw)
  To: buildroot

Dear Yann E. MORIN,

On Sun, 28 Jun 2015 22:46:22 +0200, Yann E. MORIN wrote:

> > |  35GB |  99m | master branch
> > |  37GB | 100m | patch set             
> > | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> > | 225GB | 106m | patch set not using hardlinks at all
> > 
> > Test about performances of this patch set vs master branch:
> > | 199m | HW-MED | defconfig-full | master branch | no top-level make |
> > |  99m | HW-MED | defconfig-full | master branch | top-level make    | 
> > | 100m | HW-MED | defconfig-full | patch set     | top-level make    |
> > 
> > | 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
> > |  73m | HW-HIGH | defconfig-full | master branch | top-level make    | 
> > |  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |
> > 
> > | 10m | HW-MED | defconfig-small | master branch | no top-level make |
> > |  5m | HW-MED | defconfig-small | master branch | top-level make    | 
> > |  5m | HW-MED | defconfig-samll | patch set     | top-level make    |
> > 
> > | 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
> > |  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    | 
> > |  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |
> 
> OK, so those benchmarks show that:
> 
>   - HDD are terribly slow when compared to SSDs

Yes, and the top-level parallel make gives the highest benefit on HDDs:
from 350m to 77m and from 21m to 8m on the HW-HIGH configuration.

>   - the build-time overhead is low, 5% in the worst case (73min -> 77min)

Yes.

>   - the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without
>     hardlinks

No, as you replied, the size overhead is very reasonable. You misread
Fabio's table here.

>   - we're missing the benchmarks for this patchset without top-level
>     parallel make (especially for the size overhead). Unless it no
>     longer makes sense?

Correct, I asked the same thing.

> So, I'm really a bit skeptical. About five time the size for only about
> twice the speedup, is it worth it? Sure some people will easily favour
> speed over anything else, still the size overhead is really huge.

Do you reconsider this comment now that you know the reality about the
size impact ? :-)

> Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which
> might be better as a HW_HIGH system. I can spin a test-build with this
> patchset on this machine to see what we get...

Would definitely be useful. Maybe Fabio can provide you his
defconfig-full test case (he gave the defconfig-small case).

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
  2015-06-28 20:57     ` Thomas Petazzoni
@ 2015-06-28 21:04       ` Yann E. MORIN
  0 siblings, 0 replies; 37+ messages in thread
From: Yann E. MORIN @ 2015-06-28 21:04 UTC (permalink / raw)
  To: buildroot

Thomas, Fabio, All,

On 2015-06-28 22:57 +0200, Thomas Petazzoni spake thusly:
> On Sun, 28 Jun 2015 22:49:12 +0200, Yann E. MORIN wrote:
> 
> > On 2015-06-28 22:46 +0200, Yann E. MORIN spake thusly:
> > > On 2015-06-28 21:42 +0200, Fabio Porcedda spake thusly:
> > [--SNIP--]
> > > > |  35GB |  99m | master branch
> > > > |  37GB | 100m | patch set             
> > > > | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> > > > | 225GB | 106m | patch set not using hardlinks at all
> > [--SNIP--]
> > >   - the size overhead is huge, a factor 4.4 with hardlinks, 6.5 without
> > >     hardlinks
> > 
> > Oh wait, I read the wrong line...
> > 
> > OK, disregard my comments about the size overhead. It's only 2GiB, i.e.
> > roughly 5%.
> 
> Yes, the size overhead is IMO very reasonable. So do you reconsider the
> general opinion you gave in your previous e-mail?

Yes, all I said about the size overhead, and thus my opinion, is void.

As a conclusion: yes, this is *very* good, and I think we must get that
in the tree. :-)

Sorry for the confusion... :-/

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16]  Add per-package staging feature
  2015-06-28 21:00   ` Thomas Petazzoni
@ 2015-06-28 21:25     ` Yann E. MORIN
  2015-06-29  8:35       ` Fabio Porcedda
  2015-06-29  8:38     ` Fabio Porcedda
  1 sibling, 1 reply; 37+ messages in thread
From: Yann E. MORIN @ 2015-06-28 21:25 UTC (permalink / raw)
  To: buildroot

Thomas, Fabio, All,

On 2015-06-28 23:00 +0200, Thomas Petazzoni spake thusly:
> On Sun, 28 Jun 2015 22:46:22 +0200, Yann E. MORIN wrote:
> > So, I'm really a bit skeptical. About five time the size for only about
> > twice the speedup, is it worth it? Sure some people will easily favour
> > speed over anything else, still the size overhead is really huge.
> 
> Do you reconsider this comment now that you know the reality about the
> size impact ? :-)

Hehe! ;-)

Of course, ~5% size overhead for 2+ times the speedup is totally
acceptable.

> > Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which
> > might be better as a HW_HIGH system. I can spin a test-build with this
> > patchset on this machine to see what we get...
> 
> Would definitely be useful. Maybe Fabio can provide you his
> defconfig-full test case (he gave the defconfig-small case).

Fabio, if you could share your two defconfig (or .config) files, that'd
be great!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-06-28 21:25     ` Yann E. MORIN
@ 2015-06-29  8:35       ` Fabio Porcedda
  2015-06-29 17:27         ` Yann E. MORIN
  0 siblings, 1 reply; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-29  8:35 UTC (permalink / raw)
  To: buildroot

On Sun, Jun 28, 2015 at 11:25 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Thomas, Fabio, All,
>
> On 2015-06-28 23:00 +0200, Thomas Petazzoni spake thusly:
>> On Sun, 28 Jun 2015 22:46:22 +0200, Yann E. MORIN wrote:
>> > So, I'm really a bit skeptical. About five time the size for only about
>> > twice the speedup, is it worth it? Sure some people will easily favour
>> > speed over anything else, still the size overhead is really huge.
>>
>> Do you reconsider this comment now that you know the reality about the
>> size impact ? :-)
>
> Hehe! ;-)
>
> Of course, ~5% size overhead for 2+ times the speedup is totally
> acceptable.
>
>> > Otherwise, I have a 2x4 core Xeon @3.4GHz with three SSDs in RAID0 which
>> > might be better as a HW_HIGH system. I can spin a test-build with this
>> > patchset on this machine to see what we get...

Nice machine :)
It have less cores but at higher frequency and it have three SSD in RAID0!
I'm curious to know how much faster is it.

I think the bottleneck on my HW-HIGH is the hard disk.

>> Would definitely be useful. Maybe Fabio can provide you his
>> defconfig-full test case (he gave the defconfig-small case).
>
> Fabio, if you could share your two defconfig (or .config) files, that'd
> be great!

Configs:
https://www.dropbox.com/s/tmdzhluxv69fdib/arm_external_longchaindep_defconfig?dl=0
https://www.dropbox.com/s/9ntz73wo5ua80k9/arm_external_allyespackageconfig_defconfig?dl=0

Branch with this patch set:
   https://github.com/fabio-porcedda/buildroot/tree/pps-v4

Branch without this patch set, on which this patch set was applied, it
has already all the patches for using top-level-parallel make:
   https://github.com/fabio-porcedda/buildroot/tree/pps-v4-base

Before testing the pps-v4 branch or enabling the top-level make i
suggest to test the  arm_external_allyespackageconfig_defconfig to
find if all packages builds on your machine, this defconfig builds
fine on a Fedora 22 and Ubuntu 14.04 (using gnu make 4.1 built from
source).

Command line for testing "arm_external_allyespackageconfig_defconfig"
enabling top-level parallel make:

{ time { b="make -j$((`nproc`+1)) -Orecurse -s" && $b libevas
gst1-plugins-base gst1-plugins-ugly && $b && echo Build OK; }; } 2>&1
| tee log

Pay attention to the fact that the packages "libevas"
"gst1-plugins-base" "gst1-plugins-ugly" must be built for first
otherwise they fail to build (I did not have enough time to analyze
the problem or check if using pps help on this matter).

Thanks
-- 
Fabio Porcedda

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-06-28 21:00   ` Thomas Petazzoni
  2015-06-28 21:25     ` Yann E. MORIN
@ 2015-06-29  8:38     ` Fabio Porcedda
  1 sibling, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-06-29  8:38 UTC (permalink / raw)
  To: buildroot

On Sun, Jun 28, 2015 at 11:00 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Dear Yann E. MORIN,
>
> On Sun, 28 Jun 2015 22:46:22 +0200, Yann E. MORIN wrote:
>

[snip]

>>   - we're missing the benchmarks for this patchset without top-level
>>     parallel make (especially for the size overhead). Unless it no
>>     longer makes sense?
>
> Correct, I asked the same thing.

I will test it, it just takes time.

[snip]

-- 
Fabio Porcedda

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-06-29  8:35       ` Fabio Porcedda
@ 2015-06-29 17:27         ` Yann E. MORIN
  0 siblings, 0 replies; 37+ messages in thread
From: Yann E. MORIN @ 2015-06-29 17:27 UTC (permalink / raw)
  To: buildroot

Fabio, All,

On 2015-06-29 10:35 +0200, Fabio Porcedda spake thusly:
> On Sun, Jun 28, 2015 at 11:25 PM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > Fabio, if you could share your two defconfig (or .config) files, that'd
> > be great!
> 
> Configs:
> https://www.dropbox.com/s/tmdzhluxv69fdib/arm_external_longchaindep_defconfig?dl=0
> https://www.dropbox.com/s/9ntz73wo5ua80k9/arm_external_allyespackageconfig_defconfig?dl=0
> 
> Branch with this patch set:
>    https://github.com/fabio-porcedda/buildroot/tree/pps-v4
> 
> Branch without this patch set, on which this patch set was applied, it
> has already all the patches for using top-level-parallel make:
>    https://github.com/fabio-porcedda/buildroot/tree/pps-v4-base

OK, thanks! :-)  I'll spawn a test build later tonight (UTC+2) so it
crunches during the night.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (18 preceding siblings ...)
  2015-06-28 20:46 ` Yann E. MORIN
@ 2015-07-05  8:51 ` Fabio Porcedda
  2015-07-05  9:01   ` Thomas Petazzoni
  2015-10-04 17:21 ` Arnout Vandecappelle
  20 siblings, 1 reply; 37+ messages in thread
From: Fabio Porcedda @ 2015-07-05  8:51 UTC (permalink / raw)
  To: buildroot

On Sun, Jun 28, 2015 at 9:42 PM, Fabio Porcedda
<fabio.porcedda@gmail.com> wrote:

[snip]

> To improve performance and reduce space utilization hard links are used.
>
> To evaluate performance and space utilization I've done some tests.
>
> HW-MED:
> CPU: Intel i7-4790K (4x2 @4GHz) = 8 threads
> RAM: 16GB
> SSD: 512GB
>
> HW-HIGH:
> CPU: 2 x Xeon 2620v3 (6x2 @2.40Ghz) = 24 threads
> RAM: 32GB
> SATA: RAID0 2x2TB 3.5" 7200rpm
>
>
> defconfig-small, long chain dependency, unfavorable  scenario for parallelization:
>
> BR2_arm=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_PACKAGE_XORG7=y
> BR2_PACKAGE_XSERVER_XORG_SERVER=y
> BR2_PACKAGE_LIBGTK3=y
>
> defconfig-full:
>   BR2_arm=y
>   BR2_TOOLCHAIN_EXTERNAL=y
> make allyespackageconfig
> Also i removed few packages that do not built on my system.
> make show-targets | wc -w
> 1366
>
> Test about output using hard links or note using the "defconfig-full"
> on HW-MED:
>
> |  35GB |  99m | master branch
> |  37GB | 100m | patch set
> | 153GB | 105m | patch set using hard links only for the toolchain sysroot
> | 225GB | 106m | patch set not using hardlinks at all
>
> Test about performances of this patch set vs master branch:
> | 199m | HW-MED | defconfig-full | master branch | no top-level make |
> |  99m | HW-MED | defconfig-full | master branch | top-level make    |
> | 100m | HW-MED | defconfig-full | patch set     | top-level make    |
>
> | 350m | HW-HIGH | defconfig-full | master branch | no top-level make |
> |  73m | HW-HIGH | defconfig-full | master branch | top-level make    |
> |  77m | HW-HIGH | defconfig-full | patch set     | top-level make    |
>
> | 10m | HW-MED | defconfig-small | master branch | no top-level make |
> |  5m | HW-MED | defconfig-small | master branch | top-level make    |
> |  5m | HW-MED | defconfig-samll | patch set     | top-level make    |
>
> | 21m18s | HW-HIGH | defconfig-small | master branch | no top-level make |
> |  7m53s | HW-HIGH | defconfig-small | master branch | top-level make    |
> |  7m54s | HW-HIGH | defconfig-samll | patch set     | top-level make    |

[snip]

I've redone all the tests including the case of the patch set without
using top-level make.

I've repeated the the test at least two times and taken the minimum value.

defconfig-small:
| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 21m17s    | 10m13s   | no                  | no           |
| 21m24s    | 10m17s   | no                  | yes         |
|   7m57s    |  5m25s   | yes                 | no           |
|   7m48s    |  5m22s   | yes                 | yes         |

defconfig-full:
| HW-HIGH | HW-MED | top-level make | patch set |
|----------------|----------------|-----------------------|---------------|
| 350m       | 199m       | no                  | no           |
| 355m       | 200m       | no                  | yes         |
|   74m       |  99m        | yes                 | no           |
|   76m       |  99m        | yes                 | yes         |

In these tests the penalty of using pps is 0%-3%.
Also the reduction of build time using top-level make is 50%-78%

I think they are nice numbers :)

Best regards
-- 
Fabio Porcedda

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-07-05  8:51 ` Fabio Porcedda
@ 2015-07-05  9:01   ` Thomas Petazzoni
  2015-07-05  9:26     ` Fabio Porcedda
  0 siblings, 1 reply; 37+ messages in thread
From: Thomas Petazzoni @ 2015-07-05  9:01 UTC (permalink / raw)
  To: buildroot

Hello,

On Sun, 5 Jul 2015 10:51:56 +0200, Fabio Porcedda wrote:

> I've redone all the tests including the case of the patch set without
> using top-level make.
> 
> I've repeated the the test at least two times and taken the minimum value.

How much variation did you had between the test builds for a given
scenario? Taking the minimal value is a somewhat strange choice :)

> defconfig-small:
> | HW-HIGH | HW-MED | top-level make | patch set |
> |----------------|----------------|-----------------------|---------------|
> | 21m17s    | 10m13s   | no                  | no           |
> | 21m24s    | 10m17s   | no                  | yes         |
> |   7m57s    |  5m25s   | yes                 | no           |
> |   7m48s    |  5m22s   | yes                 | yes         |
> 
> defconfig-full:
> | HW-HIGH | HW-MED | top-level make | patch set |
> |----------------|----------------|-----------------------|---------------|
> | 350m       | 199m       | no                  | no           |
> | 355m       | 200m       | no                  | yes         |
> |   74m       |  99m        | yes                 | no           |
> |   76m       |  99m        | yes                 | yes         |
> 
> In these tests the penalty of using pps is 0%-3%.
> Also the reduction of build time using top-level make is 50%-78%
> 
> I think they are nice numbers :)

Yes, they are very good. I guess that if we implement per-package host
dir there will be a small additional impact, but clearly the benefits
of top-level parallel largely outweighs the impact of per-package
staging/host directories.

Thanks a lot for having done those additional tests, it clearly means
we should progressively merge your work on this topic!

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-07-05  9:01   ` Thomas Petazzoni
@ 2015-07-05  9:26     ` Fabio Porcedda
  0 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-07-05  9:26 UTC (permalink / raw)
  To: buildroot

On Sun, Jul 5, 2015 at 11:01 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Sun, 5 Jul 2015 10:51:56 +0200, Fabio Porcedda wrote:
>
>> I've redone all the tests including the case of the patch set without
>> using top-level make.
>>
>> I've repeated the the test at least two times and taken the minimum value.
>
> How much variation did you had between the test builds for a given
> scenario? Taking the minimal value is a somewhat strange choice :)

Not very much, it was done because in some cases every time a repeat
the build the time was reduced,  i will show you the full data, build
times are sorted by their value, usually the lowest value is the last
value taken:

defconfig-small, HW-MED:
| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            | 10m13s 10m16s 10m16s 10m17s
| no                  | yes          |  10m17s 10m18s 10m20s 11m23s
| yes                 | no           |   5m25s 5m27s 5m28s 5m31s
| yes                 | yes         |  5m22s 5m23s 5m24s 5m26s

defconfig-small, HW-HIGH:
| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            | 21m17s 21m20s 21m42s
| no                  | yes          |  21m24s 21m25s 21m28s
| yes                 | no           |  7m57s 7m57s 7m58s 7m58s
| yes                 | yes         |   7m48s 7m50s 7m50s 7m55s 7m57s 7m58s

defconfig-full, HW-MED:
| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            |  199m 199m
| no                  | yes          |   200m 200m
| yes                 | no           |  99m 101m
| yes                 | yes         |  99m 100m


defconfig-full, HW-HIGH:
| top-level make | patch set | build time
|----------------------| ---------------|
--------------------------------------------------
| no                  | no            | 350m 351m
| no                  | yes          | 355m 358m
| yes                 | no           | 74m 75m
| yes                 | yes         | 76m 76m 76m

>> defconfig-small:
>> | HW-HIGH | HW-MED | top-level make | patch set |
>> |----------------|----------------|-----------------------|---------------|
>> | 21m17s    | 10m13s   | no                  | no           |
>> | 21m24s    | 10m17s   | no                  | yes         |
>> |   7m57s    |  5m25s   | yes                 | no           |
>> |   7m48s    |  5m22s   | yes                 | yes         |
>>
>> defconfig-full:
>> | HW-HIGH | HW-MED | top-level make | patch set |
>> |----------------|----------------|-----------------------|---------------|
>> | 350m       | 199m       | no                  | no           |
>> | 355m       | 200m       | no                  | yes         |
>> |   74m       |  99m        | yes                 | no           |
>> |   76m       |  99m        | yes                 | yes         |
>>
>> In these tests the penalty of using pps is 0%-3%.
>> Also the reduction of build time using top-level make is 50%-78%
>>
>> I think they are nice numbers :)
>
> Yes, they are very good. I guess that if we implement per-package host
> dir there will be a small additional impact, but clearly the benefits
> of top-level parallel largely outweighs the impact of per-package
> staging/host directories.
>
> Thanks a lot for having done those additional tests, it clearly means
> we should progressively merge your work on this topic!

Great!

I work now to use the toolchain wrapper even for the internal toolchain.

Best regards
-- 
Fabio Porcedda

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 16/16] infra: add per-package staging feature
  2015-06-28 19:42 ` [Buildroot] [RFC v4 16/16] infra: add per-package staging feature Fabio Porcedda
@ 2015-07-11 22:56   ` Romain Naour
  2015-07-13  9:42     ` Fabio Porcedda
  0 siblings, 1 reply; 37+ messages in thread
From: Romain Naour @ 2015-07-11 22:56 UTC (permalink / raw)
  To: buildroot

Hi Fabio,

Le 28/06/2015 21:42, Fabio Porcedda a ?crit :
> To improve the build reproducibility, instead of sharing a common
> staging are, every package has a private staging directory where only
> the declared dependency are present.
> 
> Having a per-package staging directory ensure that every package use
> only the declared dependency, because even if a undeclared dependency is
> build before, it will be not included in the private package directory.
> 
> This feature is useful for both top-level non-parallel make and
> top-level parallel make.
> 
> This feature is useful for target dependencies but not for host
> dependencies, to solve that a similar feature for host packages must be
> developed.
> 
> Implementation details:
> To minimize change the STAGING_DIR variable will be redefined for each
> package to point to the per-package staging directory.
> 
> At the beginning of each configuration stage the per-package staging
> directory will be created copying using hard links all the declared
> target dependencies.
> Also each path of every copied configuration files will be changed to
> point to the per-package
> 
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
> ---
>  Makefile               |  2 +-
>  package/Makefile.in    |  1 +
>  package/pkg-cmake.mk   |  2 +-
>  package/pkg-generic.mk | 34 +++++++++++++++++++++++++++++++++-
>  4 files changed, 36 insertions(+), 3 deletions(-)
> 

[snip]

> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 376b025..edac936 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -221,7 +221,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
>  
>  # Remove package sources
>  $(BUILD_DIR)/%/.stamp_dircleaned:
> -	rm -Rf $(@D)
> +	rm -Rf $(@D) $(STAGING_DIR)
>  
>  ################################################################################
>  # virt-provides-single -- check that provider-pkg is the declared provider for
> @@ -414,6 +414,33 @@ $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>  $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
>  $(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))
>  
> +ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
> +  $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
> +	$$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
> +	$$($$(call UPPERCASE,$$(dep))_STAGING_DIR)/))
> +
> +  $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
> +
> +  define $(2)_PREPARE_STAGING_DIR
> +	mkdir -p $$($(2)_STAGING_DIR)/usr/mkspecs/qws
> +	cp -rl $(STAGING_DIR)/* $$($(2)_STAGING_DIR)

I tried your series with codesourcery ARM 2014.05 with thumb2 "ARM instruction
set" enabled:

BR2_arm=y
BR2_cortex_a9=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM201405
BR2_ARM_INSTRUCTIONS_THUMB2

And the build fail immediately when copying the first stagingpkg directory due
to a symlink in staging.

>>> dash 0.5.8 Updating config.sub and config.guess
for file in config.guess config.sub; do for i in $(find /home/naourr/git/buildroot/output/build/dash-0.5.8 -name $file); do cp support/gnuconfig/$file $i; done; done
>>> dash 0.5.8 Patching libtool
>>> dash 0.5.8 Configuring
mkdir -p /home/naourr/git/buildroot/output/stagingpkg/dash/usr/mkspecs/qws
cp -rl /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/* /home/naourr/git/buildroot/output/stagingpkg/dash
cp: cannot copy cyclic symbolic link ? /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/thumb2/thumb2 ?
package/pkg-generic.mk:146: recipe for target '/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured' failed
make: *** [/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured] Error 1

Here is the content of the staging directory:
ls -l output/staging/
total 20
drwxr-xr-x 2 naourr naourr 4096 12 juil. 00:35 bin
drwxr-xr-x 2 naourr naourr 4096 30 mai    2014 etc
drwxr-xr-x 2 naourr naourr 4096 30 mai    2014 lib
lrwxrwxrwx 1 naourr naourr    3 12 juil. 00:35 lib32 -> lib
drwxr-xr-x 2 naourr naourr 4096 30 mai    2014 sbin
lrwxrwxrwx 1 naourr naourr    2 12 juil. 00:35 thumb2 -> ./
drwxr-xr-x 8 naourr naourr 4096 30 mai    2014 usr

If you use "cp -drl" or better yet "cp -al" (Thanks Yann) in order to not
follow symbolic links and preserve the link, the issue is fixed :)

Thoughts ?

Best regards,
Romain Naour

> +	$$(if $$($(2)_STAGING_DIRS), $$(foreach dir,$$($(2)_STAGING_DIRS),\
> +		rsync -au --link-dest=$$(dir) $$(dir) $$($(2)_STAGING_DIR); ))
> +	find $$($(2)_STAGING_DIR)/usr/{bin,lib,mkspecs/qws} \
> +		-ignore_readdir_race \
> +		-name "*[-_]config" -or -name "*.la" -or -name "*.pc" -or \
> +		-name "*.prl" -or -name "qmake.conf" | xargs -r sed -i -r \
> +		-e "s|$$(STAGINGPKG_DIR)/[^/]*/+usr|$$($(2)_STAGING_DIR)/usr|g" \
> +		-e "s|$$(BUILD_DIR)/[^ ]+/([^/ ]+).la|$$($(2)_STAGING_DIR)/usr/lib/\1.la|g" \
> +		-e "s|/lib/libpulsecommon|/lib/pulseaudio/libpulsecommon|g"
> +  endef
> +

[snip]

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 02/16] packages: for staging stuff use backtick instead of the shell function
  2015-06-28 19:42 ` [Buildroot] [RFC v4 02/16] packages: for staging stuff use backtick instead of the shell function Fabio Porcedda
@ 2015-07-12 14:39   ` Arnout Vandecappelle
  0 siblings, 0 replies; 37+ messages in thread
From: Arnout Vandecappelle @ 2015-07-12 14:39 UTC (permalink / raw)
  To: buildroot

On 06/28/15 21:42, Fabio Porcedda wrote:
> When the shell function execute the command before the per-package
> staging directory is created, use the backtick that is executed after
> the per-package staging directory is created.
> 
> This commit prevents build failures when the per-package staging feature
> is added.

 At first, I was pretty opposed to this change because I like make more than
shell :-)

 However, it also fixes another issue: make printvars prints a lot of errors in
addition to the real thing, because it evaluates all those $(shell ...)
variables while the programs they call or not defined yet.

 But for that, a lot more needs to be changed. So I've taken this over and
posted a patch series [1] that removes all the $(shell ...) calls.


 Regards,
 Arnout

[1] http://patchwork.ozlabs.org/patch/494127/

> 
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 16/16] infra: add per-package staging feature
  2015-07-11 22:56   ` Romain Naour
@ 2015-07-13  9:42     ` Fabio Porcedda
  0 siblings, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-07-13  9:42 UTC (permalink / raw)
  To: buildroot

On Sun, Jul 12, 2015 at 12:56 AM, Romain Naour <romain.naour@openwide.fr> wrote:
> Hi Fabio,
>
> Le 28/06/2015 21:42, Fabio Porcedda a ?crit :
>> To improve the build reproducibility, instead of sharing a common
>> staging are, every package has a private staging directory where only
>> the declared dependency are present.
>>
>> Having a per-package staging directory ensure that every package use
>> only the declared dependency, because even if a undeclared dependency is
>> build before, it will be not included in the private package directory.
>>
>> This feature is useful for both top-level non-parallel make and
>> top-level parallel make.
>>
>> This feature is useful for target dependencies but not for host
>> dependencies, to solve that a similar feature for host packages must be
>> developed.
>>
>> Implementation details:
>> To minimize change the STAGING_DIR variable will be redefined for each
>> package to point to the per-package staging directory.
>>
>> At the beginning of each configuration stage the per-package staging
>> directory will be created copying using hard links all the declared
>> target dependencies.
>> Also each path of every copied configuration files will be changed to
>> point to the per-package
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>> ---
>>  Makefile               |  2 +-
>>  package/Makefile.in    |  1 +
>>  package/pkg-cmake.mk   |  2 +-
>>  package/pkg-generic.mk | 34 +++++++++++++++++++++++++++++++++-
>>  4 files changed, 36 insertions(+), 3 deletions(-)
>>
>
> [snip]
>
>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>> index 376b025..edac936 100644
>> --- a/package/pkg-generic.mk
>> +++ b/package/pkg-generic.mk
>> @@ -221,7 +221,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
>>
>>  # Remove package sources
>>  $(BUILD_DIR)/%/.stamp_dircleaned:
>> -     rm -Rf $(@D)
>> +     rm -Rf $(@D) $(STAGING_DIR)
>>
>>  ################################################################################
>>  # virt-provides-single -- check that provider-pkg is the declared provider for
>> @@ -414,6 +414,33 @@ $(2)_FINAL_DEPENDENCIES = $$(sort $$($(2)_DEPENDENCIES))
>>  $(2)_FINAL_PATCH_DEPENDENCIES = $$(sort $$($(2)_PATCH_DEPENDENCIES))
>>  $(2)_FINAL_ALL_DEPENDENCIES = $$(sort $$($(2)_FINAL_DEPENDENCIES) $$($(2)_FINAL_PATCH_DEPENDENCIES))
>>
>> +ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
>> +  $(2)_STAGING_DIRS = $$(wildcard $$(foreach dep,\
>> +     $$(filter-out host-% toolchain,$$($(2)_FINAL_DEPENDENCIES)),\
>> +     $$($$(call UPPERCASE,$$(dep))_STAGING_DIR)/))
>> +
>> +  $(2)_STAGING_DIR = $$(STAGINGPKG_DIR)/$(1)
>> +
>> +  define $(2)_PREPARE_STAGING_DIR
>> +     mkdir -p $$($(2)_STAGING_DIR)/usr/mkspecs/qws
>> +     cp -rl $(STAGING_DIR)/* $$($(2)_STAGING_DIR)
>
> I tried your series with codesourcery ARM 2014.05 with thumb2 "ARM instruction
> set" enabled:
>
> BR2_arm=y
> BR2_cortex_a9=y
> BR2_TOOLCHAIN_EXTERNAL=y
> BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_ARM201405
> BR2_ARM_INSTRUCTIONS_THUMB2
>
> And the build fail immediately when copying the first stagingpkg directory due
> to a symlink in staging.
>
>>>> dash 0.5.8 Updating config.sub and config.guess
> for file in config.guess config.sub; do for i in $(find /home/naourr/git/buildroot/output/build/dash-0.5.8 -name $file); do cp support/gnuconfig/$file $i; done; done
>>>> dash 0.5.8 Patching libtool
>>>> dash 0.5.8 Configuring
> mkdir -p /home/naourr/git/buildroot/output/stagingpkg/dash/usr/mkspecs/qws
> cp -rl /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/* /home/naourr/git/buildroot/output/stagingpkg/dash
> cp: cannot copy cyclic symbolic link ? /home/naourr/git/buildroot/output/host/usr/arm-buildroot-linux-gnueabi/sysroot/thumb2/thumb2 ?
> package/pkg-generic.mk:146: recipe for target '/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured' failed
> make: *** [/home/naourr/git/buildroot/output/build/dash-0.5.8/.stamp_configured] Error 1
>
> Here is the content of the staging directory:
> ls -l output/staging/
> total 20
> drwxr-xr-x 2 naourr naourr 4096 12 juil. 00:35 bin
> drwxr-xr-x 2 naourr naourr 4096 30 mai    2014 etc
> drwxr-xr-x 2 naourr naourr 4096 30 mai    2014 lib
> lrwxrwxrwx 1 naourr naourr    3 12 juil. 00:35 lib32 -> lib
> drwxr-xr-x 2 naourr naourr 4096 30 mai    2014 sbin
> lrwxrwxrwx 1 naourr naourr    2 12 juil. 00:35 thumb2 -> ./
> drwxr-xr-x 8 naourr naourr 4096 30 mai    2014 usr
>
> If you use "cp -drl" or better yet "cp -al" (Thanks Yann) in order to not
> follow symbolic links and preserve the link, the issue is fixed :)
>
>
> Thoughts ?

Hi Romain,
thanks for the report and for the solution :), I will use "cp -al" in
the updated version of this patch set.

[snip]

Best regards
-- 
Fabio Porcedda

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
                   ` (19 preceding siblings ...)
  2015-07-05  8:51 ` Fabio Porcedda
@ 2015-10-04 17:21 ` Arnout Vandecappelle
  2015-10-04 17:51   ` Bjørn Forsman
  2015-10-05 21:13   ` Fabio Porcedda
  20 siblings, 2 replies; 37+ messages in thread
From: Arnout Vandecappelle @ 2015-10-04 17:21 UTC (permalink / raw)
  To: buildroot

On 28-06-15 20:42, Fabio Porcedda wrote:
> Hi all,
> this patch set aims to improve build reproducibility by using a
> per-package staging directory.
> Also this feature aims to enable the top-level parallel make.
> 
> This patch set is available at:
> https://github.com/fabio-porcedda/buildroot/tree/pps-v4
> 
> 
> Every package instead of installing the files in the shared staging
> directory (STAGING_DIR) install all files in a subdirectory of the
> STAGINGPKG_DIR.

[snip]

> TODO:
>  - use the toolchaing wrapper even for the internal toolchain
>  - rebase over master (this patch set is based on 2015.05-161-gca5f11e).
>  - more testing (tested only for arm target).

[snip]

 Hi Fabio,

 Now that we have a toolchain wrapper for the internal toolchain, would you care
to rebase this series and use the toolchain wrapper for pps support?

 Thomas commented that there should also be a per-package host directory, but
after some discussion here at the BR developer meeting we decided that we could
start with just a per-package staging and see how far we get. We anyway don't
have complete reproducibility at the moment, since the build of host packages
depends on what is installed on the system.

 Note that we are really excited about the possibilities of per-package staging
and top-level parallel build, so we hope you'll continue working on it.

 I've marked this series as Changes Requested in patchwork.

 Regards,
 Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-10-04 17:21 ` Arnout Vandecappelle
@ 2015-10-04 17:51   ` Bjørn Forsman
  2015-10-05 21:13   ` Fabio Porcedda
  1 sibling, 0 replies; 37+ messages in thread
From: Bjørn Forsman @ 2015-10-04 17:51 UTC (permalink / raw)
  To: buildroot

On 4 October 2015 at 19:21, Arnout Vandecappelle <arnout@mind.be> wrote:
> [...] We anyway don't
> have complete reproducibility at the moment, since the build of host packages
> depends on what is installed on the system.

To get a known set of host packages, Buildroot could run in an
environment setup by Nix[1].

With some extra tricks, you can get _very_ reproducible environments:

  https://garbas.si/2015/reproducible-development-environments.html

I think that's close to as reproducible as it gets, if not emulating
CPU/RAM/filesystem the build runs in :-)

[1] http://nixos.org/nix/

Disclaimer: I'm a Nix(OS) user/contributor.

- Bj?rn

^ permalink raw reply	[flat|nested] 37+ messages in thread
* [Buildroot] [RFC v4 00/16] Add per-package staging feature
  2015-10-04 17:21 ` Arnout Vandecappelle
  2015-10-04 17:51   ` Bjørn Forsman
@ 2015-10-05 21:13   ` Fabio Porcedda
  1 sibling, 0 replies; 37+ messages in thread
From: Fabio Porcedda @ 2015-10-05 21:13 UTC (permalink / raw)
  To: buildroot

On Sun, Oct 4, 2015 at 7:21 PM, Arnout Vandecappelle <arnout@mind.be> wrote:

> On 28-06-15 20:42, Fabio Porcedda wrote:
> > Hi all,
> > this patch set aims to improve build reproducibility by using a
> > per-package staging directory.
> > Also this feature aims to enable the top-level parallel make.
> >
> > This patch set is available at:
> > https://github.com/fabio-porcedda/buildroot/tree/pps-v4
> >
> >
> > Every package instead of installing the files in the shared staging
> > directory (STAGING_DIR) install all files in a subdirectory of the
> > STAGINGPKG_DIR.
>
> [snip]
>
> > TODO:
> >  - use the toolchaing wrapper even for the internal toolchain
> >  - rebase over master (this patch set is based on 2015.05-161-gca5f11e).
> >  - more testing (tested only for arm target).
>
> [snip]
>
>  Hi Fabio,
>
>  Now that we have a toolchain wrapper for the internal toolchain, would
> you care
> to rebase this series and use the toolchain wrapper for pps support?
>
>
Thanks for the toolchain wrapper for the internal toolchain.

I will rebase this patch for sure and send an updated version.



>  Thomas commented that there should also be a per-package host directory,
> but
> after some discussion here at the BR developer meeting we decided that we
> could
> start with just a per-package staging and see how far we get. We anyway
> don't
> have complete reproducibility at the moment, since the build of host
> packages
> depends on what is installed on the system.
>
>  Note that we are really excited about the possibilities of per-package
> staging
> and top-level parallel build, so we hope you'll continue working on it.
>

Great, it's nice to hear it!


>  I've marked this series as Changes Requested in patchwork.
>
Ok

BR
-- 
Fabio Porcedda
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20151005/83693684/attachment-0001.html>

^ permalink raw reply	[flat|nested] 37+ messages in thread
end of thread, other threads:[~2015-10-05 21:13 UTC | newest]

Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-06-28 19:42 [Buildroot] [RFC v4 00/16] Add per-package staging feature Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 01/16] packages: use TARGET_MAKE_ENV to add pps support Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 02/16] packages: for staging stuff use backtick instead of the shell function Fabio Porcedda
2015-07-12 14:39   ` Arnout Vandecappelle
2015-06-28 19:42 ` [Buildroot] [RFC v4 03/16] packages/scons: make avaiable the GCC_SYSROOT environment variable Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 04/16] packages/qt: read spec files from the per-package staging directory Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 05/16] pkg-perl: use TARGET_MAKE_ENV to add pps support Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 06/16] tooclahin-external: add support for GCC_SYSROOT Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 07/16] toolchain-external: move some code in a new function Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 08/16] toolchain-external: use the wrapper even for the linker "ld" Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 09/16] infra: add GCC_SYSROOT environment variable to support pps feature Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 10/16] pkg-cmake: " Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 11/16] pkg-python: add GCC_SYSROOT variable to add pps support Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 12/16] pacakge/luarocks: add GCC_SYSROOT environment variable to support pps Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 13/16] package/pkgconf: use GCC_SYSROOT to support pps feature Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 14/16] Makefile: add STAGINGNOPKG_DIR variable Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 15/16] pkg-generic: ADD_TOOLCHAIN_DEPENDENCY is true only for target packages Fabio Porcedda
2015-06-28 19:42 ` [Buildroot] [RFC v4 16/16] infra: add per-package staging feature Fabio Porcedda
2015-07-11 22:56   ` Romain Naour
2015-07-13  9:42     ` Fabio Porcedda
2015-06-28 19:53 ` [Buildroot] [RFC v4 00/16] Add " Fabio Porcedda
2015-06-28 20:34 ` Thomas Petazzoni
2015-06-28 20:46 ` Yann E. MORIN
2015-06-28 20:49   ` Yann E. MORIN
2015-06-28 20:57     ` Thomas Petazzoni
2015-06-28 21:04       ` Yann E. MORIN
2015-06-28 21:00   ` Thomas Petazzoni
2015-06-28 21:25     ` Yann E. MORIN
2015-06-29  8:35       ` Fabio Porcedda
2015-06-29 17:27         ` Yann E. MORIN
2015-06-29  8:38     ` Fabio Porcedda
2015-07-05  8:51 ` Fabio Porcedda
2015-07-05  9:01   ` Thomas Petazzoni
2015-07-05  9:26     ` Fabio Porcedda
2015-10-04 17:21 ` Arnout Vandecappelle
2015-10-04 17:51   ` Bjørn Forsman
2015-10-05 21:13   ` Fabio Porcedda

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.