[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Fabio Porcedda]

Hi all,
this is a patch set for adding support for top-level parallel make in
buildroot, the common problem scattered in buildroot's top-level
makefiles is that in the rules it relies on the order of evaluation of
the prerequisites, to be able to use top-level parallel make instead
of reling on the left to right ordering of evaluation of the
prerequisites we must add an explicit rule to describe the
dependencies.

With this patch set the top-level parallel make seems to work fine,
example:
	make clean
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))

On my quad core system the building time for qemu_x86_defconfig
is 13m versus 11m.

I've tested the qemu_x86_defconfig uclibc/eglibc/glibc.

Best regards
Fabio Porcedda

v8:
 - rebased over master
 - added patche for for base dependency
 - added patche for for glibc package
 - added patche for for uclibc package
 - removed patch already merged
 - changed some descriptions
 - modified the patch for toolchain dependency to prevent circular dependency
v7:
 - add the latest patch
 - add to the first patch the <pkgname>_TOOLCHAIN variable
 - improve the fifth patch
v6:
 - added the fifth patch
 - updated the fourth patch adding the install targets
 - updated the second patch to remove TARGETS_ALL
v5:
 - added the fourth patch
 - fixed some typos
 - rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
 - add support for top-level parallel make for the glibc package
v4:
 - rebased over master
 - add Acked-by: Thomas Petazzoni on the third patch
 - changed the orderd of the patches
v3:
 - add back the patch "package: add toolchain dependency to
    inner-generic-package" because now is working fine.
 - add Acked-by: Arnout Vandecappelle to the third patch.
 - reworked the second patch following Arnout suggestions.
v2:
 - remove patch "package: add toolchain dependency to inner-generic-package"
   because was not working fine against recent toolchain changes.

Fabio Porcedda (8):
  package: add base dependency to every package
  package: add toolchain dependency to every target package
  package: add support for top-level parallel make
  Makefile: add support for top-level parallel make
  glibc: add support for top-level parallel make
  uclibc: add support for top-level parallel make
  package: enable jobserver for recursive make
  Makefile: enable top-level parallel make

 Makefile                                           | 22 +++-----
 fs/common.mk                                       |  4 +-
 package/glibc/glibc.mk                             |  7 ++-
 package/linux-headers/linux-headers.mk             |  3 ++
 package/pkg-autotools.mk                           |  3 +-
 package/pkg-generic.mk                             | 60 +++++++++++++---------
 package/uclibc/uclibc.mk                           |  7 ++-
 .../toolchain-buildroot/toolchain-buildroot.mk     |  2 +
 toolchain/toolchain-external/toolchain-external.mk |  2 +
 toolchain/toolchain/toolchain.mk                   |  5 +-
 10 files changed, 69 insertions(+), 46 deletions(-)

-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Fabio Porcedda]

Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
every package.
This way we can build correctly every package right after the clean
stage.
As example with this commit we can build successfully the glibc right
after the clean stage:
	make clean glibc

This is also a step forward supporting top-level parallel make.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-generic.mk           | 2 ++
 toolchain/toolchain/toolchain.mk | 3 +--
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 4bba4b5..1e7154e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
 $(1)-build:		$(1)-configure \
 			$$($(2)_TARGET_BUILD)
 
+$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
+
 ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 # In the normal case (no package override), the sequence of steps is
 #  source, by downloading
diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index 44ed629..8559ac9 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -14,5 +14,4 @@ endif
 
 $(eval $(generic-package))
 
-toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-
+toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 2/8] package: add toolchain dependency to every target package

[Fabio Porcedda]

This commit makes the dependency from the target toolchain explicit.
This way we can buid from command line a package that use
innger-generic-package right after the configuration phase, example:

	make clean <package-name>

Also remove TARGETS_ALL because the only purpose was to add toolchain
dependency so it's superseded by this commit.

To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN
for not adding the toolchain dependency for toolchain packages.

This is also a step forward supporting top-level parallel make.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile                                             |  8 ++------
 package/glibc/glibc.mk                               |  3 +++
 package/linux-headers/linux-headers.mk               |  3 +++
 package/pkg-autotools.mk                             |  3 ++-
 package/pkg-generic.mk                               | 12 ++++++++++--
 package/uclibc/uclibc.mk                             |  3 +++
 toolchain/toolchain-buildroot/toolchain-buildroot.mk |  2 ++
 toolchain/toolchain-external/toolchain-external.mk   |  2 ++
 toolchain/toolchain/toolchain.mk                     |  2 ++
 9 files changed, 29 insertions(+), 9 deletions(-)

diff --git a/Makefile b/Makefile
index f266e2d..ebecec4 100644
--- a/Makefile
+++ b/Makefile
@@ -352,7 +352,6 @@ TARGETS+=target-post-image
 TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
-TARGETS_ALL:=$(patsubst %,__real_tgt_%,$(TARGETS))
 
 # host-* dependencies have to be handled specially, as those aren't
 # visible in Kconfig and hence not added to a variable like TARGETS.
@@ -375,9 +374,6 @@ HOST_SOURCE += $(addsuffix -source,$(sort $(TARGETS_HOST_DEPS) $(HOST_DEPS)))
 TARGETS_LEGAL_INFO:=$(patsubst %,%-legal-info,\
 		$(TARGETS) $(BASE_TARGETS) $(TARGETS_HOST_DEPS) $(HOST_DEPS))))
 
-# all targets depend on the crosscompiler and it's prerequisites
-$(TARGETS_ALL): __real_tgt_%: $(BASE_TARGETS) %
-
 dirs: $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
 
@@ -386,11 +382,11 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(BASE_TARGETS) $(TARGETS_ALL)
+world: $(TARGETS)
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ALL) \
+	$(BASE_TARGETS) $(TARGETS) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index 0dab492..68b98a9 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -19,6 +19,9 @@ endif
 GLIBC_LICENSE = GPLv2+ (programs), LGPLv2.1+, BSD-3c, MIT (library)
 GLIBC_LICENSE_FILES = $(addprefix $(GLIBC_SRC_SUBDIR)/,COPYING COPYING.LIB LICENSES)
 
+# glibc is part of the toolchain so disable the toolchain dependency
+GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before (e)glibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
diff --git a/package/linux-headers/linux-headers.mk b/package/linux-headers/linux-headers.mk
index 30d3076..6dac9e3 100644
--- a/package/linux-headers/linux-headers.mk
+++ b/package/linux-headers/linux-headers.mk
@@ -17,6 +17,9 @@ LINUX_HEADERS_SOURCE = linux-$(LINUX_HEADERS_VERSION).tar.xz
 
 LINUX_HEADERS_INSTALL_STAGING = YES
 
+# linux-headers is part of the toolchain so disable the toolchain dependency
+LINUX_HEADERS_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 define LINUX_HEADERS_INSTALL_STAGING_CMDS
 	(cd $(@D); \
 		$(TARGET_MAKE_ENV) $(MAKE) \
diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
index 9523529..f58570d 100644
--- a/package/pkg-autotools.mk
+++ b/package/pkg-autotools.mk
@@ -209,7 +209,8 @@ endef
 # This must be repeated from inner-generic-package, otherwise we get an empty
 # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
 # away from the non-host rule
-$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
+$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
+				host-toolchain $(1),\
     $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
 
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 1e7154e..3f19aea 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -307,8 +307,16 @@ endif
 
 $(2)_REDISTRIBUTE		?= YES
 
-
-$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+# When a target package is a toolchain dependency set this variable to
+# 'NO' so the 'toolchain' dependency is not added to prevent a circular
+# dependency
+$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+
+$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
+	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
+ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
+$(2)_DEPENDENCIES += toolchain
+endif
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index 20d3bb6..0d12afd 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -18,6 +18,9 @@ endif
 
 UCLIBC_INSTALL_STAGING = YES
 
+# uclibc is part of the toolchain so disable the toolchain dependency
+UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 # Before uClibc is configured, we must have the first stage
 # cross-compiler and the kernel headers
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
diff --git a/toolchain/toolchain-buildroot/toolchain-buildroot.mk b/toolchain/toolchain-buildroot/toolchain-buildroot.mk
index f1f07a9..d941854 100644
--- a/toolchain/toolchain-buildroot/toolchain-buildroot.mk
+++ b/toolchain/toolchain-buildroot/toolchain-buildroot.mk
@@ -14,4 +14,6 @@ BUILDROOT_LIBC = $(call qstrip,$(BR2_TOOLCHAIN_BUILDROOT_LIBC))
 
 TOOLCHAIN_BUILDROOT_DEPENDENCIES = host-gcc-final
 
+TOOLCHAIN_BUILDROOT_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
diff --git a/toolchain/toolchain-external/toolchain-external.mk b/toolchain/toolchain-external/toolchain-external.mk
index d41cc7c..2076cf7 100644
--- a/toolchain/toolchain-external/toolchain-external.mk
+++ b/toolchain/toolchain-external/toolchain-external.mk
@@ -340,6 +340,8 @@ TOOLCHAIN_EXTERNAL_SITE = $(dir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL)))
 endif
 
+TOOLCHAIN_EXTERNAL_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 TOOLCHAIN_EXTERNAL_INSTALL_STAGING = YES
 
 ifeq ($(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R1)$(BR2_TOOLCHAIN_EXTERNAL_BLACKFIN_UCLINUX_2012R2),y)
diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
index 8559ac9..7241fe7 100644
--- a/toolchain/toolchain/toolchain.mk
+++ b/toolchain/toolchain/toolchain.mk
@@ -12,6 +12,8 @@ else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
 TOOLCHAIN_DEPENDENCIES += toolchain-external
 endif
 
+TOOLCHAIN_ADD_TOOLCHAIN_DEPENDENCY = NO
+
 $(eval $(generic-package))
 
 toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 3/8] package: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relyng on
the left to right ordering of evaluation of the prerequisites add an
explicit rule to describe the dependencies.
So add explicit dependencies for the following stamp files:
   %/.stamp_extracted
   %/.stamp_patched
   %/.stamp_configured
   %/.stamp_built
   %/.stamp_host_installed
   %/.stamp_staging_installed
   %/.stamp_images_installed
   %/.stamp_target_installed

Because the %-build target is not anymore part of the dependency chain,
add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead.
This new variable is used only by uclibc and glibc packages.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/glibc/glibc.mk   |  2 +-
 package/pkg-generic.mk   | 38 +++++++++++++++++++++-----------------
 package/uclibc/uclibc.mk |  2 +-
 3 files changed, 23 insertions(+), 19 deletions(-)

diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index 68b98a9..e89c12a 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
 
 # Before (e)glibc is built, we must have the second stage cross-compiler
-glibc-build: host-gcc-intermediate
+GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
 GLIBC_SUBDIR = build
 
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 3f19aea..094868c 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
 ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
 $(2)_DEPENDENCIES += toolchain
 endif
+$(2)_BUILD_DEPENDENCIES		?=
 
 $(2)_INSTALL_STAGING		?= NO
 $(2)_INSTALL_IMAGES		?= NO
@@ -368,30 +369,34 @@ $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
 endif
 
 ifeq ($$($(2)_INSTALL_TARGET),YES)
-$(1)-install-target:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_TARGET)
+$(1)-install-target:	$$($(2)_TARGET_INSTALL_TARGET)
 else
 $(1)-install-target:
 endif
 
 ifeq ($$($(2)_INSTALL_STAGING),YES)
-$(1)-install-staging:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_STAGING)
+$(1)-install-staging:	$$($(2)_TARGET_INSTALL_STAGING)
 else
 $(1)-install-staging:
 endif
 
 ifeq ($$($(2)_INSTALL_IMAGES),YES)
-$(1)-install-images:	$(1)-build \
-			$$($(2)_TARGET_INSTALL_IMAGES)
+$(1)-install-images:	$$($(2)_TARGET_INSTALL_IMAGES)
 else
 $(1)-install-images:
 endif
 
-$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
+$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
 
-$(1)-build:		$(1)-configure \
-			$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
+$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
+	$$($(2)_TARGET_BUILD)
+
+$(1)-build:		$$($(2)_TARGET_BUILD)
+$$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES)
+
+$(1)-configure:			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES)
 
 $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
 
@@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 #  extract
 #  patch
 #  configure
-$(1)-configure:		$(1)-patch $(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_PATCH)
 
-$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
+$(1)-patch:		$$($(2)_TARGET_PATCH)
+$$($(2)_TARGET_PATCH):	$$($(2)_TARGET_EXTRACT)
 
-$(1)-extract:		$(1)-source \
-			$$($(2)_TARGET_EXTRACT)
+$(1)-extract:			$$($(2)_TARGET_EXTRACT)
+$$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
@@ -418,10 +423,9 @@ else
 #  source, by rsyncing
 #  depends
 #  configure
-$(1)-configure:		$(1)-depends \
-			$$($(2)_TARGET_CONFIGURE)
+$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_RSYNC)
 
-$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
+$(1)-depends:		$$($(2)_DEPENDENCIES)
 
 $(1)-patch:		$(1)-rsync
 $(1)-extract:		$(1)-rsync
diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index 0d12afd..dea36e7 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
 UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
 
 # Before uClibc is built, we must have the second stage cross-compiler
-uclibc-build: host-gcc-intermediate
+UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
 
 # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
 # setting.
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 4/8] Makefile: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relyng on
the left to right ordering of evaluation of the prerequisites add an
explicit rule to describe the dependencies.

Add explicit rules to describe the following dependency chain:
$(TARGETS) -> target-finalize -> rootfs-* -> target-post-image

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile     | 15 ++++++---------
 fs/common.mk |  4 ++--
 2 files changed, 8 insertions(+), 11 deletions(-)

diff --git a/Makefile b/Makefile
index ebecec4..299d34b 100644
--- a/Makefile
+++ b/Makefile
@@ -329,8 +329,6 @@ include boot/common.mk
 include linux/linux.mk
 include system/system.mk
 
-TARGETS+=target-finalize
-
 ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
 TARGETS+=target-purgelocales
 endif
@@ -347,8 +345,6 @@ endif
 
 include fs/common.mk
 
-TARGETS+=target-post-image
-
 TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
 TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
 TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
@@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
 
 prepare: $(BUILD_DIR)/buildroot-config/auto.conf
 
-world: $(TARGETS)
+world: target-post-image
 
 .PHONY: all world toolchain dirs clean distclean source outputmakefile \
 	legal-info legal-info-prepare legal-info-clean printvars \
-	$(BASE_TARGETS) $(TARGETS) \
+	target-finalize target-post-image \
+	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
 	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
 	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
 	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
@@ -443,7 +440,7 @@ endif
 STRIP_FIND_CMD += -type f -perm /111
 STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
 
-target-finalize:
+target-finalize: $(TARGETS)
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
 		$(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
@@ -550,7 +547,7 @@ target-generatelocales: host-localedef
 	done
 endif
 
-target-post-image:
+target-post-image: target-finalize $(TARGETS_ROOTFS)
 	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
 		$(call MESSAGE,"Executing post-image script $(s)"); \
 		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
@@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(REDIST_SOURCES_DIR) \
 	@rm -f $(LEGAL_WARNINGS)
 
 show-targets:
-	@echo $(TARGETS)
+	@echo $(TARGETS) $(TARGETS_ROOTFS)
 
 else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
 
diff --git a/fs/common.mk b/fs/common.mk
index 4dab7ea..7698c53 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
 # extra deps
 ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
 
-$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
+$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
 	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
 	rm -f $$(FAKEROOT_SCRIPT)
@@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
 rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
 
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
-TARGETS += rootfs-$(1)
+TARGETS_ROOTFS += rootfs-$(1)
 endif
 endef
 
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 5/8] glibc: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relyng on
the left to right ordering of evaluation of the prerequisites add
an explicit rule to describe the dependencies.

The glibc-install-target depends on glibc-install-staging so add a rule
for it.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/glibc/glibc.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
index e89c12a..838fb8c 100644
--- a/package/glibc/glibc.mk
+++ b/package/glibc/glibc.mk
@@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS
 endef
 
 $(eval $(autotools-package))
+
+$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 6/8] uclibc: add support for top-level parallel make

[Fabio Porcedda]

To be able to use top-level parallel make we must not depend in a rule
on the order of evaluation of the prerequisites, so instead of relyng on
the left to right ordering of evaluation of the prerequisites add an
explicit rule to describe the dependencies.

The uclibc-install-target depends on uclibc-install-staging so add a
rule for it.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/uclibc/uclibc.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
index dea36e7..c8eeb3d 100644
--- a/package/uclibc/uclibc.mk
+++ b/package/uclibc/uclibc.mk
@@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch
 	rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
 
 $(eval $(generic-package))
+
+$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 7/8] package: enable jobserver for recursive make

[Fabio Porcedda]

Add '+' prefix to the $($(PKG)_BUILD_CMDS) and $($(PKG)_INSTALL*_CMDS)
commands to enable jobserver for the sub-make.

Without the '+' prefix GNU make does not detect the sub-make so it
disable the jobserver for the sub-make.

From GNU make documentation:
Using the MAKE variable has the same effect as using a ?+? character
at the beginning of the recipe line.  This special feature is only
enabled if the MAKE variable appears directly in the recipe: it does
not apply if the MAKE variable is referenced through expansion of
another variable. In the latter case you must use the ?+? token to get
these special effects.

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 package/pkg-generic.mk | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 094868c..122b4f9 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -118,21 +118,21 @@ $(BUILD_DIR)/%/.stamp_configured:
 # Build
 $(BUILD_DIR)/%/.stamp_built::
 	@$(call MESSAGE,"Building")
-	$($(PKG)_BUILD_CMDS)
+	+$($(PKG)_BUILD_CMDS)
 	$(foreach hook,$($(PKG)_POST_BUILD_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to host dir
 $(BUILD_DIR)/%/.stamp_host_installed:
 	@$(call MESSAGE,"Installing to host directory")
-	$($(PKG)_INSTALL_CMDS)
+	+$($(PKG)_INSTALL_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
 # Install to staging dir
 $(BUILD_DIR)/%/.stamp_staging_installed:
 	@$(call MESSAGE,"Installing to staging directory")
-	$($(PKG)_INSTALL_STAGING_CMDS)
+	+$($(PKG)_INSTALL_STAGING_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_STAGING_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(call MESSAGE,"Fixing package configuration files") ;\
@@ -146,7 +146,7 @@ $(BUILD_DIR)/%/.stamp_staging_installed:
 # Install to images dir
 $(BUILD_DIR)/%/.stamp_images_installed:
 	@$(call MESSAGE,"Installing to images directory")
-	$($(PKG)_INSTALL_IMAGES_CMDS)
+	+$($(PKG)_INSTALL_IMAGES_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_IMAGES_HOOKS),$(call $(hook))$(sep))
 	$(Q)touch $@
 
@@ -157,7 +157,7 @@ $(BUILD_DIR)/%/.stamp_target_installed:
 		$($(PKG)_INSTALL_INIT_SYSTEMD))
 	$(if $(BR2_INIT_SYSV)$(BR2_INIT_BUSYBOX),\
 		$($(PKG)_INSTALL_INIT_SYSV))
-	$($(PKG)_INSTALL_TARGET_CMDS)
+	+$($(PKG)_INSTALL_TARGET_CMDS)
 	$(foreach hook,$($(PKG)_POST_INSTALL_TARGET_HOOKS),$(call $(hook))$(sep))
 	$(Q)if test -n "$($(PKG)_CONFIG_SCRIPTS)" ; then \
 		$(RM) -f $(addprefix $(TARGET_DIR)/usr/bin/,$($(PKG)_CONFIG_SCRIPTS)) ; \
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 8/8] Makefile: enable top-level parallel make

[Fabio Porcedda]

Enable top-level parallel make because the previous patches have fixed
all the know issue about top-level parallel make.

Example:
	make <board>_defconfig
	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
---
 Makefile | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/Makefile b/Makefile
index 299d34b..9752f96 100644
--- a/Makefile
+++ b/Makefile
@@ -42,9 +42,6 @@ export HOSTARCH := $(shell uname -m | \
 	    -e s/macppc/powerpc/\
 	    -e s/sh.*/sh/)
 
-# This top-level Makefile can *not* be executed in parallel
-.NOTPARALLEL:
-
 # absolute path
 TOPDIR:=$(shell pwd)
 CONFIG_CONFIG_IN=Config.in
-- 
1.8.4

[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> Hi all,
> this is a patch set for adding support for top-level parallel make in
> buildroot, the common problem scattered in buildroot's top-level
> makefiles is that in the rules it relies on the order of evaluation of
> the prerequisites, to be able to use top-level parallel make instead
> of reling on the left to right ordering of evaluation of the
> prerequisites we must add an explicit rule to describe the
> dependencies.
>
> With this patch set the top-level parallel make seems to work fine,
> example:
> 	make clean
> 	make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN` + 1))
>
> On my quad core system the building time for qemu_x86_defconfig
> is 13m versus 11m.
>
> I've tested the qemu_x86_defconfig uclibc/eglibc/glibc.
>
> Best regards
> Fabio Porcedda

  Hi Fabio,

  I've tested this patch series now. I'll give acks and comments for the 
individual patches later. But first some overall observations.

  When ccache is enabled, make complains about a circular dependency 
$(BUILD_DIR)/host-ccache-3.1.8/.stamp_downloaded <- dependencies
You'll find something similar for sstrip, and when the host doesn't have 
a suitable tar or xz. It's not critical because make will break the 
dependency cycle and still build these packages before the rest, but it's 
not nice. Unfortunately, I don't see a simple way to avoid it.

  In system/system.mk, there are still a few different targets 
manipulating inittab; these have to be serialized.

  The initramfs rootfs will rebuild the kernel in parallel with building 
the cpio archive. This should be fixed, which is probably a bit invasive.

  legal-info doesn't work in parallel. Not a critical concern, but 
ideally that should be fixed as well.



  Regards,
  Arnout

>
> v8:
>   - rebased over master
>   - added patche for for base dependency
>   - added patche for for glibc package
>   - added patche for for uclibc package
>   - removed patch already merged
>   - changed some descriptions
>   - modified the patch for toolchain dependency to prevent circular dependency
> v7:
>   - add the latest patch
>   - add to the first patch the <pkgname>_TOOLCHAIN variable
>   - improve the fifth patch
> v6:
>   - added the fifth patch
>   - updated the fourth patch adding the install targets
>   - updated the second patch to remove TARGETS_ALL
> v5:
>   - added the fourth patch
>   - fixed some typos
>   - rewrited the second patch to use only $$($(2)_TARGET_*) in the rules
>   - add support for top-level parallel make for the glibc package
> v4:
>   - rebased over master
>   - add Acked-by: Thomas Petazzoni on the third patch
>   - changed the orderd of the patches
> v3:
>   - add back the patch "package: add toolchain dependency to
>      inner-generic-package" because now is working fine.
>   - add Acked-by: Arnout Vandecappelle to the third patch.
>   - reworked the second patch following Arnout suggestions.
> v2:
>   - remove patch "package: add toolchain dependency to inner-generic-package"
>     because was not working fine against recent toolchain changes.
>
> Fabio Porcedda (8):
>    package: add base dependency to every package
>    package: add toolchain dependency to every target package
>    package: add support for top-level parallel make
>    Makefile: add support for top-level parallel make
>    glibc: add support for top-level parallel make
>    uclibc: add support for top-level parallel make
>    package: enable jobserver for recursive make
>    Makefile: enable top-level parallel make
>
>   Makefile                                           | 22 +++-----
>   fs/common.mk                                       |  4 +-
>   package/glibc/glibc.mk                             |  7 ++-
>   package/linux-headers/linux-headers.mk             |  3 ++
>   package/pkg-autotools.mk                           |  3 +-
>   package/pkg-generic.mk                             | 60 +++++++++++++---------
>   package/uclibc/uclibc.mk                           |  7 ++-
>   .../toolchain-buildroot/toolchain-buildroot.mk     |  2 +
>   toolchain/toolchain-external/toolchain-external.mk |  2 +
>   toolchain/toolchain/toolchain.mk                   |  5 +-
>   10 files changed, 69 insertions(+), 46 deletions(-)
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
> every package.
> This way we can build correctly every package right after the clean
> stage.
> As example with this commit we can build successfully the glibc right
> after the clean stage:
> 	make clean glibc
>
> This is also a step forward supporting top-level parallel make.
>
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>

  Although this is one of the most useful patches of the series, it is 
also the one that introduces the circular dependency. So although it 
looks good, I'm not ready to ack it.

> ---
>   package/pkg-generic.mk           | 2 ++
>   toolchain/toolchain/toolchain.mk | 3 +--
>   2 files changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 4bba4b5..1e7154e 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
>   $(1)-build:		$(1)-configure \
>   			$$($(2)_TARGET_BUILD)
>
> +$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare

  Is there any reason why you changed the order here?


  Regards,
  Arnout

> +
>   ifeq ($$($(2)_OVERRIDE_SRCDIR),)
>   # In the normal case (no package override), the sequence of steps is
>   #  source, by downloading
> diff --git a/toolchain/toolchain/toolchain.mk b/toolchain/toolchain/toolchain.mk
> index 44ed629..8559ac9 100644
> --- a/toolchain/toolchain/toolchain.mk
> +++ b/toolchain/toolchain/toolchain.mk
> @@ -14,5 +14,4 @@ endif
>
>   $(eval $(generic-package))
>
> -toolchain-source: prepare dirs dependencies $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
> -
> +toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 2/8] package: add toolchain dependency to every target package

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> This commit makes the dependency from the target toolchain explicit.
> This way we can buid from command line a package that use
> innger-generic-package right after the configuration phase, example:
>
> 	make clean <package-name>
>
> Also remove TARGETS_ALL because the only purpose was to add toolchain
> dependency so it's superseded by this commit.
>
> To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN
> for not adding the toolchain dependency for toolchain packages.
>
> This is also a step forward supporting top-level parallel make.
>
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>

  I'm not happy with this _ADD_TOOLCHAIN_DEPENDENCY variable.

  What I would prefer is to move glibc, linux-headers and uclibc back 
under the toolchain directory (why did they ever move out, Thomas?) and 
to filter on $(pkgparentdir)

ifeq ($(5),target)
ifneq ($(pkgparentdir),toolchain)
$(2)_DEPENDENCIES += toolchain
endif
endif


[snip]
> diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
> index 9523529..f58570d 100644
> --- a/package/pkg-autotools.mk
> +++ b/package/pkg-autotools.mk
> @@ -209,7 +209,8 @@ endef
>   # This must be repeated from inner-generic-package, otherwise we get an empty
>   # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of _AUTORECONF
>   # away from the non-host rule
> -$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool $(1),\
> +$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf host-libtool \
> +				host-toolchain $(1),\

  You'll also need to do this in pkg-cmake.mk.

>       $(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
>
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 1e7154e..3f19aea 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -307,8 +307,16 @@ endif
>
>   $(2)_REDISTRIBUTE		?= YES
>
> -
> -$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
> +# When a target package is a toolchain dependency set this variable to
> +# 'NO' so the 'toolchain' dependency is not added to prevent a circular
> +# dependency
> +$(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
> +
> +$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
> +	$(patsubst host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
> +ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)

  It's more readable if you split this and-ing in two separate conditions.

  For $(2)_TYPE, you can directly use $(5) because you're anyway inside 
inner-generic-package here.


  Regards,
  Arnout

> +$(2)_DEPENDENCIES += toolchain
> +endif
>
>   $(2)_INSTALL_STAGING		?= NO
>   $(2)_INSTALL_IMAGES		?= NO

[snip]


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 3/8] package: add support for top-level parallel make

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> To be able to use top-level parallel make we must not depend in a rule
> on the order of evaluation of the prerequisites, so instead of relyng on

  nitpick: relying

> the left to right ordering of evaluation of the prerequisites add an
> explicit rule to describe the dependencies.
> So add explicit dependencies for the following stamp files:
>     %/.stamp_extracted
>     %/.stamp_patched
>     %/.stamp_configured
>     %/.stamp_built
>     %/.stamp_host_installed
>     %/.stamp_staging_installed
>     %/.stamp_images_installed
>     %/.stamp_target_installed

  Your description here is not entirely accurate, because this makes it 
look as if the dependencies are added for the pattern rules.

>
> Because the %-build target is not anymore part of the dependency chain,
> add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead.
> This new variable is used only by uclibc and glibc packages.
>
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
> ---
>   package/glibc/glibc.mk   |  2 +-
>   package/pkg-generic.mk   | 38 +++++++++++++++++++++-----------------
>   package/uclibc/uclibc.mk |  2 +-
>   3 files changed, 23 insertions(+), 19 deletions(-)
>
> diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
> index 68b98a9..e89c12a 100644
> --- a/package/glibc/glibc.mk
> +++ b/package/glibc/glibc.mk
> @@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
>   GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
>
>   # Before (e)glibc is built, we must have the second stage cross-compiler
> -glibc-build: host-gcc-intermediate
> +GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate

  I don't like this. (You'll notice I generally don't like adding new 
variables :-). This build-dependency is really an exceptional situation, 
so adding generic package infrastructure for it feels wrong to me. 
Instead, you can keep the explicit dependency that was there already, 
only now it is even more explicit:

$(GLIBC_TARGET_BUILD): host-gcc-intermediate

  Note that GLIBC_TARGET_BUILD is only defined after the
$(eval $(autotools-package)) so it will have to be moved down as well.


>
>   GLIBC_SUBDIR = build
>
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 3f19aea..094868c 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
>   ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
>   $(2)_DEPENDENCIES += toolchain
>   endif
> +$(2)_BUILD_DEPENDENCIES		?=
>
>   $(2)_INSTALL_STAGING		?= NO
>   $(2)_INSTALL_IMAGES		?= NO
> @@ -368,30 +369,34 @@ $(1)-install:		$(1)-install-staging $(1)-install-target $(1)-install-images
>   endif
>

  I would like to see somewhere a comment explaining how the dependencies 
between the steps are handled. In particular, explaining that this is not 
done in the pattern rules because then make would remove the stamp files 
and do too much rebuilding.

>   ifeq ($$($(2)_INSTALL_TARGET),YES)
> -$(1)-install-target:	$(1)-build \
> -			$$($(2)_TARGET_INSTALL_TARGET)
> +$(1)-install-target:	$$($(2)_TARGET_INSTALL_TARGET)
>   else
>   $(1)-install-target:
>   endif
>
>   ifeq ($$($(2)_INSTALL_STAGING),YES)
> -$(1)-install-staging:	$(1)-build \
> -			$$($(2)_TARGET_INSTALL_STAGING)
> +$(1)-install-staging:	$$($(2)_TARGET_INSTALL_STAGING)
>   else
>   $(1)-install-staging:
>   endif
>
>   ifeq ($$($(2)_INSTALL_IMAGES),YES)
> -$(1)-install-images:	$(1)-build \
> -			$$($(2)_TARGET_INSTALL_IMAGES)
> +$(1)-install-images:	$$($(2)_TARGET_INSTALL_IMAGES)
>   else
>   $(1)-install-images:
>   endif
>
> -$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
> +$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
>
> -$(1)-build:		$(1)-configure \
> -			$$($(2)_TARGET_BUILD)
> +$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
> +$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
> +	$$($(2)_TARGET_BUILD)

  I think it would also be possible to turn this into a dependency on 
$(1)-build instead of $$($(2)_TARGET_BUILD). That would also avoid the 
need for _BUILD_DEPENDENCIES and maybe some other changes.

  Of course, to be able to do that, you have to add
.PHONY: $(1)-build
otherwise everything will always be rebuilt.


> +
> +$(1)-build:		$$($(2)_TARGET_BUILD)
> +$$($(2)_TARGET_BUILD):	$$($(2)_TARGET_CONFIGURE) | $$($(2)_BUILD_DEPENDENCIES)
> +
> +$(1)-configure:			$$($(2)_TARGET_CONFIGURE)
> +$$($(2)_TARGET_CONFIGURE):	| $$($(2)_DEPENDENCIES)

  This warrants a comment explaining why you use | here.

>
>   $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
>
> @@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
>   #  extract
>   #  patch
>   #  configure
> -$(1)-configure:		$(1)-patch $(1)-depends \
> -			$$($(2)_TARGET_CONFIGURE)
> +$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_PATCH)
>
> -$(1)-patch:		$(1)-extract $$($(2)_TARGET_PATCH)
> +$(1)-patch:		$$($(2)_TARGET_PATCH)
> +$$($(2)_TARGET_PATCH):	$$($(2)_TARGET_EXTRACT)
>
> -$(1)-extract:		$(1)-source \
> -			$$($(2)_TARGET_EXTRACT)
> +$(1)-extract:			$$($(2)_TARGET_EXTRACT)
> +$$($(2)_TARGET_EXTRACT):	$$($(2)_TARGET_SOURCE)
>
>   $(1)-depends:		$$($(2)_DEPENDENCIES)

  Is the -depends target still useful? Does anybody ever run 'make 
foo-depends'?


  Regards,
  Arnout

>
> @@ -418,10 +423,9 @@ else
>   #  source, by rsyncing
>   #  depends
>   #  configure
> -$(1)-configure:		$(1)-depends \
> -			$$($(2)_TARGET_CONFIGURE)
> +$$($(2)_TARGET_CONFIGURE):	$$($(2)_TARGET_RSYNC)
>
> -$(1)-depends:		$(1)-rsync $$($(2)_DEPENDENCIES)
> +$(1)-depends:		$$($(2)_DEPENDENCIES)
>
>   $(1)-patch:		$(1)-rsync
>   $(1)-extract:		$(1)-rsync
> diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
> index 0d12afd..dea36e7 100644
> --- a/package/uclibc/uclibc.mk
> +++ b/package/uclibc/uclibc.mk
> @@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
>   UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
>
>   # Before uClibc is built, we must have the second stage cross-compiler
> -uclibc-build: host-gcc-intermediate
> +UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
>
>   # specifying UCLIBC_CONFIG_FILE on the command-line overrides the .config
>   # setting.
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 4/8] Makefile: add support for top-level parallel make

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> To be able to use top-level parallel make we must not depend in a rule
> on the order of evaluation of the prerequisites, so instead of relyng on
> the left to right ordering of evaluation of the prerequisites add an
> explicit rule to describe the dependencies.
>
> Add explicit rules to describe the following dependency chain:
> $(TARGETS) -> target-finalize -> rootfs-* -> target-post-image
>
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

  I think this one is ready to go in, so perhaps you can reorder it as 
one of the first ones of the series. Although it serves no function 
except for the top-level parallel make, so perhaps Peter will object to 
committing it on its own.

  Regards,
  Arnout

> ---
>   Makefile     | 15 ++++++---------
>   fs/common.mk |  4 ++--
>   2 files changed, 8 insertions(+), 11 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index ebecec4..299d34b 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -329,8 +329,6 @@ include boot/common.mk
>   include linux/linux.mk
>   include system/system.mk
>
> -TARGETS+=target-finalize
> -
>   ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
>   TARGETS+=target-purgelocales
>   endif
> @@ -347,8 +345,6 @@ endif
>
>   include fs/common.mk
>
> -TARGETS+=target-post-image
> -
>   TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
>   TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
>   TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
> @@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf: $(BUILDROOT_CONFIG)
>
>   prepare: $(BUILD_DIR)/buildroot-config/auto.conf
>
> -world: $(TARGETS)
> +world: target-post-image
>
>   .PHONY: all world toolchain dirs clean distclean source outputmakefile \
>   	legal-info legal-info-prepare legal-info-clean printvars \
> -	$(BASE_TARGETS) $(TARGETS) \
> +	target-finalize target-post-image \
> +	$(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
>   	$(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE) $(TARGETS_LEGAL_INFO) \
>   	$(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
>   	$(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
> @@ -443,7 +440,7 @@ endif
>   STRIP_FIND_CMD += -type f -perm /111
>   STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
>
> -target-finalize:
> +target-finalize: $(TARGETS)
>   	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
>   		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
>   		$(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
> @@ -550,7 +547,7 @@ target-generatelocales: host-localedef
>   	done
>   endif
>
> -target-post-image:
> +target-post-image: target-finalize $(TARGETS_ROOTFS)
>   	@$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
>   		$(call MESSAGE,"Executing post-image script $(s)"); \
>   		$(s) $(BINARIES_DIR) $(call qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
> @@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(REDIST_SOURCES_DIR) \
>   	@rm -f $(LEGAL_WARNINGS)
>
>   show-targets:
> -	@echo $(TARGETS)
> +	@echo $(TARGETS) $(TARGETS_ROOTFS)
>
>   else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>
> diff --git a/fs/common.mk b/fs/common.mk
> index 4dab7ea..7698c53 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
>   # extra deps
>   ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
>
> -$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
> +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
>   	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
>   	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
>   	rm -f $$(FAKEROOT_SCRIPT)
> @@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
>   rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
>
>   ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
> -TARGETS += rootfs-$(1)
> +TARGETS_ROOTFS += rootfs-$(1)
>   endif
>   endef
>
>


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 4/8] Makefile: add support for top-level parallel make

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> diff --git a/fs/common.mk b/fs/common.mk
> index 4dab7ea..7698c53 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
>   # extra deps
>   ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
>
> -$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
> +$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
>   	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
>   	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
>   	rm -f $$(FAKEROOT_SCRIPT)
> @@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
>   rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
>
>   ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
> -TARGETS += rootfs-$(1)
> +TARGETS_ROOTFS += rootfs-$(1)
>   endif
>   endef

  Oh I forgot: you should also fix up initramfs and iso9660. They add 
themselves to TARGETS manually.

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 5/8] glibc: add support for top-level parallel make

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> To be able to use top-level parallel make we must not depend in a rule
> on the order of evaluation of the prerequisites, so instead of relyng on
> the left to right ordering of evaluation of the prerequisites add
> an explicit rule to describe the dependencies.
>
> The glibc-install-target depends on glibc-install-staging so add a rule
> for it.
>
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

> ---
>   package/glibc/glibc.mk | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
> index e89c12a..838fb8c 100644
> --- a/package/glibc/glibc.mk
> +++ b/package/glibc/glibc.mk
> @@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS
>   endef
>
>   $(eval $(autotools-package))
> +
> +$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)
>

  Nitpick: I would have added a comment why this is needed.

  Regards,
  Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 6/8] uclibc: add support for top-level parallel make

[Arnout Vandecappelle]

On 18/10/13 11:34, Fabio Porcedda wrote:
> To be able to use top-level parallel make we must not depend in a rule
> on the order of evaluation of the prerequisites, so instead of relyng on
> the left to right ordering of evaluation of the prerequisites add an
> explicit rule to describe the dependencies.
>
> The uclibc-install-target depends on uclibc-install-staging so add a
> rule for it.
>
> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>

Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

> ---
>   package/uclibc/uclibc.mk | 2 ++
>   1 file changed, 2 insertions(+)
>
> diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
> index dea36e7..c8eeb3d 100644
> --- a/package/uclibc/uclibc.mk
> +++ b/package/uclibc/uclibc.mk
> @@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch
>   	rm -f $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
>
>   $(eval $(generic-package))
> +
> +$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)
>

  Same nitpick :-)

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Fabio Porcedda]

Hi Arnout,
thanks for reviewing the patch set.

On Wed, Oct 23, 2013 at 11:12 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/10/13 11:34, Fabio Porcedda wrote:
>>
>> Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
>> every package.
>> This way we can build correctly every package right after the clean
>> stage.
>> As example with this commit we can build successfully the glibc right
>> after the clean stage:
>>         make clean glibc
>>
>> This is also a step forward supporting top-level parallel make.
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>
>
>  Although this is one of the most useful patches of the series, it is also
> the one that introduces the circular dependency. So although it looks good,
> I'm not ready to ack it.

I've found a work around:

$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare

# to prevent circular dependency
ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
endif

What do you think about that?

>
>> ---
>>   package/pkg-generic.mk           | 2 ++
>>   toolchain/toolchain/toolchain.mk | 3 +--
>>   2 files changed, 3 insertions(+), 2 deletions(-)
>>
>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>> index 4bba4b5..1e7154e 100644
>> --- a/package/pkg-generic.mk
>> +++ b/package/pkg-generic.mk
>> @@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build
>> $$($(2)_TARGET_INSTALL_HOST)
>>   $(1)-build:           $(1)-configure \
>>                         $$($(2)_TARGET_BUILD)
>>
>> +$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
>
>
>  Is there any reason why you changed the order here?

The order whit top-level parallel make does not matter, it's just that
i was thinking that it's the right logical order:
1: dependencies: requisite
2: dirs: create empty directories
3. prepare: add something to the empty directories

But now i understand that is "dirs prepare dependencies" or "dirs
dependencies prepare" because dirs can require host-ccache and so
directories,
nevertheless this does not matter anymore.

>
>  Regards,
>  Arnout
>
>
>> +
>>   ifeq ($$($(2)_OVERRIDE_SRCDIR),)
>>   # In the normal case (no package override), the sequence of steps is
>>   #  source, by downloading
>> diff --git a/toolchain/toolchain/toolchain.mk
>> b/toolchain/toolchain/toolchain.mk
>> index 44ed629..8559ac9 100644
>> --- a/toolchain/toolchain/toolchain.mk
>> +++ b/toolchain/toolchain/toolchain.mk
>> @@ -14,5 +14,4 @@ endif
>>
>>   $(eval $(generic-package))
>>
>> -toolchain-source: prepare dirs dependencies
>> $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
>> -
>> +toolchain: $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake
>>
>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Thomas De Schampheleire]

Hi,

On Thu, Oct 24, 2013 at 9:41 AM, Fabio Porcedda
<fabio.porcedda@gmail.com> wrote:
>>
>>
>>  Although this is one of the most useful patches of the series, it is also
>> the one that introduces the circular dependency. So although it looks good,
>> I'm not ready to ack it.
>
> I've found a work around:
>
> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
>
> # to prevent circular dependency
> ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
> endif
>
> What do you think about that?
>

I don't think this will work if there are more than one prerequisite:
$(1) will be one of them, and DEPENDENCIES_HOST_PREREQ will be the
entire list, so that 'ifneq' will always be true.
I think you need 'filter' here.
http://www.gnu.org/software/make/manual/make.html#index-filter-587

Best regards,
Thomas

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Arnout Vandecappelle]

On 24/10/13 09:41, Fabio Porcedda wrote:
> Hi Arnout,
> thanks for reviewing the patch set.
>
> On Wed, Oct 23, 2013 at 11:12 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
>> On 18/10/13 11:34, Fabio Porcedda wrote:
>>>
>>> Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
>>> every package.
>>> This way we can build correctly every package right after the clean
>>> stage.
>>> As example with this commit we can build successfully the glibc right
>>> after the clean stage:
>>>          make clean glibc
>>>
>>> This is also a step forward supporting top-level parallel make.
>>>
>>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>>
>>
>>   Although this is one of the most useful patches of the series, it is also
>> the one that introduces the circular dependency. So although it looks good,
>> I'm not ready to ack it.
>
> I've found a work around:
>
> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
>
> # to prevent circular dependency
> ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
> endif
>
> What do you think about that?

  Looks good except that it's incorrect :-) If HOST_PREREQ contains both 
ccache and sstrip (not to mention tar or xz), the condition won't match. 
You can try something like:

ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)


  [Just noticed now that Thomas gave the same answer.]

>
>>
>>> ---
>>>    package/pkg-generic.mk           | 2 ++
>>>    toolchain/toolchain/toolchain.mk | 3 +--
>>>    2 files changed, 3 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>>> index 4bba4b5..1e7154e 100644
>>> --- a/package/pkg-generic.mk
>>> +++ b/package/pkg-generic.mk
>>> @@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build
>>> $$($(2)_TARGET_INSTALL_HOST)
>>>    $(1)-build:           $(1)-configure \
>>>                          $$($(2)_TARGET_BUILD)
>>>
>>> +$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs prepare
>>
>>
>>   Is there any reason why you changed the order here?
>
> The order whit top-level parallel make does not matter, it's just that
> i was thinking that it's the right logical order:
> 1: dependencies: requisite
> 2: dirs: create empty directories
> 3. prepare: add something to the empty directories
>
> But now i understand that is "dirs prepare dependencies" or "dirs
> dependencies prepare" because dirs can require host-ccache and so

  You mean dependencies can require host-ccache, right? And host-ccache 
depends on both dirs and prepare itself...  So I do think the original 
order is more logical.

  Regards,
  Arnout

> directories,
> nevertheless this does not matter anymore.

[snip]


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Fabio Porcedda]

On Thu, Oct 24, 2013 at 12:37 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 24/10/13 09:41, Fabio Porcedda wrote:
>>
>> Hi Arnout,
>> thanks for reviewing the patch set.
>>
>> On Wed, Oct 23, 2013 at 11:12 PM, Arnout Vandecappelle <arnout@mind.be>
>> wrote:
>>>
>>> On 18/10/13 11:34, Fabio Porcedda wrote:
>>>>
>>>>
>>>> Move "dependencies" "dirs" "prepare" dependencies from "toolchain" to
>>>> every package.
>>>> This way we can build correctly every package right after the clean
>>>> stage.
>>>> As example with this commit we can build successfully the glibc right
>>>> after the clean stage:
>>>>          make clean glibc
>>>>
>>>> This is also a step forward supporting top-level parallel make.
>>>>
>>>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>>>
>>>
>>>
>>>   Although this is one of the most useful patches of the series, it is
>>> also
>>> the one that introduces the circular dependency. So although it looks
>>> good,
>>> I'm not ready to ack it.
>>
>>
>> I've found a work around:
>>
>> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
>>
>> # to prevent circular dependency
>> ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
>> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
>> endif
>>
>> What do you think about that?
>
>
>  Looks good except that it's incorrect :-) If HOST_PREREQ contains both
> ccache and sstrip (not to mention tar or xz), the condition won't match. You
> can try something like:
>
> ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)

Ok good, I've updated the patch according.

>
>  [Just noticed now that Thomas gave the same answer.]
>
>
>>
>>>
>>>> ---
>>>>    package/pkg-generic.mk           | 2 ++
>>>>    toolchain/toolchain/toolchain.mk | 3 +--
>>>>    2 files changed, 3 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>>>> index 4bba4b5..1e7154e 100644
>>>> --- a/package/pkg-generic.mk
>>>> +++ b/package/pkg-generic.mk
>>>> @@ -385,6 +385,8 @@ $(1)-install-host:      $(1)-build
>>>> $$($(2)_TARGET_INSTALL_HOST)
>>>>    $(1)-build:           $(1)-configure \
>>>>                          $$($(2)_TARGET_BUILD)
>>>>
>>>> +$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs
>>>> prepare
>>>
>>>
>>>
>>>   Is there any reason why you changed the order here?
>>
>>
>> The order whit top-level parallel make does not matter, it's just that
>> i was thinking that it's the right logical order:
>> 1: dependencies: requisite
>> 2: dirs: create empty directories
>> 3. prepare: add something to the empty directories
>>
>> But now i understand that is "dirs prepare dependencies" or "dirs
>> dependencies prepare" because dirs can require host-ccache and so
>
>
>  You mean dependencies can require host-ccache, right? And host-ccache
> depends on both dirs and prepare itself...  So I do think the original order
> is more logical.

Yes the final order is this: dirs prepare dependencies.

I moved dirs in front because i think creating directories it's the
first thing to do.

>  Regards,
>  Arnout
>
>
>> directories,
>> nevertheless this does not matter anymore.
>
>
> [snip]
>
>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F


Thanks and regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Fabio Porcedda]

On Thu, Oct 24, 2013 at 10:22 AM, Thomas De Schampheleire
<patrickdepinguin@gmail.com> wrote:
> Hi,
>
> On Thu, Oct 24, 2013 at 9:41 AM, Fabio Porcedda
> <fabio.porcedda@gmail.com> wrote:
>>>
>>>
>>>  Although this is one of the most useful patches of the series, it is also
>>> the one that introduces the circular dependency. So although it looks good,
>>> I'm not ready to ack it.
>>
>> I've found a work around:
>>
>> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
>>
>> # to prevent circular dependency
>> ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
>> $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
>> endif
>>
>> What do you think about that?
>>
>
> I don't think this will work if there are more than one prerequisite:
> $(1) will be one of them, and DEPENDENCIES_HOST_PREREQ will be the
> entire list, so that 'ifneq' will always be true.
> I think you need 'filter' here.
> http://www.gnu.org/software/make/manual/make.html#index-filter-587

Yes it works fine with filter, thanks for suggesting it.


> Best regards,
> Thomas


Thanks and regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Arnout Vandecappelle]

On 25/10/13 10:07, Fabio Porcedda wrote:
> On Thu, Oct 24, 2013 at 12:37 PM, Arnout Vandecappelle<arnout@mind.be>  wrote:
>> >On 24/10/13 09:41, Fabio Porcedda wrote:
>>> >>
[snip]
>>> >>I've found a work around:
>>> >>
>>> >>$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
>>> >>
>>> >># to prevent circular dependency
>>> >>ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
>>> >>$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
>>> >>endif
>>> >>
>>> >>What do you think about that?
>> >
>> >
>> >  Looks good except that it's incorrect:-)  If HOST_PREREQ contains both
>> >ccache and sstrip (not to mention tar or xz), the condition won't match. You
>> >can try something like:
>> >
>> >ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
> Ok good, I've updated the patch according.
>

  Have you tested it with various combinations of dependencies?

  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Fabio Porcedda]

On Fri, Oct 25, 2013 at 10:12 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 25/10/13 10:07, Fabio Porcedda wrote:
>>
>> On Thu, Oct 24, 2013 at 12:37 PM, Arnout Vandecappelle<arnout@mind.be>
>> wrote:
>>>
>>> >On 24/10/13 09:41, Fabio Porcedda wrote:
>>>>
>>>> >>
>
> [snip]
>
>>>> >>I've found a work around:
>>>> >>
>>>> >>$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
>>>> >>
>>>> >># to prevent circular dependency
>>>> >>ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
>>>> >>$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
>>>> >>endif
>>>> >>
>>>> >>What do you think about that?
>>>
>>> >
>>> >
>>> >  Looks good except that it's incorrect:-)  If HOST_PREREQ contains both
>>> >ccache and sstrip (not to mention tar or xz), the condition won't match.
>>> > You
>>> >can try something like:
>>> >
>>> >ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
>>
>> Ok good, I've updated the patch according.
>>
>
>  Have you tested it with various combinations of dependencies?

I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with
qemu_x86_defconfig and seems works fine.
The circular dependencies are gone.
I think the patch now does not introduce any regression.

I'm testing the whole patch set with top-level parallel make and
enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".

Thanks and regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 2/8] package: add toolchain dependency to every target package

[Thomas Petazzoni]

Dear Arnout Vandecappelle,

On Wed, 23 Oct 2013 23:53:33 +0200, Arnout Vandecappelle wrote:

>   What I would prefer is to move glibc, linux-headers and uclibc back 
> under the toolchain directory (why did they ever move out, Thomas?) and 
> to filter on $(pkgparentdir)

Well, the idea is that gcc, uclibc, glibc, linux-headers and so on are
now real, normal, packages. The only thing we keep in toolchain/ are
the "entry" point of toolchain backends: toolchain-buildroot and
toolchain-external.

Moreover, it would be strange to have glibc, uclibc and linux-headers
in toolchain/, but not gcc and binutils, for example. I do understand
that these two packages are host packages, so they don't get the
automatic dependency on the toolchain added, but that's an internal
detail. From an external point of view, all those components are part
of the toolchain, so it would be weird to have some of them in
toolchain/, and some of them in package/.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 RESEND 2/8] package: add toolchain dependency to every target package

[Arnout Vandecappelle]

On 27/10/13 18:55, Thomas Petazzoni wrote:
> Dear Arnout Vandecappelle,
>
> On Wed, 23 Oct 2013 23:53:33 +0200, Arnout Vandecappelle wrote:
>
>>    What I would prefer is to move glibc, linux-headers and uclibc back
>> under the toolchain directory (why did they ever move out, Thomas?) and
>> to filter on $(pkgparentdir)
>
> Well, the idea is that gcc, uclibc, glibc, linux-headers and so on are
> now real, normal, packages. The only thing we keep in toolchain/ are
> the "entry" point of toolchain backends: toolchain-buildroot and
> toolchain-external.
>
> Moreover, it would be strange to have glibc, uclibc and linux-headers
> in toolchain/, but not gcc and binutils, for example. I do understand
> that these two packages are host packages, so they don't get the
> automatic dependency on the toolchain added, but that's an internal
> detail. From an external point of view, all those components are part
> of the toolchain, so it would be weird to have some of them in
> toolchain/, and some of them in package/.

  So what's your conclusion now? Move all toolchain-related packages back 
to the toolchain directory, or find another solution for circumventing 
the toolchain circular dependency?


  Regards,
  Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Fabio Porcedda]

On Fri, Oct 25, 2013 at 10:45 AM, Fabio Porcedda
<fabio.porcedda@gmail.com> wrote:
> On Fri, Oct 25, 2013 at 10:12 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
>> On 25/10/13 10:07, Fabio Porcedda wrote:
>>>
>>> On Thu, Oct 24, 2013 at 12:37 PM, Arnout Vandecappelle<arnout@mind.be>
>>> wrote:
>>>>
>>>> >On 24/10/13 09:41, Fabio Porcedda wrote:
>>>>>
>>>>> >>
>>
>> [snip]
>>
>>>>> >>I've found a work around:
>>>>> >>
>>>>> >>$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dirs prepare
>>>>> >>
>>>>> >># to prevent circular dependency
>>>>> >>ifneq ($(1),$(DEPENDENCIES_HOST_PREREQ))
>>>>> >>$$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies
>>>>> >>endif
>>>>> >>
>>>>> >>What do you think about that?
>>>>
>>>> >
>>>> >
>>>> >  Looks good except that it's incorrect:-)  If HOST_PREREQ contains both
>>>> >ccache and sstrip (not to mention tar or xz), the condition won't match.
>>>> > You
>>>> >can try something like:
>>>> >
>>>> >ifeq ($(filter $(1),$(DEPENDENCIES_HOST_PREREQ)),)
>>>
>>> Ok good, I've updated the patch according.
>>>
>>
>>  Have you tested it with various combinations of dependencies?
>
> I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with
> qemu_x86_defconfig and seems works fine.
> The circular dependencies are gone.
> I think the patch now does not introduce any regression.
>
> I'm testing the whole patch set with top-level parallel make and
> enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".

The update patch seems to works fine.
I found only an issue when enabling top-level parallel make and
disabling /usr/bin/xz,
but that is not a regression and it's an issue to solve with another patch.

Regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Thomas De Schampheleire]

Hi Fabio,

On Tue, Oct 29, 2013 at 9:36 AM, Fabio Porcedda
<fabio.porcedda@gmail.com> wrote:
[..]
>>>  Have you tested it with various combinations of dependencies?
>>
>> I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with
>> qemu_x86_defconfig and seems works fine.
>> The circular dependencies are gone.
>> I think the patch now does not introduce any regression.
>>
>> I'm testing the whole patch set with top-level parallel make and
>> enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".
>
> The update patch seems to works fine.
> I found only an issue when enabling top-level parallel make and
> disabling /usr/bin/xz,
> but that is not a regression and it's an issue to solve with another patch.
>

Can you elaborate on this? What is the problem you're seeing exactly,
and how did you 'disable /usr/bin/xz' ?
When buildroot cannot find a working xzcat version, it builds one
itself through host-xz. This should still work in case of parallel
builds.
The same principle applies when the 'tar' version on host is
non-existing or too old.

Thanks,
Thomas

[Buildroot] [PATCH v8 RESEND 1/8] package: add base dependency to every package

[Fabio Porcedda]

On Tue, Oct 29, 2013 at 10:35 AM, Thomas De Schampheleire
<patrickdepinguin@gmail.com> wrote:
> Hi Fabio,
>
> On Tue, Oct 29, 2013 at 9:36 AM, Fabio Porcedda
> <fabio.porcedda@gmail.com> wrote:
> [..]
>>>>  Have you tested it with various combinations of dependencies?
>>>
>>> I've tested enabling "BR2_CCACHE" and "BR2_STRIP_sstrip" with
>>> qemu_x86_defconfig and seems works fine.
>>> The circular dependencies are gone.
>>> I think the patch now does not introduce any regression.
>>>
>>> I'm testing the whole patch set with top-level parallel make and
>>> enabling "BR2_CCACHE" and "BR2_STRIP_sstrip".
>>
>> The update patch seems to works fine.
>> I found only an issue when enabling top-level parallel make and
>> disabling /usr/bin/xz,
>> but that is not a regression and it's an issue to solve with another patch.
>>
>
> Can you elaborate on this? What is the problem you're seeing exactly,
> and how did you 'disable /usr/bin/xz' ?
To disable /usr/bin/xz i've just renamed it so buildroot doesn't find
a suitable xz.
The host-ccache source file is a tar.xz so the host-xz is a dependency
of host-ccache.
I need to add a rule to ensure the right order.

> When buildroot cannot find a working xzcat version, it builds one
> itself through host-xz. This should still work in case of parallel
> builds.
> The same principle applies when the 'tar' version on host is
> non-existing or too old.

Yes i think is affected by the same problem.

Thanks
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 2/8] package: add toolchain dependency to every target package

[Fabio Porcedda]

On Wed, Oct 23, 2013 at 11:53 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/10/13 11:34, Fabio Porcedda wrote:
>>
>> This commit makes the dependency from the target toolchain explicit.
>> This way we can buid from command line a package that use
>> innger-generic-package right after the configuration phase, example:
>>
>>         make clean <package-name>
>>
>> Also remove TARGETS_ALL because the only purpose was to add toolchain
>> dependency so it's superseded by this commit.
>>
>> To prevent circular dependency add the new variable <pkgname>_TOOLCHAIN
>> for not adding the toolchain dependency for toolchain packages.
>>
>> This is also a step forward supporting top-level parallel make.
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>
>
>  I'm not happy with this _ADD_TOOLCHAIN_DEPENDENCY variable.
>
>  What I would prefer is to move glibc, linux-headers and uclibc back under
> the toolchain directory (why did they ever move out, Thomas?) and to filter
> on $(pkgparentdir)
>
> ifeq ($(5),target)
> ifneq ($(pkgparentdir),toolchain)
> $(2)_DEPENDENCIES += toolchain
> endif
> endif
>
>
> [snip]
>
>> diff --git a/package/pkg-autotools.mk b/package/pkg-autotools.mk
>> index 9523529..f58570d 100644
>> --- a/package/pkg-autotools.mk
>> +++ b/package/pkg-autotools.mk
>> @@ -209,7 +209,8 @@ endef
>>   # This must be repeated from inner-generic-package, otherwise we get an
>> empty
>>   # _DEPENDENCIES if _AUTORECONF is YES.  Also filter the result of
>> _AUTORECONF
>>   # away from the non-host rule
>> -$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf
>> host-libtool $(1),\
>> +$(2)_DEPENDENCIES ?= $(filter-out host-automake host-autoconf
>> host-libtool \
>> +                               host-toolchain $(1),\
>
>
>  You'll also need to do this in pkg-cmake.mk.

Ok, done.

>
>>       $(patsubst host-host-%,host-%,$(addprefix
>> host-,$($(3)_DEPENDENCIES))))
>>
>>
>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>> index 1e7154e..3f19aea 100644
>> --- a/package/pkg-generic.mk
>> +++ b/package/pkg-generic.mk
>> @@ -307,8 +307,16 @@ endif
>>
>>   $(2)_REDISTRIBUTE             ?= YES
>>
>> -
>> -$(2)_DEPENDENCIES ?= $(filter-out $(1),$(patsubst
>> host-host-%,host-%,$(addprefix host-,$($(3)_DEPENDENCIES))))
>> +# When a target package is a toolchain dependency set this variable to
>> +# 'NO' so the 'toolchain' dependency is not added to prevent a circular
>> +# dependency
>> +$(2)_ADD_TOOLCHAIN_DEPENDENCY  ?= YES
>> +
>> +$(2)_DEPENDENCIES ?= $(filter-out  host-toolchain $(1),\
>> +       $(patsubst host-host-%,host-%,$(addprefix
>> host-,$($(3)_DEPENDENCIES))))
>> +ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
>
>
>  It's more readable if you split this and-ing in two separate conditions.
>
>  For $(2)_TYPE, you can directly use $(5) because you're anyway inside
> inner-generic-package here.

Ok, done.


Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 3/8] package: add support for top-level parallel make

[Fabio Porcedda]

On Thu, Oct 24, 2013 at 12:19 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/10/13 11:34, Fabio Porcedda wrote:
>>
>> To be able to use top-level parallel make we must not depend in a rule
>> on the order of evaluation of the prerequisites, so instead of relyng on
>
>
>  nitpick: relying

Ok, done.

>
>> the left to right ordering of evaluation of the prerequisites add an
>> explicit rule to describe the dependencies.
>> So add explicit dependencies for the following stamp files:
>>     %/.stamp_extracted
>>     %/.stamp_patched
>>     %/.stamp_configured
>>     %/.stamp_built
>>     %/.stamp_host_installed
>>     %/.stamp_staging_installed
>>     %/.stamp_images_installed
>>     %/.stamp_target_installed
>
>
>  Your description here is not entirely accurate, because this makes it look
> as if the dependencies are added for the pattern rules.

Ok, I'll use $(2)_TARGET_%

>>
>> Because the %-build target is not anymore part of the dependency chain,
>> add a new variable <pkgname>_BUILD_DEPENDENCIES to be used instead.
>> This new variable is used only by uclibc and glibc packages.
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>> ---
>>   package/glibc/glibc.mk   |  2 +-
>>   package/pkg-generic.mk   | 38 +++++++++++++++++++++-----------------
>>   package/uclibc/uclibc.mk |  2 +-
>>   3 files changed, 23 insertions(+), 19 deletions(-)
>>
>> diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
>> index 68b98a9..e89c12a 100644
>> --- a/package/glibc/glibc.mk
>> +++ b/package/glibc/glibc.mk
>> @@ -27,7 +27,7 @@ GLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
>>   GLIBC_DEPENDENCIES = host-gcc-initial linux-headers host-gawk
>>
>>   # Before (e)glibc is built, we must have the second stage cross-compiler
>> -glibc-build: host-gcc-intermediate
>> +GLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
>
>
>  I don't like this. (You'll notice I generally don't like adding new
> variables :-). This build-dependency is really an exceptional situation, so
> adding generic package infrastructure for it feels wrong to me. Instead, you
> can keep the explicit dependency that was there already, only now it is even
> more explicit:
>
> $(GLIBC_TARGET_BUILD): host-gcc-intermediate
>
>  Note that GLIBC_TARGET_BUILD is only defined after the
> $(eval $(autotools-package)) so it will have to be moved down as well.
>

Ok, done.


>>
>>   GLIBC_SUBDIR = build
>>
>> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
>> index 3f19aea..094868c 100644
>> --- a/package/pkg-generic.mk
>> +++ b/package/pkg-generic.mk
>> @@ -317,6 +317,7 @@ $(2)_DEPENDENCIES ?= $(filter-out  host-toolchain
>> $(1),\
>>   ifeq ($$($(2)_TYPE)-$$($(2)_ADD_TOOLCHAIN_DEPENDENCY),target-YES)
>>   $(2)_DEPENDENCIES += toolchain
>>   endif
>> +$(2)_BUILD_DEPENDENCIES                ?=
>>
>>   $(2)_INSTALL_STAGING          ?= NO
>>   $(2)_INSTALL_IMAGES           ?= NO
>> @@ -368,30 +369,34 @@ $(1)-install:             $(1)-install-staging
>> $(1)-install-target $(1)-install-images
>>   endif
>>
>
>  I would like to see somewhere a comment explaining how the dependencies
> between the steps are handled. In particular, explaining that this is not
> done in the pattern rules because then make would remove the stamp files and
> do too much rebuilding.

Do you mean to add a comment in the description of the commit?
My explanation is:

We cannot use the pattern rules because they must have the same
dependency for every package,
but we need to change the dependencies depending on
$(2)_OVERRIDE_SRCDIR variable value,
so we must use a more flexible way like $(2)_TARGET_% variables.

>
>>   ifeq ($$($(2)_INSTALL_TARGET),YES)
>> -$(1)-install-target:   $(1)-build \
>> -                       $$($(2)_TARGET_INSTALL_TARGET)
>> +$(1)-install-target:   $$($(2)_TARGET_INSTALL_TARGET)
>>   else
>>   $(1)-install-target:
>>   endif
>>
>>   ifeq ($$($(2)_INSTALL_STAGING),YES)
>> -$(1)-install-staging:  $(1)-build \
>> -                       $$($(2)_TARGET_INSTALL_STAGING)
>> +$(1)-install-staging:  $$($(2)_TARGET_INSTALL_STAGING)
>>   else
>>   $(1)-install-staging:
>>   endif
>>
>>   ifeq ($$($(2)_INSTALL_IMAGES),YES)
>> -$(1)-install-images:   $(1)-build \
>> -                       $$($(2)_TARGET_INSTALL_IMAGES)
>> +$(1)-install-images:   $$($(2)_TARGET_INSTALL_IMAGES)
>>   else
>>   $(1)-install-images:
>>   endif
>>
>> -$(1)-install-host:      $(1)-build $$($(2)_TARGET_INSTALL_HOST)
>> +$(1)-install-host:      $$($(2)_TARGET_INSTALL_HOST)
>>
>> -$(1)-build:            $(1)-configure \
>> -                       $$($(2)_TARGET_BUILD)
>> +$$($(2)_TARGET_INSTALL_TARGET) $$($(2)_TARGET_INSTALL_STAGING) \
>> +$$($(2)_TARGET_INSTALL_IMAGES) $$($(2)_TARGET_INSTALL_HOST): \
>> +       $$($(2)_TARGET_BUILD)
>
>
>  I think it would also be possible to turn this into a dependency on
> $(1)-build instead of $$($(2)_TARGET_BUILD). That would also avoid the need
> for _BUILD_DEPENDENCIES and maybe some other changes.

That's not possible because the $$($(2)_TARGET_BUILD) must depends on
$$($(2)_BUILD_DEPENDENCIES) because
if build depends on $$($(2)_TARGET_BUILD) and the contents of
$$($(2)_BUILD_DEPENDENCIES) they can be built in parallel and that's
is wrong.

>  Of course, to be able to do that, you have to add
> .PHONY: $(1)-build
> otherwise everything will always be rebuilt.
>
>
>
>> +
>> +$(1)-build:            $$($(2)_TARGET_BUILD)
>> +$$($(2)_TARGET_BUILD): $$($(2)_TARGET_CONFIGURE) |
>> $$($(2)_BUILD_DEPENDENCIES)
>> +
>> +$(1)-configure:                        $$($(2)_TARGET_CONFIGURE)
>> +$$($(2)_TARGET_CONFIGURE):     | $$($(2)_DEPENDENCIES)
>
>
>  This warrants a comment explaining why you use | here.

Ok.

>
>>
>>   $$($(2)_TARGET_SOURCE) $$($(2)_TARGET_RSYNC): | dependencies dirs
>> prepare
>>
>> @@ -402,13 +407,13 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
>>   #  extract
>>   #  patch
>>   #  configure
>> -$(1)-configure:                $(1)-patch $(1)-depends \
>> -                       $$($(2)_TARGET_CONFIGURE)
>> +$$($(2)_TARGET_CONFIGURE):     $$($(2)_TARGET_PATCH)
>>
>> -$(1)-patch:            $(1)-extract $$($(2)_TARGET_PATCH)
>> +$(1)-patch:            $$($(2)_TARGET_PATCH)
>> +$$($(2)_TARGET_PATCH): $$($(2)_TARGET_EXTRACT)
>>
>> -$(1)-extract:          $(1)-source \
>> -                       $$($(2)_TARGET_EXTRACT)
>> +$(1)-extract:                  $$($(2)_TARGET_EXTRACT)
>> +$$($(2)_TARGET_EXTRACT):       $$($(2)_TARGET_SOURCE)
>>
>>   $(1)-depends:         $$($(2)_DEPENDENCIES)
>
>
>  Is the -depends target still useful? Does anybody ever run 'make
> foo-depends'?

I don't know.

>
>  Regards,
>  Arnout
>
>
>>
>> @@ -418,10 +423,9 @@ else
>>   #  source, by rsyncing
>>   #  depends
>>   #  configure
>> -$(1)-configure:                $(1)-depends \
>> -                       $$($(2)_TARGET_CONFIGURE)
>> +$$($(2)_TARGET_CONFIGURE):     $$($(2)_TARGET_RSYNC)
>>
>> -$(1)-depends:          $(1)-rsync $$($(2)_DEPENDENCIES)
>> +$(1)-depends:          $$($(2)_DEPENDENCIES)
>>
>>   $(1)-patch:           $(1)-rsync
>>   $(1)-extract:         $(1)-rsync
>> diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
>> index 0d12afd..dea36e7 100644
>> --- a/package/uclibc/uclibc.mk
>> +++ b/package/uclibc/uclibc.mk
>> @@ -26,7 +26,7 @@ UCLIBC_ADD_TOOLCHAIN_DEPENDENCY = NO
>>   UCLIBC_DEPENDENCIES = host-gcc-initial linux-headers
>>
>>   # Before uClibc is built, we must have the second stage cross-compiler
>> -uclibc-build: host-gcc-intermediate
>> +UCLIBC_BUILD_DEPENDENCIES = host-gcc-intermediate
>>
>>   # specifying UCLIBC_CONFIG_FILE on the command-line overrides the
>> .config
>>   # setting.
>>
>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 4/8] Makefile: add support for top-level parallel make

[Fabio Porcedda]

On Thu, Oct 24, 2013 at 12:40 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/10/13 11:34, Fabio Porcedda wrote:
>>
>> diff --git a/fs/common.mk b/fs/common.mk
>> index 4dab7ea..7698c53 100644
>> --- a/fs/common.mk
>> +++ b/fs/common.mk
>> @@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
>>   # extra deps
>>   ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if
>> $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if
>> $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if
>> $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
>>
>> -$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
>> +$$(BINARIES_DIR)/rootfs.$(1): target-finalize
>> $$(ROOTFS_$(2)_DEPENDENCIES)
>>         @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
>>         $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call
>> $$(hook))$$(sep))
>>         rm -f $$(FAKEROOT_SCRIPT)
>> @@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
>>   rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
>>
>>   ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
>> -TARGETS += rootfs-$(1)
>> +TARGETS_ROOTFS += rootfs-$(1)
>>   endif
>>   endef
>
>
>  Oh I forgot: you should also fix up initramfs and iso9660. They add
> themselves to TARGETS manually.

Done, thanks.

Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 5/8] glibc: add support for top-level parallel make

[Fabio Porcedda]

On Thu, Oct 24, 2013 at 12:48 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/10/13 11:34, Fabio Porcedda wrote:
>>
>> To be able to use top-level parallel make we must not depend in a rule
>> on the order of evaluation of the prerequisites, so instead of relyng on
>> the left to right ordering of evaluation of the prerequisites add
>> an explicit rule to describe the dependencies.
>>
>> The glibc-install-target depends on glibc-install-staging so add a rule
>> for it.
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>
>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Added, thanks.

>
>> ---
>>   package/glibc/glibc.mk | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/package/glibc/glibc.mk b/package/glibc/glibc.mk
>> index e89c12a..838fb8c 100644
>> --- a/package/glibc/glibc.mk
>> +++ b/package/glibc/glibc.mk
>> @@ -130,3 +130,5 @@ define GLIBC_INSTALL_TARGET_CMDS
>>   endef
>>
>>   $(eval $(autotools-package))
>> +
>> +$(GLIBC_TARGET_INSTALL_TARGET): $(GLIBC_TARGET_INSTALL_STAGING)
>>
>
>  Nitpick: I would have added a comment why this is needed.
>  Regards,
>  Arnout

I've added:
# The glic-install-target use files from glibc-install-staging

Thanks and best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 6/8] uclibc: add support for top-level parallel make

[Fabio Porcedda]

On Thu, Oct 24, 2013 at 12:49 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/10/13 11:34, Fabio Porcedda wrote:
>>
>> To be able to use top-level parallel make we must not depend in a rule
>> on the order of evaluation of the prerequisites, so instead of relyng on
>> the left to right ordering of evaluation of the prerequisites add an
>> explicit rule to describe the dependencies.
>>
>> The uclibc-install-target depends on uclibc-install-staging so add a
>> rule for it.
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>
>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Added, thanks.

>
>> ---
>>   package/uclibc/uclibc.mk | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/package/uclibc/uclibc.mk b/package/uclibc/uclibc.mk
>> index dea36e7..c8eeb3d 100644
>> --- a/package/uclibc/uclibc.mk
>> +++ b/package/uclibc/uclibc.mk
>> @@ -559,3 +559,5 @@ uclibc-menuconfig: dirs uclibc-patch
>>         rm -f
>> $(UCLIBC_DIR)/.stamp_{configured,built,target_installed,staging_installed}
>>
>>   $(eval $(generic-package))
>> +
>> +$(UCLIBC_TARGET_INSTALL_TARGET): $(UCLIBC_TARGET_INSTALL_STAGING)
>>
>
>  Same nitpick :-)

Same comment:
# The uclibc-install-target use files from uclibc-install-staging

Thanks and best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 4/8] Makefile: add support for top-level parallel make

[Fabio Porcedda]

On Thu, Oct 24, 2013 at 12:29 AM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 18/10/13 11:34, Fabio Porcedda wrote:
>>
>> To be able to use top-level parallel make we must not depend in a rule
>> on the order of evaluation of the prerequisites, so instead of relyng on
>> the left to right ordering of evaluation of the prerequisites add an
>> explicit rule to describe the dependencies.
>>
>> Add explicit rules to describe the following dependency chain:
>> $(TARGETS) -> target-finalize -> rootfs-* -> target-post-image
>>
>> Signed-off-by: Fabio Porcedda <fabio.porcedda@gmail.com>
>
>
> Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
>
>  I think this one is ready to go in, so perhaps you can reorder it as one of
> the first ones of the series. Although it serves no function except for the
> top-level parallel make, so perhaps Peter will object to committing it on
> its own.

Is a little cumbersome to change  the order so i prefer at least for
the next iteration not to change it.

>  Regards,
>  Arnout
>
>
>> ---
>>   Makefile     | 15 ++++++---------
>>   fs/common.mk |  4 ++--
>>   2 files changed, 8 insertions(+), 11 deletions(-)
>>
>> diff --git a/Makefile b/Makefile
>> index ebecec4..299d34b 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -329,8 +329,6 @@ include boot/common.mk
>>   include linux/linux.mk
>>   include system/system.mk
>>
>> -TARGETS+=target-finalize
>> -
>>   ifeq ($(BR2_ENABLE_LOCALE_PURGE),y)
>>   TARGETS+=target-purgelocales
>>   endif
>> @@ -347,8 +345,6 @@ endif
>>
>>   include fs/common.mk
>>
>> -TARGETS+=target-post-image
>> -
>>   TARGETS_CLEAN:=$(patsubst %,%-clean,$(TARGETS))
>>   TARGETS_SOURCE:=$(patsubst %,%-source,$(TARGETS) $(BASE_TARGETS))
>>   TARGETS_DIRCLEAN:=$(patsubst %,%-dirclean,$(TARGETS))
>> @@ -382,11 +378,12 @@ $(BUILD_DIR)/buildroot-config/auto.conf:
>> $(BUILDROOT_CONFIG)
>>
>>   prepare: $(BUILD_DIR)/buildroot-config/auto.conf
>>
>> -world: $(TARGETS)
>> +world: target-post-image
>>
>>   .PHONY: all world toolchain dirs clean distclean source outputmakefile \
>>         legal-info legal-info-prepare legal-info-clean printvars \
>> -       $(BASE_TARGETS) $(TARGETS) \
>> +       target-finalize target-post-image \
>> +       $(BASE_TARGETS) $(TARGETS) $(TARGETS_ROOTFS) \
>>         $(TARGETS_CLEAN) $(TARGETS_DIRCLEAN) $(TARGETS_SOURCE)
>> $(TARGETS_LEGAL_INFO) \
>>         $(BUILD_DIR) $(STAGING_DIR) $(TARGET_DIR) \
>>         $(HOST_DIR) $(BINARIES_DIR) $(STAMP_DIR)
>> @@ -443,7 +440,7 @@ endif
>>   STRIP_FIND_CMD += -type f -perm /111
>>   STRIP_FIND_CMD += -not \( $(call findfileclauses,libpthread*.so* $(call
>> qstrip,$(BR2_STRIP_EXCLUDE_FILES))) \) -print
>>
>> -target-finalize:
>> +target-finalize: $(TARGETS)
>>         rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
>>                 $(TARGET_DIR)/usr/lib/pkgconfig
>> $(TARGET_DIR)/usr/share/pkgconfig \
>>                 $(TARGET_DIR)/usr/lib/cmake $(TARGET_DIR)/usr/share/cmake
>> @@ -550,7 +547,7 @@ target-generatelocales: host-localedef
>>         done
>>   endif
>>
>> -target-post-image:
>> +target-post-image: target-finalize $(TARGETS_ROOTFS)
>>         @$(foreach s, $(call qstrip,$(BR2_ROOTFS_POST_IMAGE_SCRIPT)), \
>>                 $(call MESSAGE,"Executing post-image script $(s)"); \
>>                 $(s) $(BINARIES_DIR) $(call
>> qstrip,$(BR2_ROOTFS_POST_SCRIPT_ARGS))$(sep))
>> @@ -586,7 +583,7 @@ legal-info: dirs legal-info-clean legal-info-prepare
>> $(REDIST_SOURCES_DIR) \
>>         @rm -f $(LEGAL_WARNINGS)
>>
>>   show-targets:
>> -       @echo $(TARGETS)
>> +       @echo $(TARGETS) $(TARGETS_ROOTFS)
>>
>>   else # ifeq ($(BR2_HAVE_DOT_CONFIG),y)
>>
>> diff --git a/fs/common.mk b/fs/common.mk
>> index 4dab7ea..7698c53 100644
>> --- a/fs/common.mk
>> +++ b/fs/common.mk
>> @@ -42,7 +42,7 @@ define ROOTFS_TARGET_INTERNAL
>>   # extra deps
>>   ROOTFS_$(2)_DEPENDENCIES += host-fakeroot host-makedevs $$(if
>> $$(BR2_TARGET_ROOTFS_$(2)_LZMA),host-lzma) $$(if
>> $$(BR2_TARGET_ROOTFS_$(2)_LZO),host-lzop) $$(if
>> $$(BR2_TARGET_ROOTFS_$(2)_XZ),host-xz)
>>
>> -$$(BINARIES_DIR)/rootfs.$(1): $$(ROOTFS_$(2)_DEPENDENCIES)
>> +$$(BINARIES_DIR)/rootfs.$(1): target-finalize
>> $$(ROOTFS_$(2)_DEPENDENCIES)
>>         @$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
>>         $$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call
>> $$(hook))$$(sep))
>>         rm -f $$(FAKEROOT_SCRIPT)
>> @@ -86,7 +86,7 @@ rootfs-$(1)-show-depends:
>>   rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
>>
>>   ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
>> -TARGETS += rootfs-$(1)
>> +TARGETS_ROOTFS += rootfs-$(1)
>>   endif
>>   endef
>>
>>
>
>
> --
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Thanks and best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Thomas Petazzoni]

Fabio, Arnout,

On Fri, 18 Oct 2013 11:34:12 +0200, Fabio Porcedda wrote:

> this is a patch set for adding support for top-level parallel make in
> buildroot, the common problem scattered in buildroot's top-level
> makefiles is that in the rules it relies on the order of evaluation of
> the prerequisites, to be able to use top-level parallel make instead
> of reling on the left to right ordering of evaluation of the
> prerequisites we must add an explicit rule to describe the
> dependencies.

I'd like to agree on a plan with this patch series. At the latest
Buildroot Developer Day in Edinburgh, I think we agreed that:

 (1) Enabling top-level parallel without having per-package sysroot was
     not desirable, as it would make the builds non-reproducible.

 (2) That being said, Fabio's patches have anyway a merit on their own
     since they make things cleaner, and that therefore, patches 1 to 6
     can be merged. Maybe patch 7 even if I'm not sure it makes sense
     to have it without the top-level parallel build.

What do you think?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Fabio Porcedda]

On Mon, Nov 11, 2013 at 2:49 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Fabio, Arnout,
>
> On Fri, 18 Oct 2013 11:34:12 +0200, Fabio Porcedda wrote:
>
>> this is a patch set for adding support for top-level parallel make in
>> buildroot, the common problem scattered in buildroot's top-level
>> makefiles is that in the rules it relies on the order of evaluation of
>> the prerequisites, to be able to use top-level parallel make instead
>> of reling on the left to right ordering of evaluation of the
>> prerequisites we must add an explicit rule to describe the
>> dependencies.
>
> I'd like to agree on a plan with this patch series. At the latest
> Buildroot Developer Day in Edinburgh, I think we agreed that:
>
>  (1) Enabling top-level parallel without having per-package sysroot was
>      not desirable, as it would make the builds non-reproducible.

What do you think if we provide an easy way to use top-level parallel make
that warns about the problems:

ifeq ($(P),1)
$(warning Not reproducible builds use at your own risk!)
BR2_JLEVEL=
else
# This top-level Makefile can *not* be executed in parallel
.NOTPARALLEL:
endif


  make P=1 -j5


>  (2) That being said, Fabio's patches have anyway a merit on their own
>      since they make things cleaner, and that therefore, patches 1 to 6
>      can be merged. Maybe patch 7 even if I'm not sure it makes sense
>      to have it without the top-level parallel build.
>
> What do you think?
>
> Best regards,
>
> Thomas
> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux, Kernel and Android engineering
> http://free-electrons.com


Best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Fabio Porcedda]

On Tue, Nov 12, 2013 at 9:39 AM, Fabio Porcedda
<fabio.porcedda@gmail.com> wrote:
> On Mon, Nov 11, 2013 at 2:49 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Fabio, Arnout,
>>
>> On Fri, 18 Oct 2013 11:34:12 +0200, Fabio Porcedda wrote:
>>
>>> this is a patch set for adding support for top-level parallel make in
>>> buildroot, the common problem scattered in buildroot's top-level
>>> makefiles is that in the rules it relies on the order of evaluation of
>>> the prerequisites, to be able to use top-level parallel make instead
>>> of reling on the left to right ordering of evaluation of the
>>> prerequisites we must add an explicit rule to describe the
>>> dependencies.
>>
>> I'd like to agree on a plan with this patch series. At the latest
>> Buildroot Developer Day in Edinburgh, I think we agreed that:
>>
>>  (1) Enabling top-level parallel without having per-package sysroot was
>>      not desirable, as it would make the builds non-reproducible.
>
> What do you think if we provide an easy way to use top-level parallel make
> that warns about the problems:
>
> ifeq ($(P),1)
> $(warning Not reproducible builds use at your own risk!)
> BR2_JLEVEL=
> else
> # This top-level Makefile can *not* be executed in parallel
> .NOTPARALLEL:
> endif
>
>
>   make P=1 -j5
>
>
>>  (2) That being said, Fabio's patches have anyway a merit on their own
>>      since they make things cleaner, and that therefore, patches 1 to 6
>>      can be merged. Maybe patch 7 even if I'm not sure it makes sense
>>      to have it without the top-level parallel build.

The patch 7 make sense only for top-level parallel build but i suggest
to accept it because does not harm the serial building and is needed
for someone that want to try to use top-level parallel make.

>> What do you think?

Is great to have most of the patch accepted, i think it's a step in
the right direction.
I will send the patch set v9 soon.

>>
>> Best regards,
>>
>> Thomas
>> --
>> Thomas Petazzoni, CTO, Free Electrons
>> Embedded Linux, Kernel and Android engineering
>> http://free-electrons.com
>
>
> Best regards
> --
> Fabio Porcedda


Thanks and best regards
-- 
Fabio Porcedda

[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Arnout Vandecappelle]

On 12/11/13 09:39, Fabio Porcedda wrote:
> On Mon, Nov 11, 2013 at 2:49 PM, Thomas Petazzoni
> <thomas.petazzoni@free-electrons.com> wrote:
>> Fabio, Arnout,
>>
>> On Fri, 18 Oct 2013 11:34:12 +0200, Fabio Porcedda wrote:
>>
>>> this is a patch set for adding support for top-level parallel make in
>>> buildroot, the common problem scattered in buildroot's top-level
>>> makefiles is that in the rules it relies on the order of evaluation of
>>> the prerequisites, to be able to use top-level parallel make instead
>>> of reling on the left to right ordering of evaluation of the
>>> prerequisites we must add an explicit rule to describe the
>>> dependencies.
>>
>> I'd like to agree on a plan with this patch series. At the latest
>> Buildroot Developer Day in Edinburgh, I think we agreed that:
>>
>>   (1) Enabling top-level parallel without having per-package sysroot was
>>       not desirable, as it would make the builds non-reproducible.
>
> What do you think if we provide an easy way to use top-level parallel make
> that warns about the problems:
>
> ifeq ($(P),1)
> $(warning Not reproducible builds use at your own risk!)

  Such a warning will not really be visible, because it appears at the 
very beginning of the build and is immediately followed by all of the 
build output...

> BR2_JLEVEL=
> else
> # This top-level Makefile can *not* be executed in parallel
> .NOTPARALLEL:
> endif

  Since you only need to make this simple change in the Makefile (remove 
.NOTPARALLEL), I think it's a lot more effective for people who want 
top-level parallel builds to remove this line in their copy of buildroot. 
We also agreed at the BR developer meeting that we would add an 
explanatory comment at that location to make it clear why we disable 
parallel builds. That way, someone who wants to enable it will be aware 
of the risks.


  Regards,
  Arnout


>
>
>    make P=1 -j5
[snip]


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] [PATCH v8 RESEND 0/8] Add support for top-level parallel make

[Fabio Porcedda]

On Wed, Nov 13, 2013 at 9:05 PM, Arnout Vandecappelle <arnout@mind.be> wrote:
> On 12/11/13 09:39, Fabio Porcedda wrote:
>>
>> On Mon, Nov 11, 2013 at 2:49 PM, Thomas Petazzoni
>> <thomas.petazzoni@free-electrons.com> wrote:
>>>
>>> Fabio, Arnout,
>>>
>>> On Fri, 18 Oct 2013 11:34:12 +0200, Fabio Porcedda wrote:
>>>
>>>> this is a patch set for adding support for top-level parallel make in
>>>> buildroot, the common problem scattered in buildroot's top-level
>>>> makefiles is that in the rules it relies on the order of evaluation of
>>>> the prerequisites, to be able to use top-level parallel make instead
>>>> of reling on the left to right ordering of evaluation of the
>>>> prerequisites we must add an explicit rule to describe the
>>>> dependencies.
>>>
>>>
>>> I'd like to agree on a plan with this patch series. At the latest
>>> Buildroot Developer Day in Edinburgh, I think we agreed that:
>>>
>>>   (1) Enabling top-level parallel without having per-package sysroot was
>>>       not desirable, as it would make the builds non-reproducible.
>>
>>
>> What do you think if we provide an easy way to use top-level parallel make
>> that warns about the problems:
>>
>> ifeq ($(P),1)
>> $(warning Not reproducible builds use at your own risk!)
>
>
>  Such a warning will not really be visible, because it appears at the very
> beginning of the build and is immediately followed by all of the build
> output...
>
>
>> BR2_JLEVEL=
>> else
>> # This top-level Makefile can *not* be executed in parallel
>> .NOTPARALLEL:
>> endif
>
>
>  Since you only need to make this simple change in the Makefile (remove
> .NOTPARALLEL), I think it's a lot more effective for people who want
> top-level parallel builds to remove this line in their copy of buildroot. We
> also agreed at the BR developer meeting that we would add an explanatory
> comment at that location to make it clear why we disable parallel builds.
> That way, someone who wants to enable it will be aware of the risks.

Ok, I will just change the comment in the latest commit, i will add
this comment:
-# This top-level Makefile can *not* be executed in parallel
+# The parallel execution of this top-level Makefile is disabled
+# because it would make builds non-reproducible also some packages
+# have unspecified optional dependencies so if those dependencies are
+# present when the package is built, they are used, otherwise they
+# aren't (but compilation happily proceeds). This means that the
+# package order is relevant in this case, and the end result will
+# differ if the order is swapped due to parallel building.
+# Taking in account the above warnings, if you still want to execute
+# this top-level Makefile in parrallel remove/comment the following
+# line and execute:
+#      make BR2_JLEVEL= -j$((`getconf _NPROCESSORS_ONLN`+1))

I've used a slightly modified description give by  Thomas De Schampheleire.
What do you think about that?

I think that there two problems:
- first: non reproducible build, that can be resolved just recording
the building steps and replaying it.
- second: the unspecified optional dependency that is harder to fix.

Best regards
-- 
Fabio Porcedda