* [Buildroot] [PATCH next v8 0/6] Top-level parallel build support
@ 2019-11-05 16:46 Thomas Petazzoni
2019-11-05 16:46 ` [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target Thomas Petazzoni
` (6 more replies)
0 siblings, 7 replies; 16+ messages in thread
From: Thomas Petazzoni @ 2019-11-05 16:46 UTC (permalink / raw)
To: buildroot
Top-level parallel build support
Hello,
Here is an eighth iteration of the per-package SDK and target
directory implementation, offering top-level parallel build
capability.
If you're interested in testing it, you can find the patch series at:
https://github.com/tpetazzoni/buildroot/commits/ppsh-v8
This is really only the core changes. It is sufficient for a number of
configuration, but we know some specific packages are broken with
per-package directories (for example: Qt5). As we cannot fix all
problems at once, we would like to first merge the core support, which
will help get broader testing for the feature, and hopefully
contributions to solve the remaining package-specific issues.
What is this all about ?
========================
See the cover letter of v3 for details:
http://lists.busybox.net/pipermail/buildroot/2017-December/208384.html
Changes v7 -> v8
================
- Drop the following patches, which have been merged upstream:
support/scripts/check-host-rpath: document existing functions
Makefile: move definition of TARGET_DIR inside .config condition
- Fix the commit log of "core: implement per-package SDK and target"
to make it clear that not just the dependencies listed in
<pkg>_DEPENDENCIES, but also their recursive dependencies. This
follows a comment and suggestion from Yann.
- Don't split the line in the main Makefile creating all the main
directories, as the split can be confusing. Suggested by Yann.
- In support/scripts/check-host-rpath, add a comment explaining why
the check using ${perpackagedir} works fine even when PPD is not
used. Suggested by Yann.
- In support/scripts/check-host-rpath and support/scripts/fix-rpath,
improve the regexp in the ${perpackagedir} check to not allow a
path like per-package//host/lib, but only allow
per-package/some-non-empty-thing/host/lib. Suggested by Yann.
- Added Acked-by from Yann on:
Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y
- In package/pkg-generic.mk, modify the regexp used in
fixup-libtool-files to not allow a path like per-package//host/lib,
but only allow per-package/some-non-empty-thing/host/lib. Suggested
by Yann.
- Improve the Buildroot manual details about the feature, from
suggestions made by Yann.
Changes v6 -> v7
================
- Make host-finalize a PHONY target, as suggested by Yann
E. Morin. Change done in "core: implement per-package SDK and
target"
- Adjust <pkg>-dirclean so that it also removes the per-package
directory for <pkg>. Change done in "core: implement per-package
SDK and target"
- Completely rework the comment that explains the .NOTPARALLEL
statement. Indeed, the existing explanation no longer made
sense. The new comment explains why .NOTPARALLEL is added when
!BR2_PER_PACKAGE_DIRECTORIES. Suggested by Yann E. Morin. Change
done in "Makefile: allow top-level parallel build with
BR2_PER_PACKAGE_DIRECTORIES=y".
- Move the libtool .la files fixup logic into a separate function,
just because it is a bit nicer. Change done in
"package/pkg-generic: make libtool .la files compatible with
per-package directories".
- Added Acked-by from Yann E. Morin on patch "package/pkg-kconfig:
handle KCONFIG_DEPENDENCIES with per-package directories".
- Fix "package/pkg-kconfig: handle KCONFIG_DEPENDENCIES with
per-package directories" to use the proper per-package-directory
macro instead of prepare-per-package-folder. This was missed during
the rename of the macro that happened in the v6 of this series.
- Drop the following patches that have been merged in master since v6
was posted:
Makefile: evaluate CCACHE and HOST{CC,CXX} at time of use
support/scripts/check-host-rpath: split condition on two statements
Makefile: rework main directory creation logic
Makefile: move .NOTPARALLEL statement after including .config file
Makefile: define TARGET_DIR_WARNING_FILE relative to TARGET_DIR
package/pkg-generic: adjust config scripts tweaks for per-package directories
- Add a preparatory patch that moves the definition of TARGET_DIR
inside the BR2_HAVE_DOT_CONFIG condition, next to the existing
HOST_DIR definition. This allows the HOST_DIR and TARGET_DIR
definition to be next to each other when introducing the
per-package directory support. Suggested by Arnout.
- Add a preparatory patch that documents the functions in
check-host-rpath.
- Add more comments in check-host-rpath to explain how it works in
relation with per-package directory support. Suggested by Arnout.
- Modify the fix-rpath script to rewrite RPATH referencing
per-package host directories so that they point to the global host
directory. This is needed for the patchelf --make-rpath-relative
logic to work, and rewrite all RPATHs as relative RPATHs. Reported
by Andreas Naumann.
- Add documentation in the Buildroot manual about top-level parallel
build support.
Changes v5 -> v6
================
- Add Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> on patch
"Makefile: move .NOTPARALLEL statement after including .config
file"
- Add Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> on patch
Makefile: define TARGET_DIR_WARNING_FILE relative to TARGET_DIR
- Add Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr> on patch
"package/pkg-generic: adjust config scripts tweaks for per-package
folders"
- Move the change that really allows top-level parallel build (by
making .NOTPARALLEL conditional) into a separate commit, as
suggested by Yann E. Morin.
- Sort $(PACKAGES) in host-finalize and target-finalize when creating
the global HOST_DIR and global TARGET_DIR. Suggested by Yann
E. Morin.
- Split too long line in the main Makefile listing all the top-level
folders as targets. Suggested by Yann E. Morin.
- Drop change in fs/common.mk adding ROOTFS_COMMON_DEPENDENCIES to
PACKAGES, since a similar change has already been merged in
master. Noticed by Yann E. Morin.
- Use "directory/directories" instead of "folder/folders", as
suggested by Arnout Vandecappelle.
- Move prepare-per-package-folder (now named
prepare-per-package-directory) to pkg-utils.mk. Suggested by both
Yann E. Morin and Arnout Vandecappelle.
- Add a comment above prepare-per-package-directory, as suggested by
Yann E. Morin.
- Update the commit log of "core: implement per-package SDK and
target" to explain why we are modifying the check-host-rpath
script. Suggested by Arnout Vandecappelle.
Changes v4 -> v5
================
- Add patch "package/pkg-generic: adjust config scripts tweaks for
per-package folders" that adjusts how pkg-generic.mk fixes *-config
scripts, to use relative paths instead of absolute paths.
- Add patch "package/pkg-generic: make libtool .la files compatible
with per-package folders" that fixes libtool.la files for
per-package folders.
- Add patch "package/pkg-kconfig: handle KCONFIG_DEPENDENCIES with
per-package folders" that properly handles KCONFIG_DEPENDENCIES in
the pkg-kconfig infrastructure for per-package folders.
- Added Acked-by from Yann E. Morin on the following patches:
support/scripts/check-host-rpath: split condition on two statements
Makefile: evaluate CCACHE and HOST{CC,CXX} at time of use
- It is not included in this series, but since the v4 was sent, the
case of the Luarocks and Meson based packages have been fixed, and
I have sent separate series for these, because they can be merged
separately. Many thanks to Fran?ois Perrad and Peter Seiderer for
providing very useful insights on Luarocks and Meson.
Changes v3 -> v4
================
- Dropped patches that have been merged upstream, mainly the "extract
dependency" patches, but also a few other preparation patches.
- Dropped the change of the RPATH handling. As discussed during
previous Buildroot Developers meeting, it is not a big problem if
during the build, the RPATH of a binary points to the library
folder of another package per-package folder. Therefore, instead of
fixing up RPATH at the end of each package installation, we keep
what Buildroot does today: fix them at the very end of the build.
- Made the support for per-package folders optional. Indeed, we
realized there was no way to make sure it would be perfect from day
1. Even if the principle works, there are lots of package-specific
details to solve, and solving all of them before merging the base
per-package folder support is not reasonable. So for now, an option
BR2_PER_PACKAGE_FOLDERS is introduced, which defaults to off. When
this option is enabled, the .NOTPARALLEL statement of the main
Makefile goes away, which allows to do top-level parallel build.
- Addition of a commit that reworks how the top-level directories are
created.
- Introduction of a prepare-per-package-folder function in
pkg-generic.mk to encapsulate the logic to create the per-package
folders from the dependencies of a package.
- Rebased on top of the latest master.
Changes v2 -> v3
================
- Dropped patches that have been merged upstream:
pkgconf: use relative path to STAGING_DIR instead of absolute path
toolchain: post-pone evaluation of TOOLCHAIN_EXTERNAL_BIN
- For now, removed the most controversial/complicated patches
(changing the rpath strategy, and switching to per-package
host/target directories). The goal for now is to merge only the
preparation patches.
Changes v1 -> v2
================
- Solved the problem of all DEPENDENCIES_HOST_PREREQ targets:
host-ccache, host-tar, host-lzip, host-xz, host-fakedate.
To achieve this, introduced <pkg>_EXTRACT_DEPENDENCIES and used
that to handle the dependencies on host-tar, host-lzip and host-xz.
Used regular dependencies for host-ccache and host-fakedate.
See below for more details.
Changes RFC -> v1
=================
- Rebased on the latest Buildroot next branch
- The pkg-config changes are reworked according to Arnout's comments:
only @STAGING_SUBDIR@ is replaced in pkg-config.in to generate
pkg-config, the rest of the logic is internal to the script.
- New patch to move the "host skeleton" logic into a proper
host-skeleton package.
- New patch to have the CMake related files generated as part of the
toolchain package.
- New patch to add a new "install" step that depends on the different
install steps (target, staging, images, host). This is needed to
later add some logic that is executed once all install steps of a
package have been done.
- Fix the approach to solve the RPATH logic: instead of using
LD_LIBRARY_PATH, we actually fix with patchelf the RPATH of host
binaries right after their installation.
- Misc other improvements in the per-package SDK implementation.
Best regards,
Thomas
Thomas Petazzoni (6):
core: implement per-package SDK and target
Makefile: allow top-level parallel build with
BR2_PER_PACKAGE_DIRECTORIES=y
package/pkg-generic: make libtool .la files compatible with
per-package directories
package/pkg-kconfig: handle KCONFIG_DEPENDENCIES with per-package
directories
docs/manual: add details about top-level parallel build support
docs/manual: document the effect of per-package directory on variables
Config.in | 18 ++++++++++
Makefile | 47 ++++++++++++-------------
docs/manual/adding-packages-generic.txt | 9 ++++-
docs/manual/common-usage.txt | 47 +++++++++++++++++++++++++
docs/manual/faq-troubleshooting.txt | 3 ++
docs/manual/quickstart.txt | 8 ++---
package/pkg-generic.mk | 24 ++++++++++++-
package/pkg-kconfig.mk | 1 +
package/pkg-utils.mk | 26 ++++++++++++++
support/scripts/check-host-rpath | 32 ++++++++++++++---
support/scripts/fix-rpath | 29 +++++++++++----
11 files changed, 201 insertions(+), 43 deletions(-)
--
2.21.0
^ permalink raw reply [flat|nested] 16+ messages in thread* [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni @ 2019-11-05 16:46 ` Thomas Petazzoni 2019-11-29 14:20 ` Peter Korsgaard 2023-11-10 14:12 ` [Buildroot] [BUG?] per-package not fully working with dependencies and rebuild John Ernberg 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 2/6] Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y Thomas Petazzoni ` (5 subsequent siblings) 6 siblings, 2 replies; 16+ messages in thread From: Thomas Petazzoni @ 2019-11-05 16:46 UTC (permalink / raw) To: buildroot This commit implements the core of the move to per-package SDK and target directories. The main idea is that instead of having a global output/host and output/target in which all packages install files, we switch to per-package host and target directories, that only contain their explicit dependencies. There are two main benefits: - Packages will now see only the dependencies they explicitly list in their <pkg>_DEPENDENCIES variable, and the recursive dependencies thereof. - We can support top-level parallel build properly, because a package only "sees" its own host directory and target directory, isolated from the build of other packages that can happen in parallel. It works as follows: - A new output/per-package/ directory is created, which will contain one sub-directory per package, and inside it, a "host" directory and a "target" directory: output/per-package/busybox/target output/per-package/busybox/host output/per-package/host-fakeroot/target output/per-package/host-fakeroot/host This output/per-package/ directory is PER_PACKAGE_DIR. - The global TARGET_DIR and HOST_DIR variable now automatically point to the per-package directory when PKG is defined. So whenever a package references $(HOST_DIR) or $(TARGET_DIR) in its build process, it effectively references the per-package host/target directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it is handled as well. - Of course, packages have dependencies, so those dependencies must be installed in the per-package host and target directories. To do so, we simply rsync (using hard links to save space and time) the host and target directories of the direct dependencies of the package to the current package host and target directories. We only need to take care of direct dependencies (and not recursively all dependencies), because we accumulate into those per-package host and target directories the files installed by the dependencies. Note that this only works because we make the assumption that one package does *not* overwrite files installed by another package. This is done for "extract dependencies" at the beginning of the extract step, and for "normal dependencies" at the beginning of the configure step. This is basically enough to make per-package SDK and target work. The only gotcha is that at the end of the build, output/target and output/host are empty, which means that: - The filesystem image creation code cannot work. - We don't have a SDK to build code outside of Buildroot. In order to fix this, this commit extends the target-finalize step so that it starts by populating output/target and output/host by rsync-ing into them the target and host directories of all packages listed in the $(PACKAGES) variable. It is necessary to do this sequentially in the target-finalize step and not in each package. Doing it in package installation means that it can be done in parallel. In that case, there is a chance that two rsyncs are creating the same hardlink or directory at the same time, which makes one of them fail. This change to per-package directories has an impact on the RPATH built into the host binaries, as those RPATH now point to various per-package host directories, and no longer to the global host directory. We do not try to rewrite such RPATHs during the build as having such RPATHs is perfectly fine, but we still need to handle two fallouts from this change: - The check-host-rpath script, which verifies at the end of each package installation that it has the appropriate RPATH, is modified to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is a correct RPAT. - The fix-rpath script, which mungles the RPATH mainly for the SDK preparation, is modified to rewrite the RPATH to not point to per-package directories. Indeed the patchelf --make-rpath-relative call only works if the RPATH points to the ROOTDIR passed as argument, and this ROOTDIR is the global host directory. Rewriting the RPATH to not point to per-package host directories prior to this is an easy solution to this issue. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- Config.in | 18 ++++++++++++++++++ Makefile | 28 +++++++++++++++++----------- package/pkg-generic.mk | 7 ++++++- package/pkg-utils.mk | 26 ++++++++++++++++++++++++++ support/scripts/check-host-rpath | 32 +++++++++++++++++++++++++++----- support/scripts/fix-rpath | 29 ++++++++++++++++++++++------- 6 files changed, 116 insertions(+), 24 deletions(-) diff --git a/Config.in b/Config.in index 010b0774e3..5a4239bc9a 100644 --- a/Config.in +++ b/Config.in @@ -690,6 +690,24 @@ config BR2_REPRODUCIBLE This is labeled as an experimental feature, as not all packages behave properly to ensure reproducibility. +config BR2_PER_PACKAGE_DIRECTORIES + bool "Use per-package directories (experimental)" + help + This option will change the build process of Buildroot + package to use per-package target and host directories. + + This is useful for two related purposes: + + - Cleanly isolate the build of each package, so that a + given package only "sees" the dependencies it has + explicitly expressed, and not other packages that may + have by chance been built before. + + - Enable top-level parallel build. + + This is labeled as an experimental feature, as not all + packages behave properly with per-package directories. + endmenu comment "Security Hardening Options" diff --git a/Makefile b/Makefile index bab34ab9b5..823c219c86 100644 --- a/Makefile +++ b/Makefile @@ -204,6 +204,7 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf) BUILD_DIR := $(BASE_DIR)/build BINARIES_DIR := $(BASE_DIR)/images BASE_TARGET_DIR := $(BASE_DIR)/target +PER_PACKAGE_DIR := $(BASE_DIR)/per-package # initial definition so that 'make clean' works for most users, even without # .config. HOST_DIR will be overwritten later when .config is included. HOST_DIR := $(BASE_DIR)/host @@ -452,12 +453,13 @@ XZCAT := $(call qstrip,$(BR2_XZCAT)) LZCAT := $(call qstrip,$(BR2_LZCAT)) TAR_OPTIONS = $(call qstrip,$(BR2_TAR_OPTIONS)) -xf -# packages compiled for the host go here +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +HOST_DIR = $(if $(PKG),$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/host,$(call qstrip,$(BR2_HOST_DIR))) +TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(if $(PKG),$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/target,$(BASE_TARGET_DIR))) +else HOST_DIR := $(call qstrip,$(BR2_HOST_DIR)) - -# The target directory is common to all packages, -# but there is one that is specific to each filesystem. TARGET_DIR = $(if $(ROOTFS),$(ROOTFS_$(ROOTFS)_TARGET_DIR),$(BASE_TARGET_DIR)) +endif ifneq ($(HOST_DIR),$(BASE_DIR)/host) HOST_DIR_SYMLINK = $(BASE_DIR)/host @@ -593,8 +595,8 @@ world: target-post-image .PHONY: prepare-sdk prepare-sdk: world @$(call MESSAGE,"Rendering the SDK relocatable") - $(TOPDIR)/support/scripts/fix-rpath host - $(TOPDIR)/support/scripts/fix-rpath staging + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath host + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath staging $(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh mkdir -p $(HOST_DIR)/share/buildroot echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location @@ -727,15 +729,19 @@ $(TARGETS_ROOTFS): target-finalize # Avoid the rootfs name leaking down the dependency chain target-finalize: ROOTFS= -host-finalize: $(HOST_DIR_SYMLINK) +.PHONY: host-finalize +host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) + @$(call MESSAGE,"Finalizing host directory") + $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR)) .PHONY: staging-finalize staging-finalize: @ln -snf $(STAGING_DIR) $(BASE_DIR)/staging .PHONY: target-finalize -target-finalize: $(PACKAGES) host-finalize +target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize @$(call MESSAGE,"Finalizing target directory") + $(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR)) # Check files that are touched by more than one package $(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep)) rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \ @@ -776,7 +782,7 @@ endif ln -sf ../usr/lib/os-release $(TARGET_DIR)/etc @$(call MESSAGE,"Sanitizing RPATH in target tree") - $(TOPDIR)/support/scripts/fix-rpath target + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) $(TOPDIR)/support/scripts/fix-rpath target # For a merged /usr, ensure that /lib, /bin and /sbin and their /usr # counterparts are appropriately setup as symlinks ones to the others. @@ -1014,7 +1020,7 @@ savedefconfig: $(BUILD_DIR)/buildroot-config/conf outputmakefile # staging and target directories do NOT list these as # dependencies anywhere else -$(BUILD_DIR) $(BASE_TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST): +$(BUILD_DIR) $(BASE_TARGET_DIR) $(HOST_DIR) $(BINARIES_DIR) $(LEGAL_INFO_DIR) $(REDIST_SOURCES_DIR_TARGET) $(REDIST_SOURCES_DIR_HOST) $(PER_PACKAGE_DIR): @mkdir -p $@ # outputmakefile generates a Makefile in the output directory, if using a @@ -1046,7 +1052,7 @@ printvars: clean: rm -rf $(BASE_TARGET_DIR) $(BINARIES_DIR) $(HOST_DIR) $(HOST_DIR_SYMLINK) \ $(BUILD_DIR) $(BASE_DIR)/staging \ - $(LEGAL_INFO_DIR) $(GRAPHS_DIR) + $(LEGAL_INFO_DIR) $(GRAPHS_DIR) $(PER_PACKAGE_DIR) .PHONY: distclean distclean: clean diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index 893faba22a..b7b211e6e4 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -106,7 +106,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += check_bin_arch # have a proper DT_RPATH or DT_RUNPATH tag define check_host_rpath $(if $(filter install-host,$(2)),\ - $(if $(filter end,$(1)),support/scripts/check-host-rpath $(3) $(HOST_DIR))) + $(if $(filter end,$(1)),support/scripts/check-host-rpath $(3) $(HOST_DIR) $(PER_PACKAGE_DIR))) endef GLOBAL_INSTRUMENTATION_HOOKS += check_host_rpath @@ -141,6 +141,7 @@ endif # Retrieve the archive $(BUILD_DIR)/%/.stamp_downloaded: @$(call step_start,download) + $(call prepare-per-package-directory,$($(PKG)_FINAL_DOWNLOAD_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_DOWNLOAD_HOOKS),$(call $(hook))$(sep)) # Only show the download message if it isn't already downloaded $(Q)for p in $($(PKG)_ALL_DOWNLOADS); do \ @@ -167,6 +168,7 @@ $(BUILD_DIR)/%/.stamp_actual_downloaded: $(BUILD_DIR)/%/.stamp_extracted: @$(call step_start,extract) @$(call MESSAGE,"Extracting") + $(call prepare-per-package-directory,$($(PKG)_FINAL_EXTRACT_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_EXTRACT_HOOKS),$(call $(hook))$(sep)) $(Q)mkdir -p $(@D) $($(PKG)_EXTRACT_CMDS) @@ -227,6 +229,7 @@ $(foreach dir,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)),\ $(BUILD_DIR)/%/.stamp_configured: @$(call step_start,configure) @$(call MESSAGE,"Configuring") + $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) @@ -349,6 +352,7 @@ $(BUILD_DIR)/%/.stamp_target_installed: # Remove package sources $(BUILD_DIR)/%/.stamp_dircleaned: + $(if $(BR2_PER_PACKAGE_DIRECTORIES),rm -Rf $(PER_PACKAGE_DIR)/$(NAME)) rm -Rf $(@D) ################################################################################ @@ -926,6 +930,7 @@ $$($(2)_TARGET_SOURCE): PKGDIR=$(pkgdir) $$($(2)_TARGET_ACTUAL_SOURCE): PKG=$(2) $$($(2)_TARGET_ACTUAL_SOURCE): PKGDIR=$(pkgdir) $$($(2)_TARGET_DIRCLEAN): PKG=$(2) +$$($(2)_TARGET_DIRCLEAN): NAME=$(1) # Compute the name of the Kconfig option that correspond to the # package being enabled. We handle three cases: the special Linux diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk index 63b19e812b..d38971caf6 100644 --- a/package/pkg-utils.mk +++ b/package/pkg-utils.mk @@ -135,6 +135,32 @@ clean-json = $(strip \ )))) \ ) +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +# rsync the contents of per-package directories +# $1: space-separated list of packages to rsync from +# $2: 'host' or 'target' +# $3: destination directory +define per-package-rsync + mkdir -p $(3) + $(foreach pkg,$(1),\ + rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ + $(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \ + $(3)$(sep)) +endef + +# prepares the per-package HOST_DIR and TARGET_DIR of the current +# package, by rsync the host and target directories of the +# dependencies of this package. The list of dependencies is passed as +# argument, so that this function can be used to prepare with +# different set of dependencies (download, extract, configure, etc.) +# +# $1: space-separated list of packages to rsync from +define prepare-per-package-directory + $(call per-package-rsync,$(1),host,$(HOST_DIR)) + $(call per-package-rsync,$(1),target,$(TARGET_DIR)) +endef +endif + # # legal-info helper functions # diff --git a/support/scripts/check-host-rpath b/support/scripts/check-host-rpath index c8939569e2..9a3866982b 100755 --- a/support/scripts/check-host-rpath +++ b/support/scripts/check-host-rpath @@ -11,6 +11,7 @@ export LC_ALL=C main() { local pkg="${1}" local hostdir="${2}" + local perpackagedir="${3}" local file ret # Remove duplicate and trailing '/' for proper match @@ -20,7 +21,7 @@ main() { while read file; do is_elf "${file}" || continue elf_needs_rpath "${file}" "${hostdir}" || continue - check_elf_has_rpath "${file}" "${hostdir}" && continue + check_elf_has_rpath "${file}" "${hostdir}" "${perpackagedir}" && continue if [ ${ret} -eq 0 ]; then ret=1 printf "***\n" @@ -44,6 +45,15 @@ is_elf() { # needs such an RPATH if at least of the libraries used by the ELF # executable is available in the host library directory. This function # returns 0 when a RPATH is needed, 1 otherwise. +# +# With per-package directory support, ${hostdir} will point to the +# current package per-package host directory, and this is where this +# function will check if the libraries needed by the executable are +# located (or not). In practice, the ELF executable RPATH may point to +# another package per-package host directory, but that is fine because +# if such an executable is within the current package per-package host +# directory, its libraries will also have been copied into the current +# package per-package host directory. elf_needs_rpath() { local file="${1}" local hostdir="${2}" @@ -62,13 +72,19 @@ elf_needs_rpath() { # This function checks whether at least one of the RPATH of the given # ELF executable (first argument) properly points to the host library # directory (second argument), either through an absolute RPATH or a -# relative RPATH. Having such a RPATH will make sure the ELF -# executable will find at runtime the shared libraries it depends -# on. This function returns 0 when a proper RPATH was found, or 1 -# otherwise. +# relative RPATH. In the context of per-package directory support, +# ${hostdir} (second argument) points to the current package host +# directory. However, it is perfectly valid for an ELF binary to have +# a RPATH pointing to another package per-package host directory, +# which is why such RPATH is also accepted (the per-package directory +# gets passed as third argument). Having a RPATH pointing to the host +# directory will make sure the ELF executable will find at runtime the +# shared libraries it depends on. This function returns 0 when a +# proper RPATH was found, or 1 otherwise. check_elf_has_rpath() { local file="${1}" local hostdir="${2}" + local perpackagedir="${3}" local rpath dir while read rpath; do @@ -77,6 +93,12 @@ check_elf_has_rpath() { dir="$( sed -r -e 's:/+:/:g; s:/$::;' <<<"${dir}" )" [ "${dir}" = "${hostdir}/lib" ] && return 0 [ "${dir}" = "\$ORIGIN/../lib" ] && return 0 + # This check is done even for builds where + # BR2_PER_PACKAGE_DIRECTORIES is disabled. In this case, + # PER_PACKAGE_DIR and therefore ${perpackagedir} points to + # a non-existent directory, and this check will always be + # false. + [[ ${dir} =~ ${perpackagedir}/[^/]+/host/lib ]] && return 0 done done < <( readelf -d "${file}" \ |sed -r -e '/.* \(R(UN)?PATH\) +Library r(un)?path: \[(.+)\]$/!d' \ diff --git a/support/scripts/fix-rpath b/support/scripts/fix-rpath index fa138ca15a..9fc9ef8514 100755 --- a/support/scripts/fix-rpath +++ b/support/scripts/fix-rpath @@ -127,14 +127,29 @@ main() { while read file ; do # check if it's an ELF file - if ${PATCHELF} --print-rpath "${file}" > /dev/null 2>&1; then - # make files writable if necessary - changed=$(chmod -c u+w "${file}") - # call patchelf to sanitize the rpath - ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}" - # restore the original permission - test "${changed}" != "" && chmod u-w "${file}" + rpath=$(${PATCHELF} --print-rpath "${file}" 2>&1) + if test $? -ne 0 ; then + continue fi + + # make files writable if necessary + changed=$(chmod -c u+w "${file}") + + # With per-package directory support, most RPATH of host + # binaries will point to per-package directories. This won't + # work with the --make-rpath-relative ${rootdir} invocation as + # the per-package host directory is not within ${rootdir}. So, + # we rewrite all RPATHs pointing to per-package directories so + # that they point to the global host directry. + changed_rpath=$(echo ${rpath} | sed "s@${PER_PACKAGE_DIR}/[^/]+/host@${HOST_DIR}@") + if test "${rpath}" != "${changed_rpath}" ; then + ${PATCHELF} --set-rpath ${changed_rpath} "${file}" + fi + + # call patchelf to sanitize the rpath + ${PATCHELF} --make-rpath-relative "${rootdir}" ${sanitize_extra_args[@]} "${file}" + # restore the original permission + test "${changed}" != "" && chmod u-w "${file}" done < <(find "${rootdir}" ${find_args[@]}) # Restore patched patchelf utility -- 2.21.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target Thomas Petazzoni @ 2019-11-29 14:20 ` Peter Korsgaard 2023-11-10 14:12 ` [Buildroot] [BUG?] per-package not fully working with dependencies and rebuild John Ernberg 1 sibling, 0 replies; 16+ messages in thread From: Peter Korsgaard @ 2019-11-29 14:20 UTC (permalink / raw) To: buildroot >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > This commit implements the core of the move to per-package SDK and > target directories. The main idea is that instead of having a global > output/host and output/target in which all packages install files, we > switch to per-package host and target directories, that only contain > their explicit dependencies. > There are two main benefits: > - Packages will now see only the dependencies they explicitly list in > their <pkg>_DEPENDENCIES variable, and the recursive dependencies > thereof. > - We can support top-level parallel build properly, because a package > only "sees" its own host directory and target directory, isolated > from the build of other packages that can happen in parallel. > It works as follows: > - A new output/per-package/ directory is created, which will contain > one sub-directory per package, and inside it, a "host" directory > and a "target" directory: > output/per-package/busybox/target > output/per-package/busybox/host > output/per-package/host-fakeroot/target > output/per-package/host-fakeroot/host > This output/per-package/ directory is PER_PACKAGE_DIR. > - The global TARGET_DIR and HOST_DIR variable now automatically point > to the per-package directory when PKG is defined. So whenever a > package references $(HOST_DIR) or $(TARGET_DIR) in its build > process, it effectively references the per-package host/target > directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it > is handled as well. > - Of course, packages have dependencies, so those dependencies must > be installed in the per-package host and target directories. To do > so, we simply rsync (using hard links to save space and time) the > host and target directories of the direct dependencies of the > package to the current package host and target directories. > We only need to take care of direct dependencies (and not > recursively all dependencies), because we accumulate into those > per-package host and target directories the files installed by the > dependencies. Note that this only works because we make the > assumption that one package does *not* overwrite files installed by > another package. > This is done for "extract dependencies" at the beginning of the > extract step, and for "normal dependencies" at the beginning of the > configure step. > This is basically enough to make per-package SDK and target work. The > only gotcha is that at the end of the build, output/target and > output/host are empty, which means that: > - The filesystem image creation code cannot work. > - We don't have a SDK to build code outside of Buildroot. > In order to fix this, this commit extends the target-finalize step so > that it starts by populating output/target and output/host by > rsync-ing into them the target and host directories of all packages > listed in the $(PACKAGES) variable. It is necessary to do this > sequentially in the target-finalize step and not in each > package. Doing it in package installation means that it can be done in > parallel. In that case, there is a chance that two rsyncs are creating > the same hardlink or directory at the same time, which makes one of > them fail. > This change to per-package directories has an impact on the RPATH > built into the host binaries, as those RPATH now point to various > per-package host directories, and no longer to the global host > directory. We do not try to rewrite such RPATHs during the build as > having such RPATHs is perfectly fine, but we still need to handle two > fallouts from this change: > - The check-host-rpath script, which verifies at the end of each > package installation that it has the appropriate RPATH, is modified > to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is > a correct RPAT. > - The fix-rpath script, which mungles the RPATH mainly for the SDK > preparation, is modified to rewrite the RPATH to not point to > per-package directories. Indeed the patchelf --make-rpath-relative > call only works if the RPATH points to the ROOTDIR passed as > argument, and this ROOTDIR is the global host directory. Rewriting > the RPATH to not point to per-package host directories prior to > this is an easy solution to this issue. > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> I think it is finally time to get this merged - Committed to next, thanks! -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [BUG?] per-package not fully working with dependencies and rebuild 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target Thomas Petazzoni 2019-11-29 14:20 ` Peter Korsgaard @ 2023-11-10 14:12 ` John Ernberg 1 sibling, 0 replies; 16+ messages in thread From: John Ernberg @ 2023-11-10 14:12 UTC (permalink / raw) To: Thomas Petazzoni, buildroot; +Cc: Jonas Blixt, Mårten Svanfeldt Hi Thomas, list, We are using buildroot 2023.2 with per-package setup and we've run into a scenario where rebuilds of a package do not propagate properly into the $(TARGET_DIR) area of the final image. We have custom closed source packages A and B, where B depends on A. If we make changes in A that do not require a rebuild of B, we just want to rebuild A and then build our image. However, since the $(TARGET_DIR) of A is rsync:ed into Bs per-package $(TARGET_DIR), the final image assembly ends up using whatever state of A that was last true when B was last rebuilt, because of naming. The $(TARGET_DIR) of A is rsync:ed into the final $(TARGET_DIR), and then the $(TARGET_DIR) of B is rsync:ed into the final $(TARGET_DIR), overwriting whatever A put there. Looking at pkg-utils.mk prepare-per-package-directory, the call with target is what is causing the issue. Thinking of this from a packaging aspect, the per-package $(TARGET_DIR) should only contain whatever the package itself is producing, as this matches the behavior of every package-based Linux distro and it is our impression of per-package that this is what it is meant to do. Removing this call however breaks multiple packages with errors of missing directories, which would need fixing. The most reliable fix is probably to have a step in target install that touches the basic directories before it runs any hooks and main CMDS. Does all this sound like a sensible assessment? Thanks! // John Ernberg _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 2/6] Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target Thomas Petazzoni @ 2019-11-05 16:46 ` Thomas Petazzoni 2019-11-29 14:20 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories Thomas Petazzoni ` (4 subsequent siblings) 6 siblings, 1 reply; 16+ messages in thread From: Thomas Petazzoni @ 2019-11-05 16:46 UTC (permalink / raw) To: buildroot With per-package folder support, top-level parallel build becomes safe, so we can enclose the .NOTPARALLEL statement in a !BR2_PER_PACKAGE_DIRECTORIES condition. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> --- Makefile | 19 +++++-------------- 1 file changed, 5 insertions(+), 14 deletions(-) diff --git a/Makefile b/Makefile index 823c219c86..c117da7e8c 100644 --- a/Makefile +++ b/Makefile @@ -227,21 +227,12 @@ ifeq ($(filter $(noconfig_targets),$(MAKECMDGOALS)),) -include $(BR2_CONFIG) endif -# Parallel execution of this Makefile is disabled because it changes -# the packages building order, that can be a problem for two reasons: -# - If a package has an unspecified optional dependency and that -# dependency is present when the package is built, it is used, -# otherwise it isn't (but compilation happily proceeds) so the end -# result will differ if the order is swapped due to parallel -# building. -# - Also changing the building order can be a problem if two packages -# manipulate the same file in the target directory. -# -# Taking into account the above considerations, if you still want to execute -# this top-level Makefile in parallel comment the ".NOTPARALLEL" line and -# use the -j<jobs> option when building, e.g: -# make -j$((`getconf _NPROCESSORS_ONLN`+1)) +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),) +# Disable top-level parallel build if per-package directories is not +# used. Indeed, per-package directories is necessary to guarantee +# determinism and reproducibility with top-level parallel build. .NOTPARALLEL: +endif # timezone and locale may affect build output ifeq ($(BR2_REPRODUCIBLE),y) -- 2.21.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 2/6] Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 2/6] Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y Thomas Petazzoni @ 2019-11-29 14:20 ` Peter Korsgaard 0 siblings, 0 replies; 16+ messages in thread From: Peter Korsgaard @ 2019-11-29 14:20 UTC (permalink / raw) To: buildroot >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > With per-package folder support, top-level parallel build becomes > safe, so we can enclose the .NOTPARALLEL statement in a > !BR2_PER_PACKAGE_DIRECTORIES condition. > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Committed, thanks. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target Thomas Petazzoni 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 2/6] Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y Thomas Petazzoni @ 2019-11-05 16:46 ` Thomas Petazzoni 2019-11-20 20:22 ` Matthew Weber 2019-11-29 14:24 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 4/6] package/pkg-kconfig: handle KCONFIG_DEPENDENCIES " Thomas Petazzoni ` (3 subsequent siblings) 6 siblings, 2 replies; 16+ messages in thread From: Thomas Petazzoni @ 2019-11-05 16:46 UTC (permalink / raw) To: buildroot Libtool .la files unfortunately contain a number of absolute paths, which now refer to per-package directories. Due to this, when building package A, .la files may contain absolute paths referring to directories in package B per-package sysroot. This causes some -L flags referring to other sysroot from being added, which doesn't work as the linker no longer realizes that such paths are within its sysroot. To fix this, we introduce a replacement step of .la files in the configure step, to make sure all paths refer to this package per-package directory. Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- package/pkg-generic.mk | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk index b7b211e6e4..71792d58a9 100644 --- a/package/pkg-generic.mk +++ b/package/pkg-generic.mk @@ -134,6 +134,21 @@ endef GLOBAL_INSTRUMENTATION_HOOKS += step_user endif +####################################### +# Helper functions + +# Make sure .la files only reference the current per-package +# directory. + +# $1: package name (lower case) +# $2: staging directory of the package +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y) +define fixup-libtool-files + $(Q)find $(2)/usr/lib* -name "*.la" | xargs --no-run-if-empty \ + $(SED) "s:$(PER_PACKAGE_DIR)/[^/]+/:$(PER_PACKAGE_DIR)/$(1)/:g" +endef +endif + ################################################################################ # Implicit targets -- produce a stamp file for each step of a package build ################################################################################ @@ -230,6 +245,7 @@ $(BUILD_DIR)/%/.stamp_configured: @$(call step_start,configure) @$(call MESSAGE,"Configuring") $(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES)) + $(call fixup-libtool-files,$(NAME),$(STAGING_DIR)) $(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep)) $($(PKG)_CONFIGURE_CMDS) $(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep)) @@ -919,6 +935,7 @@ $$($(2)_TARGET_INSTALL_IMAGES): PKG=$(2) $$($(2)_TARGET_INSTALL_HOST): PKG=$(2) $$($(2)_TARGET_BUILD): PKG=$(2) $$($(2)_TARGET_CONFIGURE): PKG=$(2) +$$($(2)_TARGET_CONFIGURE): NAME=$(1) $$($(2)_TARGET_RSYNC): SRCDIR=$$($(2)_OVERRIDE_SRCDIR) $$($(2)_TARGET_RSYNC): PKG=$(2) $$($(2)_TARGET_PATCH): PKG=$(2) -- 2.21.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories Thomas Petazzoni @ 2019-11-20 20:22 ` Matthew Weber 2019-11-29 14:24 ` Peter Korsgaard 1 sibling, 0 replies; 16+ messages in thread From: Matthew Weber @ 2019-11-20 20:22 UTC (permalink / raw) To: buildroot Thomas, On Tue, Nov 5, 2019 at 10:47 AM Thomas Petazzoni <thomas.petazzoni@bootlin.com> wrote: > > Libtool .la files unfortunately contain a number of absolute paths, > which now refer to per-package directories. Due to this, when building > package A, .la files may contain absolute paths referring to > directories in package B per-package sysroot. This causes some -L > flags referring to other sysroot from being added, which doesn't work > as the linker no longer realizes that such paths are within its > sysroot. Building on this top level parallel make series and disabling the "pkg size and bin arch hooks" first (https://pastebin.com/nUb3Qsgs) BR2_aarch64=y BR2_PER_PACKAGE_DIRECTORIES=y BR2_TOOLCHAIN_EXTERNAL=y BR2_INIT_NONE=y # BR2_TARGET_ENABLE_ROOT_LOGIN is not set # BR2_TARGET_GENERIC_GETTY is not set BR2_PACKAGE_ZLIB=y BR2_PACKAGE_LIBXSLT=y # BR2_TARGET_ROOTFS_TAR is not set Then I do a 'make -s -j 32' . Note the number of jobs doesn't seem to matter (still fails without -j set). I get the following error and it looks like all the sysroot paths are valid but it can't find the standard lib.... https://pastebin.com/2W1EKjwd Digging a bit more and I notice the failing build has an extra -L path stuck towards the end of the link line after "-lxml2". I noticed this extra -L path goes away and the build is successful with zlib disabled. I haven't tested but I believe I had a build where the external toolchain includes zlib (bootlin buildroot internal build) and it doesn't appear to include this extra path. I'm guessing the failure is related to how zlib builds with per pkg or a fixup when moving files around. I did check the zlib.pc and it was clean (below). $ cat build/libzlib-1.2.11/zlib.pc prefix=/usr exec_prefix=${prefix} libdir=${exec_prefix}/lib sharedlibdir=${libdir} includedir=${prefix}/include Name: zlib Description: zlib compression library Version: 1.2.11 Requires: Libs: -L${libdir} -L${sharedlibdir} -lz Cflags: -I${includedir} Ideas welcome :-) Regards, Matt ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories Thomas Petazzoni 2019-11-20 20:22 ` Matthew Weber @ 2019-11-29 14:24 ` Peter Korsgaard 1 sibling, 0 replies; 16+ messages in thread From: Peter Korsgaard @ 2019-11-29 14:24 UTC (permalink / raw) To: buildroot >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > Libtool .la files unfortunately contain a number of absolute paths, > which now refer to per-package directories. Due to this, when building > package A, .la files may contain absolute paths referring to > directories in package B per-package sysroot. This causes some -L > flags referring to other sysroot from being added, which doesn't work > as the linker no longer realizes that such paths are within its > sysroot. > To fix this, we introduce a replacement step of .la files in the > configure step, to make sure all paths refer to this package > per-package directory. > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Committed to next, thanks. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 4/6] package/pkg-kconfig: handle KCONFIG_DEPENDENCIES with per-package directories 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni ` (2 preceding siblings ...) 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories Thomas Petazzoni @ 2019-11-05 16:46 ` Thomas Petazzoni 2019-11-29 14:39 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 5/6] docs/manual: add details about top-level parallel build support Thomas Petazzoni ` (2 subsequent siblings) 6 siblings, 1 reply; 16+ messages in thread From: Thomas Petazzoni @ 2019-11-05 16:46 UTC (permalink / raw) To: buildroot The pkg-kconfig infrastructure hijacks the regular chain of build steps to insert its own step to prepare the configuration of kconfig packages. This additional step may have dependencies of its own, such as host-flex, host-bison or toolchain. In the context of per-package directory support, those dependencies must be copied to the per-package directory of the current package prior to doing the config preparation. This commit implements this logic by adding a call to prepare-per-package-directory at the right spot. Reported-by: Andreas Naumann <anaumann@ultratronik.de> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> --- package/pkg-kconfig.mk | 1 + 1 file changed, 1 insertion(+) diff --git a/package/pkg-kconfig.mk b/package/pkg-kconfig.mk index 86d7c14fdb..e435b95bcf 100644 --- a/package/pkg-kconfig.mk +++ b/package/pkg-kconfig.mk @@ -143,6 +143,7 @@ $(2)_KCONFIG_RULES = \ # Since the file could be a defconfig file it needs to be expanded to a # full .config first. $$($(2)_DIR)/$$($(2)_KCONFIG_STAMP_DOTCONFIG): $$($(2)_KCONFIG_FILE) $$($(2)_KCONFIG_FRAGMENT_FILES) + $$(call prepare-per-package-directory,$$($(2)_KCONFIG_DEPENDENCIES)) $$(call kconfig-package-merge-config,$(2),$$(@D)/$$($(2)_KCONFIG_DOTCONFIG),\ $$($(2)_KCONFIG_FRAGMENT_FILES)) $$(Q)touch $$(@D)/$$($(2)_KCONFIG_STAMP_DOTCONFIG) -- 2.21.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 4/6] package/pkg-kconfig: handle KCONFIG_DEPENDENCIES with per-package directories 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 4/6] package/pkg-kconfig: handle KCONFIG_DEPENDENCIES " Thomas Petazzoni @ 2019-11-29 14:39 ` Peter Korsgaard 0 siblings, 0 replies; 16+ messages in thread From: Peter Korsgaard @ 2019-11-29 14:39 UTC (permalink / raw) To: buildroot >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > The pkg-kconfig infrastructure hijacks the regular chain of build > steps to insert its own step to prepare the configuration of kconfig > packages. This additional step may have dependencies of its own, such > as host-flex, host-bison or toolchain. > In the context of per-package directory support, those dependencies > must be copied to the per-package directory of the current package > prior to doing the config preparation. This commit implements this > logic by adding a call to prepare-per-package-directory at the right > spot. > Reported-by: Andreas Naumann <anaumann@ultratronik.de> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > Acked-by: "Yann E. MORIN" <yann.morin.1998@free.fr> Committed to next, thanks. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 5/6] docs/manual: add details about top-level parallel build support 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni ` (3 preceding siblings ...) 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 4/6] package/pkg-kconfig: handle KCONFIG_DEPENDENCIES " Thomas Petazzoni @ 2019-11-05 16:46 ` Thomas Petazzoni 2019-11-29 14:43 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 6/6] docs/manual: document the effect of per-package directory on variables Thomas Petazzoni 2019-11-06 19:37 ` [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Bernd Kuhls 6 siblings, 1 reply; 16+ messages in thread From: Thomas Petazzoni @ 2019-11-05 16:46 UTC (permalink / raw) To: buildroot Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- docs/manual/common-usage.txt | 47 +++++++++++++++++++++++++++++ docs/manual/faq-troubleshooting.txt | 3 ++ docs/manual/quickstart.txt | 8 ++--- 3 files changed, 54 insertions(+), 4 deletions(-) diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt index a87980d4a2..2d315de440 100644 --- a/docs/manual/common-usage.txt +++ b/docs/manual/common-usage.txt @@ -347,6 +347,53 @@ Refer to the help text of this script for more details: utils/size-stats-compare -h ---------------- +[[top-level-parallel-build]] +=== Top-level parallel build + +.Note +This section deals with a very experimental feature, which is known to +break even in some non-unusual situations. Use at your own risk. + +Buildroot has always been capable of using parallel build on a per +package basis: each package is built by Buildroot using +make -jN+ (or +the equivalent invocation for non-make-based build systems). The level +of parallelism is by default number of CPUs + 1, but it can be +adjusted using the +BR2_JLEVEL+ configuration option. + +Until 2019.02, Buildroot was however building packages in a serial +fashion: each package was built one after the other, without +parallelization of the build between packages. As of 2019.02, +Buildroot has experimental support for *top-level parallel build*, +which allows some signicant build time savings by building packages +that have no dependency relationship in parallel. This feature is +however marked as experimental and is known not to work in some cases. + +In order to use top-level parallel build, one must: + +. Enable the option +BR2_PER_PACKAGE_DIRECTORIES+ in the Buildroot +configuration + +. Use +make -jN+ when starting the Buildroot build + +Internally, the +BR2_PER_PACKAGE_DIRECTORIES+ will enable a mechanism +called *per-package directories*, which will have the following +effects: + +* Instead of a global _target_ directory and a global _host_ directory + common to all packages, per-package _target_ and _host_ directories + will be used, in +$(O)/per-package/<pkg>/target/+ and + +$(O)/per-package/<pkg>/host/+ respectively. Those folders will be + populated from the corresponding folders of the package dependencies + at the beginning of +<pkg>+ build. The compiler and all other tools + will therefore only be able to see and access files installed by + dependencies explicitly listed by +<pkg>+. + +* At the end of the build, the global _target_ and _host_ directories + will be populated, located in +$(O)/target+ and +$(O)/host+ + respectively. This means that during the build, those folders will + be empty and it's only at the very end of the build that they will + be populated. + include::eclipse-integration.txt[] include::advanced.txt[] diff --git a/docs/manual/faq-troubleshooting.txt b/docs/manual/faq-troubleshooting.txt index b144c9e7f0..5adf3fa6ce 100644 --- a/docs/manual/faq-troubleshooting.txt +++ b/docs/manual/faq-troubleshooting.txt @@ -239,3 +239,6 @@ help reduce the build time: * Buy new hardware. SSDs and lots of RAM are key to speeding up the builds. + + * Experiment with top-level parallel build, see + xref:top-level-parallel-build[]. diff --git a/docs/manual/quickstart.txt b/docs/manual/quickstart.txt index 74158ae249..77b73ef116 100644 --- a/docs/manual/quickstart.txt +++ b/docs/manual/quickstart.txt @@ -60,10 +60,10 @@ To start the build process, simply run: $ make -------------------- -You *should never* use +make -jN+ with Buildroot: top-level parallel -make is currently not supported. Instead, use the +BR2_JLEVEL+ option -to tell Buildroot to run the compilation of each individual package -with +make -jN+. +By default, Buildroot does not support top-level parallel build, so +running +make -jN+ is not necessary. There is however experimental +support for top-level parallel build, see +xref:top-level-parallel-build[]. The `make` command will generally perform the following steps: -- 2.21.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 5/6] docs/manual: add details about top-level parallel build support 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 5/6] docs/manual: add details about top-level parallel build support Thomas Petazzoni @ 2019-11-29 14:43 ` Peter Korsgaard 0 siblings, 0 replies; 16+ messages in thread From: Peter Korsgaard @ 2019-11-29 14:43 UTC (permalink / raw) To: buildroot >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> > --- > docs/manual/common-usage.txt | 47 +++++++++++++++++++++++++++++ > docs/manual/faq-troubleshooting.txt | 3 ++ > docs/manual/quickstart.txt | 8 ++--- > 3 files changed, 54 insertions(+), 4 deletions(-) > diff --git a/docs/manual/common-usage.txt b/docs/manual/common-usage.txt > index a87980d4a2..2d315de440 100644 > --- a/docs/manual/common-usage.txt > +++ b/docs/manual/common-usage.txt > @@ -347,6 +347,53 @@ Refer to the help text of this script for more details: > utils/size-stats-compare -h > ---------------- > +[[top-level-parallel-build]] > +=== Top-level parallel build > + > +.Note > +This section deals with a very experimental feature, which is known to > +break even in some non-unusual situations. Use at your own risk. > + > +Buildroot has always been capable of using parallel build on a per > +package basis: each package is built by Buildroot using +make -jN+ (or > +the equivalent invocation for non-make-based build systems). The level > +of parallelism is by default number of CPUs + 1, but it can be > +adjusted using the +BR2_JLEVEL+ configuration option. > + > +Until 2019.02, Buildroot was however building packages in a serial Committed to next after changing the version reference to Buildroot 2020.02, thanks. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 6/6] docs/manual: document the effect of per-package directory on variables 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni ` (4 preceding siblings ...) 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 5/6] docs/manual: add details about top-level parallel build support Thomas Petazzoni @ 2019-11-05 16:46 ` Thomas Petazzoni 2019-11-29 14:43 ` Peter Korsgaard 2019-11-06 19:37 ` [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Bernd Kuhls 6 siblings, 1 reply; 16+ messages in thread From: Thomas Petazzoni @ 2019-11-05 16:46 UTC (permalink / raw) To: buildroot As suggested by Arnout Vandecappelle, this commit adjusts the generic-package documentation to document the effect of per-package directory support on HOST_DIR, STAGING_DIR and TARGET_DIR. Suggested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> --- docs/manual/adding-packages-generic.txt | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/manual/adding-packages-generic.txt b/docs/manual/adding-packages-generic.txt index b2386f2c01..86300e7460 100644 --- a/docs/manual/adding-packages-generic.txt +++ b/docs/manual/adding-packages-generic.txt @@ -586,6 +586,13 @@ In the action definitions, you can use the following variables: * +$(TARGET_CROSS)+ to get the cross-compilation toolchain prefix * Of course the +$(HOST_DIR)+, +$(STAGING_DIR)+ and +$(TARGET_DIR)+ - variables to install the packages properly. + variables to install the packages properly. Those variables point to + the global _host_, _staging_ and _target_ directories, unless + _per-package directory_ support is used, in which case they point to + the current package _host_, _staging_ and _target_ directories. In + both cases, it doesn't make any difference from the package point of + view: it should simply use +HOST_DIR+, +STAGING_DIR+ and + +TARGET_DIR+. See xref:top-level-parallel-build[] for more details + about _per-package directory_ support. Finally, you can also use hooks. See xref:hooks[] for more information. -- 2.21.0 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 6/6] docs/manual: document the effect of per-package directory on variables 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 6/6] docs/manual: document the effect of per-package directory on variables Thomas Petazzoni @ 2019-11-29 14:43 ` Peter Korsgaard 0 siblings, 0 replies; 16+ messages in thread From: Peter Korsgaard @ 2019-11-29 14:43 UTC (permalink / raw) To: buildroot >>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@bootlin.com> writes: > As suggested by Arnout Vandecappelle, this commit adjusts the > generic-package documentation to document the effect of per-package > directory support on HOST_DIR, STAGING_DIR and TARGET_DIR. > Suggested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be> > Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com> Committed, thanks. -- Bye, Peter Korsgaard ^ permalink raw reply [flat|nested] 16+ messages in thread
* [Buildroot] [PATCH next v8 0/6] Top-level parallel build support 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni ` (5 preceding siblings ...) 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 6/6] docs/manual: document the effect of per-package directory on variables Thomas Petazzoni @ 2019-11-06 19:37 ` Bernd Kuhls 6 siblings, 0 replies; 16+ messages in thread From: Bernd Kuhls @ 2019-11-06 19:37 UTC (permalink / raw) To: buildroot Am Tue, 05 Nov 2019 17:46:39 +0100 schrieb Thomas Petazzoni: > This is really only the core changes. It is sufficient for a number of > configuration, but we know some specific packages are broken with > per-package directories (for example: Qt5). Hi, are you, at this stage, interested in per-package bug reports? If yes, this defconfig does not build: BR2_x86_64=y BR2_x86_core2=y BR2_PER_PACKAGE_DIRECTORIES=y BR2_TOOLCHAIN_EXTERNAL=y BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/ tarballs/br-x86-64-core2-full-2019.05.1.tar.bz2" BR2_TOOLCHAIN_EXTERNAL_GCC_7=y BR2_TOOLCHAIN_EXTERNAL_HEADERS_5_1=y BR2_TOOLCHAIN_EXTERNAL_LOCALE=y # BR2_TOOLCHAIN_EXTERNAL_HAS_THREADS_DEBUG is not set BR2_TOOLCHAIN_EXTERNAL_CXX=y BR2_PACKAGE_APR_UTIL=y >>> apr-util 1.6.1 Building PATH="/home/buildroot/br/output/per-package/apr-util/host/bin:/home/ buildroot/br/output/per-package/apr-util/host/sbin:/usr/lib/android-ndk:/ usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games" /usr/bin/make -C /home/buildroot/br/output/build/apr-util-1.6.1/ make[1]: Entering directory '/home/buildroot/br/output/build/apr- util-1.6.1' make[2]: Entering directory '/home/buildroot/br/output/build/apr- util-1.6.1' /bin/bash /home/buildroot/br/output/per-package/apr/host/x86_64-buildroot- linux-uclibc/sysroot/usr/build-1/libtool --silent --mode=compile /home/ buildroot/br/output/per-package/apr/host/bin/x86_64-linux-gcc - D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os - DLINUX -D_GNU_SOURCE -D_REENTRANT -DHAVE_CONFIG_H -D_LARGEFILE_SOURCE - D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -I/home/buildroot/br/output/ build/apr-util-1.6.1/include -I/home/buildroot/br/output/build/apr- util-1.6.1/include/private -I/home/buildroot/br/output/per-package/apr- util/host/x86_64-buildroot-linux-uclibc/sysroot/usr/bin/../../usr/include/ apr-1 -o xml/apr_xml.lo -c xml/apr_xml.c && touch xml/apr_xml.lo xml/apr_xml.c:35:10: fatal error: expat.h: No such file or directory #include <expat.h> ^~~~~~~~~ compilation terminated. Regards, Bernd ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2023-11-10 14:12 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-11-05 16:46 [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Thomas Petazzoni 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 1/6] core: implement per-package SDK and target Thomas Petazzoni 2019-11-29 14:20 ` Peter Korsgaard 2023-11-10 14:12 ` [Buildroot] [BUG?] per-package not fully working with dependencies and rebuild John Ernberg 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 2/6] Makefile: allow top-level parallel build with BR2_PER_PACKAGE_DIRECTORIES=y Thomas Petazzoni 2019-11-29 14:20 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 3/6] package/pkg-generic: make libtool .la files compatible with per-package directories Thomas Petazzoni 2019-11-20 20:22 ` Matthew Weber 2019-11-29 14:24 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 4/6] package/pkg-kconfig: handle KCONFIG_DEPENDENCIES " Thomas Petazzoni 2019-11-29 14:39 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 5/6] docs/manual: add details about top-level parallel build support Thomas Petazzoni 2019-11-29 14:43 ` Peter Korsgaard 2019-11-05 16:46 ` [Buildroot] [PATCH next v8 6/6] docs/manual: document the effect of per-package directory on variables Thomas Petazzoni 2019-11-29 14:43 ` Peter Korsgaard 2019-11-06 19:37 ` [Buildroot] [PATCH next v8 0/6] Top-level parallel build support Bernd Kuhls
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.