[Buildroot] [PATCH 0/2 v2] core: fixup PPD paths (branch yem/ppd-fixup-paths)

[Buildroot] [PATCH 0/2 v2] core: fixup PPD paths (branch yem/ppd-fixup-paths)

[Yann E. MORIN]

Hello All!

This series builds upon previous good work from Andreas [0], Louis-Paul
[1], and Adam [2].

The first patch is the core of the change: rather than have multiple
hooks that each cater for different types of files (.py, .pc, .pri,
.cmake and so on), we jut do what we already do in relocate-sdk: we just
look for all text files that have a PPD prefix in them, and replace tht
with the current package's PPD.

The second patch ensures that we can build reliable files lists, so that
files we tweak in the infra are not believed to be installed/modified by
the current package.

[0] https://lore.kernel.org/buildroot/20200217212350.29750-21-anaumann@ultratronik.de/
[1] https://lore.kernel.org/buildroot/a339f273-33f3-f232-eac4-6e50427abf6d@cordier.org/
[2] https://lore.kernel.org/buildroot/20220106171720.12857-1-aduskett@gmail.com/

Changes v1 -> v2:
  - use --binary-files=without-match, as this *considerably* speeds up
    the fixup  (Arnout)
  - rename the python fixup  (Arnout)

Regards,
Yann E. MORIN.


----------------------------------------------------------------
Yann E. MORIN (2):
      core/pkg-generic: fixup all PPD paths in a generic fashion
      core/pkg-generic: apply post-prepare hooks before monitoring directories

 package/pkg-generic.mk | 47 +++++++++++++++++++++++------------------------
 1 file changed, 23 insertions(+), 24 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/2 v2] core/pkg-generic: fixup all PPD paths in a generic fashion

[Yann E. MORIN]

Some files contain hard-coded absolute paths that point to the host
and/or staging directories.

With per-package directories (aka. PPD), these paths point to the PPD
of the package that created the files, when we want them to point to the
PPD of the package that uses them.

Up until now, we had two hooks that attempted to fix those files:

  - a libtool-specific hook that searches for all .la files and seds
    them with the proper PPD,

  - a python-specific hook that tweaks just the sysconfigdata and
    removes the byte-compiled version of the sysconfigdata.

But now, we also have a few other kinds of files for which we need to
fix the PPD: .cmake, .pc, or .pri files, and probably a bunch of others
as well.

We solve this issue by just replacing any PPD in text files, with the
current package's PPD.

This is very similar to, and inspired from what is done when relocating
the SDK. However, we can't use the existing relocate-sdk script, because
that needs to know the original location, which we do not have when we
aggregate the PPD (we could store it, but we can easily do without it).

Furthermore, we use a construct that is way more efficient than
relocate-sdk. First, we skip binary files with grep, which means we have
way less files to check with 'file' [0]. Second, we use xargs to sed
multiple files at once: printf is a shell built-in, so it's fast, and so
we do not have to spawn a sed for each file to fixup.

[0] We still keep using 'file' as a safety net, to avoid mangling a
binary file that grep would have missed.

Finally, the existing python-specific macro is simplified to just remove
the pre-compiled sysconfigdata files. And we rename it accordingly.

And as for some timings, to see the impact, with the defconfig below,
and with the downloads already local, and with a PC mostly idle (mail
and IRC activity only):

                    Before      Now         Delta
  - without PPD :   7min 27s    7min 23s    -0.9%
  - with PPD    :   7min 51s    7min 59s    +1.7%
  - with PPD -j8:   5min 51s    5min 56s    +1.4%

So we can see a slight increase in time, but it is mostly in the noise
(some builds without this change did exceed some builds with this
change, due to background noise). Also, depending on scheduling, there
can be less parallelism; for example, python3 does not build in
parallel, and with this special defconfig, python is on the critical
path of a lot of packages that are python modules, which can negatively
impact a parallel build too. A more realistic, bigger defconfig would
probably be more parallel... YMMV...

Delta without PPD is also due to background noise, as those hooks are
not used when PPD is not enabled.

Defconfig used:
    BR2_arm=y
    BR2_cortex_a7=y
    BR2_TOOLCHAIN_EXTERNAL=y
    BR2_PACKAGE_PYTHON3=y
    BR2_PACKAGE_PYTHON_AIOBLESCAN=y
    BR2_PACKAGE_PYTHON_AIOCOAP=y
    BR2_PACKAGE_PYTHON_AIOFILES=y
    BR2_PACKAGE_PYTHON_AIOHTTP_CORS=y
    BR2_PACKAGE_PYTHON_AIOHTTP_DEBUGTOOLBAR=y
    BR2_PACKAGE_PYTHON_AIOHTTP_MAKO=y
    BR2_PACKAGE_PYTHON_AIOHTTP_REMOTES=y
    BR2_PACKAGE_PYTHON_AIOHTTP_SECURITY=y
    BR2_PACKAGE_PYTHON_AIOHTTP_SESSION=y
    BR2_PACKAGE_PYTHON_AIOHTTP_SSE=y
    BR2_PACKAGE_PYTHON_AIOJOBS=y
    BR2_PACKAGE_PYTHON_AIOLOGSTASH=y
    BR2_PACKAGE_PYTHON_AIOMONITOR=y
    BR2_PACKAGE_PYTHON_AIOPROCESSING=y
    BR2_PACKAGE_PYTHON_AIOREDIS=y
    BR2_PACKAGE_PYTHON_AIORWLOCK=y
    BR2_PACKAGE_PYTHON_AIOZIPKIN=y
    BR2_PACKAGE_LIGHTTPD=y
    BR2_PACKAGE_LIGHTTPD_OPENSSL=y
    BR2_PACKAGE_LIGHTTPD_ZLIB=y
    BR2_PACKAGE_LIGHTTPD_BZIP2=y
    BR2_PACKAGE_LIGHTTPD_PCRE=y
    BR2_PACKAGE_LIGHTTPD_WEBDAV=y
    # BR2_TARGET_ROOTFS_TAR is not set

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Cc: Adam Duskett <aduskett@gmail.com>
Cc: Louis-Paul CORDIER <lpdev@cordier.org>
Cc: Andreas Naumann <anaumann@ultratronik.de>

---
Changes v1 -> v2:
  - use grep --binary-files=without-match as it *considerably* speeds
    up the fixup  (Arnout)
  - rename the python hook  (arnout)
---
 package/pkg-generic.mk | 45 +++++++++++++++++++++---------------------
 1 file changed, 22 insertions(+), 23 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 6a5fe5507b..78226ee266 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -90,21 +90,24 @@ endif
 #######################################
 # Helper functions
 
-# Make sure .la files only reference the current per-package
-# directory.
-
-# $1: package name (lower case)
-# $2: staging directory of the package
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define fixup-libtool-files
-	$(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \
-		-name "*.la" -print0 | xargs -0 --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"
+
+# Ensure files like .la, .pc, .pri, .cmake, and so on, point to the
+# proper staging and host directories for the current package: find
+# all text files that contain the PPD root, and replace it with the
+# current package's PPD.
+define PPD_FIXUP_PATHS
+	$(Q)grep --binary-files=without-match -lrZ '$(PER_PACKAGE_DIR)/[^/]\+/' $(HOST_DIR) \
+	|while read -d '' f; do \
+		file -b --mime-type "$${f}" | grep -q '^text/' || continue; \
+		printf '%s\0' "$${f}"; \
+	done \
+	|xargs -0 --no-run-if-empty \
+		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
 endef
-endif
 
-# Make sure python _sysconfigdata*.py files only reference the current
-# per-package directory.
+# Remove python's pre-compiled "sysconfigdata", as it may contain paths to
+# the original staging or host dirs.
 #
 # Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...)
 # because those directories may be created in the same recipe this macro will
@@ -113,19 +116,15 @@ endif
 # fail.
 # So we just use HOST_DIR as a starting point, and filter on the two directories
 # of interest.
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define FIXUP_PYTHON_SYSCONFIGDATA
+define PPD_PYTHON_REMOVE_SYSCONFIGDATA_PYC
 	$(Q)find $(HOST_DIR) \
 		\(    -path '$(HOST_DIR)/lib/python*' \
 		   -o -path '$(STAGING_DIR)/usr/lib/python*' \
 		\) \
-		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
-		   -o \( -name "_sysconfigdata*.py" -print0 \) \
-		\) \
-	| xargs -0 --no-run-if-empty \
-		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
+		\( -name "_sysconfigdata*.pyc" -delete \)
 endef
-endif
+
+endif  # PPD
 
 # Functions to collect statistics about installed files
 
@@ -278,8 +277,6 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(BINARIES_DIR),-images)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
-	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
-	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
@@ -836,7 +833,9 @@ $(2)_EXTRACT_CMDS ?= \
 		$$(TAR_OPTIONS) -)
 
 # pre/post-steps hooks
-$(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA
+$(2)_POST_PREPARE_HOOKS += \
+	PPD_FIXUP_PATHS \
+	PPD_PYTHON_REMOVE_SYSCONFIGDATA_PYC
 
 ifeq ($$($(2)_TYPE),target)
 ifneq ($$(HOST_$(2)_KCONFIG_VAR),)
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2 v2] core/pkg-generic: apply post-prepare hooks before monitoring directories

[Yann E. MORIN]

Monitoring the target/ and host/ directories and so on, will serve to
generate lists of files installed by the packages. Those lists are then
used to generate graphs of the size those package take on the target
for example.

With PPD, we will also want to use those lists to only copy those files
actually installed by each dependencies of a package, recursively.

Currently, those lists are not entirely reliable, as the starting points
are established before we apply PPD fixup hooks. As such, at the end of
a package installation, fixed up files will be found to belong to the
current package, while they were in fact provided by one of its
dependency.

While this does no big harm, if at all, for the size graphs, it will
trigger overwrite detection when we eventually gather packages together
to aggregate a PPD or te final host and target. So, we better have the
lists of files be reliable.

So, we only start monitoring the directories after we apply the PPD
fixups (or seen the other way around for a smaller diff: we apply the
PPD fixups before we start monitoring the directories).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Herve Codina <herve.codina@bootlin.com>
---
 package/pkg-generic.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 78226ee266..b3a7e1d60e 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -273,11 +273,11 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call MESSAGE,"Configuring")
 	$(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
+	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(BINARIES_DIR),-images)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
-	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
-- 
2.25.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 0/2 v2] core: fixup PPD paths (branch yem/ppd-fixup-paths)

[Arnout Vandecappelle]

On 22/01/2022 19:13, Yann E. MORIN wrote:
> Hello All!
> 
> This series builds upon previous good work from Andreas [0], Louis-Paul
> [1], and Adam [2].
> 
> The first patch is the core of the change: rather than have multiple
> hooks that each cater for different types of files (.py, .pc, .pri,
> .cmake and so on), we jut do what we already do in relocate-sdk: we just
> look for all text files that have a PPD prefix in them, and replace tht
> with the current package's PPD.
> 
> The second patch ensures that we can build reliable files lists, so that
> files we tweak in the infra are not believed to be installed/modified by
> the current package.
> 
> [0] https://lore.kernel.org/buildroot/20200217212350.29750-21-anaumann@ultratronik.de/
> [1] https://lore.kernel.org/buildroot/a339f273-33f3-f232-eac4-6e50427abf6d@cordier.org/
> [2] https://lore.kernel.org/buildroot/20220106171720.12857-1-aduskett@gmail.com/
> 
> Changes v1 -> v2:
>    - use --binary-files=without-match, as this *considerably* speeds up
>      the fixup  (Arnout)
>    - rename the python fixup  (Arnout)

  Both applied to master, thanks.

  Regards,
  Arnout

> 
> Regards,
> Yann E. MORIN.
> 
> 
> ----------------------------------------------------------------
> Yann E. MORIN (2):
>        core/pkg-generic: fixup all PPD paths in a generic fashion
>        core/pkg-generic: apply post-prepare hooks before monitoring directories
> 
>   package/pkg-generic.mk | 47 +++++++++++++++++++++++------------------------
>   1 file changed, 23 insertions(+), 24 deletions(-)
> 
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot