[Buildroot] [PATCH v3 00/16] Overwritten file detection and fixes, one more step to TLP build

[Buildroot] [PATCH v3 00/16] Overwritten file detection and fixes, one more step to TLP build

[Herve Codina]

Hi,

This v3 series adds a new patch (patch 1) to fix a find failure already present
in PKG_PYTHON_FIXUP_SYSCONFIG that was highlighted in the previous version
of this series.
It also reworks the python fixup according to previous review (patches 2 and 3)
and fixes the test given in patch 12.

Two patches are added related to Qt5. The first one (patch 5) was previously
sent by Andreas Naumann and fixes a Qt5 overwrite. 
https://lore.kernel.org/buildroot/20210623083927.8278-1-anaumann@ultratronik.de/
https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
The second one (patch 6) moves the fixup function to the post-prepare hooks.

One more patch was added in this serie (patch 14) to move fixup-libtool-file to
the post-prepare hooks. This was mentionned during the review of the first
version of this series (https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/).

More changes details are given in each individual patches.

Best regards,
Hervé Codina

Andreas Naumann (1):
  qt5: Fix sporadic build failure during top-level parallel build

Herve Codina (11):
  package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA
  package/pkg-python: invalidate precompiled _sysconfigdata*.pyc
  package/pkg-generic.mk: move python fixup to generic package
    infrastructure
  package/owfs: remove Python sysconfigdata fixup
  package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook
  package/pkg-generic.mk: generate final rsync exclude file list
  Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
  Makefile: breaks hardlinks in global {TARGET,HOST}_DIR on per-package
    build
  package/pkg-generic.mk: fix per-package
    <pkg>-{reconfigure,rebuild,reinstall}
  package/pkg-generic.mk: remove .files-final-rsync.before temporary
    file
  package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook

Thomas Petazzoni (4):
  package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
    HOST_DIR
  support/testing/infra: add log_file_path() function
  support/testing/tests: add test for check_bin_arch
  support/testing/tests: add test for file overwrite detection

 Makefile                                      |  34 ++++-
 package/owfs/owfs.mk                          |   9 --
 package/pkg-generic.mk                        | 125 ++++++++++++++++--
 package/pkg-python.mk                         |  10 --
 package/pkg-qmake.mk                          |   3 +-
 package/qt5/qt5.mk                            |   1 +
 support/testing/infra/__init__.py             |  11 +-
 .../br2-external/detect-bad-arch/Config.in    |   1 +
 .../detect-bad-arch/external.desc             |   1 +
 .../br2-external/detect-bad-arch/external.mk  |   1 +
 .../package/detect-bad-arch/Config.in         |   4 +
 .../detect-bad-arch/detect-bad-arch.mk        |  15 +++
 .../br2-external/detect-overwrite/Config.in   |   1 +
 .../detect-overwrite/external.desc            |   1 +
 .../br2-external/detect-overwrite/external.mk |   1 +
 .../package/detect-overwrite/Config.in        |   5 +
 .../detect-overwrite/detect-overwrite.mk      |  19 +++
 support/testing/tests/core/test_bad_arch.py   |  19 +++
 .../testing/tests/core/test_file_overwrite.py |  48 +++++++
 19 files changed, 269 insertions(+), 40 deletions(-)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py
 create mode 100644 support/testing/tests/core/test_file_overwrite.py

-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 01/16] package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA

[Herve Codina]

Find used by PKG_PYTHON_FIXUP_SYSCONFIGDATA can fail if directories
are not present. This failure is silently ignored because find is
on the LHS of a pipe.

This commit fixes the find failure. HOST_DIR is used as the starting
point and the search is filtered on the expected directories.

This commit also adds -print0 and the $(Q) verbosity flag as minor
changes.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---

New patch in this v3 series.

The find failure was highlighted in 836528f03eb89fdc64432f7a7470145ddf553b8c
and reverted in b928074e2d83f5b873bf1ba381b212dfad9bf207.

 package/pkg-python.mk | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 59a48e5a87..cd48402e02 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -92,11 +92,22 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 	--root=/ \
 	--single-version-externally-managed
 
+# Make sure python _sysconfigdata*.py files only reference the current
+# per-package directory.
+#
+# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...)
+# because those directories may be created in the same recipe this macro will
+# be expanded in.
+# Additionally, either or both may be missing, which would make find whine and
+# fail.
+# So we just use HOST_DIR as a starting point, and filter on the two directories
+# of interest.
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
+	$(Q)find $(HOST_DIR) \
+		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
+		-name "_sysconfigdata*.py" -print0 | xargs -0 --no-run-if-empty \
+		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
 endef
 endif
 
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 02/16] package/pkg-python: invalidate precompiled _sysconfigdata*.pyc

[Herve Codina]

For per-package directories, we fixup the _sysconfigdata*.py files, so
that they get proper path pointing to the current package's direcotry
structure.

However, the corresponding, pre-compiled blobs _sysconfigdata*.pyc were
left around, and thus are inconsistent with their source. They might
also be regenerated when a package would install a python module; this
regeneration would trigger the soon-to-be-introduced overwrite
detection.

This commit simply removes _sysconfigdata*.pyc files; they will anyway
be regenerated by the PYTHON{,3}_CREATE_PYC_FILES target finalize hooks.
This is an efficient way to guarantee the consistency between the source
and precompiled versions, and to not trigger the overwrite detection.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
[yann.morin.1998@free.frs: reword the commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
Changes from v1 to v2:
 - Used separate shells for the two 'find' commands.
 - Used '-deleted' to remove files

Changes v2 to v3:
 - Reworked the find command to avoid failure highlighted in 836528f03eb89fdc64432f7a7470145ddf553b8c
   and reverted in b928074e2d83f5b873bf1ba381b212dfad9bf207.
 - Used find syntax discussed in the previous review.
   https://lore.kernel.org/buildroot/20210709084820.37830e5f@bootlin.com/
 - Kept Yann's commit log.

 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index cd48402e02..0f65bd0f75 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -106,7 +106,9 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 	$(Q)find $(HOST_DIR) \
 		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
-		-name "_sysconfigdata*.py" -print0 | xargs -0 --no-run-if-empty \
+		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
+		   -o \( -name "_sysconfigdata*.py" -print0 \) \) \
+		| xargs -0 --no-run-if-empty \
 		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
 endef
 endif
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 03/16] package/pkg-generic.mk: move python fixup to generic package infrastructure

[Herve Codina]

Fixing _sysconfigdata*.{py,pyc} was previously done by python package
infrastructure. Some packages use python stuff without using python
package infrastructure.
These packages perform overwrites and need the specific python fixup
to fix them.

In order to be sure to fix all of these packages, the python fixup
is moved to the generic package infrastructure and applied to all
packages.
This follows the same principle as for the .la libtool files fixup.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
Changes v1 to v2:
 - Used '... -print0 | xargs -0 -r ...' construction.
 - Used '-deleted' to remove files.

Changes v2 to v3:
 - Used find syntax discussed in the previous review.
   https://lore.kernel.org/buildroot/20210709084820.37830e5f@bootlin.com/

 package/pkg-generic.mk | 24 +++++++++++++++++++++++-
 package/pkg-python.mk  | 23 -----------------------
 2 files changed, 23 insertions(+), 24 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 97ee204b80..38381d93b5 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -103,6 +103,27 @@ define fixup-libtool-files
 endef
 endif
 
+# Make sure python _sysconfigdata*.py files only reference the current
+# per-package directory.
+#
+# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...)
+# because those directories may be created in the same recipe this macro will
+# be expanded in.
+# Additionally, either or both may be missing, which would make find whine and
+# fail.
+# So we just use HOST_DIR as a starting point, and filter on the two directories
+# of interest.
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define FIXUP_PYTHON_SYSCONFIGDATA
+	$(Q)find $(HOST_DIR) \
+		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
+		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
+		   -o \( -name "_sysconfigdata*.py" -print0 \) \) \
+	| xargs -0 --no-run-if-empty \
+	$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
+endef
+endif
+
 # Functions to collect statistics about installed files
 
 # $(1): base directory to search in
@@ -836,7 +857,8 @@ $(2)_PRE_LEGAL_INFO_HOOKS       ?=
 $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
 $(2)_ROOTFS_PRE_CMD_HOOKS       ?=
-$(2)_POST_PREPARE_HOOKS         ?=
+
+$(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA
 
 ifeq ($$($(2)_TYPE),target)
 ifneq ($$(HOST_$(2)_KCONFIG_VAR),)
diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 0f65bd0f75..e6b81bdfd3 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -92,27 +92,6 @@ HOST_PKG_PYTHON_SETUPTOOLS_INSTALL_OPTS = \
 	--root=/ \
 	--single-version-externally-managed
 
-# Make sure python _sysconfigdata*.py files only reference the current
-# per-package directory.
-#
-# Can't use $(foreach d, $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python*, ...)
-# because those directories may be created in the same recipe this macro will
-# be expanded in.
-# Additionally, either or both may be missing, which would make find whine and
-# fail.
-# So we just use HOST_DIR as a starting point, and filter on the two directories
-# of interest.
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define PKG_PYTHON_FIXUP_SYSCONFIGDATA
-	$(Q)find $(HOST_DIR) \
-		\( -path '$(HOST_DIR)/lib/python*' -o -path '$(STAGING_DIR)/usr/lib/python*' \) \
-		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
-		   -o \( -name "_sysconfigdata*.py" -print0 \) \) \
-		| xargs -0 --no-run-if-empty \
-		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
-endef
-endif
-
 ################################################################################
 # inner-python-package -- defines how the configuration, compilation
 # and installation of a Python package should be done, implements a
@@ -257,8 +236,6 @@ $(2)_PYTHON_INTERPRETER = $$(HOST_DIR)/bin/$$($(2)_NEEDS_HOST_PYTHON)
 endif
 endif
 
-$(2)_PRE_CONFIGURE_HOOKS += PKG_PYTHON_FIXUP_SYSCONFIGDATA
-
 #
 # Build step. Only define it if not already defined by the package .mk
 # file.
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 04/16] package/owfs: remove Python sysconfigdata fixup

[Herve Codina]

This fixup is done at pkg-generic level for all packages.
So, it is no more needed in owfs package.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
Changes v1 to v2:
 - Added 'Reviewed-by: Yann E. MORIN'

Changes v2 to v3:
None

 package/owfs/owfs.mk | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/package/owfs/owfs.mk b/package/owfs/owfs.mk
index ffc0b3098d..10543d3698 100644
--- a/package/owfs/owfs.mk
+++ b/package/owfs/owfs.mk
@@ -88,15 +88,6 @@ OWFS_DEPENDENCIES += python host-swig
 # override the PYSITEDIR variable in make.
 OWFS_EXTRA_MAKE_OPTS += PYSITEDIR=/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages
 
-ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
-		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/owfs/:g"
-endef
-OWFS_PRE_CONFIGURE_HOOKS += OWFS_FIXUP_PYTHON_SYSCONFIGDATA
-endif
-
 else
 OWFS_CONF_OPTS += --disable-owpython --without-python
 endif
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build

[Herve Codina]

From: Andreas Naumann <anaumann@ultratronik.de>

When using top level parallel build, independent qt5 packages may be
built in parallel. Because of their staging dirs being hardlinked, they
all use the same qt.conf file to manipulate during configure, while
another qt5 package might already use it. This leads to weird build failures
because the folders qmake is using are diverted in erratic ways.
Fix this by actually recreating a non-shared qt.conf file for every package.

Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
Reviewed-by: Herve Codina <herve.codina@bootlin.com>
---
New patch in this v3 series

This patch was previously send by Andreas Naumann.
https://lore.kernel.org/buildroot/20210623083927.8278-1-anaumann@ultratronik.de/
https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
As this patch makes sense in this series, it was integrated in this v3 version.

 package/qt5/qt5.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/qt5/qt5.mk b/package/qt5/qt5.mk
index db6ccd2b42..3ffb7b0063 100644
--- a/package/qt5/qt5.mk
+++ b/package/qt5/qt5.mk
@@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
 # compiled into the Qt library. We need it to make "qmake" relocatable and
 # tweak the per-package install pathes
 define QT5_INSTALL_QT_CONF
+	rm -f $(HOST_DIR)/bin/qt.conf
 	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
 		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
 endef
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 06/16] package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook

[Herve Codina]

QT5_QT_CONF_FIXUP tweaks files for per-package directory build.
This is typically the kind of operation expected to be in
post-prepare hook.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
New patch in this v3 series.
As said previously in
https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
This patch moves Qt5 fixup to <PKG>_POST_PREPARE_HOOKS (old <PKG>_PER_PACKAGE_TWEAK_HOOKS
mentioned in discussion).

 package/pkg-qmake.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/pkg-qmake.mk b/package/pkg-qmake.mk
index a77286fe3a..deb033c1d6 100644
--- a/package/pkg-qmake.mk
+++ b/package/pkg-qmake.mk
@@ -56,13 +56,14 @@ $(2)_DEPENDENCIES 		+= host-perl
 $(2)_PRE_CONFIGURE_HOOKS        += QT_HEADERS_SYNC_HOOK
 endif
 
+$(2)_POST_PREPARE_HOOKS += QT5_QT_CONF_FIXUP
+
 #
 # Configure step. Only define it if not already defined by the package
 # .mk file.
 #
 ifndef $(2)_CONFIGURE_CMDS
 define $(2)_CONFIGURE_CMDS
-	$$(QT5_QT_CONF_FIXUP)
 	cd $$($(2)_BUILDDIR) && \
 	$$(TARGET_MAKE_ENV) $$($(2)_CONF_ENV) $$(QT5_QMAKE) $$($(2)_CONF_OPTS)
 endef
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Herve Codina]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Without per-package directory support, a package can happily overwrite
files installed by other packages. Indeed, because the build order
between packages is always guaranteed, Buildroot will always produce
the same output.

However, with per-package directory support, it is absolutely critical
that a given package does not overwrite files already installed by
another package, due to how the final aggregation is done to create
the complete target/, staging/ and host/ folders. Unfortunately, we
currently don't have anything in Buildroot that detects this
situation.

We used to have check-uniq-files, but it was dropped in commit
2496189a4207173e4cd5bbab90256f911175ee57.

This commit is a new implementation of such a detection, which is
based on calculating and verifying MD5 hashes of installed files: the
calculation is done at the beginning of the configure step, the
verification during the newly introduced "install" step that takes
place after all installation steps.

Since preventing file overwrites is really only needed when
per-package directory support is used, and due to this verification
having some overhead, it is only enabled when
BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is
however not too bad as on average, with per-package directory support,
the per-package target/ and host/ directories will contain less files
than with a build that doesn't use per-package directory support. This
helps a bit in mitigating the additional cost of this verification.

Note that we are not handling separately HOST_DIR and STAGING_DIR,
like we're doing with the pkg_size_{before,after} functions. Instead,
the verification on HOST_DIR walks down into the STAGING_DIR.

During per-package build, original files are modified by
fixup-libtool-files and fixup-python-files calls.
But since these fixups modify files using sed --in-place, these
modifications are done using a temporary file and a call to rename.
Rename breaks the hardlink to the original file and leave the temporary
file in per-package TARGET dir.
As the original file is not modified, this is no longer considered as
an overwrite. This patch simply considers that what is done by
fixup-libtool-files and fixup-python-files is part of the original
snapshot used to detect overwrites. And so, the original snapshot is
taken after fixup-libtool-files and fixup-python-files calls.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
This commit is retreived from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni@bootlin.com/
This new version was not already submitted by Thomas or I missed it.
Compared to the first version, this patch has an improved commit message and
generates the md5sum snapshot using
 'LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
instead of
 'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'

Changes v1 to v2:
 - Added note about why fixup-{libtool,python}-files are not considered overwrites
   and so take the overwrite snapshot after fixup-{libtool,python}-files call.
 - Removed 'LC_ALL=C'

Changes v2 to v3:
None

 package/pkg-generic.mk | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 38381d93b5..414862e267 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -124,6 +124,25 @@ define FIXUP_PYTHON_SYSCONFIGDATA
 endef
 endif
 
+# Functions to detect overwritten files
+
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_before
+	find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_detect_overwrite_after
+	if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \
+		md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \
+		{ echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \
+	fi
+endef
+endif
+
 # Functions to collect statistics about installed files
 
 # $(1): base directory to search in
@@ -277,6 +296,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
 	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
+	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -401,6 +422,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
 
 # Remove package sources
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 08/16] package/pkg-generic.mk: generate final rsync exclude file list

[Herve Codina]

The final rsync performed at {host,target}-finalize steps
need to be done by rsyncing files generated by each packages
without looking at files generated by other packages the current
package depends on. This is needed to avoid overwrites in final
{HOST,TARGET}_DIR.

In order to prepare the final rsync, an exclusion list is generated.
This list lists files that are not generated by the current package and
so files that need to be excluded from the final rsync.

Note also that the files list was not based on .files-list.{before,after}.
During .file-list.{before,after} built for host directory, staging
sub-directory (ie <toolchain>/sysroot) is filtered out.
The final rsync exclusion list needs to take into account the full
{host,target} directory to avoid final overwrites.

Using an empty directory for per-package installation directory would be the
simplest way to find what a package installs.
However, as it has been discussed in the past, this is fraught with unworkable
issues. For example, some paths may be hard-coded at configure time and/or build
time, and thus the package would still install in the original stagin we presented
it with (for target/, this is not an isue, because target/ is never looked at
during configure or build, only at install time). Or a package installation
process just assumes that the directory structure exists (bad, but eh...).

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
Changes v1 to v2:
 - Added a note in the commit message related to why an empty directory is not
   used to find what a package installs.

Changes v2 to v3:
None

 package/pkg-generic.mk | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 414862e267..4226cfe0dc 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -168,6 +168,30 @@ define pkg_size_after
 	rm -f $($(PKG)_DIR)/.files-list$(2).after
 endef
 
+# Functions to collect final rsync exclusion files
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_final_rsync_before
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).before
+endef
+
+# $(1): base directory to search in
+# $(2): suffix of file (optional)
+define pkg_final_rsync_after
+	cd $(1); \
+	LC_ALL=C find . \( -type f -o -type l \) -printf '%T@:%i:%#m:%y:%s,%p\n' \
+		| LC_ALL=C sort > $($(PKG)_DIR)/.files-final-rsync$(2).after
+	LC_ALL=C comm -2 \
+		$($(PKG)_DIR)/.files-final-rsync$(2).before \
+		$($(PKG)_DIR)/.files-final-rsync$(2).after \
+		| sed -r -e 's/^[^,]+,./- /' \
+		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after
+endef
+
 define check_bin_arch
 	support/scripts/check-bin-arch -p $($(PKG)_NAME) \
 		-l $($(PKG)_DIR)/.files-list.txt \
@@ -290,6 +314,8 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call MESSAGE,"Configuring")
 	$(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
+	@$(call pkg_final_rsync_before,$(TARGET_DIR))
+	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
@@ -422,6 +448,8 @@ $(BUILD_DIR)/%/.stamp_installed:
 	@$(call pkg_size_after,$(STAGING_DIR),-staging)
 	@$(call pkg_size_after,$(HOST_DIR),-host)
 	@$(call check_bin_arch)
+	@$(call pkg_final_rsync_after,$(TARGET_DIR))
+	@$(call pkg_final_rsync_after,$(HOST_DIR),-host)
 	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
 	$(Q)touch $@
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 09/16] Makefile: rsync global {TARGET, HOST}_DIR using exclusion file list

[Herve Codina]

On a per-package build, rsync final {TARGET,HOST}_DIR using
exclusion file list computed for each packages.

This exclusion list allows to rsync only files generated by a given
package. This is needed to avoid any overwrites in final
{TARGET,HOST}_DIR.

As we rsync only files generated by each packages, we need to
to do this rsync recursively on each dependencies to collect
all needed files. This is done based on existing
<PKG>_FINAL_RECURSIVE_DEPENDENCIES

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
Changes v1 to v2:
 - Added a note in the commit message related to why this exclusion
   list is needed.
 - Added $(Q) to mkdir and rsync
 - Changed indentations to make calls more readable

Changes v2 to v3:
None

 Makefile | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index 3923e5875f..32fad004fe 100644
--- a/Makefile
+++ b/Makefile
@@ -725,10 +725,33 @@ TARGET_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list.txt))
 HOST_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-host.txt))
 STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.txt))
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+# rsync the contents of per-package directories
+# $1: space-separated list of packages to rsync from
+# $2: 'host' or 'target'
+# $3: destination directory
+# $4: exclude file list to use
+define per-package-rsync-delta
+	$(Q)mkdir -p $(3)
+	$(foreach pkg,$(1),\
+		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
+			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+			$(3)$(sep))
+endef
+endif
+
 .PHONY: host-finalize
 host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
 	@$(call MESSAGE,"Finalizing host directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR))
+	$(call per-package-rsync-delta, \
+		$(sort $(foreach pkg, $(PACKAGES), \
+			$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
+		), \
+		host, \
+		$(HOST_DIR), \
+		.files-final-rsync-host.exclude_rsync \
+	)
 
 .PHONY: staging-finalize
 staging-finalize: $(STAGING_DIR_SYMLINK)
@@ -736,7 +759,14 @@ staging-finalize: $(STAGING_DIR_SYMLINK)
 .PHONY: target-finalize
 target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize
 	@$(call MESSAGE,"Finalizing target directory")
-	$(call per-package-rsync,$(sort $(PACKAGES)),target,$(TARGET_DIR))
+	$(call per-package-rsync-delta, \
+		$(sort $(foreach pkg, $(PACKAGES), \
+			$(pkg) $($(call UPPERCASE,$(pkg))_FINAL_RECURSIVE_DEPENDENCIES)) \
+		), \
+		target, \
+		$(TARGET_DIR), \
+		.files-final-rsync.exclude_rsync \
+	)
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

[Herve Codina]

Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
target_finalize steps some modifications are done on installed files (ie
strip or TARGET_FINALIZE_HOOKS for instance).

In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR
and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built
using a full copy of the involved per-package files instead of hardlinks.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
Changes v1 to v2:
 - Added 'Reviewed-by: Yann E. MORIN'

Changes v2 to v2:
None

 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 32fad004fe..ccf2020565 100644
--- a/Makefile
+++ b/Makefile
@@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
 define per-package-rsync-delta
 	$(Q)mkdir -p $(3)
 	$(foreach pkg,$(1),\
-		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
+		$(Q)rsync -a \
 			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
 			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
 			$(3)$(sep))
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 11/16] package/pkg-generic.mk: fix per-package <pkg>-{reconfigure, rebuild, reinstall}

[Herve Codina]

Many overwrites are detected on <pkg>-{reconfigure,rebuild,reinstall}.
Indeed, files previously installed by a package were detected as
overwritten on next reconfigure, rebuild or reinstall.

To avoid this, we recreate per-package host and target dir from
scratch as it was done during the first configure step.

In order not to duplicate the code, this commit shares common code
in prepare-pre-configure macro.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
Changes v1 to v2:
 - Avoid code duplication using prepare-pre-configure macro

Changes v2 to v3:
None

 package/pkg-generic.mk | 40 ++++++++++++++++++++++++++++++----------
 1 file changed, 30 insertions(+), 10 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 4226cfe0dc..fb219b8ec7 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -217,6 +217,21 @@ define REMOVE_CONFLICTING_USELESS_FILES_IN_TARGET
 	$(call remove-conflicting-useless-files,$(TARGET_DIR))
 endef
 
+# Function to prepare package
+
+define prepare-pre-configure
+	@$(call pkg_final_rsync_before,$(TARGET_DIR))
+	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)
+	@$(call pkg_size_before,$(TARGET_DIR))
+	@$(call pkg_size_before,$(STAGING_DIR),-staging)
+	@$(call pkg_size_before,$(HOST_DIR),-host)
+	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
+	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
+	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
+	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
+	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
+endef
+
 ################################################################################
 # Implicit targets -- produce a stamp file for each step of a package build
 ################################################################################
@@ -314,16 +329,7 @@ $(BUILD_DIR)/%/.stamp_configured:
 	@$(call MESSAGE,"Configuring")
 	$(Q)mkdir -p $(HOST_DIR) $(TARGET_DIR) $(STAGING_DIR) $(BINARIES_DIR)
 	$(call prepare-per-package-directory,$($(PKG)_FINAL_DEPENDENCIES))
-	@$(call pkg_final_rsync_before,$(TARGET_DIR))
-	@$(call pkg_final_rsync_before,$(HOST_DIR),-host)
-	@$(call pkg_size_before,$(TARGET_DIR))
-	@$(call pkg_size_before,$(STAGING_DIR),-staging)
-	@$(call pkg_size_before,$(HOST_DIR),-host)
-	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
-	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
-	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
-	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
-	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
+	$(call prepare-pre-configure)
 	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
 	$($(PKG)_CONFIGURE_CMDS)
 	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
@@ -488,6 +494,13 @@ define pkg-graph-depends
 		$$(GRAPHS_DIR)/$$(@).dot
 endef
 
+ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
+define empty-per-package-directory
+	rm -rf $(HOST_DIR) $(TARGET_DIR)
+	mkdir -p $(HOST_DIR) $(TARGET_DIR)
+endef
+endif
+
 ################################################################################
 # inner-generic-package -- generates the make targets needed to build a
 # generic package
@@ -1092,6 +1105,8 @@ $(1)-all-legal-info:	$$(foreach p,$$($(2)_FINAL_ALL_DEPENDENCIES),$$(p)-all-lega
 
 $(1)-dirclean:		$$($(2)_TARGET_DIRCLEAN)
 
+$(1)-clean-for-reinstall: PKG=$(2)
+$(1)-clean-for-reinstall: NAME=$(1)
 $(1)-clean-for-reinstall:
 ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 			rm -f $$($(2)_TARGET_RSYNC)
@@ -1101,6 +1116,11 @@ endif
 			rm -f $$($(2)_TARGET_INSTALL_TARGET)
 			rm -f $$($(2)_TARGET_INSTALL_IMAGES)
 			rm -f $$($(2)_TARGET_INSTALL_HOST)
+			$$(call empty-per-package-directory)
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DOWNLOAD_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_EXTRACT_DEPENDENCIES))
+			$$(call prepare-per-package-directory,$$($(2)_FINAL_DEPENDENCIES))
+			$$(call prepare-pre-configure)
 
 $(1)-reinstall:		$(1)-clean-for-reinstall $(1)
 
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 12/16] package/pkg-generic.mk: remove .files-final-rsync.before temporary file

[Herve Codina]

Keeping the .files-final-rsync.before file between make invocations
was needed to ensure that this file will be present in case of
'make <pkg>_{reconfigure,rebuild,reinstall}' invocation.

The .files-final-rsync.before is now recreated on
<pkg>_{reconfigure,rebuild,reinstall} and we don't need to keep it
between make invocation.

This patch simply removes this file as soon as it is no longer needed.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
New patch in this v2 series

Changes v2 to v3:
None

 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index fb219b8ec7..bb9f395f27 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -189,6 +189,7 @@ define pkg_final_rsync_after
 		$($(PKG)_DIR)/.files-final-rsync$(2).after \
 		| sed -r -e 's/^[^,]+,./- /' \
 		> $($(PKG)_DIR)/.files-final-rsync$(2).exclude_rsync
+	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).before
 	rm -f $($(PKG)_DIR)/.files-final-rsync$(2).after
 endef
 
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 13/16] support/testing/infra: add log_file_path() function

[Herve Codina]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Some tests will need to grep through the build log to verify that some
features are working are expected. In order to allow them to open the
build log, we provide a new function called log_file_path(), which
returns the path to the log file if available.

We also use this function in open_log_file().

Note that open_log_file() cannot be used directly to grep through the
log file at the end of a build: because it opens in "a+" mode, it
greps starting from the end of the file.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
New patch in this v2 series

This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-10-thomas.petazzoni@bootlin.com/

This new version was not already submitted by Thomas or I missed it.
Compared to the first version, this patch use a more python-like
syntax as proposed by Yann in the previous review.

Changes v2 to v3:
None

 support/testing/infra/__init__.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
index b10a7601a3..1f003f24c6 100644
--- a/support/testing/infra/__init__.py
+++ b/support/testing/infra/__init__.py
@@ -10,17 +10,18 @@ ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
 BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
 
 
+def log_file_path(builddir, stage, logtofile=True):
+    """Return path to log file"""
+    return "{}-{}.log".format(builddir, stage) if logtofile else None
+
+
 def open_log_file(builddir, stage, logtofile=True):
     """
     Open a file for logging and return its handler.
     If logtofile is True, returns sys.stdout. Otherwise opens a file
     with a suitable name in the build directory.
     """
-    if logtofile:
-        fhandle = open("{}-{}.log".format(builddir, stage), 'a+')
-    else:
-        fhandle = sys.stdout
-    return fhandle
+    return open(log_file_path(builddir, stage, logtofile), 'a+') if logtofile else sys.stdout
 
 
 def basepath(relpath=""):
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 14/16] support/testing/tests: add test for check_bin_arch

[Herve Codina]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

This tests build a bogus package that installs a binary built for the
host architecture into $(TARGET_DIR), which should cause a build
failure, at least as long as the host architecture isn't ARM.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
New patch in this v2 series

This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-10-thomas.petazzoni@bootlin.com/

Compared to the first version, as proposed by Yann in the previous
review, this patch does not launch a subprocess (grep) to search the
string.

Note also that the .gitlab-ci.yml modification present in the previous
version is not present in this patch.

Changes v2 to v3:
- Fixed the test (https://lore.kernel.org/buildroot/20210707140708.259208a1@bootlin.com/)

 .../br2-external/detect-bad-arch/Config.in    |  1 +
 .../detect-bad-arch/external.desc             |  1 +
 .../br2-external/detect-bad-arch/external.mk  |  1 +
 .../package/detect-bad-arch/Config.in         |  4 ++++
 .../detect-bad-arch/detect-bad-arch.mk        | 15 +++++++++++++++
 support/testing/tests/core/test_bad_arch.py   | 19 +++++++++++++++++++
 6 files changed, 41 insertions(+)
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
 create mode 100644 support/testing/tests/core/test_bad_arch.py

diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/Config.in b/support/testing/tests/core/br2-external/detect-bad-arch/Config.in
new file mode 100644
index 0000000000..530c077bbe
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-bad-arch/Config.in
@@ -0,0 +1 @@
+source "$BR2_EXTERNAL_DETECT_BAD_ARCH_PATH/package/detect-bad-arch/Config.in"
diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/external.desc b/support/testing/tests/core/br2-external/detect-bad-arch/external.desc
new file mode 100644
index 0000000000..3c4232c90d
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-bad-arch/external.desc
@@ -0,0 +1 @@
+name: DETECT_BAD_ARCH
diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/external.mk b/support/testing/tests/core/br2-external/detect-bad-arch/external.mk
new file mode 100644
index 0000000000..71b9821ddc
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-bad-arch/external.mk
@@ -0,0 +1 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_BAD_ARCH_PATH)/package/*/*.mk))
diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
new file mode 100644
index 0000000000..9893e9afc1
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
@@ -0,0 +1,4 @@
+config BR2_PACKAGE_DETECT_BAD_ARCH
+	bool
+	default y
+
diff --git a/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
new file mode 100644
index 0000000000..5e78c55f1f
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
@@ -0,0 +1,15 @@
+################################################################################
+#
+# detect-bad-arch
+#
+################################################################################
+
+define DETECT_BAD_ARCH_BUILD_CMDS
+	echo "int main(void) { return 0; }" | $(HOSTCC) -x c -o $(@D)/foo -
+endef
+
+define DETECT_BAD_ARCH_INSTALL_TARGET_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/foo $(TARGET_DIR)/usr/bin/foo
+endef
+
+$(eval $(generic-package))
diff --git a/support/testing/tests/core/test_bad_arch.py b/support/testing/tests/core/test_bad_arch.py
new file mode 100644
index 0000000000..76acba3711
--- /dev/null
+++ b/support/testing/tests/core/test_bad_arch.py
@@ -0,0 +1,19 @@
+import infra
+import infra.basetest
+import subprocess
+
+
+class DetectBadArchTest(infra.basetest.BRConfigTest):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + infra.basetest.MINIMAL_CONFIG
+    br2_external = [infra.filepath("tests/core/br2-external/detect-bad-arch")]
+
+    def test_run(self):
+        with self.assertRaises(SystemError):
+            self.b.build()
+        logf_path = infra.log_file_path(self.b.builddir, "build",
+                                        infra.basetest.BRConfigTest.logtofile)
+        if logf_path:
+            s = 'ERROR: architecture for "/usr/bin/foo" is'
+            with open(logf_path, "r") as f:
+                lines = [l for l in f.readlines() if l.startswith(s)]
+            self.assertEqual(len(lines), 1)
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 15/16] support/testing/tests: add test for file overwrite detection

[Herve Codina]

From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
New patch in this v2 series

This patch is retrieved from Thomas's work.
The first version was discussed
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-12-thomas.petazzoni@bootlin.com/

Compared to the first version, as proposed by Yann in the previous
review, this patch does not launch a subprocess (grep) to find the
string.
Additionally, the strings searched were changed from a specific
file ('path/file: FAILED') to the "ERROR: package ..." overwrite
detection message.

Note also that the .gitlab-ci.yml modification present in the previous
version is not present in this patch.

Changes v2 to v3:
None

 .../br2-external/detect-overwrite/Config.in   |  1 +
 .../detect-overwrite/external.desc            |  1 +
 .../br2-external/detect-overwrite/external.mk |  1 +
 .../package/detect-overwrite/Config.in        |  5 ++
 .../detect-overwrite/detect-overwrite.mk      | 19 ++++++++
 .../testing/tests/core/test_file_overwrite.py | 48 +++++++++++++++++++
 6 files changed, 75 insertions(+)
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
 create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
 create mode 100644 support/testing/tests/core/test_file_overwrite.py

diff --git a/support/testing/tests/core/br2-external/detect-overwrite/Config.in b/support/testing/tests/core/br2-external/detect-overwrite/Config.in
new file mode 100644
index 0000000000..b5514510bd
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-overwrite/Config.in
@@ -0,0 +1 @@
+source "$BR2_EXTERNAL_DETECT_OVERWRITE_PATH/package/detect-overwrite/Config.in"
diff --git a/support/testing/tests/core/br2-external/detect-overwrite/external.desc b/support/testing/tests/core/br2-external/detect-overwrite/external.desc
new file mode 100644
index 0000000000..6fedc276e8
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-overwrite/external.desc
@@ -0,0 +1 @@
+name: DETECT_OVERWRITE
diff --git a/support/testing/tests/core/br2-external/detect-overwrite/external.mk b/support/testing/tests/core/br2-external/detect-overwrite/external.mk
new file mode 100644
index 0000000000..90927b33ef
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-overwrite/external.mk
@@ -0,0 +1 @@
+include $(sort $(wildcard $(BR2_EXTERNAL_DETECT_OVERWRITE_PATH)/package/*/*.mk))
diff --git a/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
new file mode 100644
index 0000000000..fff8b0320f
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
@@ -0,0 +1,5 @@
+config BR2_PACKAGE_DETECT_OVERWRITE
+	bool "detect-overwrite"
+
+config BR2_PACKAGE_HOST_DETECT_OVERWRITE
+	bool "host-detect-overwrite"
diff --git a/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
new file mode 100644
index 0000000000..c6df2a339d
--- /dev/null
+++ b/support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
@@ -0,0 +1,19 @@
+################################################################################
+#
+# detect-overwrite
+#
+################################################################################
+
+define DETECT_OVERWRITE_INSTALL_TARGET_CMDS
+	grep -q "^foo" $(TARGET_DIR)/etc/passwd || \
+		echo "foo" >> $(TARGET_DIR)/etc/passwd
+endef
+
+HOST_DETECT_OVERWRITE_DEPENDENCIES = host-pkgconf
+
+define HOST_DETECT_OVERWRITE_INSTALL_CMDS
+	$(SED) 's/manipulating/tweaking/' $(HOST_DIR)/lib/pkgconfig/libpkgconf.pc
+endef
+
+$(eval $(generic-package))
+$(eval $(host-generic-package))
diff --git a/support/testing/tests/core/test_file_overwrite.py b/support/testing/tests/core/test_file_overwrite.py
new file mode 100644
index 0000000000..96553cb483
--- /dev/null
+++ b/support/testing/tests/core/test_file_overwrite.py
@@ -0,0 +1,48 @@
+import infra
+import infra.basetest
+import subprocess
+
+
+class DetectTargetFileOverwriteTest(infra.basetest.BRConfigTest):
+    config = \
+        infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        infra.basetest.MINIMAL_CONFIG + \
+        """
+        BR2_PER_PACKAGE_DIRECTORIES=y
+        BR2_PACKAGE_DETECT_OVERWRITE=y
+        """
+    br2_external = [infra.filepath("tests/core/br2-external/detect-overwrite")]
+
+    def test_run(self):
+        with self.assertRaises(SystemError):
+            self.b.build()
+        logf_path = infra.log_file_path(self.b.builddir, "build",
+                                        infra.basetest.BRConfigTest.logtofile)
+        if logf_path:
+            s = 'ERROR: package detect-overwrite has overwritten files installed by a previous package, aborting.'
+            with open(logf_path, "r") as f:
+                lines = [l for l in f.readlines() if l.startswith(s)]
+            self.assertNotEqual(len(lines), 0)
+
+
+class DetectHostFileOverwriteTest(infra.basetest.BRConfigTest):
+    config = \
+        infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        infra.basetest.MINIMAL_CONFIG + \
+        """
+        BR2_PER_PACKAGE_DIRECTORIES=y
+        BR2_PACKAGE_HOST_DETECT_OVERWRITE=y
+        """
+    br2_external = [infra.filepath("tests/core/br2-external/detect-overwrite")]
+
+    def test_run(self):
+        with self.assertRaises(SystemError):
+            self.b.build()
+        logf_path = infra.log_file_path(self.b.builddir, "build",
+                                        infra.basetest.BRConfigTest.logtofile)
+        if logf_path:
+            s = './lib/pkgconfig/hco_libpkgconf.pc: FAILED'
+            s = 'ERROR: package host-detect-overwrite has overwritten files installed by a previous package, aborting.'
+            with open(logf_path, "r") as f:
+                lines = [l for l in f.readlines() if l.startswith(s)]
+            self.assertNotEqual(len(lines), 0)
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 16/16] package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook

[Herve Codina]

This commit moves fixup-libtool-files to the post-prepare hook
infrastructure.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
---
New patch in this v3 series.

This modification was mentionned in the previous review
https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/

 package/pkg-generic.mk | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index bb9f395f27..d8ca8aaf31 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -92,14 +92,19 @@ endif
 
 # Make sure .la files only reference the current per-package
 # directory.
-
-# $1: package name (lower case)
-# $2: staging directory of the package
+#
+# Can't use $(foreach d, $(HOST_DIR)/lib* $(STAGING_DIR)/usr/lib*, ...)
+# because those directories may be created in the same recipe this macro will
+# be expanded in.
+# Additionally, either or both may be missing, which would make find whine and
+# fail.
+# So we just use HOST_DIR as a starting point, and filter on the two directories
+# of interest.
 ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
-define fixup-libtool-files
-	$(Q)find $(2) \( -path '$(2)/lib*' -o -path '$(2)/usr/lib*' \) \
+define FIXUP_LIBTOOL_FILES
+	$(Q)find $(HOST_DIR) \( -path '$(HOST_DIR)/lib*' -o -path '$(STAGING_DIR)/usr/lib*' \) \
 		-name "*.la" -print0 | xargs -0 --no-run-if-empty \
-		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$(1)/:g"
+		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
 endef
 endif
 
@@ -226,8 +231,6 @@ define prepare-pre-configure
 	@$(call pkg_size_before,$(TARGET_DIR))
 	@$(call pkg_size_before,$(STAGING_DIR),-staging)
 	@$(call pkg_size_before,$(HOST_DIR),-host)
-	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
-	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
 	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
 	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
 	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
@@ -923,6 +926,7 @@ $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
 $(2)_ROOTFS_PRE_CMD_HOOKS       ?=
 
+$(2)_POST_PREPARE_HOOKS += FIXUP_LIBTOOL_FILES
 $(2)_POST_PREPARE_HOOKS += FIXUP_PYTHON_SYSCONFIGDATA
 
 ifeq ($$($(2)_TYPE),target)
-- 
2.31.1

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 00/16] Overwritten file detection and fixes, one more step to TLP build

[Yann E. MORIN]

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> This v3 series adds a new patch (patch 1) to fix a find failure already present
> in PKG_PYTHON_FIXUP_SYSCONFIG that was highlighted in the previous version
> of this series.
> It also reworks the python fixup according to previous review (patches 2 and 3)
> and fixes the test given in patch 12.
> 
> Two patches are added related to Qt5. The first one (patch 5) was previously
> sent by Andreas Naumann and fixes a Qt5 overwrite. 
> https://lore.kernel.org/buildroot/20210623083927.8278-1-anaumann@ultratronik.de/
> https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
> The second one (patch 6) moves the fixup function to the post-prepare hooks.
> 
> One more patch was added in this serie (patch 14) to move fixup-libtool-file to
> the post-prepare hooks. This was mentionned during the review of the first
> version of this series (https://lore.kernel.org/buildroot/20210625092704.GJ2852@scaer/).
> 
> More changes details are given in each individual patches.
> 
> Best regards,
> Hervé Codina
> 
> Andreas Naumann (1):
>   qt5: Fix sporadic build failure during top-level parallel build
> 
> Herve Codina (11):
>   package/pkg-python: fix PKG_PYTHON_FIXUP_SYSCONFIGDATA
>   package/pkg-python: invalidate precompiled _sysconfigdata*.pyc
>   package/pkg-generic.mk: move python fixup to generic package
>     infrastructure
>   package/owfs: remove Python sysconfigdata fixup

I've applied those four python-related fixes to next, thanks!

I'll resume the rest of the series later in the WE; no need to respin!

Regards,
Yann E. MORIN.

>   package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook
>   package/pkg-generic.mk: generate final rsync exclude file list
>   Makefile: rsync global {TARGET,HOST}_DIR using exclusion file list
>   Makefile: breaks hardlinks in global {TARGET,HOST}_DIR on per-package
>     build
>   package/pkg-generic.mk: fix per-package
>     <pkg>-{reconfigure,rebuild,reinstall}
>   package/pkg-generic.mk: remove .files-final-rsync.before temporary
>     file
>   package/pkg-generic.mk: move fixup-libtool-files to post-prepare hook
> 
> Thomas Petazzoni (4):
>   package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
>     HOST_DIR
>   support/testing/infra: add log_file_path() function
>   support/testing/tests: add test for check_bin_arch
>   support/testing/tests: add test for file overwrite detection
> 
>  Makefile                                      |  34 ++++-
>  package/owfs/owfs.mk                          |   9 --
>  package/pkg-generic.mk                        | 125 ++++++++++++++++--
>  package/pkg-python.mk                         |  10 --
>  package/pkg-qmake.mk                          |   3 +-
>  package/qt5/qt5.mk                            |   1 +
>  support/testing/infra/__init__.py             |  11 +-
>  .../br2-external/detect-bad-arch/Config.in    |   1 +
>  .../detect-bad-arch/external.desc             |   1 +
>  .../br2-external/detect-bad-arch/external.mk  |   1 +
>  .../package/detect-bad-arch/Config.in         |   4 +
>  .../detect-bad-arch/detect-bad-arch.mk        |  15 +++
>  .../br2-external/detect-overwrite/Config.in   |   1 +
>  .../detect-overwrite/external.desc            |   1 +
>  .../br2-external/detect-overwrite/external.mk |   1 +
>  .../package/detect-overwrite/Config.in        |   5 +
>  .../detect-overwrite/detect-overwrite.mk      |  19 +++
>  support/testing/tests/core/test_bad_arch.py   |  19 +++
>  .../testing/tests/core/test_file_overwrite.py |  48 +++++++
>  19 files changed, 269 insertions(+), 40 deletions(-)
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.desc
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/external.mk
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-bad-arch/package/detect-bad-arch/detect-bad-arch.mk
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.desc
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/external.mk
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/Config.in
>  create mode 100644 support/testing/tests/core/br2-external/detect-overwrite/package/detect-overwrite/detect-overwrite.mk
>  create mode 100644 support/testing/tests/core/test_bad_arch.py
>  create mode 100644 support/testing/tests/core/test_file_overwrite.py
> 
> -- 
> 2.31.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build

[Yann E. MORIN]

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> From: Andreas Naumann <anaumann@ultratronik.de>
> 
> When using top level parallel build, independent qt5 packages may be
> built in parallel. Because of their staging dirs being hardlinked, they
> all use the same qt.conf file to manipulate during configure, while
> another qt5 package might already use it. This leads to weird build failures
> because the folders qmake is using are diverted in erratic ways.
> Fix this by actually recreating a non-shared qt.conf file for every package.

But with patch 10 (Makefile: breaks hardlinks in global {TARGET,
HOST}_DIR on per-package build), is this patch 5 still needed?

IIUC, patch 10 makes the per-package directories actual copies rather
than hardlinks, so the per-package host directory is a complete copy,
thus this qt.conf is no longer a hardlink...

Did I miss something?

Regards,
Yann E. MORIN.

> Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
> Reviewed-by: Herve Codina <herve.codina@bootlin.com>
> ---
> New patch in this v3 series
> 
> This patch was previously send by Andreas Naumann.
> https://lore.kernel.org/buildroot/20210623083927.8278-1-anaumann@ultratronik.de/
> https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
> As this patch makes sense in this series, it was integrated in this v3 version.
> 
>  package/qt5/qt5.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/qt5/qt5.mk b/package/qt5/qt5.mk
> index db6ccd2b42..3ffb7b0063 100644
> --- a/package/qt5/qt5.mk
> +++ b/package/qt5/qt5.mk
> @@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
>  # compiled into the Qt library. We need it to make "qmake" relocatable and
>  # tweak the per-package install pathes
>  define QT5_INSTALL_QT_CONF
> +	rm -f $(HOST_DIR)/bin/qt.conf
>  	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
>  		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
>  endef
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build

[Yann E. MORIN]

Hervé, All, 

On 2021-08-28 18:38 +0200, Yann E. MORIN spake thusly:
> On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> > From: Andreas Naumann <anaumann@ultratronik.de>
> > 
> > When using top level parallel build, independent qt5 packages may be
> > built in parallel. Because of their staging dirs being hardlinked, they
> > all use the same qt.conf file to manipulate during configure, while
> > another qt5 package might already use it. This leads to weird build failures
> > because the folders qmake is using are diverted in erratic ways.
> > Fix this by actually recreating a non-shared qt.conf file for every package.
> 
> But with patch 10 (Makefile: breaks hardlinks in global {TARGET,
> HOST}_DIR on per-package build), is this patch 5 still needed?
> 
> IIUC, patch 10 makes the per-package directories actual copies rather
> than hardlinks, so the per-package host directory is a complete copy,
> thus this qt.conf is no longer a hardlink...
> 
> Did I miss something?

Yes, I missed that patch 10 is not about assembling the per-package
directories from their dependencies, but to gather those per-package
directories to assemble the final target/ and host/.

So, yes, this qt5.conf patch is still needed.

Regards,
Yann E. MORIN.

> Regards,
> Yann E. MORIN.
> 
> > Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
> > Reviewed-by: Herve Codina <herve.codina@bootlin.com>
> > ---
> > New patch in this v3 series
> > 
> > This patch was previously send by Andreas Naumann.
> > https://lore.kernel.org/buildroot/20210623083927.8278-1-anaumann@ultratronik.de/
> > https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
> > As this patch makes sense in this series, it was integrated in this v3 version.
> > 
> >  package/qt5/qt5.mk | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/package/qt5/qt5.mk b/package/qt5/qt5.mk
> > index db6ccd2b42..3ffb7b0063 100644
> > --- a/package/qt5/qt5.mk
> > +++ b/package/qt5/qt5.mk
> > @@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
> >  # compiled into the Qt library. We need it to make "qmake" relocatable and
> >  # tweak the per-package install pathes
> >  define QT5_INSTALL_QT_CONF
> > +	rm -f $(HOST_DIR)/bin/qt.conf
> >  	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
> >  		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
> >  endef
> > -- 
> > 2.31.1
> > 
> > _______________________________________________
> > buildroot mailing list
> > buildroot@busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot
> 
> -- 
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 05/16] qt5: Fix sporadic build failure during top-level parallel build

[Yann E. MORIN]

Hervé, Andreas, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> From: Andreas Naumann <anaumann@ultratronik.de>
> 
> When using top level parallel build, independent qt5 packages may be
> built in parallel. Because of their staging dirs being hardlinked, they
> all use the same qt.conf file to manipulate during configure, while
> another qt5 package might already use it. This leads to weird build failures
> because the folders qmake is using are diverted in erratic ways.
> Fix this by actually recreating a non-shared qt.conf file for every package.
> 
> Signed-off-by: Andreas Naumann <anaumann@ultratronik.de>
> Reviewed-by: Herve Codina <herve.codina@bootlin.com>

Applied to next, thanks! :-)

Regards,
Yann E. MORIN.

> ---
> New patch in this v3 series
> 
> This patch was previously send by Andreas Naumann.
> https://lore.kernel.org/buildroot/20210623083927.8278-1-anaumann@ultratronik.de/
> https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
> As this patch makes sense in this series, it was integrated in this v3 version.
> 
>  package/qt5/qt5.mk | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/package/qt5/qt5.mk b/package/qt5/qt5.mk
> index db6ccd2b42..3ffb7b0063 100644
> --- a/package/qt5/qt5.mk
> +++ b/package/qt5/qt5.mk
> @@ -15,6 +15,7 @@ include $(sort $(wildcard package/qt5/*/*.mk))
>  # compiled into the Qt library. We need it to make "qmake" relocatable and
>  # tweak the per-package install pathes
>  define QT5_INSTALL_QT_CONF
> +	rm -f $(HOST_DIR)/bin/qt.conf
>  	sed -e "s|@@HOST_DIR@@|$(HOST_DIR)|" -e "s|@@STAGING_DIR@@|$(STAGING_DIR)|" \
>  		$(QT5BASE_PKGDIR)/qt.conf.in > $(HOST_DIR)/bin/qt.conf
>  endef
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 06/16] package/pkg-qmake.mk: Move QT5_QT_CONF_FIXUP to post-prepare hook

[Yann E. MORIN]

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> QT5_QT_CONF_FIXUP tweaks files for per-package directory build.
> This is typically the kind of operation expected to be in
> post-prepare hook.
> 
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>

Applied to next, thanks.

Regards,
Yann E. MORIN.

> ---
> New patch in this v3 series.
> As said previously in
> https://lore.kernel.org/buildroot/b1d0c5a5-f76f-e38e-0257-1300f6a30eab@andin.de/
> This patch moves Qt5 fixup to <PKG>_POST_PREPARE_HOOKS (old <PKG>_PER_PACKAGE_TWEAK_HOOKS
> mentioned in discussion).
> 
>  package/pkg-qmake.mk | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/package/pkg-qmake.mk b/package/pkg-qmake.mk
> index a77286fe3a..deb033c1d6 100644
> --- a/package/pkg-qmake.mk
> +++ b/package/pkg-qmake.mk
> @@ -56,13 +56,14 @@ $(2)_DEPENDENCIES 		+= host-perl
>  $(2)_PRE_CONFIGURE_HOOKS        += QT_HEADERS_SYNC_HOOK
>  endif
>  
> +$(2)_POST_PREPARE_HOOKS += QT5_QT_CONF_FIXUP
> +
>  #
>  # Configure step. Only define it if not already defined by the package
>  # .mk file.
>  #
>  ifndef $(2)_CONFIGURE_CMDS
>  define $(2)_CONFIGURE_CMDS
> -	$$(QT5_QT_CONF_FIXUP)
>  	cd $$($(2)_BUILDDIR) && \
>  	$$(TARGET_MAKE_ENV) $$($(2)_CONF_ENV) $$(QT5_QMAKE) $$($(2)_CONF_OPTS)
>  endef
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Yann E. MORIN]

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> Without per-package directory support, a package can happily overwrite
> files installed by other packages. Indeed, because the build order
> between packages is always guaranteed, Buildroot will always produce
> the same output.
> 
> However, with per-package directory support, it is absolutely critical
> that a given package does not overwrite files already installed by
> another package, due to how the final aggregation is done to create
> the complete target/, staging/ and host/ folders. Unfortunately, we
> currently don't have anything in Buildroot that detects this
> situation.
> 
> We used to have check-uniq-files, but it was dropped in commit
> 2496189a4207173e4cd5bbab90256f911175ee57.
> 
> This commit is a new implementation of such a detection, which is
> based on calculating and verifying MD5 hashes of installed files: the
> calculation is done at the beginning of the configure step, the
> verification during the newly introduced "install" step that takes
> place after all installation steps.
> 
> Since preventing file overwrites is really only needed when
> per-package directory support is used, and due to this verification
> having some overhead, it is only enabled when
> BR2_PER_PACKAGE_DIRECTORIES=y. This additional verification cost is
> however not too bad as on average, with per-package directory support,
> the per-package target/ and host/ directories will contain less files
> than with a build that doesn't use per-package directory support. This
> helps a bit in mitigating the additional cost of this verification.
> 
> Note that we are not handling separately HOST_DIR and STAGING_DIR,
> like we're doing with the pkg_size_{before,after} functions. Instead,
> the verification on HOST_DIR walks down into the STAGING_DIR.
> 
> During per-package build, original files are modified by
> fixup-libtool-files and fixup-python-files calls.
> But since these fixups modify files using sed --in-place, these
> modifications are done using a temporary file and a call to rename.
> Rename breaks the hardlink to the original file and leave the temporary
> file in per-package TARGET dir.
> As the original file is not modified, this is no longer considered as
> an overwrite. This patch simply considers that what is done by
> fixup-libtool-files and fixup-python-files is part of the original
> snapshot used to detect overwrites. And so, the original snapshot is
> taken after fixup-libtool-files and fixup-python-files calls.

This paragraph should take into consideration the fact that we also now
have those _POST_PREPARE_HOOKS (and first moving the libtool fixups to
be hooks would have simplified this paragraph).

However, what prompted me from applying for now, is that this new
detection is a hard error.

Previously, check-uniq-files was just emitting warnings, but would not
prevent the build from failing. Now, with this patch, even an innocuous
overwrite (e.g. because a post-build script deletes the file, or the
content of the file really does not matter at runtime), the build will
fail.

I.e. configurations that are currently working with PPD, despite the
overwrite, will suddenly no longer build.

OTOH, if we do not make that a hard-error, we will never detect most
issues, because users will never spot those warnings and wil enver
report issues, and the autobuilders will not fail and we will not
notice either...

One solution is to add a configuration knob to make that a hard-error,
like we have the paranoid libs/headers check:

    config BR2_PPD_OVERWRITE_STRICT
        bool "Strict file overwrite detection"
        depends on BR2_PER_PACKAGE_DIRECTORIES
        help
          Say 'y' here to turn the file overwrite detection
          to a hard error. By default, only warnings will be
          printed.

And use that to exit in error only if set:

    [...]
    { \
        echo "ERROR: [...]"; \
        $(if $(BR2_PPD_OVERWRITE_STRICT),exit 1;) \
    } ; \
    [...]

And then, ensure it is always enabled in our autobuilders, so in
utils/genrandconfig:

    diff --git a/utils/genrandconfig b/utils/genrandconfig index 622cfd4891..a891f93b6d 100755
    --- a/utils/genrandconfig
    +++ b/utils/genrandconfig
    @@ -353,6 +353,7 @@ def gen_config(args):
         # Per-package folder
         if randint(0, 15) == 0:
             configlines.append("BR2_PER_PACKAGE_DIRECTORIES=y\n")
    +        configlines.append("BR2_PPD_OVERWRITE_STRICT=y\n")
     
         # Amend the configuration with a few things.
         if randint(0, 20) == 0:

I am not totally sold to the new config knob either, but I'd still
rather have it than unconditionally break the build...

After all, we know that there are some people using PPD (and TLPB) and
are happy with how it works for them, so breaking the build on purpose
would probably not be seen as an improvement...

Regards,
Yann E. MORIN.

> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> ---
> This commit is retreived from Thomas's work.
> The first version was discussed
> https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni@bootlin.com/
> This new version was not already submitted by Thomas or I missed it.
> Compared to the first version, this patch has an improved commit message and
> generates the md5sum snapshot using
>  'LC_ALL=C find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;'
> instead of
>  'cd $(1); LC_ALL=C find . -type f -exec md5sum {} \; > $($(PKG)_DIR)/.files$(2).md5'
> 
> Changes v1 to v2:
>  - Added note about why fixup-{libtool,python}-files are not considered overwrites
>    and so take the overwrite snapshot after fixup-{libtool,python}-files call.
>  - Removed 'LC_ALL=C'
> 
> Changes v2 to v3:
> None
> 
>  package/pkg-generic.mk | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
> index 38381d93b5..414862e267 100644
> --- a/package/pkg-generic.mk
> +++ b/package/pkg-generic.mk
> @@ -124,6 +124,25 @@ define FIXUP_PYTHON_SYSCONFIGDATA
>  endef
>  endif
>  
> +# Functions to detect overwritten files
> +
> +ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
> +# $(1): base directory to search in
> +# $(2): suffix of file (optional)
> +define pkg_detect_overwrite_before
> +	find $(1) -type f -print0 | xargs -0 -r md5sum > $($(PKG)_DIR)/.files$(2).md5;
> +endef
> +
> +# $(1): base directory to search in
> +# $(2): suffix of file (optional)
> +define pkg_detect_overwrite_after
> +	if test -s $($(PKG)_DIR)/.files$(2).md5 ; then \
> +		md5sum --quiet -c $($(PKG)_DIR)/.files$(2).md5 || \
> +		{ echo "ERROR: package $($(PKG)_NAME) has overwritten files installed by a previous package, aborting."; exit 1; } ; \
> +	fi
> +endef
> +endif
> +
>  # Functions to collect statistics about installed files
>  
>  # $(1): base directory to search in
> @@ -277,6 +296,8 @@ $(BUILD_DIR)/%/.stamp_configured:
>  	$(call fixup-libtool-files,$(NAME),$(HOST_DIR))
>  	$(call fixup-libtool-files,$(NAME),$(STAGING_DIR))
>  	$(foreach hook,$($(PKG)_POST_PREPARE_HOOKS),$(call $(hook))$(sep))
> +	@$(call pkg_detect_overwrite_before,$(TARGET_DIR))
> +	@$(call pkg_detect_overwrite_before,$(HOST_DIR),-host)
>  	$(foreach hook,$($(PKG)_PRE_CONFIGURE_HOOKS),$(call $(hook))$(sep))
>  	$($(PKG)_CONFIGURE_CMDS)
>  	$(foreach hook,$($(PKG)_POST_CONFIGURE_HOOKS),$(call $(hook))$(sep))
> @@ -401,6 +422,8 @@ $(BUILD_DIR)/%/.stamp_installed:
>  	@$(call pkg_size_after,$(STAGING_DIR),-staging)
>  	@$(call pkg_size_after,$(HOST_DIR),-host)
>  	@$(call check_bin_arch)
> +	@$(call pkg_detect_overwrite_after,$(TARGET_DIR))
> +	@$(call pkg_detect_overwrite_after,$(HOST_DIR),-host)
>  	$(Q)touch $@
>  
>  # Remove package sources
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 13/16] support/testing/infra: add log_file_path() function

[Yann E. MORIN]

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> Some tests will need to grep through the build log to verify that some
> features are working are expected. In order to allow them to open the
> build log, we provide a new function called log_file_path(), which
> returns the path to the log file if available.
> 
> We also use this function in open_log_file().
> 
> Note that open_log_file() cannot be used directly to grep through the
> log file at the end of a build: because it opens in "a+" mode, it
> greps starting from the end of the file.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>

This was not really part of the PPD/TLPB stuff, and did not seem to
depend on it in any way, so: applied to next, thanks.

Regards,
Yann E. MORIN.

> ---
> New patch in this v2 series
> 
> This patch is retrieved from Thomas's work.
> The first version was discussed
> https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-10-thomas.petazzoni@bootlin.com/
> 
> This new version was not already submitted by Thomas or I missed it.
> Compared to the first version, this patch use a more python-like
> syntax as proposed by Yann in the previous review.
> 
> Changes v2 to v3:
> None
> 
>  support/testing/infra/__init__.py | 11 ++++++-----
>  1 file changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/support/testing/infra/__init__.py b/support/testing/infra/__init__.py
> index b10a7601a3..1f003f24c6 100644
> --- a/support/testing/infra/__init__.py
> +++ b/support/testing/infra/__init__.py
> @@ -10,17 +10,18 @@ ARTIFACTS_URL = "http://autobuild.buildroot.net/artefacts/"
>  BASE_DIR = os.path.realpath(os.path.join(os.path.dirname(__file__), "../../.."))
>  
>  
> +def log_file_path(builddir, stage, logtofile=True):
> +    """Return path to log file"""
> +    return "{}-{}.log".format(builddir, stage) if logtofile else None
> +
> +
>  def open_log_file(builddir, stage, logtofile=True):
>      """
>      Open a file for logging and return its handler.
>      If logtofile is True, returns sys.stdout. Otherwise opens a file
>      with a suitable name in the build directory.
>      """
> -    if logtofile:
> -        fhandle = open("{}-{}.log".format(builddir, stage), 'a+')
> -    else:
> -        fhandle = sys.stdout
> -    return fhandle
> +    return open(log_file_path(builddir, stage, logtofile), 'a+') if logtofile else sys.stdout
>  
>  
>  def basepath(relpath=""):
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 14/16] support/testing/tests: add test for check_bin_arch

[Yann E. MORIN]

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> 
> This tests build a bogus package that installs a binary built for the
> host architecture into $(TARGET_DIR), which should cause a build
> failure, at least as long as the host architecture isn't ARM.
> 
> Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>

This was not really part of the PPD/TLPB stuff, and did not seem to
depend on it in any way, so: applied to next, thanks.

Seem a small nit below, though...

> ---
[--SNIP--]
> diff --git a/support/testing/tests/core/test_bad_arch.py b/support/testing/tests/core/test_bad_arch.py
> new file mode 100644
> index 0000000000..76acba3711
> --- /dev/null
> +++ b/support/testing/tests/core/test_bad_arch.py
> @@ -0,0 +1,19 @@
> +import infra
> +import infra.basetest
> +import subprocess

    $ make check-flake8
    support/testing/tests/core/test_bad_arch.py:3:1: F401 'subprocess' imported but unused
    1     F401 'subprocess' imported but unused

I fixed that when applying.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Thomas Petazzoni]

Hello Yann,

First of all, thanks a lot for reviewing and merging bits of this patch
series, I'm glad to see we're making progress with the TLP stuff.

On Sun, 29 Aug 2021 00:47:40 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> However, what prompted me from applying for now, is that this new
> detection is a hard error.
> 
> Previously, check-uniq-files was just emitting warnings, but would not
> prevent the build from failing. Now, with this patch, even an innocuous
> overwrite (e.g. because a post-build script deletes the file, or the
> content of the file really does not matter at runtime), the build will
> fail.
> 
> I.e. configurations that are currently working with PPD, despite the
> overwrite, will suddenly no longer build.
> 
> OTOH, if we do not make that a hard-error, we will never detect most
> issues, because users will never spot those warnings and wil enver
> report issues, and the autobuilders will not fail and we will not
> notice either...
> 
> One solution is to add a configuration knob to make that a hard-error,
> like we have the paranoid libs/headers check:
> 
>     config BR2_PPD_OVERWRITE_STRICT
>         bool "Strict file overwrite detection"
>         depends on BR2_PER_PACKAGE_DIRECTORIES
>         help
>           Say 'y' here to turn the file overwrite detection
>           to a hard error. By default, only warnings will be
>           printed.

We had some discussion with Hervé back when he worked on this, and I
disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a
file overwrite must be a hard error, as the result of the build is
incorrect if there is an overwrite. It's not the "latest" package that
wins in an overwrite situation, like it does in a non-PPD case.

So I really think this must be a hard error for PPD builds, and just a
warning for non-PPD builds.

Yes, for PPD builds, it means users will get failures, but those
failures are pointing to real problems.

So, my preference would be to merge as an unconditional check, and see
how it goes. Perhaps the situation will be so bad that we will have to
make it conditional, but I would prefer to have it unconditional first
and see the impact.

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Yann E. MORIN]

Thomas, Hervé, All,

On 2021-08-29 13:39 +0200, Thomas Petazzoni spake thusly:
> On Sun, 29 Aug 2021 00:47:40 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> > However, what prompted me from applying for now, is that this new
> > detection is a hard error.
[--SNIP--]
> We had some discussion with Hervé back when he worked on this, and I
> disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a
> file overwrite must be a hard error, as the result of the build is
> incorrect if there is an overwrite. It's not the "latest" package that
> wins in an overwrite situation, like it does in a non-PPD case.

So, existing builds that succeed with PPD, and for which an overwrite
is not an issue even at runtime, will now be broken at build time.

After all, we _know_ there are users that already use PPD (for TLPB),
and those users have working systems, but I highly doubt these builds
have no file overwrite whatsoever...

> So I really think this must be a hard error for PPD builds, and just a
> warning for non-PPD builds.

For non-PPD, that code is not even used, because the macros are only
defined if BR2_PER_PACKAGE_DIRECTORIES=y; so for non-PPD, we will not
even get a warning.

> Yes, for PPD builds, it means users will get failures, but those
> failures are pointing to real problems.

I fully agree, but in practice, some of those overwrites may be totally
innocusous when running the system. Think the gnu info db: we 100% don't
care about it at runtime, but this is an overwrite (yes, we already
fixed that one, but it was in this cycle, so users of previous releases
who use PPD do have overwrites that they don't care about).

> So, my preference would be to merge as an unconditional check, and see
> how it goes.

OK, I am aligning with that position. Unless someone else beats me to
it, I'll resume work on this series later in the WE, or early next
week...

> Perhaps the situation will be so bad that we will have to
> make it conditional, but I would prefer to have it unconditional first
> and see the impact.

Note that my proposal would have unconditionally enabled it in the
autobuilders, so we would have noticed as well, so we can still do that
if we later decide to make the hard error conditional.

Finally, as a side note: there is still a case of file overwrite that we
do not detect, even with this series: if two packages that are not part
of the same dependency chain (i.e. they do not depend one on the other),
and they both install the same file, then the file-overwrite will only
happen when we eventually assemble the global target/ and host/ from the
individual PPD target/ and host/ and we still have to add a detection
for that case (which is not a pre-requisite before we apply the current
series, of course).

Regards,
Yann E. MORIN.

> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, co-owner and CEO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Arnout Vandecappelle]

On 29/08/2021 13:39, Thomas Petazzoni wrote:
> Hello Yann,
> 
> First of all, thanks a lot for reviewing and merging bits of this patch
> series, I'm glad to see we're making progress with the TLP stuff.
> 
> On Sun, 29 Aug 2021 00:47:40 +0200
> "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
>> However, what prompted me from applying for now, is that this new
>> detection is a hard error.
>>
>> Previously, check-uniq-files was just emitting warnings, but would not
>> prevent the build from failing. Now, with this patch, even an innocuous
>> overwrite (e.g. because a post-build script deletes the file, or the
>> content of the file really does not matter at runtime), the build will
>> fail.
>>
>> I.e. configurations that are currently working with PPD, despite the
>> overwrite, will suddenly no longer build.
>>
>> OTOH, if we do not make that a hard-error, we will never detect most
>> issues, because users will never spot those warnings and wil enver
>> report issues, and the autobuilders will not fail and we will not
>> notice either...
>>
>> One solution is to add a configuration knob to make that a hard-error,
>> like we have the paranoid libs/headers check:
>>
>>     config BR2_PPD_OVERWRITE_STRICT
>>         bool "Strict file overwrite detection"
>>         depends on BR2_PER_PACKAGE_DIRECTORIES
>>         help
>>           Say 'y' here to turn the file overwrite detection
>>           to a hard error. By default, only warnings will be
>>           printed.
> 
> We had some discussion with Hervé back when he worked on this, and I
> disagreed with adding an option. When BR2_PER_PACKAGE_DIRECTORIES=y, a
> file overwrite must be a hard error, as the result of the build is
> incorrect if there is an overwrite. It's not the "latest" package that
> wins in an overwrite situation, like it does in a non-PPD case.
> 
> So I really think this must be a hard error for PPD builds, and just a
> warning for non-PPD builds.
> 
> Yes, for PPD builds, it means users will get failures, but those
> failures are pointing to real problems.
> 
> So, my preference would be to merge as an unconditional check, and see
> how it goes. Perhaps the situation will be so bad that we will have to
> make it conditional, but I would prefer to have it unconditional first
> and see the impact.

 I was originally with Yann, but these arguments convinced that it is indeed
better to not have the option (for now).

 Regards,
 Arnout

_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Yann E. MORIN]

Thomas, All,

On 2021-08-29 14:51 +0200, Yann E. MORIN spake thusly:
[--SNIP--]
> Finally, as a side note: there is still a case of file overwrite that we
> do not detect, even with this series: if two packages that are not part
> of the same dependency chain (i.e. they do not depend one on the other),
> and they both install the same file, then the file-overwrite will only
> happen when we eventually assemble the global target/ and host/ from the
> individual PPD target/ and host/ and we still have to add a detection
> for that case (which is not a pre-requisite before we apply the current
> series, of course).

And of course, this will also cause an overwrite when two such packages
are also dependencies of a third patckage, and so when the PPD of that
third package is prepared, we would have to detect file overwrite.

Note: let's assume the following hypotetical dependency graph (where the
notation A --<-- B means that B depends on A):

      .-<-- A --<-.                 S: skeleton
     /             \                F: target-finalize
    S --<-- B --<-- C
     \               \
      `-<-- D --<-----<-- F

If A and B both install the same file, then we do not have a sane PPD
to build C.

If D and any package in the dependency chain of C (i.e. S, A, B, or C
itself) install the same file, then we can only detect it in
target-finalize.

So, let's take a step back, and have a wider look. I wonder if the end
of a package installation step is the best location to detected file
overwrite...

Indeed, let's assume that we do file overwrite detection:

 1. in PPD preparation (when we rsync the dependencies, in the macro
    per-package-rsync): we can also detect that a package overwrites one
    or more files of another package, report those files, *and* prevent
    building a package with unsound dependencies;

 2. in {target,host}-finalize: we can detect that a package overwrites
    one or more files provided by another package, and report those
    files.

So, the above (still theoretical implementation) is all we would need,
in the end, and we would not need to do the detection at the end of a
package installation step of each package, because it would not provide
any information that we couldn't have with the above.

However, a benefit of also doing it at the end of the installation step,
of a package, is that we can cut short on the build time. However, I am
not sure that benefit is that important or major...

So, I'd like we think a bit harder before we apply this patch...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Arnout Vandecappelle]

On 29/08/2021 18:40, Yann E. MORIN wrote:
> Thomas, All,
> 
> On 2021-08-29 14:51 +0200, Yann E. MORIN spake thusly:
> [--SNIP--]
>> Finally, as a side note: there is still a case of file overwrite that we
>> do not detect, even with this series: if two packages that are not part
>> of the same dependency chain (i.e. they do not depend one on the other),
>> and they both install the same file, then the file-overwrite will only
>> happen when we eventually assemble the global target/ and host/ from the
>> individual PPD target/ and host/ and we still have to add a detection
>> for that case (which is not a pre-requisite before we apply the current
>> series, of course).
> 
> And of course, this will also cause an overwrite when two such packages
> are also dependencies of a third patckage, and so when the PPD of that
> third package is prepared, we would have to detect file overwrite.
> 
> Note: let's assume the following hypotetical dependency graph (where the
> notation A --<-- B means that B depends on A):
> 
>       .-<-- A --<-.                 S: skeleton
>      /             \                F: target-finalize
>     S --<-- B --<-- C
>      \               \
>       `-<-- D --<-----<-- F
> 
> If A and B both install the same file, then we do not have a sane PPD
> to build C.
> 
> If D and any package in the dependency chain of C (i.e. S, A, B, or C
> itself) install the same file, then we can only detect it in
> target-finalize.
> 
> So, let's take a step back, and have a wider look. I wonder if the end
> of a package installation step is the best location to detected file
> overwrite...
> 
> Indeed, let's assume that we do file overwrite detection:
> 
>  1. in PPD preparation (when we rsync the dependencies, in the macro
>     per-package-rsync): we can also detect that a package overwrites one
>     or more files of another package, report those files, *and* prevent
>     building a package with unsound dependencies;
> 
>  2. in {target,host}-finalize: we can detect that a package overwrites
>     one or more files provided by another package, and report those
>     files.
> 
> So, the above (still theoretical implementation) is all we would need,
> in the end, and we would not need to do the detection at the end of a
> package installation step of each package, because it would not provide
> any information that we couldn't have with the above.

 Fabrice's patches have shown that the overwritten file detection is too
aggressive in the sense to that it *will* cause (many?) failures in the current
situation.

 So, taking a step further back, what really is the problem we're trying to
solve here? Why can't we overwrite files from other packages?

 I think there are three distinct problems. Let's take Yann's example from above.

1. A file written in A and overwritten in C is only a problem because the
finalize step doesn't take into account dependencies when combining the trees.
We could solve that by taking into the account the dependencies.

2. A file written by both A and B is a problem for package C, because it is not
predictable which version it will see. This problem can be solved by adding a
dependency between A and B.

3. A file written by S and overwritten by A - this, I think, is the big problem.
At first sight it's like case 1 - by taking into account the dependencies, the
finalize stage will see the version from A. However, the problem is for stuff
installed into staging, where B sees the version from S while C sees the version
from A or from S, depending on order of A/B installation. This has two problems:
3a. C's version depends on the order of A/B installation;
3b. B sees a different version of the file than A - this could be a problem for
e.g. header files that affect compilation.

 The original scheme we discussed three or four years ago already took into
account situations 1 and 2, I think, but not yet situation 3. Note that back
then, the overwritten file detection was assumed to happen only in finalize and
accept overwrites in case there's a dependency.

 The curses and crypt issues that Fabrice detected fall in category 3. And
they're extra nasty, because one of the packages involved is the toolchain that
everybody depends on, so you can't just add a dependency to break it.

 I haven't given sufficient thought to it yet, but one idea is to allow packages
to add a post-install hook to another package. That way, ncurses could remove
curses.h from the toolchain, and libxcrypt could remove crypt.h. Still tricky
though because it would mean that any package that uses crypt.h would need to
optionally depend on libxcrypt - so that particular case would probably have to
be solved by making libxcrypt be part of the toolchain... Or maybe making sure
it doesn't overwrite crypt.h and teach its dependencies to include the
libxcrypt-specific crypt.h.

 One thing is certain: it's complicated :-)

 Regards,
 Arnout


> 
> However, a benefit of also doing it at the end of the installation step,
> of a package, is that we can cut short on the build time. However, I am
> not sure that benefit is that important or major...
> 
> So, I'd like we think a bit harder before we apply this patch...
> 
> Regards,
> Yann E. MORIN.
> 
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Andreas Naumann]

Hi all,

On 29.08.21 17:01, Arnout Vandecappelle wrote:
> 
> 
> On 29/08/2021 13:39, Thomas Petazzoni wrote:
>> Hello Yann,
>> 
>> First of all, thanks a lot for reviewing and merging bits of this
>> patch series, I'm glad to see we're making progress with the TLP
>> stuff.
>> 
>> On Sun, 29 Aug 2021 00:47:40 +0200 "Yann E. MORIN"
>> <yann.morin.1998@free.fr> wrote:
>> 
>>> However, what prompted me from applying for now, is that this
>>> new detection is a hard error.
>>> 
>>> Previously, check-uniq-files was just emitting warnings, but
>>> would not prevent the build from failing. Now, with this patch,
>>> even an innocuous overwrite (e.g. because a post-build script
>>> deletes the file, or the content of the file really does not
>>> matter at runtime), the build will fail.
>>> 
>>> I.e. configurations that are currently working with PPD, despite
>>> the overwrite, will suddenly no longer build.
>>> 
>>> OTOH, if we do not make that a hard-error, we will never detect
>>> most issues, because users will never spot those warnings and wil
>>> enver report issues, and the autobuilders will not fail and we
>>> will not notice either...
>>> 
>>> One solution is to add a configuration knob to make that a
>>> hard-error, like we have the paranoid libs/headers check:
>>> 
>>> config BR2_PPD_OVERWRITE_STRICT bool "Strict file overwrite
>>> detection" depends on BR2_PER_PACKAGE_DIRECTORIES help Say 'y'
>>> here to turn the file overwrite detection to a hard error. By
>>> default, only warnings will be printed.
>> 
>> We had some discussion with Hervé back when he worked on this, and
>> I disagreed with adding an option. When
>> BR2_PER_PACKAGE_DIRECTORIES=y, a file overwrite must be a hard
>> error, as the result of the build is incorrect if there is an
>> overwrite. It's not the "latest" package that wins in an overwrite
>> situation, like it does in a non-PPD case.
>> 
>> So I really think this must be a hard error for PPD builds, and
>> just a warning for non-PPD builds.
>> 
>> Yes, for PPD builds, it means users will get failures, but those 
>> failures are pointing to real problems.
>> 
>> So, my preference would be to merge as an unconditional check, and
>> see how it goes. Perhaps the situation will be so bad that we will
>> have to make it conditional, but I would prefer to have it
>> unconditional first and see the impact.
> 
> I was originally with Yann, but these arguments convinced that it is
> indeed better to not have the option (for now).

So I'm one of those actively using TLPD and I'm happy to have this
producing hard errors to find possible issues in my configs as well as
helping to fix them.

So I gave this a spin and immediately found util-linux recompiling some
libs which util-linux-libs already installed. Looking for the reason I
came upon Carlos Santos Commit 8bafc6dc "package/util-linux: build
programs and libraries in separate packages" where the commit msg says
".... Installing util-linux overrides files installed by util-linux-libs
but this is not a problem: it's allowed for a package to overwrite files
from another package, as long as there is a dependency between the two."

So if this is still true, the logic should be changed to take
dependencies into account like suggested in the other discussion.

However, the check also found a conflict where GL/glext.h of mesa3d is
overwritten by gst1-plugins-base. It is a dependency, thus it's
deterministic, but still we probably end up with an unexpected version
glext.h in the SDK, so it's nice that things like this are brought to
attention.

Now the build of our standard qt-demo config is almost done and another
conflict pops up when some kernel modules of Wifi-Sticks are installed.
Of course modules.dep, modules.alias and so on are modified. That will 
have to be taken care of.

However, only 3 problems up to just before the target-finalize step. So
far that's not too aggressive for me.


regards,
Andreas



> 
> Regards, Arnout
> 
> _______________________________________________ buildroot mailing
> list buildroot@busybox.net 
> http://lists.busybox.net/mailman/listinfo/buildroot
> 
_______________________________________________
buildroot mailing list
buildroot@busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 07/16] package/pkg-generic.mk: detect files overwritten in TARGET_DIR and HOST_DIR

[Yann E. MORIN]

Hervé, Thomas, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> From: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Without per-package directory support, a package can happily overwrite
> files installed by other packages. Indeed, because the build order
> between packages is always guaranteed, Buildroot will always produce
> the same output.
[--SNIP--]
> This commit is a new implementation of such a detection, which is
> based on calculating and verifying MD5 hashes of installed files: the
> calculation is done at the beginning of the configure step, the
> verification during the newly introduced "install" step that takes
> place after all installation steps.

Following the various mails in the thread, and the discussion we had on
Tuesday, I'm markign this patch as Changes Requested.

I'll try to scavenge all I can from the remaining patches, and mark the
rest as Changes Requested as well.

So, to summarise the discussion, taking my previous example as a basis,
and with Arnout's comment in mind (Aha, pun!):

      .-<-- A --<-.                 S: skeleton
     /             \                F: target-finalize
    S --<-- B --<-- C
     \               \
      `-<-- D --<-----<-- F

 1. If C overwrites a file from (e.g.) A: we don't know the order in
    which the target-finalize will copy files.

    - we can guarantee that ordering by taking the dependency chain into
      account when doing target-finalize.

 2. If A and B install the same package: we don't know what to provide
    to package C:

    - adding an artificial dependency between A and B is not acceptable,
      or we could end up with a lot of such unmanageable dependencies,
      not even  accounting for packages in a br32-external tree;

    - if the file is a DB (like the GNU info DB), stuff is "added" to
      that DB; only providing an arbitrary version of the file (from A
      or from B) is incorrect.

 3. If a package in the C dependency chain (C included), and package D
    install the same file, then we're in the same boat as the above: we
    do not know what version to use in tergaet-finalize.

If I missed something, then feel free to complete. ;-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

[Yann E. MORIN]

Hervé, All,

On 2021-08-17 10:39 +0200, Herve Codina spake thusly:
> Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
> target_finalize steps some modifications are done on installed files (ie
> strip or TARGET_FINALIZE_HOOKS for instance).
> 
> In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR
> and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built
> using a full copy of the involved per-package files instead of hardlinks.
> 
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
> Changes v1 to v2:
>  - Added 'Reviewed-by: Yann E. MORIN'
> 
> Changes v2 to v2:
> None
> 
>  Makefile | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 32fad004fe..ccf2020565 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
>  define per-package-rsync-delta
>  	$(Q)mkdir -p $(3)
>  	$(foreach pkg,$(1),\
> -		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
> +		$(Q)rsync -a \

I was about to apply that one, because it does not look like it
(semantically) depends on the previous patches, but of course it does
not apply cleanly... I found it too dangerous to apply and tweak locally
without further consideration. So it follows the fate of the rest of the
series: marked Changes Requested.

Regards,
Yann E. MORIN.

>  			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
>  			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
>  			$(3)$(sep))
> -- 
> 2.31.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

[Peter Korsgaard]

On 17/08/2021 10.39, Herve Codina wrote:

Hi,

Sorry, very old patch.


> Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
> target_finalize steps some modifications are done on installed files (ie
> strip or TARGET_FINALIZE_HOOKS for instance).
> 
> In order to avoid these modifications seen from per-package {TARGET,HOST}_DIR
> and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built
> using a full copy of the involved per-package files instead of hardlinks.
> 
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
> Changes v1 to v2:
>   - Added 'Reviewed-by: Yann E. MORIN'

We have similar issues if you have a post-build script making changes 
with sed or similar, so committed to master after fixing it up (the 
logic was in the mean time moved to package/pkg-utils.mk), thanks.


> 
> Changes v2 to v2:
> None
> 
>   Makefile | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile b/Makefile
> index 32fad004fe..ccf2020565 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -734,7 +734,7 @@ ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
>   define per-package-rsync-delta
>   	$(Q)mkdir -p $(3)
>   	$(foreach pkg,$(1),\
> -		$(Q)rsync -a --link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
> +		$(Q)rsync -a \
>   			--filter='merge $($(call UPPERCASE,$(pkg))_DIR)/$(strip $(4))' \
>   			$(PER_PACKAGE_DIR)/$(pkg)/$(strip $(2))/ \
>   			$(3)$(sep))

-- 
Bye, Peter Korsgaard

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 10/16] Makefile: breaks hardlinks in global {TARGET, HOST}_DIR on per-package build

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <peter@korsgaard.com> writes:

 > On 17/08/2021 10.39, Herve Codina wrote:
 > Hi,

 > Sorry, very old patch.


 >> Without this patch, a make <pkg>_rebuild detects overwrites. Indeed, in
 >> target_finalize steps some modifications are done on installed files (ie
 >> strip or TARGET_FINALIZE_HOOKS for instance).
 >> In order to avoid these modifications seen from per-package
 >> {TARGET,HOST}_DIR
 >> and so been analyzed as some overwrites, global {TARGET,HOST}_DIR is built
 >> using a full copy of the involved per-package files instead of hardlinks.
 >> Signed-off-by: Herve Codina <herve.codina@bootlin.com>
 >> Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>
 >> ---
 >> Changes v1 to v2:
 >> - Added 'Reviewed-by: Yann E. MORIN'

 > We have similar issues if you have a post-build script making changes
 > with sed or similar, so committed to master after fixing it up (the 
 > logic was in the mean time moved to package/pkg-utils.mk), thanks.

Committed to 2023.02.x and 2023.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot