[Buildroot] [PATCH 00/15] Overwritten file detection and fixes, one more step to TLP build

[Andreas Naumann <dev@andin.de>]

Hi Herve, all,

On 21.06.21 16:11, Herve Codina wrote:
> Hi,
> 
> This series fixes some issues in the TLP build feature.
> The starting point was the work done by Thomas Petazzoni on overwrites
> detection (PATCH 1).

after a long time I have recently picked up playing with TLP build 
again. As part of that I found a problem in qt5 where I sent a patch two 
days ago "qt5: Fix sporadic build failure during top-level parallel 
build" which is caused by manipulation of the hard-linked qt.conf in 
HOST_DIR from different qt5 packages.

Now I see your promising set, but the matter is quite complex and so I 
wonder if the "overwritten file detection" would a) uncover the qt.conf 
problem and b)if your patch series somehow fixes it in a generic way?


regards,
Andreas



> 
> Based on this work, this series fixes some overwrites, introduces and uses
> <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific
> per-package path tweaks needed by per-package build.
> With this tweaks collected in this variable, they can be re-applied
> when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).
> 
> It also reworks the final {HOST,TARGET}_DIR rsync.
> For each packages only files generated by the package are rsynced (PATCH 13).
> This is done using an exclusion list computed in (PATCH 12).
> In order to avoid modifications performed in target-finalize step to be seen
> as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any
> hardlink anymore and are a full copy of expected files and directories (PATCH 14).
> 
> The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating
> per-package {HOST,TARGET}_DIR from scratch.
> Indeed, for a given package, re-generating the same file is detected as an overwrite.
> The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR
> directory and recreate them.
> 
> To have a TLP build fully fonctionnal, several things are still missing:
>    - Be sure that all buildroot packages do not perform any overwrites.
>    - Undo tweak stuff that have be done for per-package build (fixup-libtool-files,
>      fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files
>      in final {HOST,STAGING}_DIR.
>      This mainly means reverting .../per-package/xxxxx/... path to point to
>      the correct ones in final {HOST,STAGING}_DIR.
> 
> I hope this series will help in TLP build feature.
> 
> Best regards,
> Herv? Codina
> 
> Herve Codina (14):
>    package/e2fsprogs: fix fsck overwrite in HOST_DIR
>    package/pkg-generic.mk: Remove Info documents dir entry
>    package/pkg-generic.mk: Fix .la files overwrite detection
>    package/pkg-generic.mk: Perform .la files fixup in per-package
>      HOST_DIR
>    package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
>    package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS
>    package/apache: Move APACHE_FIXUP_APR_LIBTOOL to
>      <PKG>_PER_PACKAGE_TWEAK_HOOKS
>    package/pkg-python: Remove _sysconfigdata*.pyc files when
>      _sysconfigdata*.py are changed
>    package/pkg-generic.mk: Move python fixup to generic package
>      infrastructure
>    package/owfs: Remove Python sysconfigdata fixup
>    package/pkg-generic.mk: Generate final rsync exclude file list
>    Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
>    Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
>      build
>    package/pkg-generic.mk: Fix per-package
>      <pkg>-{reconfigure,rebuild,reinstall}
> 
> Thomas Petazzoni (1):
>    package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
>      HOST_DIR
> 
>   Makefile                       |  24 +++++++-
>   package/apache/apache.mk       |   2 +-
>   package/apr-util/apr-util.mk   |  12 ++--
>   package/e2fsprogs/e2fsprogs.mk |   1 +
>   package/owfs/owfs.mk           |   9 ---
>   package/pkg-generic.mk         | 103 ++++++++++++++++++++++++++++++++-
>   package/pkg-python.mk          |  10 ----
>   7 files changed, 134 insertions(+), 27 deletions(-)
>