[Buildroot] [PATCH 00/15] Overwritten file detection and fixes, one more step to TLP build

[Arnout Vandecappelle <arnout@mind.be>]

 Hi Herve?,

 Great that someone is picking thuis up!

On 21/06/2021 16:11, Herve Codina wrote:
> Hi,
> 
> This series fixes some issues in the TLP build feature.
> The starting point was the work done by Thomas Petazzoni on overwrites
> detection (PATCH 1).

 I've marked [1] as Changes Requested.

 However, it seems the tests [2][3][4] are missing from this series. Could you
include them next time you post it?

> Based on this work, this series fixes some overwrites, introduces and uses
> <PKG>_PER_PACKAGE_TWEAK_HOOKS. This variable is used to collect specific
> per-package path tweaks needed by per-package build.
> With this tweaks collected in this variable, they can be re-applied
> when they are needed (<pkg>_{reconfigure,rebuild,reinstall}).

 That's excellent!


> It also reworks the final {HOST,TARGET}_DIR rsync.
> For each packages only files generated by the package are rsynced (PATCH 13).
> This is done using an exclusion list computed in (PATCH 12).
> In order to avoid modifications performed in target-finalize step to be seen
> as overwrites on next build, the final {HOST,TARGET}_DIR do not contain any
> hardlink anymore and are a full copy of expected files and directories (PATCH 14).

 Hm, that will have quite the impact on disk size, no? In non-PPS builds,
HOST_DIR is typically about 1/3 of the total size (the other 2/3 is BUILD_DIR,
the rest is negligible). By doing a non-hardlinked copy, the size of HOST_DIR is
basically going to double, right?

 I still think it's acceptable, but it's something to take into account...


> The last patch (PATCH 15) fixes <pkg>_{reconfigure,rebuild,reinstall} recreating
> per-package {HOST,TARGET}_DIR from scratch.
> Indeed, for a given package, re-generating the same file is detected as an overwrite.
> The simplest way to avoid this is to fully discard previous per-package {HOST,TARGET}_DIR
> directory and recreate them.

 +1 to that!


> To have a TLP build fully fonctionnal, several things are still missing:
>   - Be sure that all buildroot packages do not perform any overwrites.

 I guess that's easy: increase the chance of PER_PACKAGE enabled in
genrandconfig, and see what happens to the autobuilders. Or is there anything
else missing?

>   - Undo tweak stuff that have be done for per-package build (fixup-libtool-files,
>     fixup-python-files, <PKG>_PER_PACKAGE_TWEAK_HOOKS) after collecting files
>     in final {HOST,STAGING}_DIR.
>     This mainly means reverting .../per-package/xxxxx/... path to point to
>     the correct ones in final {HOST,STAGING}_DIR.
> 
> I hope this series will help in TLP build feature.


 Regards,
 Arnout

[1]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-9-thomas.petazzoni at bootlin.com/
[2]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-10-thomas.petazzoni at bootlin.com/
[3]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-11-thomas.petazzoni at bootlin.com/
[4]
https://patchwork.ozlabs.org/project/buildroot/patch/20200430095249.782597-12-thomas.petazzoni at bootlin.com/



> 
> Best regards,
> Herv? Codina
> 
> Herve Codina (14):
>   package/e2fsprogs: fix fsck overwrite in HOST_DIR
>   package/pkg-generic.mk: Remove Info documents dir entry
>   package/pkg-generic.mk: Fix .la files overwrite detection
>   package/pkg-generic.mk: Perform .la files fixup in per-package
>     HOST_DIR
>   package/pkg-generic.mk: Introduce <PKG>_PER_PACKAGE_TWEAK_HOOKS
>   package/apr-util: Use <PKG>_PER_PACKAGE_TWEAK_HOOKS
>   package/apache: Move APACHE_FIXUP_APR_LIBTOOL to
>     <PKG>_PER_PACKAGE_TWEAK_HOOKS
>   package/pkg-python: Remove _sysconfigdata*.pyc files when
>     _sysconfigdata*.py are changed
>   package/pkg-generic.mk: Move python fixup to generic package
>     infrastructure
>   package/owfs: Remove Python sysconfigdata fixup
>   package/pkg-generic.mk: Generate final rsync exclude file list
>   Makefile: Rsync global {TARGET,HOST}_DIR using exclusion file list
>   Makefile: Breaks hardlinks in global {TARGET,HOST}_DIR on per-package
>     build
>   package/pkg-generic.mk: Fix per-package
>     <pkg>-{reconfigure,rebuild,reinstall}
> 
> Thomas Petazzoni (1):
>   package/pkg-generic.mk: detect files overwritten in TARGET_DIR and
>     HOST_DIR
> 
>  Makefile                       |  24 +++++++-
>  package/apache/apache.mk       |   2 +-
>  package/apr-util/apr-util.mk   |  12 ++--
>  package/e2fsprogs/e2fsprogs.mk |   1 +
>  package/owfs/owfs.mk           |   9 ---
>  package/pkg-generic.mk         | 103 ++++++++++++++++++++++++++++++++-
>  package/pkg-python.mk          |  10 ----
>  7 files changed, 134 insertions(+), 27 deletions(-)
>