[Buildroot] [PATCH 0/4 v2] core/pkg-generic: store package<->files for staging and host

[Buildroot] [PATCH 0/4 v2] core/pkg-generic: store package<->files for staging and host

[Yann E. MORIN]

Hello All!

This small series does for staging and host as we do for target:
store the list of installed files and relate them to the package
that installed each.

In the end, it also reports files that are touched by more than one
package, for target, staging and host files.

---
Changes v1 -> v2:
  - split in three patches
  - in addition to staging, also store for host
  - report files touched by more than one package


Regards,
Yann E. MORIN.


The following changes since commit c2aa34d8cede20d4c8701f20ade9a016fb478b87

  CHANGES: update after netsnmp changes (2017-10-22 16:46:33 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 64a5b7666e47d87d58845028f9576ab68a186f44

  core: check files are nto touched by more than on package (2017-10-22 16:55:40 +0200)


----------------------------------------------------------------
Yann E. MORIN (4):
      core/pkg-generic: remove intermediate file-list files
      core/pkg-generic: redirect only when listing package's installed files
      core/pkg-generic: store file->package list for staging and host too
      core: check files are nto touched by more than on package

 Makefile                            |  4 ++++
 package/pkg-generic.mk              | 45 ++++++++++++++++++++++++-------------
 support/scripts/check-uniq-files.py | 40 +++++++++++++++++++++++++++++++++
 3 files changed, 73 insertions(+), 16 deletions(-)
 create mode 100755 support/scripts/check-uniq-files.py

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4 v2] core/pkg-generic: remove intermediate file-list files

[Yann E. MORIN]

To compute the list of files added by a package, we first store the list
of files before the install, do the install, list the files after the
install, and finally compare the two lists. The two lists are stored in
dot-files, hidden in the package's build dir.

We currently keep those two files, and only list the files installed in
target/

In followup patches, we'll also list files installed in staging/ as well
as files installed in host/.

Rather than add even more internal, hidden files in the package build
dir, we'll just re-use the same two temporary files to store the bfore
and after lists.

So, remove them after the comparison is done.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-generic.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index cca94ba338..c81a701834 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -85,6 +85,7 @@ define step_pkg_size_end
 		while read hash file ; do \
 			echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
 		done
+	rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after
 endef
 
 define step_pkg_size
-- 
2.11.0

[Buildroot] [PATCH 2/4 v2] core/pkg-generic: redirect only when listing package's installed files

[Yann E. MORIN]

There is no need to redirect again and again for each new file added to
the list; we can just redirect once and for all.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-generic.mk | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index c81a701834..0baab661c7 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -83,8 +83,8 @@ define step_pkg_size_end
 	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after)
 	comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \
 		while read hash file ; do \
-			echo "$(1),$${file}" >> $(BUILD_DIR)/packages-file-list.txt ; \
-		done
+			echo "$(1),$${file}" ; \
+		done >> $(BUILD_DIR)/packages-file-list.txt
 	rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after
 endef
 
-- 
2.11.0

[Buildroot] [PATCH 3/4 v2] core/pkg-generic: store file->package list for staging and host too

[Yann E. MORIN]

Currently, we store the list of files installed in target/ and associate
each of them to the package that installed it.

However, we sometime may need to know what package installed which file
in staging/, for example to debug header collision, or in host/, to
debug what package installed what host tool.

Enhance the step instrumentation to also generate the list for staging/
and host/.

We maintain backward compatibility, for external script that wanted to
parse the previously existing list, by not renaming the target-related
package list. Only the staging- and host-related lists are named after
staging and host.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/pkg-generic.mk | 42 +++++++++++++++++++++++++++---------------
 1 file changed, 27 insertions(+), 15 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 0baab661c7..e50fad906c 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -58,7 +58,7 @@ GLOBAL_INSTRUMENTATION_HOOKS += step_time
 # Hooks to collect statistics about installed files
 
 define _step_pkg_size_get_file_list
-	(cd $(TARGET_DIR) ; \
+	(cd $(2) ; \
 		( \
 			find . -xtype f -print0 | xargs -0 md5sum ; \
 			find . -xtype d -print0 | xargs -0 -I{} printf 'directory  {}\n'; \
@@ -66,32 +66,44 @@ define _step_pkg_size_get_file_list
 	) | sort > $1
 endef
 
-# This hook will be called before the target installation of a
-# package. We store in a file named .br_filelist_before the list of
-# files currently installed in the target. Note that the MD5 is also
-# stored, in order to identify if the files are overwritten.
+# This hook will be called before the installation of a package. We store in
+# a file named .br_filelist_before the list of files currently installed.
+# Note that the MD5 is also stored, in order to identify if the files are
+# overwritten.
+# $(1): package name (ignored)
+# $(2): base directory to search in
 define step_pkg_size_start
-	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before)
+	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_before,$(2))
 endef
 
-# This hook will be called after the target installation of a
-# package. We store in a file named .br_filelist_after the list of
-# files (and their MD5) currently installed in the target. We then do
-# a diff with the .br_filelist_before to compute the list of files
-# installed by this package.
+# This hook will be called after the installation of a package. We store in
+# a file named .br_filelist_after the list of files (and their MD5) currently
+# installed. We then do a diff with the .br_filelist_before to compute the
+# list of files installed by this package.
+# The suffix is typically for the target variant, for legacy backward
+# compatibility.
+# $(1): package name (ignored)
+# $(2): base directory to search in
+# $(3): suffix of file  (optional)
 define step_pkg_size_end
-	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after)
+	$(call _step_pkg_size_get_file_list,$($(PKG)_DIR)/.br_filelist_after,$(2))
 	comm -13 $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after | \
 		while read hash file ; do \
 			echo "$(1),$${file}" ; \
-		done >> $(BUILD_DIR)/packages-file-list.txt
+		done >> $(BUILD_DIR)/packages-file-list$(3).txt
 	rm -f $($(PKG)_DIR)/.br_filelist_before $($(PKG)_DIR)/.br_filelist_after
 endef
 
 define step_pkg_size
 	$(if $(filter install-target,$(2)),\
-		$(if $(filter start,$(1)),$(call step_pkg_size_start,$(3))) \
-		$(if $(filter end,$(1)),$(call step_pkg_size_end,$(3))))
+		$(if $(filter start,$(1)),$(call step_pkg_size_start,$(3),$(TARGET_DIR))) \
+		$(if $(filter end,$(1)),$(call step_pkg_size_end,$(3),$(TARGET_DIR))))
+	$(if $(filter install-staging,$(2)),\
+		$(if $(filter start,$(1)),$(call step_pkg_size_start,$(3),$(STAGING_DIR),-staging)) \
+		$(if $(filter end,$(1)),$(call step_pkg_size_end,$(3),$(STAGING_DIR),-staging)))
+	$(if $(filter install-host,$(2)),\
+		$(if $(filter start,$(1)),$(call step_pkg_size_start,$(3),$(HOST_DIR),-host)) \
+		$(if $(filter end,$(1)),$(call step_pkg_size_end,$(3),$(HOST_DIR),-host)))
 endef
 GLOBAL_INSTRUMENTATION_HOOKS += step_pkg_size
 
-- 
2.11.0

[Buildroot] [PATCH 4/4 v2] core: check files are nto touched by more than on package

[Yann E. MORIN]

Currently, we do nothing about packages that touch the same file: given
a specific configuration, the result is reproducible (even though it
might not be what the user expected) because the build order is
guaranteed.

Hwever, when we later introduce top-level parallel buil, we will no
longer be able to guarantee a build order, by the mere way of it being
parallel. Reconcialliting all those modified files will be impossible to
do automatically. The only way will be to  refuse such situations.

As a preliminary step, introduce a helper script that detects files that
are being moified by two or more packages, and reports them and the
impacted packages, at the end of the build.

The list being reported at the end of the build will make it prominently
visible in autobuilder results, so we can assess the problem, if any.

Later on, calling that helper script can be done right after the package
installation step, to bail out early.

Thanks Arnout for the pythonist way to write default dictionaries! ;-)

Note: doing it in python rather than a shell script is impressively
faster: where the shell script takes ~1.2s on a minimalist build, the
python script only takes ~0.015s, that is about 80 times faster.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
---
 Makefile                            |  4 ++++
 support/scripts/check-uniq-files.py | 40 +++++++++++++++++++++++++++++++++++++
 2 files changed, 44 insertions(+)
 create mode 100755 support/scripts/check-uniq-files.py

diff --git a/Makefile b/Makefile
index 79db7fe48a..f0b60bae5c 100644
--- a/Makefile
+++ b/Makefile
@@ -679,6 +679,10 @@ $(TARGETS_ROOTFS): target-finalize
 .PHONY: target-finalize
 target-finalize: $(PACKAGES)
 	@$(call MESSAGE,"Finalizing target directory")
+	# Check files that are touched by more than one package
+	./support/scripts/check-uniq-files.py -t target $(BUILD_DIR)/packages-file-list.txt
+	./support/scripts/check-uniq-files.py -t staging $(BUILD_DIR)/packages-file-list-staging.txt
+	./support/scripts/check-uniq-files.py -t host $(BUILD_DIR)/packages-file-list-host.txt
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
diff --git a/support/scripts/check-uniq-files.py b/support/scripts/check-uniq-files.py
new file mode 100755
index 0000000000..e77472cf80
--- /dev/null
+++ b/support/scripts/check-uniq-files.py
@@ -0,0 +1,40 @@
+#!/usr/bin/env python2
+
+import sys
+import csv
+import argparse
+from collections import defaultdict
+
+warn = 'Warnng: {}s file "{}" is touched by more than one package: {}\n'
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument('packages_file_list', nargs='*',
+                    help='The packages-file-list to check from')
+    parser.add_argument('-t', '--type', metavar="TYPE",
+                        help='Report as a TYPE file (TYPE is either target, staging, or host)')
+
+    args = parser.parse_args()
+
+    if not len(args.packages_file_list) == 1:
+        sys.stderr.write('No packages-file-list was provided.\n')
+        return False
+
+    if args.type is None:
+        sys.stderr.write('No type was provided\n')
+        return False
+
+    file_to_pkg = defaultdict(list)
+    with open(args.packages_file_list[0], 'rb') as pkg_file_list:
+        r = csv.reader(pkg_file_list, delimiter=',')
+        for row in r:
+            pkg = row[0]
+            file = row[1]
+            file_to_pkg[file].append(pkg)
+
+    for file in file_to_pkg:
+        if len(file_to_pkg[file]) > 1:
+            sys.stderr.write(warn.format(args.type, file, file_to_pkg[file]))
+
+if __name__ == "__main__":
+    sys.exit(main())
-- 
2.11.0

[Buildroot] [PATCH 4/4 v2] core: check files are nto touched by more than on package

[Peter Seiderer]

Hello Yann,

typo in the patch title s/nto/not/...

On Sun, 22 Oct 2017 17:15:40 +0200, "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:

> Currently, we do nothing about packages that touch the same file: given
> a specific configuration, the result is reproducible (even though it
> might not be what the user expected) because the build order is
> guaranteed.
> 
> Hwever, when we later introduce top-level parallel buil, we will no

s/Hwever/However/, s/buil/build/

> longer be able to guarantee a build order, by the mere way of it being
> parallel. Reconcialliting all those modified files will be impossible to
> do automatically. The only way will be to  refuse such situations.

s/  / /

Regards,
Peter

> 
> As a preliminary step, introduce a helper script that detects files that
> are being moified by two or more packages, and reports them and the
> impacted packages, at the end of the build.
> 
> The list being reported at the end of the build will make it prominently
> visible in autobuilder results, so we can assess the problem, if any.
> 
> Later on, calling that helper script can be done right after the package
> installation step, to bail out early.
> 
> Thanks Arnout for the pythonist way to write default dictionaries! ;-)
> 
> Note: doing it in python rather than a shell script is impressively
> faster: where the shell script takes ~1.2s on a minimalist build, the
> python script only takes ~0.015s, that is about 80 times faster.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Peter Korsgaard <peter@korsgaard.com>
> ---
>  Makefile                            |  4 ++++
>  support/scripts/check-uniq-files.py | 40 +++++++++++++++++++++++++++++++++++++
>  2 files changed, 44 insertions(+)
>  create mode 100755 support/scripts/check-uniq-files.py
> 
> diff --git a/Makefile b/Makefile
> index 79db7fe48a..f0b60bae5c 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -679,6 +679,10 @@ $(TARGETS_ROOTFS): target-finalize
>  .PHONY: target-finalize
>  target-finalize: $(PACKAGES)
>  	@$(call MESSAGE,"Finalizing target directory")
> +	# Check files that are touched by more than one package
> +	./support/scripts/check-uniq-files.py -t target $(BUILD_DIR)/packages-file-list.txt
> +	./support/scripts/check-uniq-files.py -t staging $(BUILD_DIR)/packages-file-list-staging.txt
> +	./support/scripts/check-uniq-files.py -t host $(BUILD_DIR)/packages-file-list-host.txt
>  	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
>  	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
>  		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
> diff --git a/support/scripts/check-uniq-files.py b/support/scripts/check-uniq-files.py
> new file mode 100755
> index 0000000000..e77472cf80
> --- /dev/null
> +++ b/support/scripts/check-uniq-files.py
> @@ -0,0 +1,40 @@
> +#!/usr/bin/env python2
> +
> +import sys
> +import csv
> +import argparse
> +from collections import defaultdict
> +
> +warn = 'Warnng: {}s file "{}" is touched by more than one package: {}\n'
> +
> +def main():
> +    parser = argparse.ArgumentParser()
> +    parser.add_argument('packages_file_list', nargs='*',
> +                    help='The packages-file-list to check from')
> +    parser.add_argument('-t', '--type', metavar="TYPE",
> +                        help='Report as a TYPE file (TYPE is either target, staging, or host)')
> +
> +    args = parser.parse_args()
> +
> +    if not len(args.packages_file_list) == 1:
> +        sys.stderr.write('No packages-file-list was provided.\n')
> +        return False
> +
> +    if args.type is None:
> +        sys.stderr.write('No type was provided\n')
> +        return False
> +
> +    file_to_pkg = defaultdict(list)
> +    with open(args.packages_file_list[0], 'rb') as pkg_file_list:
> +        r = csv.reader(pkg_file_list, delimiter=',')
> +        for row in r:
> +            pkg = row[0]
> +            file = row[1]
> +            file_to_pkg[file].append(pkg)
> +
> +    for file in file_to_pkg:
> +        if len(file_to_pkg[file]) > 1:
> +            sys.stderr.write(warn.format(args.type, file, file_to_pkg[file]))
> +
> +if __name__ == "__main__":
> +    sys.exit(main())

[Buildroot] [PATCH 4/4 v2] core: check files are nto touched by more than on package

[Yann E. MORIN]

Peter, All,

On 2017-10-23 19:56 +0200, Peter Seiderer spake thusly:
> Hello Yann,
> 
> typo in the patch title s/nto/not/...

Yup, fixed here for a respin.

> On Sun, 22 Oct 2017 17:15:40 +0200, "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> 
> > Currently, we do nothing about packages that touch the same file: given
> > a specific configuration, the result is reproducible (even though it
> > might not be what the user expected) because the build order is
> > guaranteed.
> > 
> > Hwever, when we later introduce top-level parallel buil, we will no
> 
> s/Hwever/However/, s/buil/build/

Ack.

> > longer be able to guarantee a build order, by the mere way of it being
> > parallel. Reconcialliting all those modified files will be impossible to
> > do automatically. The only way will be to  refuse such situations.
> 
> s/  / /

Duh... ;-)

[--SNIP--]
> > diff --git a/support/scripts/check-uniq-files.py b/support/scripts/check-uniq-files.py
> > new file mode 100755
> > index 0000000000..e77472cf80
> > --- /dev/null
> > +++ b/support/scripts/check-uniq-files.py
> > @@ -0,0 +1,40 @@
> > +#!/usr/bin/env python2

As spotted by Baruch:

s/python2/python/

> > +import sys
> > +import csv
> > +import argparse
> > +from collections import defaultdict
> > +
> > +warn = 'Warnng: {}s file "{}" is touched by more than one package: {}\n'
> > +
> > +def main():
> > +    parser = argparse.ArgumentParser()
> > +    parser.add_argument('packages_file_list', nargs='*',
> > +                    help='The packages-file-list to check from')
> > +    parser.add_argument('-t', '--type', metavar="TYPE",
> > +                        help='Report as a TYPE file (TYPE is either target, staging, or host)')
> > +
> > +    args = parser.parse_args()
> > +
> > +    if not len(args.packages_file_list) == 1:
> > +        sys.stderr.write('No packages-file-list was provided.\n')
> > +        return False
> > +
> > +    if args.type is None:
> > +        sys.stderr.write('No type was provided\n')
> > +        return False
> > +
> > +    file_to_pkg = defaultdict(list)
> > +    with open(args.packages_file_list[0], 'rb') as pkg_file_list:

'rb' is not working for python3... :-/

I'll fix the python2 vs. python3 issue and respin the series...

Regards,
Yann E. MORIN.

> > +        r = csv.reader(pkg_file_list, delimiter=',')
> > +        for row in r:
> > +            pkg = row[0]
> > +            file = row[1]
> > +            file_to_pkg[file].append(pkg)
> > +
> > +    for file in file_to_pkg:
> > +        if len(file_to_pkg[file]) > 1:
> > +            sys.stderr.write(warn.format(args.type, file, file_to_pkg[file]))
> > +
> > +if __name__ == "__main__":
> > +    sys.exit(main())
> 
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'