[Buildroot] [PATCH 01/14] core: sort packages and eliminate duplicates in show-targets

[Buildroot] [PATCH 01/14] core: sort packages and eliminate duplicates in show-targets

[Yann E. MORIN]

Currently, enabling more than one filesystem image will make
'show-targets' list a few host packages more than once.

This is because all filesystem images add the same set of
host-packages to their dependencies, which are then added as-is
to the package list.

Thus, host-fakeroot, host-makedevs and, if needed, host-mkpasswd will
appear as many times as there are filesystem images enabled.

Fix that by sorting the package list, thus eliminating duplicates from
that list. Also sort the rootfs list for good measure. Sort the two
separately, so that rootfses are last.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <jacmet@uclibc.org>
Reviewed-by: Romain Naour <romain.naour@gmail.com>

---
Changes v1 -> v2:
  - don't toch the variable; just sort the list when dumping it
---
 Makefile | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile b/Makefile
index 9240d0aeeb..19bfcfbee3 100644
--- a/Makefile
+++ b/Makefile
@@ -789,7 +789,7 @@ legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p
 
 .PHONY: show-targets
 show-targets:
-	@echo $(PACKAGES) $(TARGETS_ROOTFS)
+	@echo $(sort $(PACKAGES)) $(sort $(TARGETS_ROOTFS))
 
 .PHONY: show-build-order
 show-build-order: $(patsubst %,%-show-build-order,$(PACKAGES))
-- 
2.11.0

[Buildroot] [PATCH 02/14] linux: split overly-long dependency line for readability

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>

---
This is going to be useful to better understand the initramfs patch
coming next.
---
 linux/linux.mk | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/linux/linux.mk b/linux/linux.mk
index bd5589bae0..1c5631f937 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -485,7 +485,10 @@ $(eval $(kconfig-package))
 
 # Support for rebuilding the kernel after the cpio archive has
 # been generated in $(BINARIES_DIR)/rootfs.cpio.
-$(LINUX_DIR)/.stamp_initramfs_rebuilt: $(LINUX_DIR)/.stamp_target_installed $(LINUX_DIR)/.stamp_images_installed $(BINARIES_DIR)/rootfs.cpio
+$(LINUX_DIR)/.stamp_initramfs_rebuilt: $(LINUX_DIR)/.stamp_target_installed
+$(LINUX_DIR)/.stamp_initramfs_rebuilt: $(LINUX_DIR)/.stamp_images_installed
+$(LINUX_DIR)/.stamp_initramfs_rebuilt: $(BINARIES_DIR)/rootfs.cpio
+$(LINUX_DIR)/.stamp_initramfs_rebuilt:
 	@$(call MESSAGE,"Rebuilding kernel with initramfs")
 	# Build the kernel.
 	$(LINUX_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) $(LINUX_TARGET_NAME)
-- 
2.11.0

[Buildroot] [PATCH 03/14] linux: meddle not in the internals of filesystems

[Yann E. MORIN]

Currently, the rule to rebuild the Linux kernel with an initramfs
directly depends on the path of the file of the intermediate cpio image.

This is inherently "bad" from a purity point of view; linux.mk should
not have to delve into the fs internals.

Rather, make it directly depend on the "frontal" rule that generates the
cpio image.

Drop the comment for linux-rebuild-with-initramfs, it was misleading
(talking about generating "the initramfs list of files", which is not
what was done, since we use a cpio as source of initramfs, not a list of
files).

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Romain Naour <romain.naour@openwide.fr>

---
Changes v2 -> v3:
  - use $(LINUX_DIR) instead of $(@D) since there's no longer any
    directory component in the target rule  (Romain)

Changes v1 -> v2:
  - drop intermediate, unneeded stamp-file
  - drop misleading comment

---
This, too, will be useful for the initramfs patch coming next.
---
 linux/linux.mk | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)

diff --git a/linux/linux.mk b/linux/linux.mk
index 1c5631f937..5300b9cfc5 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -484,21 +484,17 @@ endif # BR_BUILDING
 $(eval $(kconfig-package))
 
 # Support for rebuilding the kernel after the cpio archive has
-# been generated in $(BINARIES_DIR)/rootfs.cpio.
-$(LINUX_DIR)/.stamp_initramfs_rebuilt: $(LINUX_DIR)/.stamp_target_installed
-$(LINUX_DIR)/.stamp_initramfs_rebuilt: $(LINUX_DIR)/.stamp_images_installed
-$(LINUX_DIR)/.stamp_initramfs_rebuilt: $(BINARIES_DIR)/rootfs.cpio
-$(LINUX_DIR)/.stamp_initramfs_rebuilt:
+# been generated.
+.PHONY: linux-rebuild-with-initramfs
+linux-rebuild-with-initramfs: $(LINUX_DIR)/.stamp_target_installed
+linux-rebuild-with-initramfs: $(LINUX_DIR)/.stamp_images_installed
+linux-rebuild-with-initramfs: rootfs-cpio
+linux-rebuild-with-initramfs:
 	@$(call MESSAGE,"Rebuilding kernel with initramfs")
 	# Build the kernel.
-	$(LINUX_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(@D) $(LINUX_TARGET_NAME)
+	$(LINUX_MAKE_ENV) $(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) $(LINUX_TARGET_NAME)
 	$(LINUX_APPEND_DTB)
 	# Copy the kernel image(s) to its(their) final destination
 	$(call LINUX_INSTALL_IMAGE,$(BINARIES_DIR))
 	# If there is a .ub file copy it to the final destination
 	test ! -f $(LINUX_IMAGE_PATH).ub || cp $(LINUX_IMAGE_PATH).ub $(BINARIES_DIR)
-	$(Q)touch $@
-
-# The initramfs building code must make sure this target gets called
-# after it generated the initramfs list of files.
-linux-rebuild-with-initramfs: $(LINUX_DIR)/.stamp_initramfs_rebuilt
-- 
2.11.0

[Buildroot] [PATCH 04/14] fs/initramfs: cleanups, enhance comments

[Yann E. MORIN]

rootfs-initramfs is not using the generic fs infrastructure, because
there is virtually nothing to do to build the initramfs image: there is
no actual image to be built to begin with.

The only purpose of rootfs-initramfs is to ensure the rootfs.cpio image
is built and then that the Linux kernel is rebuilt with that rootfs.cpio
as initramfs source.

Using variables of the fs infra like if it were used is misleading. It
looked nice as long as there was the possibility that rootfs-initramfs
would one day use the fs infra. But there's no way that will happen any
time soon.

Furthermore, the linux' rule linux-rebuild-with-initramfs now already
depends on rootfs-cpio by itself, so we need not duplicate this
dependency in rootfs-initramfs.

Still, we want to advertise that the dependency is on rootfs-cpio, so
we get nice dependency graphs (and not expose the internal
linux-rebuild-with-initramfs rule to the users).

So, remove the variables and directly define the rules.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Romain Naour <romain.naour@openwide.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>

---
Changes v2 -> v3:
  - fix comment about now non-existent stamp file  (Romain)
---
 fs/initramfs/initramfs.mk | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/fs/initramfs/initramfs.mk b/fs/initramfs/initramfs.mk
index db5081224c..b8dee18ec5 100644
--- a/fs/initramfs/initramfs.mk
+++ b/fs/initramfs/initramfs.mk
@@ -5,17 +5,25 @@
 #
 ################################################################################
 
-ROOTFS_INITRAMFS_DEPENDENCIES += rootfs-cpio
-
-ROOTFS_INITRAMFS_POST_TARGETS += linux-rebuild-with-initramfs
-
-
 # The generic fs infrastructure isn't very useful here.
+#
+# The initramfs image does not actually build an image; its only purpose is:
+# 1- to ensure rootfs.cpio is generated,
+# 2- to then rebuild the kernel with rootfs.cpio as initramfs
+#
+# Note: ordering of the dependencies is not guaranteed here, but in
+# linux/linux.mk, via the linux-rebuild-with-initramfs rule, which depends
+# on the rootfs-cpio filesystem rule.
+#
+# Note: the trick here is that we directly depend on rebuilding the Linux
+# kernel image (which itself depends on the rootfs-cpio rule), while we
+# advertise that our dependency is on the rootfs-cpio rule, which is
+# cleaner in the dependency graph.
 
-rootfs-initramfs: $(ROOTFS_INITRAMFS_DEPENDENCIES) $(ROOTFS_INITRAMFS_POST_TARGETS)
+rootfs-initramfs: linux-rebuild-with-initramfs
 
 rootfs-initramfs-show-depends:
-	@echo $(ROOTFS_INITRAMFS_DEPENDENCIES)
+	@echo rootfs-cpio
 
 .PHONY: rootfs-initramfs rootfs-initramfs-show-depends
 
-- 
2.11.0

[Buildroot] [PATCH 05/14] fs/ext2: use a post-gen hook rather than a post-target rule

[Yann E. MORIN]

post-target rules are probably not resilient to parallel builds, given
that they do not depend on the image being generated first.

Beside, we already have a mechanism for running stuff after the
filesystem is generated, and that's called post-gen hooks.

Use those hooks.

Note: this basically reverts 75b6303 (rootfs-ext2: make the symlink as a
_POST_TARGET) since we've now re-introduced post-gen hooks.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
---
 fs/ext2/ext2.mk | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/fs/ext2/ext2.mk b/fs/ext2/ext2.mk
index 5439e2b993..12b87a722c 100644
--- a/fs/ext2/ext2.mk
+++ b/fs/ext2/ext2.mk
@@ -36,13 +36,11 @@ define ROOTFS_EXT2_CMD
 	}
 endef
 
-rootfs-ext2-symlink:
-	ln -sf rootfs.ext2$(ROOTFS_EXT2_COMPRESS_EXT) $(BINARIES_DIR)/rootfs.ext$(BR2_TARGET_ROOTFS_EXT2_GEN)$(ROOTFS_EXT2_COMPRESS_EXT)
-
-.PHONY: rootfs-ext2-symlink
-
 ifneq ($(BR2_TARGET_ROOTFS_EXT2_GEN),2)
-ROOTFS_EXT2_POST_TARGETS += rootfs-ext2-symlink
+define ROOTFS_EXT2_SYMLINK
+	ln -sf rootfs.ext2$(ROOTFS_EXT2_COMPRESS_EXT) $(BINARIES_DIR)/rootfs.ext$(BR2_TARGET_ROOTFS_EXT2_GEN)$(ROOTFS_EXT2_COMPRESS_EXT)
+endef
+ROOTFS_EXT2_POST_GEN_HOOKS += ROOTFS_EXT2_SYMLINK
 endif
 
 $(eval $(call ROOTFS_TARGET,ext2))
-- 
2.11.0

[Buildroot] [PATCH 06/14] fs/cpio: use a post-gen hook rather than a post-target rule

[Yann E. MORIN]

We already have a mechanism for running stuff after the filesystem is
generated, and that's called post-gen hooks.

Use those hooks.

Note: for cpio (and unlike ext2 previously), the dependency chain was
correct, in that the post-target rule correctly depended on the image
rule. Nonetheless, we still want to fix it for consistency.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
---
 fs/cpio/cpio.mk | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
index e82167e512..c68e0bfb97 100644
--- a/fs/cpio/cpio.mk
+++ b/fs/cpio/cpio.mk
@@ -31,12 +31,13 @@ define ROOTFS_CPIO_CMD
 	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
 endef
 
-$(BINARIES_DIR)/rootfs.cpio.uboot: $(BINARIES_DIR)/rootfs.cpio host-uboot-tools
-	$(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \
-		-C none -d $<$(ROOTFS_CPIO_COMPRESS_EXT) $@
-
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
-ROOTFS_CPIO_POST_TARGETS += $(BINARIES_DIR)/rootfs.cpio.uboot
+ROOTFS_CPIO_DEPENDENCIES += host-uboot-tools
+define ROOTFS_CPIO_UBOOT_MKIMAGE
+	$(MKIMAGE) -A $(MKIMAGE_ARCH) -T ramdisk \
+		-C none -d $@$(ROOTFS_CPIO_COMPRESS_EXT) $@.uboot
+endef
+ROOTFS_CPIO_POST_GEN_HOOKS += ROOTFS_CPIO_UBOOT_MKIMAGE
 endif
 
 $(eval $(call ROOTFS_TARGET,cpio))
-- 
2.11.0

[Buildroot] [PATCH 07/14] fs/common: get rid of post-target rules

[Yann E. MORIN]

The only users of post-target rules were ext2, cpio and initramfs.

Of those, ext2 and cpio were changed to use post-gen hooks, while
initramfs was not even using the generic rootfs infra and was fixed
to no longer reference post-target rules.

Besides, the comment in the infra was really misleading: it referenced
initramfs implying it was the sole user of that feature, even though
initramfs was not using the fs infra.

Furthermore, using post-target rules was inherently broken for top-level
parallel builds, because filesystems had to ensure the ordering by
themselves. Of the two real users of post-target rules (cpio and ext2),
one did enforce rules ordering (apparently correctly), while the other
forgot to do so.

We can get rid of post-target rules altogether, now.

Add a legacy check, to catch out-of-tree (e.g. br2-external) users of
post-target rules, and instruct them to switch to post-gen hooks instead.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
---
 fs/common.mk | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/fs/common.mk b/fs/common.mk
index 5b612a3f41..378907e9ed 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -19,10 +19,6 @@
 #  ROOTFS_$(FSTYPE)_POST_GEN_HOOKS, a list of hooks to call after
 #  generating the filesystem image
 #
-#  ROOTFS_$(FSTYPE)_POST_TARGETS, the list of targets that should be
-#  run after running the main filesystem target. This is useful for
-#  initramfs, to rebuild the kernel once the initramfs is generated.
-#
 # In terms of configuration option, this macro assumes that the
 # BR2_TARGET_ROOTFS_$(FSTYPE) config option allows to enable/disable
 # the generation of a filesystem image of a particular type. If
@@ -115,7 +111,7 @@ endif
 rootfs-$(1)-show-depends:
 	@echo $$(ROOTFS_$(2)_DEPENDENCIES)
 
-rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1) $$(ROOTFS_$(2)_POST_TARGETS)
+rootfs-$(1): $$(BINARIES_DIR)/rootfs.$(1)
 
 .PHONY: rootfs-$(1) rootfs-$(1)-show-depends
 
@@ -123,6 +119,13 @@ ifeq ($$(BR2_TARGET_ROOTFS_$(2)),y)
 TARGETS_ROOTFS += rootfs-$(1)
 PACKAGES += $$(filter-out rootfs-%,$$(ROOTFS_$(2)_DEPENDENCIES))
 endif
+
+# Check for legacy POST_TARGETS rules
+ifneq ($$(ROOTFS_$(2)_POST_TARGETS),)
+$$(error Filesystem $(1) uses post-target rules, which are no longer supported.\
+	Update $(1) to use post-gen hooks instead)
+endif
+
 endef
 
 define ROOTFS_TARGET
-- 
2.11.0

[Buildroot] [PATCH 08/14] fs: remove TARGET_DIR_WARNING_FILE late

[Yann E. MORIN]

Remove it just before generating the filesystem image.

This way, removing-and-recreating the file encloses the actual
image generation as tightly as possible.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 fs/common.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/common.mk b/fs/common.mk
index 378907e9ed..d4d5c183a5 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -70,7 +70,6 @@ $$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
 	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
 	rm -f $$(FAKEROOT_SCRIPT)
-	rm -f $$(TARGET_DIR_WARNING_FILE)
 	rm -f $$(USERS_TABLE)
 	echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT)
 	echo "set -e" >> $$(FAKEROOT_SCRIPT)
@@ -100,6 +99,7 @@ endif
 	$$(foreach hook,$$(ROOTFS_POST_CMD_HOOKS),\
 		$$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep))
 	chmod a+x $$(FAKEROOT_SCRIPT)
+	rm -f $$(TARGET_DIR_WARNING_FILE)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(FAKEROOT_SCRIPT)
 	$$(INSTALL) -m 0644 support/misc/target-dir-warning.txt $$(TARGET_DIR_WARNING_FILE)
 	- at rm -f $$(FAKEROOT_SCRIPT) $$(FULL_DEVICE_TABLE)
-- 
2.11.0

[Buildroot] [PATCH 00/14] fs: cleanups and enhancements

[Yann E. MORIN]

Hello All!

This series brings a few cleanups, fixes and enhancements to the
filesystem infra. More is to come, but this is already large enough
and self-contained, so that it warrants being pushed.

The underlying reason for this series if many-fold.

First, some comments and code in the fs infra have started to bit-rot
and are now wrong, misleading, sub-optimal. So, we update those to the
current state with patches 1..7, essentially to get rid of the
duplication between post-gen hooks and post-target rules, which
basically serve the exact same purpose.

The second part of the series deals with re-organising the infra, so
that all actions common to all image generations are done only once, not
once per image. As a side effect, the scripts used to do the generation
are moved to their own directory, and kept afterward to help debugging
if needed.

As a side effect, this common part is now done only once, so it will
automatically act as a synchronisation point when we later add support
for top-level aparallel build: no two filesystem images will have to
modify the content of tartget/ at the same time anymore.

Now, since we have pre- and post-fs hooks registered by packages, the
target directory is no longer immutable during the creation of
filesystem images. So, when two filesystem images are enabled, they
will both try to run the pre-rootfs hooks, generate the image, then run
the post-rootfs hooks; none of this is atomic, obviously, so they will
conflict. At best, there will be a failure, at worst there will be
silent corruption.

So, the last patch makes the target-finalize step act on a copy of
target/, so as to leave target/ in the state it was set by installing
packages, and each filesystem image generator only sees that and can
tweak it as needed without impacting the others. That copy is deleted
after the filesystem image is generated.


Regards,
Yann E. MORIN.


The following changes since commit 7feaac43718bdfff056e35ffdeec424bf03f1e9c

  wpa_supplicant: mesh support needs openssl (2017-11-12 10:12:08 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 8a74ac07aaa7e2f745ab75dde70f42ee0bff5cb1

  TEST-ME (2017-11-12 12:19:34 +0100)


----------------------------------------------------------------
Yann E. MORIN (14):
      core: sort packages and eliminate duplicates in show-targets
      linux: split overly-long dependency line for readability
      linux: meddle not in the internals of filesystems
      fs/initramfs: cleanups, enhance comments
      fs/ext2: use a post-gen hook rather than a post-target rule
      fs/cpio: use a post-gen hook rather than a post-target rule
      fs/common: get rid of post-target rules
      fs: remove TARGET_DIR_WARNING_FILE late
      fs: don't pollute $(BUILD_DIR) with temp files
      fs/iso9660: don't pollute $(BUILD_DIR) with temp dir
      fs: don't remove intermediate files
      fs: move actions common to all filesystems to their own rule
      core: finalise target in its own location
      TEST-ME

 .gitlab-ci.yml                                     | 204 ---------------------
 Makefile                                           |  30 ++-
 fs/common.mk                                       |  89 +++++----
 fs/cpio/cpio.mk                                    |  11 +-
 fs/ext2/ext2.mk                                    |  10 +-
 fs/initramfs/initramfs.mk                          |  22 ++-
 fs/iso9660/iso9660.mk                              |   2 +-
 linux/linux.mk                                     |  15 +-
 package/pkg-generic.mk                             |   2 -
 .../skeleton-init-systemd/skeleton-init-systemd.mk |   6 -
 support/testing/tests/core/test_post_scripts.py    |   8 +-
 support/testing/tests/core/test_rootfs_overlay.py  |   4 +-
 12 files changed, 117 insertions(+), 286 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 09/14] fs: don't pollute $(BUILD_DIR) with temp files

[Yann E. MORIN]

Currently, we create a bunch of temporary files in $(BUILD_DIR), while
assembling the filesystem images.

Move those files to their own sub-directory.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 fs/common.mk | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/fs/common.mk b/fs/common.mk
index d4d5c183a5..7bbc143892 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -27,11 +27,12 @@
 # BR2_TARGET_ROOTFS_$(FSTYPE)_LZMA exist and are enabled, then the
 # macro will automatically generate a compressed filesystem image.
 
-FAKEROOT_SCRIPT = $(BUILD_DIR)/_fakeroot.fs
-FULL_DEVICE_TABLE = $(BUILD_DIR)/_device_table.txt
+FS_DIR = $(BUILD_DIR)/buildroot-fs
+FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs
+FULL_DEVICE_TABLE = $(FS_DIR)/device_table.txt
 ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \
 	$(BR2_ROOTFS_STATIC_DEVICE_TABLE))
-USERS_TABLE = $(BUILD_DIR)/_users_table.txt
+USERS_TABLE = $(FS_DIR)/users_table.txt
 ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES))
 
 # Since this function will be called from within an $(eval ...)
@@ -69,8 +70,8 @@ endif
 $$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
 	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
-	rm -f $$(FAKEROOT_SCRIPT)
-	rm -f $$(USERS_TABLE)
+	rm -rf $(FS_DIR)
+	mkdir -p $(FS_DIR)
 	echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT)
 	echo "set -e" >> $$(FAKEROOT_SCRIPT)
 	echo "chown -h -R 0:0 $$(TARGET_DIR)" >> $$(FAKEROOT_SCRIPT)
-- 
2.11.0

[Buildroot] [PATCH 10/14] fs/iso9660: don't pollute $(BUILD_DIR) with temp dir

[Yann E. MORIN]

Use the newly-introdued $(FS_DIR) location to store temporary files.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 fs/iso9660/iso9660.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index d49d593bd6..c2de27101a 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -35,7 +35,7 @@ ROOTFS_ISO9660_USE_INITRD = YES
 endif
 
 ifeq ($(ROOTFS_ISO9660_USE_INITRD),YES)
-ROOTFS_ISO9660_TARGET_DIR = $(BUILD_DIR)/rootfs.iso9660.tmp
+ROOTFS_ISO9660_TARGET_DIR = $(FS_DIR)/rootfs.iso9660.tmp
 define ROOTFS_ISO9660_CREATE_TEMPDIR
 	$(RM) -rf $(ROOTFS_ISO9660_TARGET_DIR)
 	mkdir -p $(ROOTFS_ISO9660_TARGET_DIR)
-- 
2.11.0

[Buildroot] [PATCH 11/14] fs: don't remove intermediate files

[Yann E. MORIN]

Now that they are in their own directory and no longer pollute the build
dir, there is no point in removing them.

Furthermore, a follow-up patch will require that those files survive
when more than one filesystem image is generated.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 fs/common.mk | 1 -
 1 file changed, 1 deletion(-)

diff --git a/fs/common.mk b/fs/common.mk
index 7bbc143892..c17c4585ff 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -103,7 +103,6 @@ endif
 	rm -f $$(TARGET_DIR_WARNING_FILE)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(FAKEROOT_SCRIPT)
 	$$(INSTALL) -m 0644 support/misc/target-dir-warning.txt $$(TARGET_DIR_WARNING_FILE)
-	- at rm -f $$(FAKEROOT_SCRIPT) $$(FULL_DEVICE_TABLE)
 ifneq ($$(ROOTFS_$(2)_COMPRESS_CMD),)
 	PATH=$$(BR_PATH) $$(ROOTFS_$(2)_COMPRESS_CMD) $$@ > $$@$$(ROOTFS_$(2)_COMPRESS_EXT)
 endif
-- 
2.11.0

[Buildroot] [PATCH 12/14] fs: move actions common to all filesystems to their own rule

[Yann E. MORIN]

A lot of actions are common to generating the various images.
Currently, they are all done for each image being generated.

However, we can do them once and for all.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 fs/common.mk | 67 +++++++++++++++++++++++++++++++++---------------------------
 1 file changed, 37 insertions(+), 30 deletions(-)

diff --git a/fs/common.mk b/fs/common.mk
index c17c4585ff..67925c20c7 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -28,13 +28,42 @@
 # macro will automatically generate a compressed filesystem image.
 
 FS_DIR = $(BUILD_DIR)/buildroot-fs
-FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.fs
+FAKEROOT_SCRIPT = $(FS_DIR)/fakeroot.common
 FULL_DEVICE_TABLE = $(FS_DIR)/device_table.txt
 ROOTFS_DEVICE_TABLES = $(call qstrip,$(BR2_ROOTFS_DEVICE_TABLE) \
 	$(BR2_ROOTFS_STATIC_DEVICE_TABLE))
 USERS_TABLE = $(FS_DIR)/users_table.txt
 ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES))
 
+.PHONY: rootfs-common
+rootfs-common: target-finalize
+	@$(call MESSAGE,"Preparing rootfs generation script")
+	rm -f $(FAKEROOT_SCRIPT)
+	rm -f $(USERS_TABLE)
+	mkdir -p $(FS_DIR)
+	echo '#!/bin/sh' > $(FAKEROOT_SCRIPT)
+	echo "set -e" >> $(FAKEROOT_SCRIPT)
+	echo "chown -h -R 0:0 $(TARGET_DIR)" >> $(FAKEROOT_SCRIPT)
+ifneq ($(ROOTFS_USERS_TABLES),)
+	cat $(ROOTFS_USERS_TABLES) >> $(USERS_TABLE)
+endif
+	$(call PRINTF,$(PACKAGES_USERS)) >> $(USERS_TABLE)
+	PATH=$(BR_PATH) $(TOPDIR)/support/scripts/mkusers $(USERS_TABLE) $(TARGET_DIR) >> $(FAKEROOT_SCRIPT)
+ifneq ($(ROOTFS_DEVICE_TABLES),)
+	cat $(ROOTFS_DEVICE_TABLES) > $(FULL_DEVICE_TABLE)
+ifeq ($(BR2_ROOTFS_DEVICE_CREATION_STATIC),y)
+	$(call PRINTF,$(PACKAGES_DEVICES_TABLE)) >> $(FULL_DEVICE_TABLE)
+endif
+endif
+	$(call PRINTF,$(PACKAGES_PERMISSIONS_TABLE)) >> $(FULL_DEVICE_TABLE)
+	echo "$(HOST_DIR)/bin/makedevs -d $(FULL_DEVICE_TABLE) $(TARGET_DIR)" >> $(FAKEROOT_SCRIPT)
+	$(foreach s,$(call qstrip,$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
+		echo "echo '$(TERM_BOLD)>>>   Executing fakeroot script $(s)$(TERM_RESET)'" >> $(FAKEROOT_SCRIPT); \
+		echo $(s) $(TARGET_DIR) $(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $(FAKEROOT_SCRIPT)$(sep))
+ifeq ($(BR2_REPRODUCIBLE),y)
+	echo "find $(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$(SOURCE_DATE_EPOCH)" >> $(FAKEROOT_SCRIPT)
+endif
+
 # Since this function will be called from within an $(eval ...)
 # all variable references except the arguments must be $$-quoted.
 define ROOTFS_TARGET_INTERNAL
@@ -67,42 +96,20 @@ ROOTFS_$(2)_COMPRESS_EXT = .xz
 ROOTFS_$(2)_COMPRESS_CMD = xz -9 -C crc32 -c
 endif
 
-$$(BINARIES_DIR)/rootfs.$(1): target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
+ROOTFS_$(2)_FAKEROOT_SCRIPT = $$(FS_DIR)/fakeroot.$(1)
+
+$$(BINARIES_DIR)/rootfs.$(1): TARGET_DIR=$(FINAL_TARGET_DIR)
+$$(BINARIES_DIR)/rootfs.$(1): rootfs-common target-finalize $$(ROOTFS_$(2)_DEPENDENCIES)
 	@$$(call MESSAGE,"Generating root filesystem image rootfs.$(1)")
 	$$(foreach hook,$$(ROOTFS_$(2)_PRE_GEN_HOOKS),$$(call $$(hook))$$(sep))
-	rm -rf $(FS_DIR)
-	mkdir -p $(FS_DIR)
-	echo '#!/bin/sh' > $$(FAKEROOT_SCRIPT)
-	echo "set -e" >> $$(FAKEROOT_SCRIPT)
-	echo "chown -h -R 0:0 $$(TARGET_DIR)" >> $$(FAKEROOT_SCRIPT)
-ifneq ($$(ROOTFS_USERS_TABLES),)
-	cat $$(ROOTFS_USERS_TABLES) >> $$(USERS_TABLE)
-endif
-	$$(call PRINTF,$$(PACKAGES_USERS)) >> $$(USERS_TABLE)
-	PATH=$$(BR_PATH) $$(TOPDIR)/support/scripts/mkusers $$(USERS_TABLE) $$(TARGET_DIR) >> $$(FAKEROOT_SCRIPT)
-ifneq ($$(ROOTFS_DEVICE_TABLES),)
-	cat $$(ROOTFS_DEVICE_TABLES) > $$(FULL_DEVICE_TABLE)
-ifeq ($$(BR2_ROOTFS_DEVICE_CREATION_STATIC),y)
-	$$(call PRINTF,$$(PACKAGES_DEVICES_TABLE)) >> $$(FULL_DEVICE_TABLE)
-endif
-endif
-	$$(call PRINTF,$$(PACKAGES_PERMISSIONS_TABLE)) >> $$(FULL_DEVICE_TABLE)
-	echo "$$(HOST_DIR)/bin/makedevs -d $$(FULL_DEVICE_TABLE) $$(TARGET_DIR)" >> $$(FAKEROOT_SCRIPT)
-	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
-		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
-		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
+	$$(INSTALL) -m 0755 $$(FAKEROOT_SCRIPT) $$(ROOTFS_$(2)_FAKEROOT_SCRIPT)
 	$$(foreach hook,$$(ROOTFS_PRE_CMD_HOOKS),\
 		$$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep))
-ifeq ($$(BR2_REPRODUCIBLE),y)
-	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)
-endif
-	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
+	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(ROOTFS_$(2)_FAKEROOT_SCRIPT)
 	$$(foreach hook,$$(ROOTFS_POST_CMD_HOOKS),\
 		$$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT)$$(sep))
-	chmod a+x $$(FAKEROOT_SCRIPT)
 	rm -f $$(TARGET_DIR_WARNING_FILE)
-	PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(FAKEROOT_SCRIPT)
-	$$(INSTALL) -m 0644 support/misc/target-dir-warning.txt $$(TARGET_DIR_WARNING_FILE)
+	PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(ROOTFS_$(2)_FAKEROOT_SCRIPT)
 ifneq ($$(ROOTFS_$(2)_COMPRESS_CMD),)
 	PATH=$$(BR_PATH) $$(ROOTFS_$(2)_COMPRESS_CMD) $$@ > $$@$$(ROOTFS_$(2)_COMPRESS_EXT)
 endif
-- 
2.11.0

[Buildroot] [PATCH 13/14] core: finalise target in its own location

[Yann E. MORIN]

Currently, after all packages have been installed into target/ , we run
a sanitising pass called 'target-finalize' on that directory, to:
  - apply overlays
  - remove unnecessary files (man, .h, .a ...)
  - strip files
as well as a few other miscellanous cleanups.

This means that target/ no longer contains only package-installed files,
and that target-finalize might not be idempotent (i.e. sucessive runs of
target-finalize may yield different results in target/ ). We're trying
pretty hard that all the internal target-finalize hooks are idempotent,
whether they are from the core (e.g. installing glibc locales) or
provided by packages (e.g. cleaning up perl files).

However, that might not be the case for packages from br2-external for
example, or under complex situations where a combination of packages
does not yield an idempotent sequence (quoting Wikipedia: "a combination
of idempotent methods or subroutines is not necessrily idempotent"; see:
https://en.wikipedia.org/wiki/Idempotence#Examples ).

A second issue is that we further need to further change the layout of
target/ just prior to assembling the filesystem image(s), and then
restore it just after. This is the case to support systemd on a
read-only filesystem, for example (the only example in fact).
This means that doing so is not parallel safe.

Address this issue by copying target/ to a "landing" area where both
finalising actions (target-finalize and pre-image hooks) take place.

This keeps the user-visible target/ directory to contain only what
packages have installed, helps keeping target-finalize be idempotent by
allowing packages to provide simpler target-finalize hooks, and ensures
that target/ is never left in an incorrect state in case a filesystem
image generator fails.

For simplicity for packages, we have to allow them to use $(TARGET_DIR)
everywhere, be it in target install commands or target finalize
commands, without requiring them to know whether to use $(TARGET_DIR)
or $(FINAL_TARGET_DIR). $(TARGET_DIR) always points to the directory in
which to act.

So, for target-finalize, we override TARGET_DIR to point to the landing
area. But since packages are dependencies of target-finalize, they
would also inherit from this override. So, we over-override TARGET_DIR
for packages, to point to the usual and currently used $(O)/target/
directory.

Finally, filesystem images are generated from that landing area, of
course. Similarly, we need to override TARGET_DIR for them.

This also means that we no longer need the post-filesystem command hooks
from skeleton-systemd, since we no longer need to restore target/ into
the state it was before assembling the image(s).

As a consequence, update the testsing infra to reflect the fact that the
target directoryy has moved, and is no longer pointing to the same
location at various points in the build steps.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 Makefile                                           | 28 +++++++++++++++++++++-
 fs/common.mk                                       |  1 +
 package/pkg-generic.mk                             |  2 --
 .../skeleton-init-systemd/skeleton-init-systemd.mk |  6 -----
 support/testing/tests/core/test_post_scripts.py    |  8 +++----
 support/testing/tests/core/test_rootfs_overlay.py  |  4 ++--
 6 files changed, 34 insertions(+), 15 deletions(-)

diff --git a/Makefile b/Makefile
index 19bfcfbee3..f38c97074b 100644
--- a/Makefile
+++ b/Makefile
@@ -215,7 +215,9 @@ BR_GRAPH_OUT := $(or $(BR2_GRAPH_OUT),pdf)
 
 BUILD_DIR := $(BASE_DIR)/build
 BINARIES_DIR := $(BASE_DIR)/images
-TARGET_DIR := $(BASE_DIR)/target
+BUILD_TARGET_DIR := $(BASE_DIR)/target
+TARGET_DIR := $(BUILD_TARGET_DIR)
+FINAL_TARGET_DIR := $(BUILD_DIR)/target-finalize
 # initial definition so that 'make clean' works for most users, even without
 # .config. HOST_DIR will be overwritten later when .config is included.
 HOST_DIR := $(BASE_DIR)/host
@@ -676,9 +678,33 @@ endif
 
 $(TARGETS_ROOTFS): target-finalize
 
+# Finalizing the target directory involves:
+#  - applying overlays,
+#  - removing unecessary files (man, .h, .a ...)
+#  - tweaking files installed by packages (like stripping)
+# and miscellanous cleanups.
+#
+# We want to keep the $(O)/target/ directory as a perfect image of
+# what packages have actually installed, so we copy it to a landing
+# location where we'll do those cleanups.
+#
+# Still, we only document $(TARGET_DIR) so packages that want to provide
+# target-finalize hooks will be using that. Thus, we just override that
+# variable with the landing location just for target-finalize.
+#
+# However, since rule-overriden variables are inherited by the dependencies
+# of that rule, we must re-override TARGET_DIR with its original location
+# for packages.
+#
+$(PACKAGES): TARGET_DIR=$(BUILD_TARGET_DIR)
+
 .PHONY: target-finalize
+target-finalize: TARGET_DIR=$(FINAL_TARGET_DIR)
 target-finalize: $(PACKAGES)
 	@$(call MESSAGE,"Finalizing target directory")
+	$(Q)rm -rf $(TARGET_DIR)
+	$(Q)cp -a $(BUILD_TARGET_DIR) $(TARGET_DIR)
+	$(Q)rm -f $(TARGET_DIR_WARNING_FILE)
 	$(foreach hook,$(TARGET_FINALIZE_HOOKS),$($(hook))$(sep))
 	rm -rf $(TARGET_DIR)/usr/include $(TARGET_DIR)/usr/share/aclocal \
 		$(TARGET_DIR)/usr/lib/pkgconfig $(TARGET_DIR)/usr/share/pkgconfig \
diff --git a/fs/common.mk b/fs/common.mk
index 67925c20c7..878b81e43f 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -36,6 +36,7 @@ USERS_TABLE = $(FS_DIR)/users_table.txt
 ROOTFS_USERS_TABLES = $(call qstrip,$(BR2_ROOTFS_USERS_TABLES))
 
 .PHONY: rootfs-common
+rootfs-common: TARGET_DIR=$(FINAL_TARGET_DIR)
 rootfs-common: target-finalize
 	@$(call MESSAGE,"Preparing rootfs generation script")
 	rm -f $(FAKEROOT_SCRIPT)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 0e28675fbe..19527dcaf7 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -620,7 +620,6 @@ $(2)_PRE_LEGAL_INFO_HOOKS       ?=
 $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
 $(2)_ROOTFS_PRE_CMD_HOOKS       ?=
-$(2)_ROOTFS_POST_CMD_HOOKS      ?=
 
 # human-friendly targets and target sequencing
 $(1):			$(1)-install
@@ -941,7 +940,6 @@ PACKAGES_USERS += $$($(2)_USERS)$$(sep)
 endif
 TARGET_FINALIZE_HOOKS += $$($(2)_TARGET_FINALIZE_HOOKS)
 ROOTFS_PRE_CMD_HOOKS += $$($(2)_ROOTFS_PRE_CMD_HOOKS)
-ROOTFS_POST_CMD_HOOKS += $$($(2)_ROOTFS_POST_CMD_HOOKS)
 
 ifeq ($$($(2)_SITE_METHOD),svn)
 DL_TOOLS_DEPENDENCIES += svn
diff --git a/package/skeleton-init-systemd/skeleton-init-systemd.mk b/package/skeleton-init-systemd/skeleton-init-systemd.mk
index a2d4e8c4b3..7da801ac4e 100644
--- a/package/skeleton-init-systemd/skeleton-init-systemd.mk
+++ b/package/skeleton-init-systemd/skeleton-init-systemd.mk
@@ -54,12 +54,6 @@ define SKELETON_INIT_SYSTEMD_PRE_ROOTFS_VAR
 endef
 SKELETON_INIT_SYSTEMD_ROOTFS_PRE_CMD_HOOKS += SKELETON_INIT_SYSTEMD_PRE_ROOTFS_VAR
 
-define SKELETON_INIT_SYSTEMD_POST_ROOTFS_VAR
-	rm -rf $(TARGET_DIR)/var
-	ln -s usr/share/factory/var $(TARGET_DIR)/var
-endef
-SKELETON_INIT_SYSTEMD_ROOTFS_POST_CMD_HOOKS += SKELETON_INIT_SYSTEMD_POST_ROOTFS_VAR
-
 endif
 
 define SKELETON_INIT_SYSTEMD_INSTALL_TARGET_CMDS
diff --git a/support/testing/tests/core/test_post_scripts.py b/support/testing/tests/core/test_post_scripts.py
index 1db568b0d6..3377088cca 100644
--- a/support/testing/tests/core/test_post_scripts.py
+++ b/support/testing/tests/core/test_post_scripts.py
@@ -16,7 +16,7 @@ class TestPostScripts(infra.basetest.BRTest):
         """.format(infra.filepath("tests/core/post-build.sh"),
                    infra.filepath("tests/core/post-image.sh"))
 
-    def check_post_log_file(self, path, what):
+    def check_post_log_file(self, path, what, target_dir):
         lines = {}
         with open(path, 'rb') as csvfile:
             r = csv.reader(csvfile, delimiter=',')
@@ -26,7 +26,7 @@ class TestPostScripts(infra.basetest.BRTest):
         self.assertEqual(lines["arg1"], os.path.join(self.builddir, what))
         self.assertEqual(lines["arg2"], "foobar")
         self.assertEqual(lines["arg3"], "baz")
-        self.assertEqual(lines["TARGET_DIR"], os.path.join(self.builddir, "target"))
+        self.assertEqual(lines["TARGET_DIR"], target_dir)
         self.assertEqual(lines["BUILD_DIR"], os.path.join(self.builddir, "build"))
         self.assertEqual(lines["HOST_DIR"], os.path.join(self.builddir, "host"))
         staging = os.readlink(os.path.join(self.builddir, "staging"))
@@ -36,6 +36,6 @@ class TestPostScripts(infra.basetest.BRTest):
 
     def test_run(self):
         f = os.path.join(self.builddir, "build", "post-build.log")
-        self.check_post_log_file(f, "target")
+        self.check_post_log_file(f, "build/target-finalize", os.path.join(self.builddir, "build/target-finalize"))
         f = os.path.join(self.builddir, "build", "post-image.log")
-        self.check_post_log_file(f, "images")
+        self.check_post_log_file(f, "images", os.path.join(self.builddir, "target"))
diff --git a/support/testing/tests/core/test_rootfs_overlay.py b/support/testing/tests/core/test_rootfs_overlay.py
index fedd40d8ac..119169094a 100644
--- a/support/testing/tests/core/test_rootfs_overlay.py
+++ b/support/testing/tests/core/test_rootfs_overlay.py
@@ -19,12 +19,12 @@ class TestRootfsOverlay(infra.basetest.BRTest):
         """.format(rootfs_overlay_path)
 
     def test_run(self):
-        target_file = os.path.join(self.builddir, "target", "test-file1")
+        target_file = os.path.join(self.builddir, "build/target-finalize/test-file1")
         overlay_file = "{}1/test-file1".format(self.rootfs_overlay_path)
         ret = compare_file(overlay_file, target_file)
         self.assertEqual(ret, 0)
 
-        target_file = os.path.join(self.builddir, "target", "etc", "test-file2")
+        target_file = os.path.join(self.builddir, "build/target-finalize/etc/test-file2")
         overlay_file = "{}2/etc/test-file2".format(self.rootfs_overlay_path)
         ret = compare_file(overlay_file, target_file)
         self.assertEqual(ret, 0)
-- 
2.11.0

[Buildroot] [PATCH 14/14] TEST-ME

[Yann E. MORIN]

---
 .gitlab-ci.yml | 204 ---------------------------------------------------------
 1 file changed, 204 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 0dddb22f12..41ab248229 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -17,19 +17,6 @@ image: buildroot/base
             exit 1
         }
 
-check-gitlab-ci.yml:
-    script:
-        - mv .gitlab-ci.yml .gitlab-ci.yml.orig
-        - make .gitlab-ci.yml
-        - diff -u .gitlab-ci.yml.orig .gitlab-ci.yml
-
-check-DEVELOPERS:
-    # get-developers should print just "No action specified"; if it prints
-    # anything else, it's a parse error.
-    # The initial ! is removed by YAML so we need to quote it.
-    script:
-        - "! utils/get-developers | grep -v 'No action specified'"
-
 .defconfig: &defconfig
     # Running the defconfigs for every push is too much, so limit to
     # explicit triggers through the API.
@@ -58,175 +45,8 @@ check-DEVELOPERS:
         paths:
             - test-output/*.log
             - test-output/*/images/*
-acmesystems_aria_g25_128mb_defconfig: *defconfig
-acmesystems_aria_g25_256mb_defconfig: *defconfig
-acmesystems_arietta_g25_128mb_defconfig: *defconfig
-acmesystems_arietta_g25_256mb_defconfig: *defconfig
-arcturus_ucp1020_defconfig: *defconfig
-arm_foundationv8_defconfig: *defconfig
-arm_juno_defconfig: *defconfig
-armadeus_apf27_defconfig: *defconfig
-armadeus_apf28_defconfig: *defconfig
-armadeus_apf51_defconfig: *defconfig
-at91sam9260eknf_defconfig: *defconfig
-at91sam9g20dfc_defconfig: *defconfig
-at91sam9g45m10ek_defconfig: *defconfig
-at91sam9rlek_defconfig: *defconfig
-at91sam9x5ek_defconfig: *defconfig
-at91sam9x5ek_dev_defconfig: *defconfig
-at91sam9x5ek_mmc_defconfig: *defconfig
-at91sam9x5ek_mmc_dev_defconfig: *defconfig
-atmel_sama5d27_som1_ek_mmc_dev_defconfig: *defconfig
-atmel_sama5d2_xplained_mmc_defconfig: *defconfig
-atmel_sama5d2_xplained_mmc_dev_defconfig: *defconfig
-atmel_sama5d3_xplained_defconfig: *defconfig
-atmel_sama5d3_xplained_dev_defconfig: *defconfig
-atmel_sama5d3_xplained_mmc_defconfig: *defconfig
-atmel_sama5d3_xplained_mmc_dev_defconfig: *defconfig
-atmel_sama5d3xek_defconfig: *defconfig
-atmel_sama5d4_xplained_defconfig: *defconfig
-atmel_sama5d4_xplained_dev_defconfig: *defconfig
-atmel_sama5d4_xplained_mmc_defconfig: *defconfig
-atmel_sama5d4_xplained_mmc_dev_defconfig: *defconfig
-bananapi_m1_defconfig: *defconfig
-bananapi_m2_plus_defconfig: *defconfig
-bananapro_defconfig: *defconfig
-beagleboardx15_defconfig: *defconfig
-beaglebone_defconfig: *defconfig
-beaglebone_qt5_defconfig: *defconfig
-chromebook_snow_defconfig: *defconfig
-ci20_defconfig: *defconfig
-ci40_defconfig: *defconfig
-csky_gx6605s_defconfig: *defconfig
-cubieboard2_defconfig: *defconfig
-engicam_imx6qdl_icore_defconfig: *defconfig
-engicam_imx6qdl_icore_qt5_defconfig: *defconfig
-engicam_imx6qdl_icore_rqs_defconfig: *defconfig
-engicam_imx6ul_geam_defconfig: *defconfig
-engicam_imx6ul_isiot_defconfig: *defconfig
-firefly_rk3288_defconfig: *defconfig
-firefly_rk3288_demo_defconfig: *defconfig
-freescale_imx28evk_defconfig: *defconfig
-freescale_imx31_3stack_defconfig: *defconfig
-freescale_imx6dlsabreauto_defconfig: *defconfig
-freescale_imx6dlsabresd_defconfig: *defconfig
-freescale_imx6qsabreauto_defconfig: *defconfig
-freescale_imx6qsabresd_defconfig: *defconfig
-freescale_imx6sololiteevk_defconfig: *defconfig
-freescale_imx6sxsabresd_defconfig: *defconfig
-freescale_imx6ulevk_defconfig: *defconfig
-freescale_imx7dsabresd_defconfig: *defconfig
-freescale_mpc8315erdb_defconfig: *defconfig
-freescale_p1010rdb_pa_defconfig: *defconfig
-galileo_defconfig: *defconfig
-gdb_bfin_bf512_defconfig: *defconfig
-grinn_chiliboard_defconfig: *defconfig
-grinn_liteboard_defconfig: *defconfig
-imx23evk_defconfig: *defconfig
-imx6-sabreauto_defconfig: *defconfig
-imx6-sabresd_defconfig: *defconfig
-imx6-sabresd_qt5_defconfig: *defconfig
-imx6ulpico_defconfig: *defconfig
-imx7dpico_defconfig: *defconfig
-lego_ev3_defconfig: *defconfig
-linksprite_pcduino_defconfig: *defconfig
-minnowboard_max-graphical_defconfig: *defconfig
-minnowboard_max_defconfig: *defconfig
-mx25pdk_defconfig: *defconfig
-mx51evk_defconfig: *defconfig
-mx53loco_defconfig: *defconfig
-mx6cubox_defconfig: *defconfig
-mx6sx_udoo_neo_defconfig: *defconfig
-mx6udoo_defconfig: *defconfig
-nanopi_m1_defconfig: *defconfig
-nanopi_m1_plus_defconfig: *defconfig
-nanopi_neo_defconfig: *defconfig
-nexbox_a95x_defconfig: *defconfig
-nitrogen6sx_defconfig: *defconfig
-nitrogen6x_defconfig: *defconfig
-nitrogen7_defconfig: *defconfig
-odroidc2_defconfig: *defconfig
-olimex_a13_olinuxino_defconfig: *defconfig
-olimex_a20_olinuxino_lime2_defconfig: *defconfig
-olimex_a20_olinuxino_lime_defconfig: *defconfig
-olimex_a20_olinuxino_lime_mali_defconfig: *defconfig
-olimex_a20_olinuxino_micro_defconfig: *defconfig
-olimex_imx233_olinuxino_defconfig: *defconfig
-openblocks_a6_defconfig: *defconfig
-orangepi_one_defconfig: *defconfig
-orangepi_pc_defconfig: *defconfig
-orangepi_plus_defconfig: *defconfig
-orangepi_zero_defconfig: *defconfig
-pandaboard_defconfig: *defconfig
-pc_x86_64_bios_defconfig: *defconfig
-pc_x86_64_efi_defconfig: *defconfig
-qemu_aarch64_virt_defconfig: *defconfig
-qemu_arm_versatile_defconfig: *defconfig
-qemu_arm_versatile_nommu_defconfig: *defconfig
-qemu_arm_vexpress_defconfig: *defconfig
-qemu_m68k_mcf5208_defconfig: *defconfig
-qemu_m68k_q800_defconfig: *defconfig
-qemu_microblazebe_mmu_defconfig: *defconfig
-qemu_microblazeel_mmu_defconfig: *defconfig
-qemu_mips32r2_malta_defconfig: *defconfig
-qemu_mips32r2el_malta_defconfig: *defconfig
-qemu_mips32r6_malta_defconfig: *defconfig
-qemu_mips32r6el_malta_defconfig: *defconfig
-qemu_mips64_malta_defconfig: *defconfig
-qemu_mips64el_malta_defconfig: *defconfig
-qemu_mips64r6_malta_defconfig: *defconfig
-qemu_mips64r6el_malta_defconfig: *defconfig
-qemu_nios2_10m50_defconfig: *defconfig
-qemu_or1k_defconfig: *defconfig
-qemu_ppc64_pseries_defconfig: *defconfig
-qemu_ppc64le_pseries_defconfig: *defconfig
-qemu_ppc_g3beige_defconfig: *defconfig
-qemu_ppc_mpc8544ds_defconfig: *defconfig
-qemu_ppc_virtex_ml507_defconfig: *defconfig
-qemu_sh4_r2d_defconfig: *defconfig
-qemu_sh4eb_r2d_defconfig: *defconfig
-qemu_sparc64_sun4u_defconfig: *defconfig
-qemu_sparc_ss10_defconfig: *defconfig
-qemu_x86_64_defconfig: *defconfig
-qemu_x86_defconfig: *defconfig
-qemu_xtensa_lx60_defconfig: *defconfig
-qemu_xtensa_lx60_nommu_defconfig: *defconfig
-raspberrypi0_defconfig: *defconfig
-raspberrypi2_defconfig: *defconfig
-raspberrypi3_64_defconfig: *defconfig
-raspberrypi3_defconfig: *defconfig
-raspberrypi3_qt5we_defconfig: *defconfig
-raspberrypi_defconfig: *defconfig
-riotboard_defconfig: *defconfig
-roseapplepi_defconfig: *defconfig
-s6lx9_microboard_defconfig: *defconfig
-sheevaplug_defconfig: *defconfig
-snps_aarch64_vdk_defconfig: *defconfig
-snps_arc700_axs101_defconfig: *defconfig
-snps_archs38_axs103_defconfig: *defconfig
-snps_archs38_haps_defconfig: *defconfig
-snps_archs38_vdk_defconfig: *defconfig
-socrates_cyclone5_defconfig: *defconfig
-stm32f429_disco_defconfig: *defconfig
-stm32f469_disco_defconfig: *defconfig
-telit_evk_pro3_defconfig: *defconfig
-toradex_apalis_imx6_defconfig: *defconfig
-ts4800_defconfig: *defconfig
-ts4900_defconfig: *defconfig
-ts5x00_defconfig: *defconfig
-ts7680_defconfig: *defconfig
-wandboard_defconfig: *defconfig
-warp7_defconfig: *defconfig
-warpboard_defconfig: *defconfig
-zynq_microzed_defconfig: *defconfig
-zynq_zc706_defconfig: *defconfig
-zynq_zed_defconfig: *defconfig
-zynq_zybo_defconfig: *defconfig
 tests.core.test_post_scripts.TestPostScripts: *runtime_test
 tests.core.test_rootfs_overlay.TestRootfsOverlay: *runtime_test
-tests.core.test_timezone.TestGlibcAllTimezone: *runtime_test
-tests.core.test_timezone.TestGlibcNonDefaultLimitedTimezone: *runtime_test
-tests.core.test_timezone.TestNoTimezone: *runtime_test
 tests.fs.test_ext.TestExt2: *runtime_test
 tests.fs.test_ext.TestExt2r1: *runtime_test
 tests.fs.test_ext.TestExt3: *runtime_test
@@ -239,27 +59,3 @@ tests.fs.test_jffs2.TestJffs2: *runtime_test
 tests.fs.test_squashfs.TestSquashfs: *runtime_test
 tests.fs.test_ubi.TestUbi: *runtime_test
 tests.fs.test_yaffs2.TestYaffs2: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRo: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRoNet: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRw: *runtime_test
-tests.init.test_busybox.TestInitSystemBusyboxRwNet: *runtime_test
-tests.init.test_none.TestInitSystemNone: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoFull: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoIfupdown: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRoNetworkd: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwFull: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwIfupdown: *runtime_test
-tests.init.test_systemd.TestInitSystemSystemdRwNetworkd: *runtime_test
-tests.package.test_dropbear.TestDropbear: *runtime_test
-tests.package.test_ipython.TestIPythonPy2: *runtime_test
-tests.package.test_ipython.TestIPythonPy3: *runtime_test
-tests.package.test_python.TestPython2: *runtime_test
-tests.package.test_python.TestPython3: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainBuildrootMusl: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainCCache: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainCtngMusl: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainLinaroArm: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv4: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv5: *runtime_test
-tests.toolchain.test_external.TestExternalToolchainSourceryArmv7: *runtime_test
-- 
2.11.0

[Buildroot] [PATCH 14/14] TEST-ME

[Yann E. MORIN]

All,

On 2017-11-12 18:45 +0100, Yann E. MORIN spake thusly:
>  .gitlab-ci.yml | 204 ---------------------------------------------------------
>  1 file changed, 204 deletions(-)

Sorry, I should not have sent that patch as part of the series. It was
there so that gitlab-ci would only test my changes.

I've marked it as rejected on patchwork. Sorry for the noise...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 00/14] fs: cleanups and enhancements

[Thomas Petazzoni]

Hello,

On Sun, 12 Nov 2017 18:45:47 +0100, Yann E. MORIN wrote:

> Yann E. MORIN (14):
>       core: sort packages and eliminate duplicates in show-targets
>       linux: split overly-long dependency line for readability
>       linux: meddle not in the internals of filesystems
>       fs/initramfs: cleanups, enhance comments
>       fs/ext2: use a post-gen hook rather than a post-target rule
>       fs/cpio: use a post-gen hook rather than a post-target rule
>       fs/common: get rid of post-target rules
>       fs: remove TARGET_DIR_WARNING_FILE late
>       fs: don't pollute $(BUILD_DIR) with temp files
>       fs/iso9660: don't pollute $(BUILD_DIR) with temp dir
>       fs: don't remove intermediate files

I have applied the patch series up to this point to the next branch.
Besides minor typo fixes here and there, I didn't change anything.

>       fs: move actions common to all filesystems to their own rule
>       core: finalise target in its own location

I will continue to apply the remainder of this series soon. I need it
for the TLP work :-)

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com