[Buildroot] [PATCH 01/20] support/tests: allow properly indented config fragment

[Buildroot] [PATCH 01/20] support/tests: allow properly indented config fragment

[Yann E. MORIN]

Currently, defining a config fragment in the runtime test infra requires
that the fragment not to be indented. This is beark, and causes grievance
when looking at the code (e.g. to fix it).

Just strip out all leading spaces/tabs when writing the configuration
lines into the config file, allowing in-line indented config fragments,
like so:

    class TestFoo(bla):
        config = bla.config + \
            """
            FOO=y
            # BAR is not set
            """

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

---
Changes v1 -> v2:
  - add example in commit log  (Thomas)

---
Note: this is only tangentially related to this series, because it is
needed by the last patch in the series. It wasalready submitted before:
    https://patchwork.ozlabs.org/patch/788985/
---
 support/testing/infra/builder.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py
index a475bb0a30..81735dec96 100644
--- a/support/testing/infra/builder.py
+++ b/support/testing/infra/builder.py
@@ -16,7 +16,8 @@ class Builder(object):
 
         config_file = os.path.join(self.builddir, ".config")
         with open(config_file, "w+") as cf:
-            cf.write(self.config)
+            for line in self.config.splitlines():
+                cf.write("{}\n".format(line.lstrip()))
 
         cmd = ["make",
                "O={}".format(self.builddir),
-- 
2.11.0

[Buildroot] [PATCH 02/20] core/pkg-generic: add variable to skip skeleton dependency

[Yann E. MORIN]

We currently skip the skeleton dependency by checking if the current
package is the skeleton.

We are going to introduce more skeleton-related packages, so we
need a way to exclude the skeleton dependency for those, or we'd
get a circular dependency, for the same reason we need to skip
the toolchain dependency.

Instead of checking for all the skeleton-providing packages in the core
infra, add a new package options so that packages can express they do
not need the dependency on the skeleton, like we have an option to avoid
the depednency on the toolchain. The only packages that will use that
option are probably the skeletons, so we need not document this
variable, like we did not document the option to exclude the dependency
on the toolchain.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
---
 package/pkg-generic.mk       | 7 +++++--
 package/skeleton/skeleton.mk | 1 +
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index 2916a7bbad..b2b3af7d8f 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -542,11 +542,14 @@ $(2)_REDIST_SOURCES_DIR = $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))/$$($(2)
 
 # When a target package is a toolchain dependency set this variable to
 # 'NO' so the 'toolchain' dependency is not added to prevent a circular
-# dependency
+# dependency.
+# Similarly for the skeleton.
 $(2)_ADD_TOOLCHAIN_DEPENDENCY	?= YES
+$(2)_ADD_SKELETON_DEPENDENCY	?= YES
+
 
 ifeq ($(4),target)
-ifneq ($(1),skeleton)
+ifeq ($$($(2)_ADD_SKELETON_DEPENDENCY),YES)
 $(2)_DEPENDENCIES += skeleton
 endif
 ifeq ($$($(2)_ADD_TOOLCHAIN_DEPENDENCY),YES)
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index a310a314a5..9b7407a236 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -9,6 +9,7 @@
 # Hence, skeleton would depends on the toolchain and the toolchain would depend
 # on skeleton.
 SKELETON_ADD_TOOLCHAIN_DEPENDENCY = NO
+SKELETON_ADD_SKELETON_DEPENDENCY = NO
 
 # The skeleton also handles the merged /usr case in the sysroot
 SKELETON_INSTALL_STAGING = YES
-- 
2.11.0

[Buildroot] [PATCH 03/20] package/skeleton: add macro to rsync skeleton directory

[Yann E. MORIN]

The rsync command is currently called from a single location. In
follow-up patches, it will be called from various places then.

It will be easier to update and fix this command if it is defined in a
single location, rather than if duplicated in many places.

Define a macro that skeleton packages can call to install their skeleton
directories. Since at least one occurence of it will want to install in
staging, the macro must accept a destination path, rather than hard-code
target as the destination.

Since it is to be used by other skeleton packages, define it early,
outside of any conditional block.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>
---
 package/skeleton/skeleton.mk | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)

diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 9b7407a236..017479a990 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -14,6 +14,18 @@ SKELETON_ADD_SKELETON_DEPENDENCY = NO
 # The skeleton also handles the merged /usr case in the sysroot
 SKELETON_INSTALL_STAGING = YES
 
+############
+# Macros available for use by any skeleton package:
+# - SKELETON_RSYNC
+
+# This function rsyncs the skeleton directory in $(1) to the destination
+# in $(2), which should be either $(TARTGET_DIR) or $(STAGING_DIR)
+define SKELETON_RSYNC
+	rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
+		--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
+		$(1)/ $(2)/
+endef
+
 ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 
 SKELETON_PATH = $(call qstrip,$(BR2_ROOTFS_SKELETON_CUSTOM_PATH))
@@ -89,9 +101,7 @@ SKELETON_LIB_SYMLINK = lib32
 endif
 
 define SKELETON_INSTALL_TARGET_CMDS
-	rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
-		--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
-		$(SKELETON_PATH)/ $(TARGET_DIR)/
+	$(call SKELETON_RSYNC,$(SKELETON_PATH),$(TARGET_DIR))
 	$(call SKELETON_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
 	ln -snf lib $(TARGET_DIR)/$(SKELETON_LIB_SYMLINK)
 	ln -snf lib $(TARGET_DIR)/usr/$(SKELETON_LIB_SYMLINK)
-- 
2.11.0

[Buildroot] [PATCH 04/20] package/skeleton: make SKELETON_LIB_SYMLINK a macro

[Yann E. MORIN]

Rather than duplicate that code in all skeleton packages, make this a
macro, so that it is easier to use.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/skeleton/skeleton.mk | 30 ++++++++++++++++++------------
 1 file changed, 18 insertions(+), 12 deletions(-)

diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 017479a990..0a8add66dc 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -17,6 +17,7 @@ SKELETON_INSTALL_STAGING = YES
 ############
 # Macros available for use by any skeleton package:
 # - SKELETON_RSYNC
+# - SKELETON_LIB_SYMLINK
 
 # This function rsyncs the skeleton directory in $(1) to the destination
 # in $(2), which should be either $(TARTGET_DIR) or $(STAGING_DIR)
@@ -26,6 +27,21 @@ define SKELETON_RSYNC
 		$(1)/ $(2)/
 endef
 
+# Make a symlink lib32->lib or lib64->lib as appropriate.
+# MIPS64/n32 requires lib32 even though it's a 64-bit arch.
+# $(1): base dir (either staging or target)
+ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)
+define SKELETON_LIB_SYMLINK
+	ln -snf lib $(1)/lib64
+	ln -snf lib $(1)/usr/lib64
+endef
+else
+define SKELETON_LIB_SYMLINK
+	ln -snf lib $(1)/lib32
+	ln -snf lib $(1)/usr/lib32
+endef
+endif
+
 ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 
 SKELETON_PATH = $(call qstrip,$(BR2_ROOTFS_SKELETON_CUSTOM_PATH))
@@ -92,19 +108,10 @@ define SKELETON_USR_SYMLINKS_OR_DIRS
 endef
 endif
 
-# We make a symlink lib32->lib or lib64->lib as appropriate
-# MIPS64/n32 requires lib32 even though it's a 64-bit arch.
-ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)
-SKELETON_LIB_SYMLINK = lib64
-else
-SKELETON_LIB_SYMLINK = lib32
-endif
-
 define SKELETON_INSTALL_TARGET_CMDS
 	$(call SKELETON_RSYNC,$(SKELETON_PATH),$(TARGET_DIR))
 	$(call SKELETON_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
-	ln -snf lib $(TARGET_DIR)/$(SKELETON_LIB_SYMLINK)
-	ln -snf lib $(TARGET_DIR)/usr/$(SKELETON_LIB_SYMLINK)
+	$(call SKELETON_LIB_SYMLINK,$(TARGET_DIR))
 	$(INSTALL) -m 0644 support/misc/target-dir-warning.txt \
 		$(TARGET_DIR_WARNING_FILE)
 endef
@@ -120,8 +127,7 @@ define SKELETON_INSTALL_STAGING_CMDS
 	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/sbin
 	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
 	$(call SKELETON_USR_SYMLINKS_OR_DIRS,$(STAGING_DIR))
-	ln -snf lib $(STAGING_DIR)/$(SKELETON_LIB_SYMLINK)
-	ln -snf lib $(STAGING_DIR)/usr/$(SKELETON_LIB_SYMLINK)
+	$(call SKELETON_LIB_SYMLINK,$(STAGING_DIR))
 endef
 
 # The TARGET_FINALIZE_HOOKS must be sourced only if the users choose to use the
-- 
2.11.0

[Buildroot] [PATCH 05/20] system: provide package-wide system variables and macros

[Yann E. MORIN]

Some macros, soon some variables, currently defined in the skeleton are
going to be used by other packages.

Some of those variables will be used as Makefile conditions (e.g. in
ifeq() conditiopns), so they *must* be defined before being used.

Since the skeleton package, starting with an 's', is included quite
late, those variables would not be available to most packages.

Offload the existing macros into the new system/system/mk file, that is
included early, before any package is. Rename the macros to appropriate
names.

Future commits will add new macros and variables in that file.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 Makefile                     |  1 +
 package/skeleton/skeleton.mk | 53 ++++---------------------------------
 system/system.mk             | 62 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 68 insertions(+), 48 deletions(-)
 create mode 100644 system/system.mk

diff --git a/Makefile b/Makefile
index 188ce9adc7..41cae8baa5 100644
--- a/Makefile
+++ b/Makefile
@@ -479,6 +479,7 @@ all: world
 # may rely on it.
 include Makefile.legacy
 
+include system/system.mk
 include package/Makefile.in
 # arch/arch.mk.* must be after package/Makefile.in because it may need to
 # complement variables defined therein, like BR_NO_CHECK_HASH_FOR.
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 0a8add66dc..3ae37c76a9 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -14,34 +14,6 @@ SKELETON_ADD_SKELETON_DEPENDENCY = NO
 # The skeleton also handles the merged /usr case in the sysroot
 SKELETON_INSTALL_STAGING = YES
 
-############
-# Macros available for use by any skeleton package:
-# - SKELETON_RSYNC
-# - SKELETON_LIB_SYMLINK
-
-# This function rsyncs the skeleton directory in $(1) to the destination
-# in $(2), which should be either $(TARTGET_DIR) or $(STAGING_DIR)
-define SKELETON_RSYNC
-	rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
-		--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
-		$(1)/ $(2)/
-endef
-
-# Make a symlink lib32->lib or lib64->lib as appropriate.
-# MIPS64/n32 requires lib32 even though it's a 64-bit arch.
-# $(1): base dir (either staging or target)
-ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)
-define SKELETON_LIB_SYMLINK
-	ln -snf lib $(1)/lib64
-	ln -snf lib $(1)/usr/lib64
-endef
-else
-define SKELETON_LIB_SYMLINK
-	ln -snf lib $(1)/lib32
-	ln -snf lib $(1)/usr/lib32
-endef
-endif
-
 ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
 
 SKELETON_PATH = $(call qstrip,$(BR2_ROOTFS_SKELETON_CUSTOM_PATH))
@@ -93,25 +65,10 @@ SKELETON_PATH = system/skeleton
 
 endif # ! custom skeleton
 
-# This function handles the merged or non-merged /usr cases
-ifeq ($(BR2_ROOTFS_MERGED_USR),y)
-define SKELETON_USR_SYMLINKS_OR_DIRS
-	ln -snf usr/bin $(1)/bin
-	ln -snf usr/sbin $(1)/sbin
-	ln -snf usr/lib $(1)/lib
-endef
-else
-define SKELETON_USR_SYMLINKS_OR_DIRS
-	$(INSTALL) -d -m 0755 $(1)/bin
-	$(INSTALL) -d -m 0755 $(1)/sbin
-	$(INSTALL) -d -m 0755 $(1)/lib
-endef
-endif
-
 define SKELETON_INSTALL_TARGET_CMDS
-	$(call SKELETON_RSYNC,$(SKELETON_PATH),$(TARGET_DIR))
-	$(call SKELETON_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
-	$(call SKELETON_LIB_SYMLINK,$(TARGET_DIR))
+	$(call SYSTEM_RSYNC,$(SKELETON_PATH),$(TARGET_DIR))
+	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
+	$(call SYSTEM_LIB_SYMLINK,$(TARGET_DIR))
 	$(INSTALL) -m 0644 support/misc/target-dir-warning.txt \
 		$(TARGET_DIR_WARNING_FILE)
 endef
@@ -126,8 +83,8 @@ define SKELETON_INSTALL_STAGING_CMDS
 	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/bin
 	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/sbin
 	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
-	$(call SKELETON_USR_SYMLINKS_OR_DIRS,$(STAGING_DIR))
-	$(call SKELETON_LIB_SYMLINK,$(STAGING_DIR))
+	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(STAGING_DIR))
+	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
 endef
 
 # The TARGET_FINALIZE_HOOKS must be sourced only if the users choose to use the
diff --git a/system/system.mk b/system/system.mk
new file mode 100644
index 0000000000..cab5c1df5d
--- /dev/null
+++ b/system/system.mk
@@ -0,0 +1,62 @@
+################################################################################
+#
+# system-related variables and macros
+#
+################################################################################
+
+# This file exists to define variables and macros that pertain to the system
+# settings, like rsyncing a directory for skeletons, or the /lib vs. /usr/lib
+# symlink handling.
+#
+# Some variables may be used as conditions in Makefile code, so they must be
+# defined properly before they are used; this file is included early, before
+# any package is.
+
+# - SYSTEM_USR_SYMLINKS_OR_DIRS
+#   create /lib, /bin and /sbin, either as directories or as symlinks to
+#   their /usr conterparts
+#
+# - SYSTEM_RSYNC
+#   rsync $(1) to $(2), with proper exclusions and rights
+#
+# - SYSTEM_LIB_SYMLINK
+#   create the appropriate /lib{32,64} symlinks
+#
+
+# This function handles the merged or non-merged /usr cases
+ifeq ($(BR2_ROOTFS_MERGED_USR),y)
+define SYSTEM_USR_SYMLINKS_OR_DIRS
+	ln -snf usr/bin $(1)/bin
+	ln -snf usr/sbin $(1)/sbin
+	ln -snf usr/lib $(1)/lib
+endef
+else
+define SYSTEM_USR_SYMLINKS_OR_DIRS
+	$(INSTALL) -d -m 0755 $(1)/bin
+	$(INSTALL) -d -m 0755 $(1)/sbin
+	$(INSTALL) -d -m 0755 $(1)/lib
+endef
+endif
+
+# This function rsyncs the skeleton directory in $(1) to the destination
+# in $(2), which should be either $(TARTGET_DIR) or $(STAGING_DIR)
+define SYSTEM_RSYNC
+	rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
+		--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
+		$(1)/ $(2)/
+endef
+
+# Make a symlink lib32->lib or lib64->lib as appropriate.
+# MIPS64/n32 requires lib32 even though it's a 64-bit arch.
+# $(1): base dir (either staging or target)
+ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)
+define SYSTEM_LIB_SYMLINK
+	ln -snf lib $(1)/lib64
+	ln -snf lib $(1)/usr/lib64
+endef
+else
+define SYSTEM_LIB_SYMLINK
+	ln -snf lib $(1)/lib32
+	ln -snf lib $(1)/usr/lib32
+endef
+endif
-- 
2.11.0

[Buildroot] [PATCH 06/20] system: move setting getty to the corresponding init systems

[Yann E. MORIN]

Currently, setting the getty is done:
  - by the skeleton package when the init system is either busybox or
    sysvinit;
  - by the systemd package when the init system is systemd;
both by registering a target-finalize hook.

This is not very consistent.

Move setting the getty out of the skeleton and into the package that
provides the init system, by registering a per-package target-fialize
hook.

This offloads yet a bit more out of the skeleton, so that it is easier
to properly separate the skeletons for the various init systems.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/busybox/busybox.mk   | 12 +++++++++++-
 package/skeleton/skeleton.mk | 22 ----------------------
 package/sysvinit/sysvinit.mk |  8 ++++++++
 system/system.mk             | 11 +++++++++++
 4 files changed, 30 insertions(+), 23 deletions(-)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index 2231730aa8..2001c2dbdb 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -181,10 +181,20 @@ define BUSYBOX_INSTALL_UDHCPC_SCRIPT
 endef
 
 ifeq ($(BR2_INIT_BUSYBOX),y)
+
 define BUSYBOX_SET_INIT
 	$(call KCONFIG_ENABLE_OPT,CONFIG_INIT,$(BUSYBOX_BUILD_CONFIG))
 endef
-endif
+
+ifeq ($(BR2_TARGET_GENERIC_GETTY),y)
+define BUSYBOX_SET_GETTY
+	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(SYSTEM_GETTY_PORT)::respawn:/sbin/getty -L $(SYSTEM_GETTY_OPTIONS) $(SYSTEM_GETTY_PORT) $(SYSTEM_GETTY_BAUDRATE) $(SYSTEM_GETTY_TERM) #~' \
+		$(TARGET_DIR)/etc/inittab
+endef
+BUSYBOX_TARGET_FINALIZE_HOOKS += BUSYBOX_SET_GETTY
+endif # BR2_TARGET_GENERIC_GETTY
+
+endif # BR2_INIT_BUSYBOX
 
 ifeq ($(BR2_PACKAGE_BUSYBOX_SELINUX),y)
 BUSYBOX_DEPENDENCIES += host-pkgconf libselinux libsepol
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 3ae37c76a9..7ad2cb1a66 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -96,10 +96,6 @@ SKELETON_TARGET_GENERIC_ISSUE = $(call qstrip,$(BR2_TARGET_GENERIC_ISSUE))
 SKELETON_TARGET_GENERIC_ROOT_PASSWD = $(call qstrip,$(BR2_TARGET_GENERIC_ROOT_PASSWD))
 SKELETON_TARGET_GENERIC_PASSWD_METHOD = $(call qstrip,$(BR2_TARGET_GENERIC_PASSWD_METHOD))
 SKELETON_TARGET_GENERIC_BIN_SH = $(call qstrip,$(BR2_SYSTEM_BIN_SH))
-SKELETON_TARGET_GENERIC_GETTY_PORT = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_PORT))
-SKELETON_TARGET_GENERIC_GETTY_BAUDRATE = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_BAUDRATE))
-SKELETON_TARGET_GENERIC_GETTY_TERM = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_TERM))
-SKELETON_TARGET_GENERIC_GETTY_OPTIONS = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_OPTIONS))
 
 ifneq ($(SKELETON_TARGET_GENERIC_HOSTNAME),)
 define SKELETON_SET_HOSTNAME
@@ -152,24 +148,6 @@ endif
 endif
 TARGET_FINALIZE_HOOKS += SKELETON_BIN_SH
 
-ifeq ($(BR2_TARGET_GENERIC_GETTY),y)
-ifeq ($(BR2_INIT_SYSV),y)
-# In sysvinit inittab, the "id" must not be longer than 4 bytes, so we
-# skip the "tty" part and keep only the remaining.
-define SKELETON_SET_GETTY
-	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(shell echo $(SKELETON_TARGET_GENERIC_GETTY_PORT) | tail -c+4)::respawn:/sbin/getty -L $(SKELETON_TARGET_GENERIC_GETTY_OPTIONS) $(SKELETON_TARGET_GENERIC_GETTY_PORT) $(SKELETON_TARGET_GENERIC_GETTY_BAUDRATE) $(SKELETON_TARGET_GENERIC_GETTY_TERM) #~' \
-		$(TARGET_DIR)/etc/inittab
-endef
-else ifeq ($(BR2_INIT_BUSYBOX),y)
-# Add getty to busybox inittab
-define SKELETON_SET_GETTY
-	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(SKELETON_TARGET_GENERIC_GETTY_PORT)::respawn:/sbin/getty -L $(SKELETON_TARGET_GENERIC_GETTY_OPTIONS) $(SKELETON_TARGET_GENERIC_GETTY_PORT) $(SKELETON_TARGET_GENERIC_GETTY_BAUDRATE) $(SKELETON_TARGET_GENERIC_GETTY_TERM) #~' \
-		$(TARGET_DIR)/etc/inittab
-endef
-endif
-TARGET_FINALIZE_HOOKS += SKELETON_SET_GETTY
-endif
-
 ifeq ($(BR2_INIT_BUSYBOX)$(BR2_INIT_SYSV),y)
 ifeq ($(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),y)
 # Find commented line, if any, and remove leading '#'s
diff --git a/package/sysvinit/sysvinit.mk b/package/sysvinit/sysvinit.mk
index ca7d06b61d..e9aa69b27b 100644
--- a/package/sysvinit/sysvinit.mk
+++ b/package/sysvinit/sysvinit.mk
@@ -47,4 +47,12 @@ define SYSVINIT_INSTALL_TARGET_CMDS
 	ln -sf killall5 $(TARGET_DIR)/sbin/pidof
 endef
 
+ifeq ($(BR2_TARGET_GENERIC_GETTY),y)
+define SYSVINIT_SET_GETTY
+	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(shell echo $(SYSTEM_GETTY_PORT) | tail -c+4)::respawn:/sbin/getty -L $(SYSTEM_GETTY_OPTIONS) $(SYSTEM_GETTY_PORT) $(SYSTEM_GETTY_BAUDRATE) $(SYSTEM_GETTY_TERM) #~' \
+		$(TARGET_DIR)/etc/inittab
+endef
+SYSVINIT_TARGET_FINALIZE_HOOKS += SYSVINIT_SET_GETTY
+endif # BR2_TARGET_GENERIC_GETTY
+
 $(eval $(generic-package))
diff --git a/system/system.mk b/system/system.mk
index cab5c1df5d..7b843dad8e 100644
--- a/system/system.mk
+++ b/system/system.mk
@@ -22,6 +22,12 @@
 # - SYSTEM_LIB_SYMLINK
 #   create the appropriate /lib{32,64} symlinks
 #
+# - SYSTEM_GETTY_PORT
+# - SYSTEM_GETTY_BAUDRATE
+# - SYSTEM_GETTY_TERM
+# - SYSTEM_GETTY_OPTIONS
+#   the un-quoted getty setting
+#
 
 # This function handles the merged or non-merged /usr cases
 ifeq ($(BR2_ROOTFS_MERGED_USR),y)
@@ -60,3 +66,8 @@ define SYSTEM_LIB_SYMLINK
 	ln -snf lib $(1)/usr/lib32
 endef
 endif
+
+SYSTEM_GETTY_PORT = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_PORT))
+SYSTEM_GETTY_BAUDRATE = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_BAUDRATE))
+SYSTEM_GETTY_TERM = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_TERM))
+SYSTEM_GETTY_OPTIONS = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_OPTIONS))
-- 
2.11.0

[Buildroot] [PATCH 07/20] system: move remounting / to the corresponding init systems

[Yann E. MORIN]

Currently, remounting / read-write (or not) is done by the skeleton
package when the init system is eitehr busybox or sysvinit, by
registering a target-finalize hook; it is not done at all for systemd.

Move registering this target-finalize hook to both of busybox and
sysvinit. Leave systemd alone, we'll take care of it later.

Rename the macro to a more meaningful name, and move it to system.mk
with the other such macros.

Yet a little bit less init-system knowledge in the skeleton.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/busybox/busybox.mk   |  3 +++
 package/skeleton/skeleton.mk | 15 ---------------
 package/sysvinit/sysvinit.mk |  3 +++
 system/system.mk             | 15 +++++++++++++++
 4 files changed, 21 insertions(+), 15 deletions(-)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index 2001c2dbdb..0550275a38 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -194,6 +194,9 @@ endef
 BUSYBOX_TARGET_FINALIZE_HOOKS += BUSYBOX_SET_GETTY
 endif # BR2_TARGET_GENERIC_GETTY
 
+# Provided by the skeleton package
+BUSYBOX_TARGET_FINALIZE_HOOKS += SYSTEM_REMOUNT_ROOT_INITTAB
+
 endif # BR2_INIT_BUSYBOX
 
 ifeq ($(BR2_PACKAGE_BUSYBOX_SELINUX),y)
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 7ad2cb1a66..4a66f6ceca 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -148,21 +148,6 @@ endif
 endif
 TARGET_FINALIZE_HOOKS += SKELETON_BIN_SH
 
-ifeq ($(BR2_INIT_BUSYBOX)$(BR2_INIT_SYSV),y)
-ifeq ($(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),y)
-# Find commented line, if any, and remove leading '#'s
-define SKELETON_REMOUNT_RW
-	$(SED) '/^#.*-o remount,rw \/$$/s~^#\+~~' $(TARGET_DIR)/etc/inittab
-endef
-else
-# Find uncommented line, if any, and add a leading '#'
-define SKELETON_REMOUNT_RW
-	$(SED) '/^[^#].*-o remount,rw \/$$/s~^~#~' $(TARGET_DIR)/etc/inittab
-endef
-endif
-TARGET_FINALIZE_HOOKS += SKELETON_REMOUNT_RW
-endif # BR2_INIT_BUSYBOX || BR2_INIT_SYSV
-
 endif # BR2_ROOTFS_SKELETON_DEFAULT
 
 $(eval $(generic-package))
diff --git a/package/sysvinit/sysvinit.mk b/package/sysvinit/sysvinit.mk
index e9aa69b27b..d3312820b1 100644
--- a/package/sysvinit/sysvinit.mk
+++ b/package/sysvinit/sysvinit.mk
@@ -55,4 +55,7 @@ endef
 SYSVINIT_TARGET_FINALIZE_HOOKS += SYSVINIT_SET_GETTY
 endif # BR2_TARGET_GENERIC_GETTY
 
+# Provided by the skeleton package
+SYSVINIT_TARGET_FINALIZE_HOOKS += SYSTEM_REMOUNT_ROOT_INITTAB
+
 $(eval $(generic-package))
diff --git a/system/system.mk b/system/system.mk
index 7b843dad8e..ca6bf1388f 100644
--- a/system/system.mk
+++ b/system/system.mk
@@ -28,6 +28,9 @@
 # - SYSTEM_GETTY_OPTIONS
 #   the un-quoted getty setting
 #
+# - SYSTEM_REMOUNT_ROOT_INITTAB
+#   set inittab to remount root read-write or read-only
+#
 
 # This function handles the merged or non-merged /usr cases
 ifeq ($(BR2_ROOTFS_MERGED_USR),y)
@@ -71,3 +74,15 @@ SYSTEM_GETTY_PORT = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_PORT))
 SYSTEM_GETTY_BAUDRATE = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_BAUDRATE))
 SYSTEM_GETTY_TERM = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_TERM))
 SYSTEM_GETTY_OPTIONS = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_OPTIONS))
+
+ifeq ($(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),y)
+# Find commented line, if any, and remove leading '#'s
+define SYSTEM_REMOUNT_ROOT_INITTAB
+	$(SED) '/^#.*-o remount,rw \/$$/s~^#\+~~' $(TARGET_DIR)/etc/inittab
+endef
+else
+# Find uncommented line, if any, and add a leading '#'
+define SYSTEM_REMOUNT_ROOT_INITTAB
+	$(SED) '/^[^#].*-o remount,rw \/$$/s~^~#~' $(TARGET_DIR)/etc/inittab
+endef
+endif
-- 
2.11.0

[Buildroot] [PATCH 08/20] system: with no init system, only allow custom skeleton

[Yann E. MORIN]

When there is no init system (i.e. a custom one), our bundled default
skeleton is most probably un-fit for that (non-)init system.

This will be the case when we introduce per init system skeletons. The
systemd skeleton will be unfit except for a systemd init; the sysv
skeleton will be unfit except for a sysv-like init system.

In case they are using no init system (really, a custom one), the user
should be responsible for providing their own, custom skeleton as well.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 system/Config.in | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/system/Config.in b/system/Config.in
index 828df4217e..ed539dcbe0 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -3,11 +3,17 @@ menu "System configuration"
 choice
 	prompt "Root FS skeleton"
 
+# When not using an init system (i.e. a custom one), our default skeleton
+# is most probably unfit; so, allow only the custom skeleton in that case.
 config BR2_ROOTFS_SKELETON_DEFAULT
 	bool "default target skeleton"
+	depends on !BR2_INIT_NONE
 	help
 	  Use default target skeleton
 
+comment "default target skeleton needs an init system"
+	depends on BR2_INIT_NONE
+
 config BR2_ROOTFS_SKELETON_CUSTOM
 	bool "custom target skeleton"
 	help
-- 
2.11.0

[Buildroot] [PATCH 09/20] package/skeleton: drop dependency on host-mkpasswd

[Yann E. MORIN]

Setting the root pasword is done in a target-finalize hook, so we do not
need to enforce a dependency from the skeleton onto host-mkpasswd.

Dropping that dependency will simplify making skeleton a virtual
package (in up-coming changes).

This however introduces a slight change in behaviour: previously,
host-mkpasswd would only be built if we needed to hash the root password
from its plain-text value. Now, host-mkpasswd is always built as soon as
the root password is non-empty, even if already pre-hashed.

Since host-mkpasswd is a really tiny weeny package bundled in Buidlroot,
with only two C files, built as a single unit with a single gcc call,
the overhead is really minimal. Compared to the simplifications this
will allow in the skeleton packages (plural: common, sysv, systemd,
custom) to come, this overhead is acceptable.

Yet another simplification, even if small, to ease providing multiple
skeletons.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/skeleton/skeleton.mk | 1 -
 system/Config.in             | 1 +
 2 files changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 4a66f6ceca..4ec1ecbb51 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -121,7 +121,6 @@ SKELETON_ROOT_PASSWORD =
 else ifneq ($(filter $$1$$% $$5$$% $$6$$%,$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)),)
 SKELETON_ROOT_PASSWORD = '$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)'
 else
-SKELETON_DEPENDENCIES += host-mkpasswd
 # This variable will only be evaluated in the finalize stage, so we can
 # be sure that host-mkpasswd will have already been built by that time.
 SKELETON_ROOT_PASSWORD = "`$(MKPASSWD) -m "$(SKELETON_TARGET_GENERIC_PASSWD_METHOD)" "$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)"`"
diff --git a/system/Config.in b/system/Config.in
index ed539dcbe0..5cb921f108 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -224,6 +224,7 @@ config BR2_ROOTFS_MERGED_USR
 config BR2_TARGET_ENABLE_ROOT_LOGIN
 	bool "Enable root login with password"
 	default y
+	select BR2_PACKAGE_HOST_MKPASSWD if BR2_TARGET_GENERIC_ROOT_PASSWD != ""
 	help
 	  Allow root to log in with a password.
 
-- 
2.11.0

[Buildroot] [PATCH 10/20] package/skeleton: select it rather than default to y

[Yann E. MORIN]

Our current skeleton is tailored to sysv-like init systems; it is not
fit for systemd-based systems. So, in upcoming changes, we'll add
another skeleton for systemd.

This means we can no longer have the current skeleton default to 'y', or
it would be enabled also for systemd, which would be incorrect.

So, we remove the default to 'y' but have it selected by the default
skeleton choice.

However, we do not yet have a way to directly build (really, install)
the custom skeleton, it is built (really, installed) as a dependency of
the default skeleton. So we must also forcibly select the default
skeleton when using a custom one.

Until we have the mean to do only one or the other; i.e. when we have a
virtual skeleton.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/skeleton/Config.in | 1 -
 system/Config.in           | 2 ++
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/package/skeleton/Config.in b/package/skeleton/Config.in
index d25147bd92..b22ac66b35 100644
--- a/package/skeleton/Config.in
+++ b/package/skeleton/Config.in
@@ -1,5 +1,4 @@
 config BR2_PACKAGE_SKELETON
 	bool
-	default y
 	help
 	  The basic skeleton for your rootfs.
diff --git a/system/Config.in b/system/Config.in
index 5cb921f108..9b42fbbbac 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -8,6 +8,7 @@ choice
 config BR2_ROOTFS_SKELETON_DEFAULT
 	bool "default target skeleton"
 	depends on !BR2_INIT_NONE
+	select BR2_PACKAGE_SKELETON
 	help
 	  Use default target skeleton
 
@@ -16,6 +17,7 @@ comment "default target skeleton needs an init system"
 
 config BR2_ROOTFS_SKELETON_CUSTOM
 	bool "custom target skeleton"
+	select BR2_PACKAGE_SKELETON
 	help
 	  Use custom target skeleton.
 
-- 
2.11.0

[Buildroot] [PATCH 11/20] package/skeleton: split out into skeleton-custom

[Yann E. MORIN]

For the custom skeleton, we practicaly do nothing, except ensure it
contains the basic, required directories, and that those are properly
setup wrt. merged /usr.

Furthermore, our current skeleton is not fit for systemd, and we'll
have to split things out into various skeletons.

So, off-load the custom skeleton into its own package.

Thus, the existing skeleton package is now limited to:

  - when using our default skeleton, install and tweak it properly;

  - when using a custom skeleton, do nothing except for depending on
    the skeleton-custom package.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

---
Note: this calls for making skeleton a virtual package. This will come
in follwup patches, after skeleton has lost all its code to the various
skeleton packages.
---
 package/Config.in                          |   1 +
 package/skeleton-custom/Config.in          |   3 +
 package/skeleton-custom/skeleton-custom.mk | 105 +++++++++++++++++++++++++++++
 package/skeleton/skeleton.mk               |  51 +-------------
 system/Config.in                           |   2 +-
 5 files changed, 112 insertions(+), 50 deletions(-)
 create mode 100644 package/skeleton-custom/Config.in
 create mode 100644 package/skeleton-custom/skeleton-custom.mk

diff --git a/package/Config.in b/package/Config.in
index b48ab5e0dd..5fb9560ba9 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -2,6 +2,7 @@ menu "Target packages"
 
 	source "package/busybox/Config.in"
 	source "package/skeleton/Config.in"
+	source "package/skeleton-custom/Config.in"
 
 menu "Audio and video applications"
 	source "package/alsa-utils/Config.in"
diff --git a/package/skeleton-custom/Config.in b/package/skeleton-custom/Config.in
new file mode 100644
index 0000000000..b12bd8f73c
--- /dev/null
+++ b/package/skeleton-custom/Config.in
@@ -0,0 +1,3 @@
+config BR2_PACKAGE_SKELETON_CUSTOM
+	bool
+	select BR2_PACKAGE_SKELETON
diff --git a/package/skeleton-custom/skeleton-custom.mk b/package/skeleton-custom/skeleton-custom.mk
new file mode 100644
index 0000000000..6f2cd2a42d
--- /dev/null
+++ b/package/skeleton-custom/skeleton-custom.mk
@@ -0,0 +1,105 @@
+################################################################################
+#
+# skeleton-custom
+#
+################################################################################
+
+SKELETON_CUSTOM_ADD_TOOLCHAIN_DEPENDENCY = NO
+SKELETON_CUSTOM_ADD_SKELETON_DEPENDENCY = NO
+
+SKELETON_CUSTOM_INSTALL_STAGING = YES
+
+SKELETON_CUSTOM_PATH = $(call qstrip,$(BR2_ROOTFS_SKELETON_CUSTOM_PATH))
+
+ifeq ($(BR2_PACKAGE_SKELETON_CUSTOM)$(BR_BUILDING),yy)
+ifeq ($(SKELETON_CUSTOM_PATH),)
+$(error No path specified for the custom skeleton)
+endif
+endif
+
+# Extract the inode numbers for all of those directories. In case any is
+# a symlink, we want to get the inode of the pointed-to directory, so we
+# append '/.' to be sure we get the target directory. Since the symlinks
+# can be anyway (/bin -> /usr/bin or /usr/bin -> /bin), we do that for
+# all of them.
+#
+SKELETON_CUSTOM_LIB_INODE = $(shell stat -c '%i' $(SKELETON_CUSTOM_PATH)/lib/. 2>/dev/null)
+SKELETON_CUSTOM_BIN_INODE = $(shell stat -c '%i' $(SKELETON_CUSTOM_PATH)/bin/. 2>/dev/null)
+SKELETON_CUSTOM_SBIN_INODE = $(shell stat -c '%i' $(SKELETON_CUSTOM_PATH)/sbin/. 2>/dev/null)
+SKELETON_CUSTOM_USR_LIB_INODE = $(shell stat -c '%i' $(SKELETON_CUSTOM_PATH)/usr/lib/. 2>/dev/null)
+SKELETON_CUSTOM_USR_BIN_INODE = $(shell stat -c '%i' $(SKELETON_CUSTOM_PATH)/usr/bin/. 2>/dev/null)
+SKELETON_CUSTOM_USR_SBIN_INODE = $(shell stat -c '%i' $(SKELETON_CUSTOM_PATH)/usr/sbin/. 2>/dev/null)
+
+# Ensure that the custom skeleton has /lib, /bin and /sbin, and their
+# /usr counterparts
+ifeq ($(SKELETON_CUSTOM_LIB_INODE),)
+SKELETON_CUSTOM_MISSING_DIRS += /lib
+endif
+ifeq ($(SKELETON_CUSTOM_USR_LIB_INODE),)
+SKELETON_CUSTOM_MISSING_DIRS += /usr/lib
+endif
+ifeq ($(SKELETON_CUSTOM_BIN_INODE),)
+SKELETON_CUSTOM_MISSING_DIRS += /bin
+endif
+ifeq ($(SKELETON_CUSTOM_USR_BIN_INODE),)
+SKELETON_CUSTOM_MISSING_DIRS += /usr/bin
+endif
+ifeq ($(SKELETON_CUSTOM_SBIN_INODE),)
+SKELETON_CUSTOM_MISSING_DIRS += /sbin
+endif
+ifeq ($(SKELETON_CUSTOM_USR_SBIN_INODE),)
+SKELETON_CUSTOM_MISSING_DIRS += /usr/sbin
+endif
+
+# For a merged /usr, ensure that /lib, /bin and /sbin and their /usr
+# counterparts are appropriately setup as symlinks ones to the others.
+ifeq ($(BR2_ROOTFS_MERGED_USR),y)
+
+ifneq ($(SKELETON_CUSTOM_LIB_INODE),$(SKELETON_CUSTOM_USR_LIB_INODE))
+SKELETON_CUSTOM_NOT_MERGED_USR_DIRS += /lib
+endif
+ifneq ($(SKELETON_CUSTOM_BIN_INODE),$(SKELETON_CUSTOM_USR_BIN_INODE))
+SKELETON_CUSTOM_NOT_MERGED_USR_DIRS += /bin
+endif
+ifneq ($(SKELETON_CUSTOM_SBIN_INODE),$(SKELETON_CUSTOM_USR_SBIN_INODE))
+SKELETON_CUSTOM_NOT_MERGED_USR_DIRS += /sbin
+endif
+
+endif # merged /usr
+
+ifeq ($(BR2_PACKAGE_SKELETON_CUSTOM)$(BR_BUILDING),yy)
+ifneq ($(SKELETON_CUSTOM_MISSING_DIRS),)
+$(error The custom skeleton in $(SKELETON_CUSTOM_PATH) is \
+	missing those directories or symlinks: \
+	$(SKELETON_CUSTOM_MISSING_DIRS))
+endif
+ifneq ($(SKELETON_CUSTOM_NOT_MERGED_USR_DIRS),)
+$(error The custom skeleton in $(SKELETON_PATH) is not \
+       using a merged /usr for the following directories: \
+       $(SKELETON_CUSTOM_NOT_MERGED_USR_DIRS))
+endif
+endif
+
+# Provided by the 'skeleton' package:
+# - SKELETON_RSYNC
+# - SKELETON_LIB_SYMLINK
+
+# The target-dir-warning file is the only thing we customise in the
+# custom skeleton.
+define SKELETON_CUSTOM_INSTALL_TARGET_CMDS
+	$(call SYSTEM_RSYNC,$(SKELETON_CUSTOM_PATH),$(TARGET_DIR))
+	$(call SYSTEM_LIB_SYMLINK,$(TARGET_DIR))
+	$(INSTALL) -m 0644 support/misc/target-dir-warning.txt \
+		$(TARGET_DIR_WARNING_FILE)
+endef
+
+# For the staging dir, we don't really care what we install, but we
+# need the /lib and /usr/lib appropriately setup. Since we ensure,
+# above, that they are correct in the skeleton, we can simply copy the
+# skeleton to staging.
+define SKELETON_CUSTOM_INSTALL_STAGING_CMDS
+	$(call SYSTEM_RSYNC,$(SKELETON_CUSTOM_PATH),$(STAGING_DIR))
+	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
+endef
+
+$(eval $(generic-package))
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 4ec1ecbb51..717bd65ada 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -14,57 +14,14 @@ SKELETON_ADD_SKELETON_DEPENDENCY = NO
 # The skeleton also handles the merged /usr case in the sysroot
 SKELETON_INSTALL_STAGING = YES
 
-ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
+ifeq ($(BR2_PACKAGE_SKELETON_CUSTOM),y)
 
-SKELETON_PATH = $(call qstrip,$(BR2_ROOTFS_SKELETON_CUSTOM_PATH))
-
-ifeq ($(BR_BUILDING),y)
-ifeq ($(SKELETON_PATH),)
-$(error No path specified for the custom skeleton)
-endif
-endif
-
-ifeq ($(BR2_ROOTFS_MERGED_USR),y)
-
-# Ensure the user has prepared a merged /usr.
-#
-# Extract the inode numbers for all of those directories. In case any is
-# a symlink, we want to get the inode of the pointed-to directory, so we
-# append '/.' to be sure we get the target directory. Since the symlinks
-# can be anyway (/bin -> /usr/bin or /usr/bin -> /bin), we do that for
-# all of them.
-#
-SKELETON_LIB_INODE = $(shell stat -c '%i' $(SKELETON_PATH)/lib/.)
-SKELETON_BIN_INODE = $(shell stat -c '%i' $(SKELETON_PATH)/bin/.)
-SKELETON_SBIN_INODE = $(shell stat -c '%i' $(SKELETON_PATH)/sbin/.)
-SKELETON_USR_LIB_INODE = $(shell stat -c '%i' $(SKELETON_PATH)/usr/lib/.)
-SKELETON_USR_BIN_INODE = $(shell stat -c '%i' $(SKELETON_PATH)/usr/bin/.)
-SKELETON_USR_SBIN_INODE = $(shell stat -c '%i' $(SKELETON_PATH)/usr/sbin/.)
-
-ifneq ($(SKELETON_LIB_INODE),$(SKELETON_USR_LIB_INODE))
-SKELETON_CUSTOM_NOT_MERGED_USR += /lib
-endif
-ifneq ($(SKELETON_BIN_INODE),$(SKELETON_USR_BIN_INODE))
-SKELETON_CUSTOM_NOT_MERGED_USR += /bin
-endif
-ifneq ($(SKELETON_SBIN_INODE),$(SKELETON_USR_SBIN_INODE))
-SKELETON_CUSTOM_NOT_MERGED_USR += /sbin
-endif
-
-ifneq ($(SKELETON_CUSTOM_NOT_MERGED_USR),)
-$(error The custom skeleton in $(SKELETON_PATH) is not \
-	using a merged /usr for the following directories: \
-	$(SKELETON_CUSTOM_NOT_MERGED_USR))
-endif
-
-endif # merged /usr
+SKELETON_DEPENDENCIES = skeleton-custom
 
 else # ! custom skeleton
 
 SKELETON_PATH = system/skeleton
 
-endif # ! custom skeleton
-
 define SKELETON_INSTALL_TARGET_CMDS
 	$(call SYSTEM_RSYNC,$(SKELETON_PATH),$(TARGET_DIR))
 	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
@@ -87,10 +44,6 @@ define SKELETON_INSTALL_STAGING_CMDS
 	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
 endef
 
-# The TARGET_FINALIZE_HOOKS must be sourced only if the users choose to use the
-# default skeleton.
-ifeq ($(BR2_ROOTFS_SKELETON_DEFAULT),y)
-
 SKELETON_TARGET_GENERIC_HOSTNAME = $(call qstrip,$(BR2_TARGET_GENERIC_HOSTNAME))
 SKELETON_TARGET_GENERIC_ISSUE = $(call qstrip,$(BR2_TARGET_GENERIC_ISSUE))
 SKELETON_TARGET_GENERIC_ROOT_PASSWD = $(call qstrip,$(BR2_TARGET_GENERIC_ROOT_PASSWD))
diff --git a/system/Config.in b/system/Config.in
index 9b42fbbbac..5f8770d6c9 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -17,7 +17,7 @@ comment "default target skeleton needs an init system"
 
 config BR2_ROOTFS_SKELETON_CUSTOM
 	bool "custom target skeleton"
-	select BR2_PACKAGE_SKELETON
+	select BR2_PACKAGE_SKELETON_CUSTOM
 	help
 	  Use custom target skeleton.
 
-- 
2.11.0

[Buildroot] [PATCH 12/20] package/skeleton: split out into skeleton-common

[Yann E. MORIN]

Move all the handling of the default skeleton into a new package,
skeleton-common.

We don;t name it skeleton-default, because it will be further split
later, into a skeleton for sysv and another for systemd, with some parts
still common between the two. So just name it skeleton-common right now;
this will save us a rename later.

We keep in the skeleton package all definitions of macros and variables
that can be used by other skeleton packages and/or init systems, and
just offload the actual feeding of the skeleton to the new package.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/Config.in                          |  1 +
 package/skeleton-common/Config.in          |  3 +
 package/skeleton-common/skeleton-common.mk | 96 ++++++++++++++++++++++++++++++
 package/skeleton/skeleton.mk               | 87 +--------------------------
 system/Config.in                           |  2 +-
 5 files changed, 102 insertions(+), 87 deletions(-)
 create mode 100644 package/skeleton-common/Config.in
 create mode 100644 package/skeleton-common/skeleton-common.mk

diff --git a/package/Config.in b/package/Config.in
index 5fb9560ba9..bb068b7de8 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -2,6 +2,7 @@ menu "Target packages"
 
 	source "package/busybox/Config.in"
 	source "package/skeleton/Config.in"
+	source "package/skeleton-common/Config.in"
 	source "package/skeleton-custom/Config.in"
 
 menu "Audio and video applications"
diff --git a/package/skeleton-common/Config.in b/package/skeleton-common/Config.in
new file mode 100644
index 0000000000..6c094f6466
--- /dev/null
+++ b/package/skeleton-common/Config.in
@@ -0,0 +1,3 @@
+config BR2_PACKAGE_SKELETON_COMMON
+	bool
+	select BR2_PACKAGE_SKELETON
diff --git a/package/skeleton-common/skeleton-common.mk b/package/skeleton-common/skeleton-common.mk
new file mode 100644
index 0000000000..788a17dd61
--- /dev/null
+++ b/package/skeleton-common/skeleton-common.mk
@@ -0,0 +1,96 @@
+################################################################################
+#
+# skeleton-common
+#
+################################################################################
+
+# The skeleton can't depend on the toolchain, since all packages depends on the
+# skeleton and the toolchain is a target package, as is skeleton.
+# Hence, skeleton would depends on the toolchain and the toolchain would depend
+# on skeleton.
+SKELETON_COMMON_ADD_TOOLCHAIN_DEPENDENCY = NO
+SKELETON_COMMON_ADD_SKELETON_DEPENDENCY = NO
+
+# The skeleton also handles the merged /usr case in the sysroot
+SKELETON_COMMON_INSTALL_STAGING = YES
+
+SKELETON_COMMON_PATH = system/skeleton
+
+define SKELETON_COMMON_INSTALL_TARGET_CMDS
+	$(call SYSTEM_RSYNC,$(SKELETON_COMMON_PATH),$(TARGET_DIR))
+	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
+	$(call SYSTEM_LIB_SYMLINK,$(TARGET_DIR))
+	$(INSTALL) -m 0644 support/misc/target-dir-warning.txt \
+		$(TARGET_DIR_WARNING_FILE)
+endef
+
+# For the staging dir, we don't really care about /bin and /sbin.
+# But for consistency with the target dir, and to simplify the code,
+# we still handle them for the merged or non-merged /usr cases.
+# Since the toolchain is not yet available, the staging is not yet
+# populated, so we need to create the directories in /usr
+define SKELETON_COMMON_INSTALL_STAGING_CMDS
+	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/lib
+	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/bin
+	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/sbin
+	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
+	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(STAGING_DIR))
+	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
+endef
+
+SKELETON_COMMON_HOSTNAME = $(call qstrip,$(BR2_TARGET_GENERIC_HOSTNAME))
+SKELETON_COMMON_ISSUE = $(call qstrip,$(BR2_TARGET_GENERIC_ISSUE))
+SKELETON_COMMON_ROOT_PASSWD = $(call qstrip,$(BR2_TARGET_GENERIC_ROOT_PASSWD))
+SKELETON_COMMON_PASSWD_METHOD = $(call qstrip,$(BR2_TARGET_GENERIC_PASSWD_METHOD))
+SKELETON_COMMON_BIN_SH = $(call qstrip,$(BR2_SYSTEM_BIN_SH))
+
+ifneq ($(SKELETON_COMMON_HOSTNAME),)
+define SKELETON_COMMON_SET_HOSTNAME
+	mkdir -p $(TARGET_DIR)/etc
+	echo "$(SKELETON_COMMON_HOSTNAME)" > $(TARGET_DIR)/etc/hostname
+	$(SED) '$$a \127.0.1.1\t$(SKELETON_COMMON_HOSTNAME)' \
+		-e '/^127.0.1.1/d' $(TARGET_DIR)/etc/hosts
+endef
+SKELETON_COMMON_TARGET_FINALIZE_HOOKS += SKELETON_COMMON_SET_HOSTNAME
+endif
+
+ifneq ($(SKELETON_COMMON_ISSUE),)
+define SKELETON_COMMON_SET_ISSUE
+	mkdir -p $(TARGET_DIR)/etc
+	echo "$(SKELETON_COMMON_ISSUE)" > $(TARGET_DIR)/etc/issue
+endef
+SKELETON_COMMON_TARGET_FINALIZE_HOOKS += SKELETON_COMMON_SET_ISSUE
+endif
+
+ifeq ($(BR2_TARGET_ENABLE_ROOT_LOGIN),y)
+ifeq ($(SKELETON_COMMON_ROOT_PASSWD),)
+SKELETON_ROOT_PASSWORD =
+else ifneq ($(filter $$1$$% $$5$$% $$6$$%,$(SKELETON_COMMON_ROOT_PASSWD)),)
+SKELETON_ROOT_PASSWORD = '$(SKELETON_COMMON_ROOT_PASSWD)'
+else
+# This variable will only be evaluated in the finalize stage, so we can
+# be sure that host-mkpasswd will have already been built by that time.
+SKELETON_ROOT_PASSWORD = "`$(MKPASSWD) -m "$(SKELETON_COMMON_PASSWD_METHOD)" "$(SKELETON_COMMON_ROOT_PASSWD)"`"
+endif
+else # !BR2_TARGET_ENABLE_ROOT_LOGIN
+SKELETON_ROOT_PASSWORD = "*"
+endif
+define SKELETON_COMMON_SET_ROOT_PASSWD
+	$(SED) s,^root:[^:]*:,root:$(SKELETON_ROOT_PASSWORD):, $(TARGET_DIR)/etc/shadow
+endef
+SKELETON_COMMON_TARGET_FINALIZE_HOOKS += SKELETON_COMMON_SET_ROOT_PASSWD
+
+ifeq ($(BR2_SYSTEM_BIN_SH_NONE),y)
+define SKELETON_COMMON_BIN_SH
+	rm -f $(TARGET_DIR)/bin/sh
+endef
+else
+ifneq ($(SKELETON_COMMON_BIN_SH),)
+define SKELETON_COMMON_BIN_SH
+	ln -sf $(SKELETON_COMMON_BIN_SH) $(TARGET_DIR)/bin/sh
+endef
+endif
+endif
+SKELETON_COMMON_TARGET_FINALIZE_HOOKS += SKELETON_COMMON_BIN_SH
+
+$(eval $(generic-package))
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 717bd65ada..1d3e4fbe3c 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -11,95 +11,10 @@
 SKELETON_ADD_TOOLCHAIN_DEPENDENCY = NO
 SKELETON_ADD_SKELETON_DEPENDENCY = NO
 
-# The skeleton also handles the merged /usr case in the sysroot
-SKELETON_INSTALL_STAGING = YES
-
 ifeq ($(BR2_PACKAGE_SKELETON_CUSTOM),y)
-
 SKELETON_DEPENDENCIES = skeleton-custom
-
-else # ! custom skeleton
-
-SKELETON_PATH = system/skeleton
-
-define SKELETON_INSTALL_TARGET_CMDS
-	$(call SYSTEM_RSYNC,$(SKELETON_PATH),$(TARGET_DIR))
-	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
-	$(call SYSTEM_LIB_SYMLINK,$(TARGET_DIR))
-	$(INSTALL) -m 0644 support/misc/target-dir-warning.txt \
-		$(TARGET_DIR_WARNING_FILE)
-endef
-
-# For the staging dir, we don't really care about /bin and /sbin.
-# But for consistency with the target dir, and to simplify the code,
-# we still handle them for the merged or non-merged /usr cases.
-# Since the toolchain is not yet available, the staging is not yet
-# populated, so we need to create the directories in /usr
-define SKELETON_INSTALL_STAGING_CMDS
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/lib
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/bin
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/sbin
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
-	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(STAGING_DIR))
-	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
-endef
-
-SKELETON_TARGET_GENERIC_HOSTNAME = $(call qstrip,$(BR2_TARGET_GENERIC_HOSTNAME))
-SKELETON_TARGET_GENERIC_ISSUE = $(call qstrip,$(BR2_TARGET_GENERIC_ISSUE))
-SKELETON_TARGET_GENERIC_ROOT_PASSWD = $(call qstrip,$(BR2_TARGET_GENERIC_ROOT_PASSWD))
-SKELETON_TARGET_GENERIC_PASSWD_METHOD = $(call qstrip,$(BR2_TARGET_GENERIC_PASSWD_METHOD))
-SKELETON_TARGET_GENERIC_BIN_SH = $(call qstrip,$(BR2_SYSTEM_BIN_SH))
-
-ifneq ($(SKELETON_TARGET_GENERIC_HOSTNAME),)
-define SKELETON_SET_HOSTNAME
-	mkdir -p $(TARGET_DIR)/etc
-	echo "$(SKELETON_TARGET_GENERIC_HOSTNAME)" > $(TARGET_DIR)/etc/hostname
-	$(SED) '$$a \127.0.1.1\t$(SKELETON_TARGET_GENERIC_HOSTNAME)' \
-		-e '/^127.0.1.1/d' $(TARGET_DIR)/etc/hosts
-endef
-TARGET_FINALIZE_HOOKS += SKELETON_SET_HOSTNAME
-endif
-
-ifneq ($(SKELETON_TARGET_GENERIC_ISSUE),)
-define SKELETON_SET_ISSUE
-	mkdir -p $(TARGET_DIR)/etc
-	echo "$(SKELETON_TARGET_GENERIC_ISSUE)" > $(TARGET_DIR)/etc/issue
-endef
-TARGET_FINALIZE_HOOKS += SKELETON_SET_ISSUE
-endif
-
-ifeq ($(BR2_TARGET_ENABLE_ROOT_LOGIN),y)
-ifeq ($(SKELETON_TARGET_GENERIC_ROOT_PASSWD),)
-SKELETON_ROOT_PASSWORD =
-else ifneq ($(filter $$1$$% $$5$$% $$6$$%,$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)),)
-SKELETON_ROOT_PASSWORD = '$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)'
-else
-# This variable will only be evaluated in the finalize stage, so we can
-# be sure that host-mkpasswd will have already been built by that time.
-SKELETON_ROOT_PASSWORD = "`$(MKPASSWD) -m "$(SKELETON_TARGET_GENERIC_PASSWD_METHOD)" "$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)"`"
-endif
-else # !BR2_TARGET_ENABLE_ROOT_LOGIN
-SKELETON_ROOT_PASSWORD = "*"
-endif
-
-define SKELETON_SET_ROOT_PASSWD
-	$(SED) s,^root:[^:]*:,root:$(SKELETON_ROOT_PASSWORD):, $(TARGET_DIR)/etc/shadow
-endef
-TARGET_FINALIZE_HOOKS += SKELETON_SET_ROOT_PASSWD
-
-ifeq ($(BR2_SYSTEM_BIN_SH_NONE),y)
-define SKELETON_BIN_SH
-	rm -f $(TARGET_DIR)/bin/sh
-endef
 else
-ifneq ($(SKELETON_TARGET_GENERIC_BIN_SH),)
-define SKELETON_BIN_SH
-	ln -sf $(SKELETON_TARGET_GENERIC_BIN_SH) $(TARGET_DIR)/bin/sh
-endef
-endif
+SKELETON_DEPENDENCIES = skeleton-common
 endif
-TARGET_FINALIZE_HOOKS += SKELETON_BIN_SH
-
-endif # BR2_ROOTFS_SKELETON_DEFAULT
 
 $(eval $(generic-package))
diff --git a/system/Config.in b/system/Config.in
index 5f8770d6c9..c27b013784 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -8,7 +8,7 @@ choice
 config BR2_ROOTFS_SKELETON_DEFAULT
 	bool "default target skeleton"
 	depends on !BR2_INIT_NONE
-	select BR2_PACKAGE_SKELETON
+	select BR2_PACKAGE_SKELETON_COMMON
 	help
 	  Use default target skeleton
 
-- 
2.11.0

[Buildroot] [PATCH 13/20] package/skeleton: make it a virtual package

[Yann E. MORIN]

We now have two packages that can act as a skeleton, skeleton-common,
also known as our default skeleton, and skeleton-custom.

This means that the skeleton package can be a standard virtual package
now.

Well, except that, besides being a virtual package, it also provides
variables and macros that can be used by the other skeletons and/or the
init systems.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/skeleton-common/Config.in          | 5 ++++-
 package/skeleton-common/skeleton-common.mk | 2 ++
 package/skeleton-custom/Config.in          | 5 ++++-
 package/skeleton-custom/skeleton-custom.mk | 2 ++
 package/skeleton/Config.in                 | 9 +++++++--
 package/skeleton/skeleton.mk               | 8 +-------
 system/Config.in                           | 4 ++++
 7 files changed, 24 insertions(+), 11 deletions(-)

diff --git a/package/skeleton-common/Config.in b/package/skeleton-common/Config.in
index 6c094f6466..5675e873cf 100644
--- a/package/skeleton-common/Config.in
+++ b/package/skeleton-common/Config.in
@@ -1,3 +1,6 @@
 config BR2_PACKAGE_SKELETON_COMMON
 	bool
-	select BR2_PACKAGE_SKELETON
+	select BR2_PACKAGE_HAS_SKELETON
+
+config BR2_PACKAGE_PROVIDES_SKELETON
+	default "skeleton-common" if BR2_PACKAGE_SKELETON_COMMON
diff --git a/package/skeleton-common/skeleton-common.mk b/package/skeleton-common/skeleton-common.mk
index 788a17dd61..ee44eaf810 100644
--- a/package/skeleton-common/skeleton-common.mk
+++ b/package/skeleton-common/skeleton-common.mk
@@ -11,6 +11,8 @@
 SKELETON_COMMON_ADD_TOOLCHAIN_DEPENDENCY = NO
 SKELETON_COMMON_ADD_SKELETON_DEPENDENCY = NO
 
+SKELETON_COMMON_PROVIDES = skeleton
+
 # The skeleton also handles the merged /usr case in the sysroot
 SKELETON_COMMON_INSTALL_STAGING = YES
 
diff --git a/package/skeleton-custom/Config.in b/package/skeleton-custom/Config.in
index b12bd8f73c..601c3b247e 100644
--- a/package/skeleton-custom/Config.in
+++ b/package/skeleton-custom/Config.in
@@ -1,3 +1,6 @@
 config BR2_PACKAGE_SKELETON_CUSTOM
 	bool
-	select BR2_PACKAGE_SKELETON
+	select BR2_PACKAGE_HAS_SKELETON
+
+config BR2_PACKAGE_PROVIDES_SKELETON
+	default "skeleton-custom" if BR2_PACKAGE_SKELETON_CUSTOM
diff --git a/package/skeleton-custom/skeleton-custom.mk b/package/skeleton-custom/skeleton-custom.mk
index 6f2cd2a42d..9b51fe93c1 100644
--- a/package/skeleton-custom/skeleton-custom.mk
+++ b/package/skeleton-custom/skeleton-custom.mk
@@ -7,6 +7,8 @@
 SKELETON_CUSTOM_ADD_TOOLCHAIN_DEPENDENCY = NO
 SKELETON_CUSTOM_ADD_SKELETON_DEPENDENCY = NO
 
+SKELETON_CUSTOM_PROVIDES = skeleton
+
 SKELETON_CUSTOM_INSTALL_STAGING = YES
 
 SKELETON_CUSTOM_PATH = $(call qstrip,$(BR2_ROOTFS_SKELETON_CUSTOM_PATH))
diff --git a/package/skeleton/Config.in b/package/skeleton/Config.in
index b22ac66b35..efaa1e135f 100644
--- a/package/skeleton/Config.in
+++ b/package/skeleton/Config.in
@@ -1,4 +1,9 @@
 config BR2_PACKAGE_SKELETON
 	bool
-	help
-	  The basic skeleton for your rootfs.
+	default y
+
+config BR2_PACKAGE_HAS_SKELETON
+	bool
+
+config BR2_PACKAGE_PROVIDES_SKELETON
+	string
diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
index 1d3e4fbe3c..d380f41649 100644
--- a/package/skeleton/skeleton.mk
+++ b/package/skeleton/skeleton.mk
@@ -11,10 +11,4 @@
 SKELETON_ADD_TOOLCHAIN_DEPENDENCY = NO
 SKELETON_ADD_SKELETON_DEPENDENCY = NO
 
-ifeq ($(BR2_PACKAGE_SKELETON_CUSTOM),y)
-SKELETON_DEPENDENCIES = skeleton-custom
-else
-SKELETON_DEPENDENCIES = skeleton-common
-endif
-
-$(eval $(generic-package))
+$(eval $(virtual-package))
diff --git a/system/Config.in b/system/Config.in
index c27b013784..75b6a8edac 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -1,5 +1,9 @@
 menu "System configuration"
 
+# Note: usually, it is not possible to select a provider of a virtual
+# package. But here we have an exception: there are only two providers
+# and they only get selected each by separate entries in this choice.
+# So this is a safe situation.
 choice
 	prompt "Root FS skeleton"
 
-- 
2.11.0

[Buildroot] [PATCH 14/20] package/skeleton-common: simplify staging install

[Yann E. MORIN]

We don't really care what's going in staging, as long as it is properly
setup for merged/non-merged /usr, especially for the lib/ directory.

So we can just copy the skeleton as-is.

This simplify the maintenance, should we ever need to tweak the layout:
we'd just have to do it once in the skeleton directory to have it
propagated to both target and staging.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/skeleton-common/skeleton-common.mk | 14 +++++---------
 1 file changed, 5 insertions(+), 9 deletions(-)

diff --git a/package/skeleton-common/skeleton-common.mk b/package/skeleton-common/skeleton-common.mk
index ee44eaf810..475c07e32d 100644
--- a/package/skeleton-common/skeleton-common.mk
+++ b/package/skeleton-common/skeleton-common.mk
@@ -26,18 +26,14 @@ define SKELETON_COMMON_INSTALL_TARGET_CMDS
 		$(TARGET_DIR_WARNING_FILE)
 endef
 
-# For the staging dir, we don't really care about /bin and /sbin.
-# But for consistency with the target dir, and to simplify the code,
-# we still handle them for the merged or non-merged /usr cases.
-# Since the toolchain is not yet available, the staging is not yet
-# populated, so we need to create the directories in /usr
+# We don't care much about what goes in staging, as long as it is
+# correctly setup for merged/non-merged /usr. The simplest is to
+# fill it in with the content of the skeleton.
 define SKELETON_COMMON_INSTALL_STAGING_CMDS
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/lib
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/bin
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/sbin
-	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
+	$(call SYSTEM_RSYNC,$(SKELETON_COMMON_PATH),$(STAGING_DIR))
 	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(STAGING_DIR))
 	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
+	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
 endef
 
 SKELETON_COMMON_HOSTNAME = $(call qstrip,$(BR2_TARGET_GENERIC_HOSTNAME))
-- 
2.11.0

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Yann E. MORIN]

Hello All!

This series is a proposal to fix our handling of systemd as an init
system.

Our default skeleton is not well suited for systemd:

  - we have /var/log a symlink to /tmp/log, but the way systemd starts
    hides the journals:
    - start systemd-journald, stores journals in /tmp/log/ (because of
      the redirection above)
    - mounts a tmpfs over /tmp, thus hidding the journals.

  - on a read-only rootfs, systemd expects /var to be read-write, which
    we do not provide.

All of this sounds trivial, but fixing it is definitely not.

The overall idea is that we need different skeletons, one for each type
of init systems: sysv, systemd, others (aka custom).

The skeleton package becomes a virtual package, that is provided by one
of four implementations:

  - skeleton-sysv or skeleton-systemd, as their names imply, for either
    SysV- or systemd-based systems, and which depend on skeleton-common;

  - skeleton-common, that provides base files and directories comon to
    all skeletons (except custom, below); skeleton-common is also a
    provider for the skeleton virtual package, for use with no-init (aka
    custom init) systems;

  - skeleton-custom, for user-provided custom skeleton, and of which we
    expect it is fully prepared; skeleton-custom does not depend on
    skeleton-common.

A series of runtime tests have also been added, running on the QEMU
vexpress:

    init system         rootfs      DHCP?
    -------------------------------------
    busybox             ext4        no
    busybox             ext4        eth0
    busybox             squashfs    no
    busybox             squashfs    eth0
  * systemd-ifupdown    ext4        eth0
    systemd-ifupdown    squashfs    eth0
 ** systemd-networkd    ext4        eth0
    systemd-networkd    squashfs    eth0
*** systemd-full        ext4        eth0
    systemd-full        squashfs    eth0
  + no-init             squashfs    no

    *   systemd-networkd: only what gets selected automatically when
        systemd is used as init system, with all other options unset;

    **  systemd-ifupdown: like systemd-networkd, but with networkd
        disabled, thus using ifupdown instead;

    *** systemd-full: all systemd options enabled, implies networkd;

    +   a shell is launched as init

Those new tests are not exhaustive, especially the systemd one; instead,
they just test basic features: what init is being used, and did we get
an IP adress for eth0? More tests can be added later on.

Changes v2 -> v3:
  - too much to note, mostly rewriten from scratch

Changes v1 -> v2:
  - enhance the commit logs


Regards,
Yann E. MORIN.


The following changes since commit bc801efacb298e2297024f83e960eaf8a45dd39f

  python-coherence: bump to the latest upstream commit (2017-07-17 11:28:58 +0200)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 95f6965c48dbf2f754818e2bec1f2d3891015475

  support/testing: add runtime testing for init systems (2017-07-18 18:59:05 +0200)


----------------------------------------------------------------
Yann E. MORIN (20):
      support/tests: allow properly indented config fragment
      core/pkg-generic: add variable to skip skeleton dependency
      package/skeleton: add macro to rsync skeleton directory
      package/skeleton: make SKELETON_LIB_SYMLINK a macro
      system: provide package-wide system variables and macros
      system: move setting getty to the corresponding init systems
      system: move remounting / to the corresponding init systems
      system: with no init system, only allow custom skeleton
      package/skeleton: drop dependency on host-mkpasswd
      package/skeleton: select it rather than default to y
      package/skeleton: split out into skeleton-custom
      package/skeleton: split out into skeleton-common
      package/skeleton: make it a virtual package
      package/skeleton-common: simplify staging install
      package/skeleton: introduce sysv- and systemd-specific skeletons
      system: separate sysv and systemd parts of the skeleton
      sytem: no-init systems may use our default, common skeleton
      fs: add pre- and post-command hooks
      system: make systemd work on a read-only rootfs
      support/testing: add runtime testing for init systems

 Makefile                                           |   1 +
 fs/common.mk                                       |   4 +
 package/Config.in                                  |   4 +
 package/busybox/busybox.mk                         |  15 +-
 package/pkg-generic.mk                             |  11 +-
 package/skeleton-common/Config.in                  |  31 ++++
 package/skeleton-common/skeleton-common.mk         |  92 +++++++++
 package/skeleton-custom/Config.in                  |   6 +
 package/skeleton-custom/skeleton-custom.mk         | 107 +++++++++++
 package/skeleton-systemd/Config.in                 |   7 +
 package/skeleton-systemd/skeleton-systemd.mk       |  71 +++++++
 package/skeleton-sysv/Config.in                    |   7 +
 package/skeleton-sysv/skeleton-sysv.mk             |  22 +++
 {system => package/skeleton-sysv}/skeleton/dev/log |   0
 .../skeleton-sysv}/skeleton/dev/pts/.empty         |   0
 .../skeleton-sysv}/skeleton/dev/shm/.empty         |   0
 .../skeleton-sysv}/skeleton/etc/fstab              |   0
 .../skeleton-sysv}/skeleton/var/cache              |   0
 .../skeleton-sysv}/skeleton/var/lib/misc           |   0
 .../skeleton-sysv}/skeleton/var/lock               |   0
 {system => package/skeleton-sysv}/skeleton/var/log |   0
 {system => package/skeleton-sysv}/skeleton/var/run |   0
 .../skeleton-sysv}/skeleton/var/spool              |   0
 {system => package/skeleton-sysv}/skeleton/var/tmp |   0
 package/skeleton/Config.in                         |   8 +-
 package/skeleton/skeleton.mk                       | 206 +--------------------
 package/sysvinit/sysvinit.mk                       |  11 ++
 support/testing/infra/builder.py                   |   3 +-
 .../testing/tests/init/__init__.py                 |   0
 support/testing/tests/init/base.py                 |  47 +++++
 support/testing/tests/init/test_busybox.py         |  67 +++++++
 support/testing/tests/init/test_none.py            |  32 ++++
 support/testing/tests/init/test_systemd.py         | 145 +++++++++++++++
 system/Config.in                                   |  13 +-
 system/skeleton/dev/{pts => }/.empty               |   0
 system/system.mk                                   |  88 +++++++++
 36 files changed, 787 insertions(+), 211 deletions(-)
 create mode 100644 package/skeleton-common/Config.in
 create mode 100644 package/skeleton-common/skeleton-common.mk
 create mode 100644 package/skeleton-custom/Config.in
 create mode 100644 package/skeleton-custom/skeleton-custom.mk
 create mode 100644 package/skeleton-systemd/Config.in
 create mode 100644 package/skeleton-systemd/skeleton-systemd.mk
 create mode 100644 package/skeleton-sysv/Config.in
 create mode 100644 package/skeleton-sysv/skeleton-sysv.mk
 rename {system => package/skeleton-sysv}/skeleton/dev/log (100%)
 copy {system => package/skeleton-sysv}/skeleton/dev/pts/.empty (100%)
 rename {system => package/skeleton-sysv}/skeleton/dev/shm/.empty (100%)
 rename {system => package/skeleton-sysv}/skeleton/etc/fstab (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/cache (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/lib/misc (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/lock (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/log (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/run (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/spool (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/tmp (100%)
 copy system/skeleton/dev/pts/.empty => support/testing/tests/init/__init__.py (100%)
 create mode 100644 support/testing/tests/init/base.py
 create mode 100644 support/testing/tests/init/test_busybox.py
 create mode 100644 support/testing/tests/init/test_none.py
 create mode 100644 support/testing/tests/init/test_systemd.py
 rename system/skeleton/dev/{pts => }/.empty (100%)
 create mode 100644 system/system.mk

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 15/20] package/skeleton: introduce sysv- and systemd-specific skeletons

[Yann E. MORIN]

Currently, we use the same skeleton for sysv-like init systems and
systemd, even though systemd has some peculiarities that makes out
default skeleton unfit.

So, we'll need to provide different skeletons (really, only part of it)
for sysv-like and systemd.

Introduce two new skeleton packages, aptly named skeleton-sysv and
skeleton-systemd, one for each, that are both providers of the skeleton
package, in lieue of the skeleton-common, which is now a simple
dependency of both the two new skeletons.

Those packages are empty for now. In followup changes:
  - sysv-specific stuff will be moved out of skeleton-common and into
    skeleton-sysv;
  - systemd-specific stuff will be added to skeleton-systemd.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/Config.in                            |  2 ++
 package/skeleton-common/Config.in            |  4 ----
 package/skeleton-common/skeleton-common.mk   |  2 --
 package/skeleton-systemd/Config.in           |  7 +++++++
 package/skeleton-systemd/skeleton-systemd.mk | 18 ++++++++++++++++++
 package/skeleton-sysv/Config.in              |  7 +++++++
 package/skeleton-sysv/skeleton-sysv.mk       | 18 ++++++++++++++++++
 system/Config.in                             | 10 ++++++----
 8 files changed, 58 insertions(+), 10 deletions(-)
 create mode 100644 package/skeleton-systemd/Config.in
 create mode 100644 package/skeleton-systemd/skeleton-systemd.mk
 create mode 100644 package/skeleton-sysv/Config.in
 create mode 100644 package/skeleton-sysv/skeleton-sysv.mk

diff --git a/package/Config.in b/package/Config.in
index bb068b7de8..d830cf743f 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -4,6 +4,8 @@ menu "Target packages"
 	source "package/skeleton/Config.in"
 	source "package/skeleton-common/Config.in"
 	source "package/skeleton-custom/Config.in"
+	source "package/skeleton-systemd/Config.in"
+	source "package/skeleton-sysv/Config.in"
 
 menu "Audio and video applications"
 	source "package/alsa-utils/Config.in"
diff --git a/package/skeleton-common/Config.in b/package/skeleton-common/Config.in
index 5675e873cf..9abed4292a 100644
--- a/package/skeleton-common/Config.in
+++ b/package/skeleton-common/Config.in
@@ -1,6 +1,2 @@
 config BR2_PACKAGE_SKELETON_COMMON
 	bool
-	select BR2_PACKAGE_HAS_SKELETON
-
-config BR2_PACKAGE_PROVIDES_SKELETON
-	default "skeleton-common" if BR2_PACKAGE_SKELETON_COMMON
diff --git a/package/skeleton-common/skeleton-common.mk b/package/skeleton-common/skeleton-common.mk
index 475c07e32d..57cee48931 100644
--- a/package/skeleton-common/skeleton-common.mk
+++ b/package/skeleton-common/skeleton-common.mk
@@ -11,8 +11,6 @@
 SKELETON_COMMON_ADD_TOOLCHAIN_DEPENDENCY = NO
 SKELETON_COMMON_ADD_SKELETON_DEPENDENCY = NO
 
-SKELETON_COMMON_PROVIDES = skeleton
-
 # The skeleton also handles the merged /usr case in the sysroot
 SKELETON_COMMON_INSTALL_STAGING = YES
 
diff --git a/package/skeleton-systemd/Config.in b/package/skeleton-systemd/Config.in
new file mode 100644
index 0000000000..c507264598
--- /dev/null
+++ b/package/skeleton-systemd/Config.in
@@ -0,0 +1,7 @@
+config BR2_PACKAGE_SKELETON_SYSTEMD
+	bool
+	select BR2_PACKAGE_HAS_SKELETON
+	select BR2_PACKAGE_SKELETON_COMMON
+
+config BR2_PACKAGE_PROVIDES_SKELETON
+	default "skeleton-systemd" if BR2_PACKAGE_SKELETON_SYSTEMD
diff --git a/package/skeleton-systemd/skeleton-systemd.mk b/package/skeleton-systemd/skeleton-systemd.mk
new file mode 100644
index 0000000000..cec6359007
--- /dev/null
+++ b/package/skeleton-systemd/skeleton-systemd.mk
@@ -0,0 +1,18 @@
+################################################################################
+#
+# skeleton-systemd
+#
+################################################################################
+
+# The skeleton can't depend on the toolchain, since all packages depends on the
+# skeleton and the toolchain is a target package, as is skeleton.
+# Hence, skeleton would depends on the toolchain and the toolchain would depend
+# on skeleton.
+SKELETON_SYSTEMD_ADD_TOOLCHAIN_DEPENDENCY = NO
+SKELETON_SYSTEMD_ADD_SKELETON_DEPENDENCY = NO
+
+SKELETON_SYSTEMD_DEPENDENCIES = skeleton-common
+
+SKELETON_SYSTEMD_PROVIDES = skeleton
+
+$(eval $(generic-package))
diff --git a/package/skeleton-sysv/Config.in b/package/skeleton-sysv/Config.in
new file mode 100644
index 0000000000..2f6dbd9673
--- /dev/null
+++ b/package/skeleton-sysv/Config.in
@@ -0,0 +1,7 @@
+config BR2_PACKAGE_SKELETON_SYSV
+	bool
+	select BR2_PACKAGE_HAS_SKELETON
+	select BR2_PACKAGE_SKELETON_COMMON
+
+config BR2_PACKAGE_PROVIDES_SKELETON
+	default "skeleton-sysv" if BR2_PACKAGE_SKELETON_SYSV
diff --git a/package/skeleton-sysv/skeleton-sysv.mk b/package/skeleton-sysv/skeleton-sysv.mk
new file mode 100644
index 0000000000..b0c2b6bac1
--- /dev/null
+++ b/package/skeleton-sysv/skeleton-sysv.mk
@@ -0,0 +1,18 @@
+################################################################################
+#
+# skeleton-sysv
+#
+################################################################################
+
+# The skeleton can't depend on the toolchain, since all packages depends on the
+# skeleton and the toolchain is a target package, as is skeleton.
+# Hence, skeleton would depends on the toolchain and the toolchain would depend
+# on skeleton.
+SKELETON_SYSV_ADD_TOOLCHAIN_DEPENDENCY = NO
+SKELETON_SYSV_ADD_SKELETON_DEPENDENCY = NO
+
+SKELETON_SYSV_DEPENDENCIES = skeleton-common
+
+SKELETON_SYSV_PROVIDES = skeleton
+
+$(eval $(generic-package))
diff --git a/system/Config.in b/system/Config.in
index 75b6a8edac..65139c0bfd 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -1,9 +1,9 @@
 menu "System configuration"
 
 # Note: usually, it is not possible to select a provider of a virtual
-# package. But here we have an exception: there are only two providers
-# and they only get selected each by separate entries in this choice.
-# So this is a safe situation.
+# package. But here we have an exception: there are only three providers
+# and they only get selected by separate entries in this choice and
+# under different, exclusive conditions. So this is a safe situation.
 choice
 	prompt "Root FS skeleton"
 
@@ -12,7 +12,9 @@ choice
 config BR2_ROOTFS_SKELETON_DEFAULT
 	bool "default target skeleton"
 	depends on !BR2_INIT_NONE
-	select BR2_PACKAGE_SKELETON_COMMON
+	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_SYSV
+	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_BUSYBOX
+	select BR2_PACKAGE_SKELETON_SYSTEMD if BR2_INIT_SYSTEMD
 	help
 	  Use default target skeleton
 
-- 
2.11.0

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Yann E. MORIN]

systems does not like that the content of /var be symlinks to /tmp,
especially journald that starts before /tmp is mounted, and thus the
journal files are hidden from view, which causes wuite a bit if fuss...

Instead, move the current /var to a sysv-only skeleton.

systemd at install time will create the /var content it needs, so we
just create an empty /var for systemd. Yet, systemd really requires a
bunch of extra directories to exist, so we also create those.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>

---
Note: /home and /srv will be created at runtime if missing, but it is
better to create them right now, to simplify supporting systemd on a
RO filesystem in the (near) future.
---
 package/skeleton-systemd/skeleton-systemd.mk              | 7 +++++++
 package/skeleton-sysv/skeleton-sysv.mk                    | 4 ++++
 {system => package/skeleton-sysv}/skeleton/dev/log        | 0
 {system => package/skeleton-sysv}/skeleton/dev/pts/.empty | 0
 {system => package/skeleton-sysv}/skeleton/dev/shm/.empty | 0
 {system => package/skeleton-sysv}/skeleton/etc/fstab      | 0
 {system => package/skeleton-sysv}/skeleton/var/cache      | 0
 {system => package/skeleton-sysv}/skeleton/var/lib/misc   | 0
 {system => package/skeleton-sysv}/skeleton/var/lock       | 0
 {system => package/skeleton-sysv}/skeleton/var/log        | 0
 {system => package/skeleton-sysv}/skeleton/var/run        | 0
 {system => package/skeleton-sysv}/skeleton/var/spool      | 0
 {system => package/skeleton-sysv}/skeleton/var/tmp        | 0
 system/skeleton/dev/.empty                                | 0
 14 files changed, 11 insertions(+)
 rename {system => package/skeleton-sysv}/skeleton/dev/log (100%)
 rename {system => package/skeleton-sysv}/skeleton/dev/pts/.empty (100%)
 rename {system => package/skeleton-sysv}/skeleton/dev/shm/.empty (100%)
 rename {system => package/skeleton-sysv}/skeleton/etc/fstab (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/cache (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/lib/misc (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/lock (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/log (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/run (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/spool (100%)
 rename {system => package/skeleton-sysv}/skeleton/var/tmp (100%)
 create mode 100644 system/skeleton/dev/.empty

diff --git a/package/skeleton-systemd/skeleton-systemd.mk b/package/skeleton-systemd/skeleton-systemd.mk
index cec6359007..384715e1c9 100644
--- a/package/skeleton-systemd/skeleton-systemd.mk
+++ b/package/skeleton-systemd/skeleton-systemd.mk
@@ -15,4 +15,11 @@ SKELETON_SYSTEMD_DEPENDENCIES = skeleton-common
 
 SKELETON_SYSTEMD_PROVIDES = skeleton
 
+define SKELETON_SYSTEMD_INSTALL_TARGET_CMDS
+	mkdir -p $(TARGET_DIR)/home
+	mkdir -p $(TARGET_DIR)/srv
+	mkdir -p $(TARGET_DIR)/var
+	echo "/dev/root / auto rw 0 1" >$(TARGET_DIR)/etc/fstab
+endef
+
 $(eval $(generic-package))
diff --git a/package/skeleton-sysv/skeleton-sysv.mk b/package/skeleton-sysv/skeleton-sysv.mk
index b0c2b6bac1..b5fa396ac2 100644
--- a/package/skeleton-sysv/skeleton-sysv.mk
+++ b/package/skeleton-sysv/skeleton-sysv.mk
@@ -15,4 +15,8 @@ SKELETON_SYSV_DEPENDENCIES = skeleton-common
 
 SKELETON_SYSV_PROVIDES = skeleton
 
+define SKELETON_SYSV_INSTALL_TARGET_CMDS
+	$(call SYSTEM_RSYNC,$(SKELETON_SYSV_PKGDIR)/skeleton,$(TARGET_DIR))
+endef
+
 $(eval $(generic-package))
diff --git a/system/skeleton/dev/log b/package/skeleton-sysv/skeleton/dev/log
similarity index 100%
rename from system/skeleton/dev/log
rename to package/skeleton-sysv/skeleton/dev/log
diff --git a/system/skeleton/dev/pts/.empty b/package/skeleton-sysv/skeleton/dev/pts/.empty
similarity index 100%
rename from system/skeleton/dev/pts/.empty
rename to package/skeleton-sysv/skeleton/dev/pts/.empty
diff --git a/system/skeleton/dev/shm/.empty b/package/skeleton-sysv/skeleton/dev/shm/.empty
similarity index 100%
rename from system/skeleton/dev/shm/.empty
rename to package/skeleton-sysv/skeleton/dev/shm/.empty
diff --git a/system/skeleton/etc/fstab b/package/skeleton-sysv/skeleton/etc/fstab
similarity index 100%
rename from system/skeleton/etc/fstab
rename to package/skeleton-sysv/skeleton/etc/fstab
diff --git a/system/skeleton/var/cache b/package/skeleton-sysv/skeleton/var/cache
similarity index 100%
rename from system/skeleton/var/cache
rename to package/skeleton-sysv/skeleton/var/cache
diff --git a/system/skeleton/var/lib/misc b/package/skeleton-sysv/skeleton/var/lib/misc
similarity index 100%
rename from system/skeleton/var/lib/misc
rename to package/skeleton-sysv/skeleton/var/lib/misc
diff --git a/system/skeleton/var/lock b/package/skeleton-sysv/skeleton/var/lock
similarity index 100%
rename from system/skeleton/var/lock
rename to package/skeleton-sysv/skeleton/var/lock
diff --git a/system/skeleton/var/log b/package/skeleton-sysv/skeleton/var/log
similarity index 100%
rename from system/skeleton/var/log
rename to package/skeleton-sysv/skeleton/var/log
diff --git a/system/skeleton/var/run b/package/skeleton-sysv/skeleton/var/run
similarity index 100%
rename from system/skeleton/var/run
rename to package/skeleton-sysv/skeleton/var/run
diff --git a/system/skeleton/var/spool b/package/skeleton-sysv/skeleton/var/spool
similarity index 100%
rename from system/skeleton/var/spool
rename to package/skeleton-sysv/skeleton/var/spool
diff --git a/system/skeleton/var/tmp b/package/skeleton-sysv/skeleton/var/tmp
similarity index 100%
rename from system/skeleton/var/tmp
rename to package/skeleton-sysv/skeleton/var/tmp
diff --git a/system/skeleton/dev/.empty b/system/skeleton/dev/.empty
new file mode 100644
index 0000000000..e69de29bb2
-- 
2.11.0

[Buildroot] [PATCH 17/20] sytem: no-init systems may use our default, common skeleton

[Yann E. MORIN]

Requiring that no-init system indeed provide their own custom skeletons
is a bit too-high a burden, especially for quick one-off tests when
testing our packages.

Allow no-init systems to use our default skeleton, but only use the
common part, for the specific sysv or systemd ones are unfit.

Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 package/skeleton-common/Config.in | 29 +++++++++++++++++++++++++++++
 system/Config.in                  |  5 +----
 2 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/package/skeleton-common/Config.in b/package/skeleton-common/Config.in
index 9abed4292a..ba0abe9d35 100644
--- a/package/skeleton-common/Config.in
+++ b/package/skeleton-common/Config.in
@@ -1,2 +1,31 @@
+# Normally, skeleton-common is a dependency of either the sysv or
+# the systemd ones, so it should not declare itself as a provider
+# for the skeleton virtual package.
+#
+# However, when one uses a no-init system (BR2_INIT_NONE), neither
+# the sysv not the systemd skeleton are enabled, so skeleton-common
+# is the one and only skeleton available, so it must declare itself
+# as a provider.
+#
+# We can safely select _HAS_SKELETON, because either we are the only
+# provider (in which case we must select it) or we were selected by
+# another skeleton which would have selected it by itself (in which
+# case us selecting _HAS_SKELETON is harmless).
+#
 config BR2_PACKAGE_SKELETON_COMMON
 	bool
+	select BR2_PACKAGE_HAS_SKELETON
+
+if BR2_PACKAGE_SKELETON_COMMON
+
+config BR2_PACKAGE_SKELETON_COMMON_ONLY
+	bool
+	default y
+	depends on !BR2_PACKAGE_SKELETON_SYSV
+	depends on !BR2_PACKAGE_SKELETON_SYSTEMD
+	depends on !BR2_PACKAGE_SKELETON_CUSTOM
+
+config BR2_PACKAGE_PROVIDES_SKELETON
+	default "skeleton-common" if BR2_PACKAGE_SKELETON_COMMON_ONLY
+
+endif
diff --git a/system/Config.in b/system/Config.in
index 65139c0bfd..4d417a93b3 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -11,16 +11,13 @@ choice
 # is most probably unfit; so, allow only the custom skeleton in that case.
 config BR2_ROOTFS_SKELETON_DEFAULT
 	bool "default target skeleton"
-	depends on !BR2_INIT_NONE
 	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_SYSV
 	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_BUSYBOX
 	select BR2_PACKAGE_SKELETON_SYSTEMD if BR2_INIT_SYSTEMD
+	select BR2_PACKAGE_SKELETON_COMMON if BR2_INIT_NONE
 	help
 	  Use default target skeleton
 
-comment "default target skeleton needs an init system"
-	depends on BR2_INIT_NONE
-
 config BR2_ROOTFS_SKELETON_CUSTOM
 	bool "custom target skeleton"
 	select BR2_PACKAGE_SKELETON_CUSTOM
-- 
2.11.0

[Buildroot] [PATCH 18/20] fs: add pre- and post-command hooks

[Yann E. MORIN]

In some cases, the directory structure we want in the filesystem is not
exactly what we have in target/

For example, when systemd is used on a read-only rootfs, /var must be a
tmpfs. However, we may have packages that install stuff in there, and
set important rights (via the permission-table). So, at build time, we
need /var to be a symlink to the remanent location (/usr/share/factory)
while at runtime we need /var to be a directory.

One option would have seen to have /var as a real directory even during
build time, and in a target-finalize hook, move everything out of there
and into the "factory" location. However, that's not possible because
it's too early: some packages may want to set ownership and/or acces
rights on directories or files in /var, and this is only done in the
fakeroot script, which is called only later during the assembling of the
filesystem images.

Also, there would have been no way to undo the tweak (i.e. we need to
restore the /var symlink so that subsequent builds continue to work) if
it were done as a target-finalize hook.

The only solution is to allow packages to register pre- and post-hooks
that are called right before and right after the rootfs commands are
executed, and inside in the fakeroot script.

We can however not re-use the BR2_ROOTFS_POST_FAKEROOT_SCRIPT feature
either because it is done before the filesystem command, but there is
nothing that is done after. Also, we don't want to add to, and modify a
user-supplied variable.

So, we introduce two new variables that packages can set to add the
commands they need to run to tweak the filesystem right at the last
moment.

Those hooks are not documented on-purpose; they are probably going to
only ever be used by systemd.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>

---
Changes v2 -> v3:
  - add space between >> and $$   (Romain)
---
 fs/common.mk           | 4 ++++
 package/pkg-generic.mk | 4 ++++
 2 files changed, 8 insertions(+)

diff --git a/fs/common.mk b/fs/common.mk
index 14e0a6b868..9a7758ff49 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -95,10 +95,14 @@ endif
 	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
 		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
 		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
+	$$(foreach hook,$$(ROOTFS_PRE_CMD_HOOKS),\
+		$$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT))
 ifeq ($$(BR2_REPRODUCIBLE),y)
 	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)
 endif
 	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
+	$$(foreach hook,$$(ROOTFS_POST_CMD_HOOKS),\
+		$$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT))
 	chmod a+x $$(FAKEROOT_SCRIPT)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/bin/fakeroot -- $$(FAKEROOT_SCRIPT)
 	$$(INSTALL) -m 0644 support/misc/target-dir-warning.txt $$(TARGET_DIR_WARNING_FILE)
diff --git a/package/pkg-generic.mk b/package/pkg-generic.mk
index b2b3af7d8f..e57aaaeeea 100644
--- a/package/pkg-generic.mk
+++ b/package/pkg-generic.mk
@@ -612,6 +612,8 @@ $(2)_POST_INSTALL_IMAGES_HOOKS  ?=
 $(2)_PRE_LEGAL_INFO_HOOKS       ?=
 $(2)_POST_LEGAL_INFO_HOOKS      ?=
 $(2)_TARGET_FINALIZE_HOOKS      ?=
+$(2)_ROOTFS_PRE_CMD_HOOKS       ?=
+$(2)_ROOTFS_POST_CMD_HOOKS      ?=
 
 # human-friendly targets and target sequencing
 $(1):			$(1)-install
@@ -931,6 +933,8 @@ ifneq ($$($(2)_USERS),)
 PACKAGES_USERS += $$($(2)_USERS)$$(sep)
 endif
 TARGET_FINALIZE_HOOKS += $$($(2)_TARGET_FINALIZE_HOOKS)
+ROOTFS_PRE_CMD_HOOKS += $$($(2)_ROOTFS_PRE_CMD_HOOKS)
+ROOTFS_POST_CMD_HOOKS += $$($(2)_ROOTFS_POST_CMD_HOOKS)
 
 ifeq ($$($(2)_SITE_METHOD),svn)
 DL_TOOLS_DEPENDENCIES += svn
-- 
2.11.0

[Buildroot] [PATCH 19/20] system: make systemd work on a read-only rootfs

[Yann E. MORIN]

When the rootfs is readonly, systemd will expect /var to be writable.
Because we do not really have a R/W filesystem to mount on /var, we make
it a tmpfs [*], and use the systemd-tmpfiles feature to populate it with
"factory" defaults.

We obtain those factory defaults by redirecting /var to that location at
build time, using a symlink /var -> /usr/share/factory which is the
location in which systemd-tmpfiles will look for when instructed to
"recursively copy" a directory.

With a line like:

    C /var/something - - - -

it will look for /usr/share/factory/something and copy it (recursively
if it is a directory) to /var/something, but only if it does not already
exist there.

We also mark this copy with the exclamation mark, as it is only safe to
copy on boot, not when changing targets.

To be noted: the real format for such lines are:

    C /var/something - - - - /from/where/to/copy/something

But if the source is not given, then it is implicitly taken from
/usr/share/factory (which in our case is as-good a location as whatever
else, so we use it, and thus we need not specify the source of the
copy).

Note that we treat symlinks a little bit specially, by creating symlinks
to the factory defaults rather than copying them.

Finally, /var at build time is a symlink, but at runtime, it must be a
directory (so we can mount the tmpfs over there). We can't change that
as a target-finalize hook, because:

  - some packages may want to set ownership and/or access rights on
    files or directories in /var, and that only happens while assembling
    the filesystem images; changing /var from a symlink to a (then
    empty) directory would break this;

  - /var would be a directory on sub-sequent builds (until the next
    "make clean").

Instead, we use the newly-introduce pre- and post-rootfs command hooks,
to turn /var into a directory before assembling the image, and back to a
symlink after assembling the image.

[*] People who want the factory-defaults only on first boot will have
    to tweak the fstab to mount something else than a tmpfs on /var.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Romain Naour <romain.naour@gmail.com>

---
Changes v2 -> v3;
  - fix typoes in commit log  (Romain)
---
 package/skeleton-systemd/skeleton-systemd.mk | 50 ++++++++++++++++++++++++++--
 system/Config.in                             |  1 -
 2 files changed, 48 insertions(+), 3 deletions(-)

diff --git a/package/skeleton-systemd/skeleton-systemd.mk b/package/skeleton-systemd/skeleton-systemd.mk
index 384715e1c9..d9c5c76a7f 100644
--- a/package/skeleton-systemd/skeleton-systemd.mk
+++ b/package/skeleton-systemd/skeleton-systemd.mk
@@ -15,11 +15,57 @@ SKELETON_SYSTEMD_DEPENDENCIES = skeleton-common
 
 SKELETON_SYSTEMD_PROVIDES = skeleton
 
+ifeq ($(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),y)
+
+define SKELETON_SYSTEMD_ROOT_RO_OR_RW
+	echo "/dev/root / auto rw 0 1" >$(TARGET_DIR)/etc/fstab
+	mkdir -p $(TARGET_DIR)/var
+endef
+
+else
+
+# On a R/O rootfs, /var is a tmpfs filesystem. So, at build time, we
+# redirect /var to the "factory settings" location. Just before the
+# filesystem gets created, the /var symlink will be replaced with
+# a real (but empty) directory, and the "factory files" will be copied
+# back there by the tmpfiles.d mechanism.
+define SKELETON_SYSTEMD_ROOT_RO_OR_RW
+	mkdir -p $(TARGET_DIR)/etc/systemd/tmpfiles.d
+	mkdir -p $(TARGET_DIR)/usr/share/factory
+	ln -s usr/share/factory $(TARGET_DIR)/var
+	echo "/dev/root / auto ro 0 1" >$(TARGET_DIR)/etc/fstab
+	echo "tmpfs /var tmpfs mode=1777 0 0" >>$(TARGET_DIR)/etc/fstab
+endef
+
+define SKELETON_SYSTEMD_PRE_ROOTFS_VAR
+	rm -f $(TARGET_DIR)/var
+	mkdir $(TARGET_DIR)/var
+	for i in $(TARGET_DIR)/usr/share/factory/*; do \
+		j="$${i##*/}"; \
+		if [ -L "$${i}" ]; then \
+			printf "L+! /var/%s - - - - %s\n" \
+				"$${j}" "../usr/share/factory/$${j}" \
+			|| exit 1; \
+		else \
+			printf "C! /var/%s - - - -\n" "$${j}" \
+			|| exit 1; \
+		fi; \
+	done >$(TARGET_DIR)/etc/systemd/tmpfiles.d/var-factory.conf
+endef
+SKELETON_SYSTEMD_ROOTFS_PRE_CMD_HOOKS += SKELETON_SYSTEMD_PRE_ROOTFS_VAR
+
+define SKELETON_SYSTEMD_POST_ROOTFS_VAR
+	rm -rf $(TARGET_DIR)/var
+	ln -s usr/share/factory $(TARGET_DIR)/var
+endef
+SKELETON_SYSTEMD_ROOTFS_POST_CMD_HOOKS += SKELETON_SYSTEMD_POST_ROOTFS_VAR
+
+endif
+
 define SKELETON_SYSTEMD_INSTALL_TARGET_CMDS
 	mkdir -p $(TARGET_DIR)/home
 	mkdir -p $(TARGET_DIR)/srv
-	mkdir -p $(TARGET_DIR)/var
-	echo "/dev/root / auto rw 0 1" >$(TARGET_DIR)/etc/fstab
+	$(SKELETON_SYSTEMD_ROOT_RO_OR_RW)
 endef
 
 $(eval $(generic-package))
diff --git a/system/Config.in b/system/Config.in
index 4d417a93b3..0235897735 100644
--- a/system/Config.in
+++ b/system/Config.in
@@ -138,7 +138,6 @@ config BR2_INIT_SYSTEMD
 	depends on BR2_TOOLCHAIN_HEADERS_AT_LEAST_3_10
 	select BR2_ROOTFS_MERGED_USR
 	select BR2_PACKAGE_SYSTEMD
-	select BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW if BR2_ROOTFS_SKELETON_DEFAULT
 
 comment "systemd needs a glibc toolchain, headers >= 3.10"
 	depends on !(BR2_TOOLCHAIN_USES_GLIBC \
-- 
2.11.0

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Yann E. MORIN]

The "builtin" kernel does not boot a systemd-based system, so
we resort to building the same one as currently used by our
qemu_arm_vexpress_defconfig.

We test the 11 following combinations:

  - busybox, read-only, without network
  - busybox, read-only, with network
  - busybox, read-write, without network
  - busybox, read-write, with network

  - basic systemd, read-only, network w/ ifupdown
  - basic systemd, read-only, network w/ networkd
  - basic systemd, read-write, network w/ ifupdown
  - basic systemd, read-write, network w/ networkd

  - full systemd, read-only, network w/ networkd
  - full systemd, read-write, network w/ networkd

  - no init system, read-only, without network

The tests just verify what the /sbin/init binary is, and that we were
able to grab an IP address. More tests can be added later, for example
to check each systemd features (journal, tmpfiles...)

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 support/testing/tests/init/__init__.py     |   0
 support/testing/tests/init/base.py         |  47 ++++++++++
 support/testing/tests/init/test_busybox.py |  67 +++++++++++++
 support/testing/tests/init/test_none.py    |  32 +++++++
 support/testing/tests/init/test_systemd.py | 145 +++++++++++++++++++++++++++++
 5 files changed, 291 insertions(+)
 create mode 100644 support/testing/tests/init/__init__.py
 create mode 100644 support/testing/tests/init/base.py
 create mode 100644 support/testing/tests/init/test_busybox.py
 create mode 100644 support/testing/tests/init/test_none.py
 create mode 100644 support/testing/tests/init/test_systemd.py

diff --git a/support/testing/tests/init/__init__.py b/support/testing/tests/init/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/support/testing/tests/init/base.py b/support/testing/tests/init/base.py
new file mode 100644
index 0000000000..a261d7dd46
--- /dev/null
+++ b/support/testing/tests/init/base.py
@@ -0,0 +1,47 @@
+import os
+import subprocess
+import infra.basetest
+
+class InitSystemBase(infra.basetest.BRTest):
+
+    def startEmulator(self, fs_type, kernel=None, dtb=None, init=None):
+        img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
+        subprocess.call(["truncate", "-s", "%1M", img])
+
+        options = ["-drive",
+                   "file={},if=sd,format=raw".format(img),
+                   "-M", "vexpress-a9"]
+
+        if kernel is None:
+            kernel = "builtin"
+        else:
+            kernel = os.path.join(self.builddir, "images", kernel)
+            options.extend(["-dtb", os.path.join(self.builddir, "images",
+                                             "{}.dtb".format(dtb))])
+
+        kernel_cmdline = ["root=/dev/mmcblk0",
+                          "rootfstype={}".format(fs_type),
+                          "rootwait",
+                          "ro",
+                          "console=ttyAMA0"]
+
+        if not init is None:
+            kernel_cmdline.extend(["init={}".format(init)])
+
+        self.emulator.boot(arch="armv7",
+                           kernel=kernel,
+                           kernel_cmdline=kernel_cmdline,
+                           options=options)
+
+        if init is None:
+            self.emulator.login()
+
+    def checkInit(self, path):
+        cmd = "cmp /proc/1/exe {}".format(path)
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, 0)
+
+    def checkNetwork(self, interface, exitCode=0):
+        cmd = "ip addr show {} |grep inet".format(interface)
+        _, exit_code = self.emulator.run(cmd)
+        self.assertEqual(exit_code, exitCode)
diff --git a/support/testing/tests/init/test_busybox.py b/support/testing/tests/init/test_busybox.py
new file mode 100644
index 0000000000..c3e425bf5d
--- /dev/null
+++ b/support/testing/tests/init/test_busybox.py
@@ -0,0 +1,67 @@
+import infra.basetest
+from tests.init.base import InitSystemBase as InitSystemBase
+
+class InitSystemBusyboxBase(InitSystemBase):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        """
+        # BR2_TARGET_ROOTFS_TAR is not set
+        """
+
+    def checkInit(self):
+        super(InitSystemBusyboxBase, self).checkInit("/bin/busybox")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemBusyboxRo(InitSystemBusyboxBase):
+    config = InitSystemBusyboxBase.config + \
+        """
+        # BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set
+        BR2_TARGET_ROOTFS_SQUASHFS=y
+        """
+
+    def test_run(self):
+        self.startEmulator("squashfs")
+        self.checkInit()
+        self.checkNetwork("eth0", 1)
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemBusyboxRw(InitSystemBusyboxBase):
+    config = InitSystemBusyboxBase.config + \
+        """
+        BR2_TARGET_ROOTFS_EXT2=y
+        """
+
+    def test_run(self):
+        self.startEmulator("ext2")
+        self.checkInit()
+        self.checkNetwork("eth0", 1)
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemBusyboxRoNet(InitSystemBusyboxBase):
+    config = InitSystemBusyboxBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        # BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set
+        BR2_TARGET_ROOTFS_SQUASHFS=y
+        """
+
+    def test_run(self):
+        self.startEmulator("squashfs")
+        self.checkInit()
+        self.checkNetwork("eth0")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemBusyboxRwNet(InitSystemBusyboxBase):
+    config = InitSystemBusyboxBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        BR2_TARGET_ROOTFS_EXT2=y
+        """
+
+    def test_run(self):
+        self.startEmulator("ext2")
+        self.checkInit()
+        self.checkNetwork("eth0")
diff --git a/support/testing/tests/init/test_none.py b/support/testing/tests/init/test_none.py
new file mode 100644
index 0000000000..f55db5d3e0
--- /dev/null
+++ b/support/testing/tests/init/test_none.py
@@ -0,0 +1,32 @@
+import pexpect
+
+import infra.basetest
+from tests.init.base import InitSystemBase as InitSystemBase
+
+class TestInitSystemNone(InitSystemBase):
+    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
+        """
+        BR2_INIT_NONE=y
+        # BR2_TARGET_ROOTFS_TAR is not set
+        BR2_TARGET_ROOTFS_SQUASHFS=y
+        """
+
+    def test_run(self):
+        self.startEmulator(fs_type="squashfs", init="/bin/sh")
+        index = self.emulator.qemu.expect(["/bin/sh: can't access tty; job control turned off", pexpect.TIMEOUT], timeout=60)
+        if index != 0:
+            self.emulator.logfile.write("==> System does not boot")
+            raise SystemError("System does not boot")
+        index = self.emulator.qemu.expect(["#", pexpect.TIMEOUT], timeout=60)
+        if index != 0:
+            self.emulator.logfile.write("==> System does not boot")
+            raise SystemError("System does not boot")
+
+        out, exit_code = self.emulator.run("sh -c 'echo $PPID'")
+        self.assertEqual(exit_code, 0)
+        self.assertEqual(out[0], "1")
+
+        _, exit_code = self.emulator.run("mount -t proc none /proc")
+        self.assertEqual(exit_code, 0)
+
+        self.checkInit("/bin/sh")
diff --git a/support/testing/tests/init/test_systemd.py b/support/testing/tests/init/test_systemd.py
new file mode 100644
index 0000000000..010e1f6514
--- /dev/null
+++ b/support/testing/tests/init/test_systemd.py
@@ -0,0 +1,145 @@
+import infra.basetest
+from tests.init.base import InitSystemBase as InitSystemBase
+
+class InitSystemSystemdBase(InitSystemBase):
+    config = \
+        """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_INIT_SYSTEMD=y
+        BR2_TARGET_GENERIC_GETTY_PORT="ttyAMA0"
+        BR2_LINUX_KERNEL=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION=y
+        BR2_LINUX_KERNEL_CUSTOM_VERSION_VALUE="4.11.3"
+        BR2_LINUX_KERNEL_DEFCONFIG="vexpress"
+        BR2_LINUX_KERNEL_DTS_SUPPORT=y
+        BR2_LINUX_KERNEL_INTREE_DTS_NAME="vexpress-v2p-ca9"
+        # BR2_TARGET_ROOTFS_TAR is not set
+        """
+
+    def checkInit(self):
+        super(InitSystemSystemdBase, self).checkInit("/lib/systemd/systemd")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemSystemdRoNetworkd(InitSystemSystemdBase):
+    config = InitSystemSystemdBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        # BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set
+        BR2_TARGET_ROOTFS_SQUASHFS=y
+        """
+
+    def test_run(self):
+        self.startEmulator("squashfs", "zImage", "vexpress-v2p-ca9")
+        self.checkInit()
+        self.checkNetwork("eth0")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemSystemdRwNetworkd(InitSystemSystemdBase):
+    config = InitSystemSystemdBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        BR2_TARGET_ROOTFS_EXT2=y
+        """
+
+    def test_run(self):
+        self.startEmulator("ext2", "zImage", "vexpress-v2p-ca9")
+        self.checkInit()
+        self.checkNetwork("eth0")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemSystemdRoIfupdown(InitSystemSystemdBase):
+    config = InitSystemSystemdBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        # BR2_PACKAGE_SYSTEMD_NETWORKD is not set
+        # BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set
+        BR2_TARGET_ROOTFS_SQUASHFS=y
+        """
+
+    def test_run(self):
+        self.startEmulator("squashfs", "zImage", "vexpress-v2p-ca9")
+        self.checkInit()
+        self.checkNetwork("eth0")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemSystemdRwIfupdown(InitSystemSystemdBase):
+    config = InitSystemSystemdBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        # BR2_PACKAGE_SYSTEMD_NETWORKD is not set
+        # BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set
+        BR2_TARGET_ROOTFS_EXT2=y
+        """
+
+    def test_run(self):
+        self.startEmulator("ext2", "zImage", "vexpress-v2p-ca9")
+        self.checkInit()
+        self.checkNetwork("eth0")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemSystemdRoFull(InitSystemSystemdBase):
+    config = InitSystemSystemdBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        # BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set
+        BR2_PACKAGE_SYSTEMD_JOURNAL_GATEWAY=y
+        BR2_PACKAGE_SYSTEMD_BACKLIGHT=y
+        BR2_PACKAGE_SYSTEMD_BINFMT=y
+        BR2_PACKAGE_SYSTEMD_COREDUMP=y
+        BR2_PACKAGE_SYSTEMD_FIRSTBOOT=y
+        BR2_PACKAGE_SYSTEMD_HIBERNATE=y
+        BR2_PACKAGE_SYSTEMD_IMPORTD=y
+        BR2_PACKAGE_SYSTEMD_LOCALED=y
+        BR2_PACKAGE_SYSTEMD_LOGIND=y
+        BR2_PACKAGE_SYSTEMD_MACHINED=y
+        BR2_PACKAGE_SYSTEMD_POLKIT=y
+        BR2_PACKAGE_SYSTEMD_QUOTACHECK=y
+        BR2_PACKAGE_SYSTEMD_RANDOMSEED=y
+        BR2_PACKAGE_SYSTEMD_RFKILL=y
+        BR2_PACKAGE_SYSTEMD_SMACK_SUPPORT=y
+        BR2_PACKAGE_SYSTEMD_SYSUSERS=y
+        BR2_PACKAGE_SYSTEMD_VCONSOLE=y
+        BR2_TARGET_ROOTFS_SQUASHFS=y
+        """
+
+    def test_run(self):
+        self.startEmulator("squashfs", "zImage", "vexpress-v2p-ca9")
+        self.checkInit()
+        self.checkNetwork("eth0")
+
+
+#-------------------------------------------------------------------------------
+class TestInitSystemSystemdRwFull(InitSystemSystemdBase):
+    config = InitSystemSystemdBase.config + \
+        """
+        BR2_SYSTEM_DHCP="eth0"
+        BR2_PACKAGE_SYSTEMD_JOURNAL_GATEWAY=y
+        BR2_PACKAGE_SYSTEMD_BACKLIGHT=y
+        BR2_PACKAGE_SYSTEMD_BINFMT=y
+        BR2_PACKAGE_SYSTEMD_COREDUMP=y
+        BR2_PACKAGE_SYSTEMD_FIRSTBOOT=y
+        BR2_PACKAGE_SYSTEMD_HIBERNATE=y
+        BR2_PACKAGE_SYSTEMD_IMPORTD=y
+        BR2_PACKAGE_SYSTEMD_LOCALED=y
+        BR2_PACKAGE_SYSTEMD_LOGIND=y
+        BR2_PACKAGE_SYSTEMD_MACHINED=y
+        BR2_PACKAGE_SYSTEMD_POLKIT=y
+        BR2_PACKAGE_SYSTEMD_QUOTACHECK=y
+        BR2_PACKAGE_SYSTEMD_RANDOMSEED=y
+        BR2_PACKAGE_SYSTEMD_RFKILL=y
+        BR2_PACKAGE_SYSTEMD_SMACK_SUPPORT=y
+        BR2_PACKAGE_SYSTEMD_SYSUSERS=y
+        BR2_PACKAGE_SYSTEMD_VCONSOLE=y
+        BR2_TARGET_ROOTFS_EXT2=y
+        """
+
+    def test_run(self):
+        self.startEmulator("ext2", "zImage", "vexpress-v2p-ca9")
+        self.checkInit()
+        self.checkNetwork("eth0")
-- 
2.11.0

[Buildroot] [PATCH 01/20] support/tests: allow properly indented config fragment

[Yann E. MORIN]

All,

This is version 2 of the patch. Version 1 has been just reviewed now by
Arnout, after which I sent v3:
    https://patchwork.ozlabs.org/patch/790525/

So, please disregard this one; use v3 instead (but still before the rest
of this series).

Thanks!

Regards,
Yann E. MORIN.

On 2017-07-18 19:25 +0200, Yann E. MORIN spake thusly:
> Currently, defining a config fragment in the runtime test infra requires
> that the fragment not to be indented. This is beark, and causes grievance
> when looking at the code (e.g. to fix it).
> 
> Just strip out all leading spaces/tabs when writing the configuration
> lines into the config file, allowing in-line indented config fragments,
> like so:
> 
>     class TestFoo(bla):
>         config = bla.config + \
>             """
>             FOO=y
>             # BAR is not set
>             """
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> 
> ---
> Changes v1 -> v2:
>   - add example in commit log  (Thomas)
> 
> ---
> Note: this is only tangentially related to this series, because it is
> needed by the last patch in the series. It wasalready submitted before:
>     https://patchwork.ozlabs.org/patch/788985/
> ---
>  support/testing/infra/builder.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/support/testing/infra/builder.py b/support/testing/infra/builder.py
> index a475bb0a30..81735dec96 100644
> --- a/support/testing/infra/builder.py
> +++ b/support/testing/infra/builder.py
> @@ -16,7 +16,8 @@ class Builder(object):
>  
>          config_file = os.path.join(self.builddir, ".config")
>          with open(config_file, "w+") as cf:
> -            cf.write(self.config)
> +            for line in self.config.splitlines():
> +                cf.write("{}\n".format(line.lstrip()))
>  
>          cmd = ["make",
>                 "O={}".format(self.builddir),
> -- 
> 2.11.0
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Andrey Yurovsky]

On Tue, Jul 18, 2017 at 10:25 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> Hello All!
>
> This series is a proposal to fix our handling of systemd as an init
> system.
>
> Our default skeleton is not well suited for systemd:
>
>   - we have /var/log a symlink to /tmp/log, but the way systemd starts
>     hides the journals:
>     - start systemd-journald, stores journals in /tmp/log/ (because of
>       the redirection above)
>     - mounts a tmpfs over /tmp, thus hidding the journals.
>
>   - on a read-only rootfs, systemd expects /var to be read-write, which
>     we do not provide.
>
> All of this sounds trivial, but fixing it is definitely not.
>
> The overall idea is that we need different skeletons, one for each type
> of init systems: sysv, systemd, others (aka custom).
>
> The skeleton package becomes a virtual package, that is provided by one
> of four implementations:
>
>   - skeleton-sysv or skeleton-systemd, as their names imply, for either
>     SysV- or systemd-based systems, and which depend on skeleton-common;
>
>   - skeleton-common, that provides base files and directories comon to
>     all skeletons (except custom, below); skeleton-common is also a
>     provider for the skeleton virtual package, for use with no-init (aka
>     custom init) systems;
>
>   - skeleton-custom, for user-provided custom skeleton, and of which we
>     expect it is fully prepared; skeleton-custom does not depend on
>     skeleton-common.
>
> A series of runtime tests have also been added, running on the QEMU
> vexpress:
>
>     init system         rootfs      DHCP?
>     -------------------------------------
>     busybox             ext4        no
>     busybox             ext4        eth0
>     busybox             squashfs    no
>     busybox             squashfs    eth0
>   * systemd-ifupdown    ext4        eth0
>     systemd-ifupdown    squashfs    eth0
>  ** systemd-networkd    ext4        eth0
>     systemd-networkd    squashfs    eth0
> *** systemd-full        ext4        eth0
>     systemd-full        squashfs    eth0
>   + no-init             squashfs    no
>
>     *   systemd-networkd: only what gets selected automatically when
>         systemd is used as init system, with all other options unset;
>
>     **  systemd-ifupdown: like systemd-networkd, but with networkd
>         disabled, thus using ifupdown instead;
>
>     *** systemd-full: all systemd options enabled, implies networkd;
>
>     +   a shell is launched as init
>
> Those new tests are not exhaustive, especially the systemd one; instead,
> they just test basic features: what init is being used, and did we get
> an IP adress for eth0? More tests can be added later on.
>
> Changes v2 -> v3:
>   - too much to note, mostly rewriten from scratch
>
> Changes v1 -> v2:
>   - enhance the commit logs
>
>
> Regards,
> Yann E. MORIN.
>
>
> The following changes since commit bc801efacb298e2297024f83e960eaf8a45dd39f
>
>   python-coherence: bump to the latest upstream commit (2017-07-17 11:28:58 +0200)
>
>
> are available in the git repository at:
>
>   git://git.buildroot.org/~ymorin/git/buildroot.git
>
> for you to fetch changes up to 95f6965c48dbf2f754818e2bec1f2d3891015475
>
>   support/testing: add runtime testing for init systems (2017-07-18 18:59:05 +0200)
>
>
> ----------------------------------------------------------------
> Yann E. MORIN (20):
>       support/tests: allow properly indented config fragment
>       core/pkg-generic: add variable to skip skeleton dependency
>       package/skeleton: add macro to rsync skeleton directory
>       package/skeleton: make SKELETON_LIB_SYMLINK a macro
>       system: provide package-wide system variables and macros
>       system: move setting getty to the corresponding init systems
>       system: move remounting / to the corresponding init systems
>       system: with no init system, only allow custom skeleton
>       package/skeleton: drop dependency on host-mkpasswd
>       package/skeleton: select it rather than default to y
>       package/skeleton: split out into skeleton-custom
>       package/skeleton: split out into skeleton-common
>       package/skeleton: make it a virtual package
>       package/skeleton-common: simplify staging install
>       package/skeleton: introduce sysv- and systemd-specific skeletons
>       system: separate sysv and systemd parts of the skeleton
>       sytem: no-init systems may use our default, common skeleton
>       fs: add pre- and post-command hooks
>       system: make systemd work on a read-only rootfs
>       support/testing: add runtime testing for init systems
>
>  Makefile                                           |   1 +
>  fs/common.mk                                       |   4 +
>  package/Config.in                                  |   4 +
>  package/busybox/busybox.mk                         |  15 +-
>  package/pkg-generic.mk                             |  11 +-
>  package/skeleton-common/Config.in                  |  31 ++++
>  package/skeleton-common/skeleton-common.mk         |  92 +++++++++
>  package/skeleton-custom/Config.in                  |   6 +
>  package/skeleton-custom/skeleton-custom.mk         | 107 +++++++++++
>  package/skeleton-systemd/Config.in                 |   7 +
>  package/skeleton-systemd/skeleton-systemd.mk       |  71 +++++++
>  package/skeleton-sysv/Config.in                    |   7 +
>  package/skeleton-sysv/skeleton-sysv.mk             |  22 +++
>  {system => package/skeleton-sysv}/skeleton/dev/log |   0
>  .../skeleton-sysv}/skeleton/dev/pts/.empty         |   0
>  .../skeleton-sysv}/skeleton/dev/shm/.empty         |   0
>  .../skeleton-sysv}/skeleton/etc/fstab              |   0
>  .../skeleton-sysv}/skeleton/var/cache              |   0
>  .../skeleton-sysv}/skeleton/var/lib/misc           |   0
>  .../skeleton-sysv}/skeleton/var/lock               |   0
>  {system => package/skeleton-sysv}/skeleton/var/log |   0
>  {system => package/skeleton-sysv}/skeleton/var/run |   0
>  .../skeleton-sysv}/skeleton/var/spool              |   0
>  {system => package/skeleton-sysv}/skeleton/var/tmp |   0
>  package/skeleton/Config.in                         |   8 +-
>  package/skeleton/skeleton.mk                       | 206 +--------------------
>  package/sysvinit/sysvinit.mk                       |  11 ++
>  support/testing/infra/builder.py                   |   3 +-
>  .../testing/tests/init/__init__.py                 |   0
>  support/testing/tests/init/base.py                 |  47 +++++
>  support/testing/tests/init/test_busybox.py         |  67 +++++++
>  support/testing/tests/init/test_none.py            |  32 ++++
>  support/testing/tests/init/test_systemd.py         | 145 +++++++++++++++
>  system/Config.in                                   |  13 +-
>  system/skeleton/dev/{pts => }/.empty               |   0
>  system/system.mk                                   |  88 +++++++++
>  36 files changed, 787 insertions(+), 211 deletions(-)
>  create mode 100644 package/skeleton-common/Config.in
>  create mode 100644 package/skeleton-common/skeleton-common.mk
>  create mode 100644 package/skeleton-custom/Config.in
>  create mode 100644 package/skeleton-custom/skeleton-custom.mk
>  create mode 100644 package/skeleton-systemd/Config.in
>  create mode 100644 package/skeleton-systemd/skeleton-systemd.mk
>  create mode 100644 package/skeleton-sysv/Config.in
>  create mode 100644 package/skeleton-sysv/skeleton-sysv.mk
>  rename {system => package/skeleton-sysv}/skeleton/dev/log (100%)
>  copy {system => package/skeleton-sysv}/skeleton/dev/pts/.empty (100%)
>  rename {system => package/skeleton-sysv}/skeleton/dev/shm/.empty (100%)
>  rename {system => package/skeleton-sysv}/skeleton/etc/fstab (100%)
>  rename {system => package/skeleton-sysv}/skeleton/var/cache (100%)
>  rename {system => package/skeleton-sysv}/skeleton/var/lib/misc (100%)
>  rename {system => package/skeleton-sysv}/skeleton/var/lock (100%)
>  rename {system => package/skeleton-sysv}/skeleton/var/log (100%)
>  rename {system => package/skeleton-sysv}/skeleton/var/run (100%)
>  rename {system => package/skeleton-sysv}/skeleton/var/spool (100%)
>  rename {system => package/skeleton-sysv}/skeleton/var/tmp (100%)
>  copy system/skeleton/dev/pts/.empty => support/testing/tests/init/__init__.py (100%)
>  create mode 100644 support/testing/tests/init/base.py
>  create mode 100644 support/testing/tests/init/test_busybox.py
>  create mode 100644 support/testing/tests/init/test_none.py
>  create mode 100644 support/testing/tests/init/test_systemd.py
>  rename system/skeleton/dev/{pts => }/.empty (100%)
>  create mode 100644 system/system.mk
>
> --
> .-----------------.--------------------.------------------.--------------------.
> |  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
> | +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
> | +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
> | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
> '------------------------------^-------^------------------^--------------------'
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

For what it's worth,

Tested-by: Andrey Yurovsky <yurovsky@gmail.com>

using some in-house configs (armv5 and armv7) with and without
systemd-networkd, verified journald functionality and the state of
/var and so on.

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Thomas Petazzoni]

Hello,

On Wed, 19 Jul 2017 13:14:58 -0700, Andrey Yurovsky wrote:

> For what it's worth,
> 
> Tested-by: Andrey Yurovsky <yurovsky@gmail.com>
> 
> using some in-house configs (armv5 and armv7) with and without
> systemd-networkd, verified journald functionality and the state of
> /var and so on.

Thanks a lot for testing, very useful to get some feedback!

Can you confirm that it fixes the journald issue reported in bug
https://bugs.busybox.net/show_bug.cgi?id=7892 ?

Also, it would be useful if someone interested in systemd could look at
bugs https://bugs.busybox.net/show_bug.cgi?id=9881 and
https://bugs.busybox.net/show_bug.cgi?id=9926.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Andrey Yurovsky]

On Wed, Jul 19, 2017 at 1:53 PM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Hello,
>
> On Wed, 19 Jul 2017 13:14:58 -0700, Andrey Yurovsky wrote:
>
>> For what it's worth,
>>
>> Tested-by: Andrey Yurovsky <yurovsky@gmail.com>
>>
>> using some in-house configs (armv5 and armv7) with and without
>> systemd-networkd, verified journald functionality and the state of
>> /var and so on.
>
> Thanks a lot for testing, very useful to get some feedback!
>
> Can you confirm that it fixes the journald issue reported in bug
> https://bugs.busybox.net/show_bug.cgi?id=7892 ?

It does.

> Also, it would be useful if someone interested in systemd could look at
> bugs https://bugs.busybox.net/show_bug.cgi?id=9881 and

I did an ARMv7 build with networkd+resolvd and couldn't reproduce the
described problem with these patches applied. /etc/resolv.conf points
to ../tmp/resolv.conf which in turn is properly populated and names do
resolve. systemd-hostnamed is acting like it does on Fedora (for what
it's worth), I'm still learning about how to test it properly (that
is, I just have a static hostname at the moment).

> https://bugs.busybox.net/show_bug.cgi?id=9926.

Ok, I'll do a clean rebuild without networkd and with NetworkManager.

> --
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Thomas Petazzoni]

Hello,

On Wed, 19 Jul 2017 15:17:21 -0700, Andrey Yurovsky wrote:

> > Thanks a lot for testing, very useful to get some feedback!
> >
> > Can you confirm that it fixes the journald issue reported in bug
> > https://bugs.busybox.net/show_bug.cgi?id=7892 ?  
> 
> It does.

Awesome, thanks.

> > Also, it would be useful if someone interested in systemd could look at
> > bugs https://bugs.busybox.net/show_bug.cgi?id=9881 and  
> 
> I did an ARMv7 build with networkd+resolvd and couldn't reproduce the
> described problem with these patches applied.

Can you reproduce the problem *without* the patches ?

> /etc/resolv.conf points
> to ../tmp/resolv.conf which in turn is properly populated and names do
> resolve. systemd-hostnamed is acting like it does on Fedora (for what
> it's worth), I'm still learning about how to test it properly (that
> is, I just have a static hostname at the moment).
> 
> > https://bugs.busybox.net/show_bug.cgi?id=9926.  
> 
> Ok, I'll do a clean rebuild without networkd and with NetworkManager.

Thanks a lot!

BTW, for these bugs, if there is something easy we can add to the
testing infrastructure to make sure it keeps working, it would be nice!

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Marcus Hoffmann]

Hey,

On 20.07.2017 09:32, Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 19 Jul 2017 15:17:21 -0700, Andrey Yurovsky wrote:
> 
>>> Thanks a lot for testing, very useful to get some feedback!
>>>
>>> Can you confirm that it fixes the journald issue reported in bug
>>> https://bugs.busybox.net/show_bug.cgi?id=7892 ?  
>>
>> It does.
> 
> Awesome, thanks.
> 
>>> Also, it would be useful if someone interested in systemd could look at
>>> bugs https://bugs.busybox.net/show_bug.cgi?id=9881 and  
>>
>> I did an ARMv7 build with networkd+resolvd and couldn't reproduce the
>> described problem with these patches applied.
> 
> Can you reproduce the problem *without* the patches ?

We are experiencing this problem. I'll try to test it with this patch
series on Monday.

> 
>> /etc/resolv.conf points
>> to ../tmp/resolv.conf which in turn is properly populated and names do
>> resolve. systemd-hostnamed is acting like it does on Fedora (for what
>> it's worth), I'm still learning about how to test it properly (that
>> is, I just have a static hostname at the moment).
>>
>>> https://bugs.busybox.net/show_bug.cgi?id=9926.  
>>
>> Ok, I'll do a clean rebuild without networkd and with NetworkManager.
> 
> Thanks a lot!
> 
> BTW, for these bugs, if there is something easy we can add to the
> testing infrastructure to make sure it keeps working, it would be nice!
> 
> Best regards,
> 
> Thomas
> 


Best wishes,
Marcus

[Buildroot] [PATCH 02/20] core/pkg-generic: add variable to skip skeleton dependency

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> We currently skip the skeleton dependency by checking if the current
> package is the skeleton.
> 
> We are going to introduce more skeleton-related packages, so we
> need a way to exclude the skeleton dependency for those, or we'd
> get a circular dependency, for the same reason we need to skip
> the toolchain dependency.
> 
> Instead of checking for all the skeleton-providing packages in the core
> infra, add a new package options so that packages can express they do
> not need the dependency on the skeleton, like we have an option to avoid
> the depednency on the toolchain. The only packages that will use that
> option are probably the skeletons, so we need not document this
> variable, like we did not document the option to exclude the dependency
> on the toolchain.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Reviewed-by: Romain Naour <romain.naour@gmail.com>

 I absolutely hate adding additional variables that are used for only a few
packages, but I have no better ideas, so

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

[snip]
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 03/20] package/skeleton: add macro to rsync skeleton directory

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> The rsync command is currently called from a single location. In
> follow-up patches, it will be called from various places then.
> 
> It will be easier to update and fix this command if it is defined in a
> single location, rather than if duplicated in many places.
> 
> Define a macro that skeleton packages can call to install their skeleton
> directories. Since at least one occurence of it will want to install in
> staging, the macro must accept a destination path, rather than hard-code
> target as the destination.
> 
> Since it is to be used by other skeleton packages, define it early,
> outside of any conditional block.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Reviewed-by: Romain Naour <romain.naour@gmail.com>
> ---
>  package/skeleton/skeleton.mk | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
> index 9b7407a236..017479a990 100644
> --- a/package/skeleton/skeleton.mk
> +++ b/package/skeleton/skeleton.mk
> @@ -14,6 +14,18 @@ SKELETON_ADD_SKELETON_DEPENDENCY = NO
>  # The skeleton also handles the merged /usr case in the sysroot
>  SKELETON_INSTALL_STAGING = YES
>  
> +############
> +# Macros available for use by any skeleton package:
> +# - SKELETON_RSYNC
> +
> +# This function rsyncs the skeleton directory in $(1) to the destination
> +# in $(2), which should be either $(TARTGET_DIR) or $(STAGING_DIR)
> +define SKELETON_RSYNC
> +	rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
> +		--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
> +		$(1)/ $(2)/
> +endef

 Another thing I hate is make functions - they're difficult to understand. So
instead, I'd just define

SKELETON_RSYNC_ARGS = -a --ignore-times ...

 Adding the / after $(1) and $(2) is the only reason to claim this should be a
macro, but really I don't think that that's enough.


 The above is bikeshedding though, so feel free to commit the current solution.

 Regards,
 Arnout

> +
>  ifeq ($(BR2_ROOTFS_SKELETON_CUSTOM),y)
>  
>  SKELETON_PATH = $(call qstrip,$(BR2_ROOTFS_SKELETON_CUSTOM_PATH))
> @@ -89,9 +101,7 @@ SKELETON_LIB_SYMLINK = lib32
>  endif
>  
>  define SKELETON_INSTALL_TARGET_CMDS
> -	rsync -a --ignore-times $(RSYNC_VCS_EXCLUSIONS) \
> -		--chmod=u=rwX,go=rX --exclude .empty --exclude '*~' \
> -		$(SKELETON_PATH)/ $(TARGET_DIR)/
> +	$(call SKELETON_RSYNC,$(SKELETON_PATH),$(TARGET_DIR))
>  	$(call SKELETON_USR_SYMLINKS_OR_DIRS,$(TARGET_DIR))
>  	ln -snf lib $(TARGET_DIR)/$(SKELETON_LIB_SYMLINK)
>  	ln -snf lib $(TARGET_DIR)/usr/$(SKELETON_LIB_SYMLINK)
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 04/20] package/skeleton: make SKELETON_LIB_SYMLINK a macro

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Rather than duplicate that code in all skeleton packages, make this a
> macro, so that it is easier to use.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 However...

> ---
>  package/skeleton/skeleton.mk | 30 ++++++++++++++++++------------
>  1 file changed, 18 insertions(+), 12 deletions(-)
> 
> diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
> index 017479a990..0a8add66dc 100644
> --- a/package/skeleton/skeleton.mk
> +++ b/package/skeleton/skeleton.mk
> @@ -17,6 +17,7 @@ SKELETON_INSTALL_STAGING = YES
>  ############
>  # Macros available for use by any skeleton package:
>  # - SKELETON_RSYNC
> +# - SKELETON_LIB_SYMLINK
>  
>  # This function rsyncs the skeleton directory in $(1) to the destination
>  # in $(2), which should be either $(TARTGET_DIR) or $(STAGING_DIR)
> @@ -26,6 +27,21 @@ define SKELETON_RSYNC
>  		$(1)/ $(2)/
>  endef
>  
> +# Make a symlink lib32->lib or lib64->lib as appropriate.
> +# MIPS64/n32 requires lib32 even though it's a 64-bit arch.
> +# $(1): base dir (either staging or target)
> +ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)

 This makes me wonder: are we doing the right thing with NABI32? What is the
semantics of BR2_ARCH_IS_64? I would say, it doesn't specify that you have a
32-bit or 64-bit capable CPU; instead, it says that you have a 32-bit or 64-bit
userspace. I.e., that void* and long are 32 resp. 64 bits. Indeed, I don't think
it's really relevant for Buildroot to know about the CPU capabilities, the
important thing is the userpace ABI, i.e. how we compile packages.

 So, with that in mind, I think we should NOT select BR2_ARCH_IS_64 for the n32
ABI. Similarly, when we add x32, it should also not be BR2_ARCH_IS_64.

 It's a kind of big change though, that also affects how the architectures are
selected, and possibly with legacy implications...

 Vicente, what do you think?


 Regards,
 Arnout

 PS when replying to this, better change the subject :-)


> +define SKELETON_LIB_SYMLINK
> +	ln -snf lib $(1)/lib64
> +	ln -snf lib $(1)/usr/lib64
> +endef
> +else
> +define SKELETON_LIB_SYMLINK
> +	ln -snf lib $(1)/lib32
> +	ln -snf lib $(1)/usr/lib32
> +endef
> +endif
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 05/20] system: provide package-wide system variables and macros

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Some macros, soon some variables, currently defined in the skeleton are
> going to be used by other packages.
> 
> Some of those variables will be used as Makefile conditions (e.g. in
> ifeq() conditiopns), so they *must* be defined before being used.
> 
> Since the skeleton package, starting with an 's', is included quite
> late, those variables would not be available to most packages.
> 
> Offload the existing macros into the new system/system/mk file, that is
                                                        ^.
> included early, before any package is. Rename the macros to appropriate
> names.
> 
> Future commits will add new macros and variables in that file.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 03/20] package/skeleton: add macro to rsync skeleton directory

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Define a macro that skeleton packages can call to install their skeleton
> directories. Since at least one occurence of it will want to install in
                                  occurrence
> staging, the macro must accept a destination path, rather than hard-code
> target as the destination.

 By the way, the overlay copying uses almost the same rsync arguments, so they
can be used there as well.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 06/20] system: move setting getty to the corresponding init systems

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Currently, setting the getty is done:
>   - by the skeleton package when the init system is either busybox or
>     sysvinit;
>   - by the systemd package when the init system is systemd;
> both by registering a target-finalize hook.
> 
> This is not very consistent.
> 
> Move setting the getty out of the skeleton and into the package that
> provides the init system, by registering a per-package target-fialize
                                                                finalize
> hook.
> 
> This offloads yet a bit more out of the skeleton, so that it is easier
> to properly separate the skeletons for the various init systems.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

[snip]
> +define BUSYBOX_SET_GETTY
> +	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(SYSTEM_GETTY_PORT)::respawn:/sbin/getty -L $(SYSTEM_GETTY_OPTIONS) $(SYSTEM_GETTY_PORT) $(SYSTEM_GETTY_BAUDRATE) $(SYSTEM_GETTY_TERM) #~' \
> +		$(TARGET_DIR)/etc/inittab
> +endef
[snip]
> +define SYSVINIT_SET_GETTY
> +	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(shell echo $(SYSTEM_GETTY_PORT) | tail -c+4)::respawn:/sbin/getty -L $(SYSTEM_GETTY_OPTIONS) $(SYSTEM_GETTY_PORT) $(SYSTEM_GETTY_BAUDRATE) $(SYSTEM_GETTY_TERM) #~' \
> +		$(TARGET_DIR)/etc/inittab
> +endef

 The only difference between these two is that for sysvinit we have to make sure
it's only 4 characters. But it actually doesn't hurt to do the same for busybox
as well. So perhaps we can factor the two in a single SYSTEM_SET_GETTY ?

[snip]
> +SYSTEM_GETTY_PORT = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_PORT))
> +SYSTEM_GETTY_BAUDRATE = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_BAUDRATE))
> +SYSTEM_GETTY_TERM = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_TERM))
> +SYSTEM_GETTY_OPTIONS = $(call qstrip,$(BR2_TARGET_GENERIC_GETTY_OPTIONS))

 Perhaps systemd.mk should also start making use of these?

 Both of these are of course separate patches.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 07/20] system: move remounting / to the corresponding init systems

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Currently, remounting / read-write (or not) is done by the skeleton
> package when the init system is eitehr busybox or sysvinit, by
                                  either
> registering a target-finalize hook; it is not done at all for systemd.
> 
> Move registering this target-finalize hook to both of busybox and
> sysvinit. Leave systemd alone, we'll take care of it later.
> 
> Rename the macro to a more meaningful name, and move it to system.mk
> with the other such macros.
> 
> Yet a little bit less init-system knowledge in the skeleton.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

 Small details to fix...

[snip]
> +# Provided by the skeleton package

 Not really, it's provided by system.mk - which is kind of obvious from the
name. So I'd just remove this comment.

> +BUSYBOX_TARGET_FINALIZE_HOOKS += SYSTEM_REMOUNT_ROOT_INITTAB

 Excellent choice in variable name!

[snip]
> +# Provided by the skeleton package

 Same here of course.

> +SYSVINIT_TARGET_FINALIZE_HOOKS += SYSTEM_REMOUNT_ROOT_INITTAB
> +
[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 08/20] system: with no init system, only allow custom skeleton

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> When there is no init system (i.e. a custom one), our bundled default
> skeleton is most probably un-fit for that (non-)init system.
> 
> This will be the case when we introduce per init system skeletons. The
> systemd skeleton will be unfit except for a systemd init; the sysv
> skeleton will be unfit except for a sysv-like init system.
> 
> In case they are using no init system (really, a custom one), the user
> should be responsible for providing their own, custom skeleton as well.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

 NACK on this one. Our skeleton (as in, the contents of system/skeleton) is
already init-agnostic. If you use another init system (shell script, or no init
at all because you just use it as a chroot, or whatever), you will *still* need
that skelecton. Indeed, the skeleton contains things like the /sys and /dev
directories, a passwd file, ....

 If there are things in that skeleton that you don't need for a custom init
system, then they probably shouldn't even be in the skeleton.

 Regards,
 Arnout

> ---
>  system/Config.in | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/system/Config.in b/system/Config.in
> index 828df4217e..ed539dcbe0 100644
> --- a/system/Config.in
> +++ b/system/Config.in
> @@ -3,11 +3,17 @@ menu "System configuration"
>  choice
>  	prompt "Root FS skeleton"
>  
> +# When not using an init system (i.e. a custom one), our default skeleton
> +# is most probably unfit; so, allow only the custom skeleton in that case.
>  config BR2_ROOTFS_SKELETON_DEFAULT
>  	bool "default target skeleton"
> +	depends on !BR2_INIT_NONE
>  	help
>  	  Use default target skeleton
>  
> +comment "default target skeleton needs an init system"
> +	depends on BR2_INIT_NONE
> +
>  config BR2_ROOTFS_SKELETON_CUSTOM
>  	bool "custom target skeleton"
>  	help
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 02/20] core/pkg-generic: add variable to skip skeleton dependency

[Thomas Petazzoni]

Hello,

On Tue, 18 Jul 2017 19:25:28 +0200, Yann E. MORIN wrote:
> We currently skip the skeleton dependency by checking if the current
> package is the skeleton.
> 
> We are going to introduce more skeleton-related packages, so we
> need a way to exclude the skeleton dependency for those, or we'd
> get a circular dependency, for the same reason we need to skip
> the toolchain dependency.
> 
> Instead of checking for all the skeleton-providing packages in the core
> infra, add a new package options so that packages can express they do
> not need the dependency on the skeleton, like we have an option to avoid
> the depednency on the toolchain. The only packages that will use that
> option are probably the skeletons, so we need not document this
> variable, like we did not document the option to exclude the dependency
> on the toolchain.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Reviewed-by: Romain Naour <romain.naour@gmail.com>
> ---
>  package/pkg-generic.mk       | 7 +++++--
>  package/skeleton/skeleton.mk | 1 +
>  2 files changed, 6 insertions(+), 2 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 03/20] package/skeleton: add macro to rsync skeleton directory

[Thomas Petazzoni]

Hello,

On Tue, 18 Jul 2017 19:25:29 +0200, Yann E. MORIN wrote:
> The rsync command is currently called from a single location. In
> follow-up patches, it will be called from various places then.
> 
> It will be easier to update and fix this command if it is defined in a
> single location, rather than if duplicated in many places.
> 
> Define a macro that skeleton packages can call to install their skeleton
> directories. Since at least one occurence of it will want to install in
> staging, the macro must accept a destination path, rather than hard-code
> target as the destination.
> 
> Since it is to be used by other skeleton packages, define it early,
> outside of any conditional block.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Reviewed-by: Romain Naour <romain.naour@gmail.com>
> ---
>  package/skeleton/skeleton.mk | 16 +++++++++++++---
>  1 file changed, 13 insertions(+), 3 deletions(-)

I know Arnout was not entirely happy about this solution, but Arnout
comments can be addressed later on, they were not really crucial, and I
want to move forward with applying the rest of the series. Therefore:
patch applied, after fixing the typo pointed by Arnout in the commit
log.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 04/20] package/skeleton: make SKELETON_LIB_SYMLINK a macro

[Thomas Petazzoni]

Hello,

On Tue, 18 Jul 2017 19:25:30 +0200, Yann E. MORIN wrote:
> Rather than duplicate that code in all skeleton packages, make this a
> macro, so that it is easier to use.

Is it really going to be used in *all* skeleton packages ? Because if
it is, it means it's the exact same call that will be duplicated
everywhere, which would mean it should be factored out.

But in practice, it seems it's not used in all skeleton packages, but
just in skeleton-common.mk and skeleton-custom.mk, correct ?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 08/20] system: with no init system, only allow custom skeleton

[Yann E. MORIN]

Ranout, All,

On 2017-07-22 15:28 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > When there is no init system (i.e. a custom one), our bundled default
> > skeleton is most probably un-fit for that (non-)init system.
> > 
> > This will be the case when we introduce per init system skeletons. The
> > systemd skeleton will be unfit except for a systemd init; the sysv
> > skeleton will be unfit except for a sysv-like init system.
> > 
> > In case they are using no init system (really, a custom one), the user
> > should be responsible for providing their own, custom skeleton as well.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
>  NACK on this one. Our skeleton (as in, the contents of system/skeleton) is
> already init-agnostic. If you use another init system (shell script, or no init
> at all because you just use it as a chroot, or whatever), you will *still* need
> that skelecton. Indeed, the skeleton contains things like the /sys and /dev
> directories, a passwd file, ....
> 
>  If there are things in that skeleton that you don't need for a custom init
> system, then they probably shouldn't even be in the skeleton.

This patch is reverted later in the series, once all the types of
skeleton are in place. I could not see how to do it sanely and cleanly
otherwise.

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> > ---
> >  system/Config.in | 6 ++++++
> >  1 file changed, 6 insertions(+)
> > 
> > diff --git a/system/Config.in b/system/Config.in
> > index 828df4217e..ed539dcbe0 100644
> > --- a/system/Config.in
> > +++ b/system/Config.in
> > @@ -3,11 +3,17 @@ menu "System configuration"
> >  choice
> >  	prompt "Root FS skeleton"
> >  
> > +# When not using an init system (i.e. a custom one), our default skeleton
> > +# is most probably unfit; so, allow only the custom skeleton in that case.
> >  config BR2_ROOTFS_SKELETON_DEFAULT
> >  	bool "default target skeleton"
> > +	depends on !BR2_INIT_NONE
> >  	help
> >  	  Use default target skeleton
> >  
> > +comment "default target skeleton needs an init system"
> > +	depends on BR2_INIT_NONE
> > +
> >  config BR2_ROOTFS_SKELETON_CUSTOM
> >  	bool "custom target skeleton"
> >  	help
> > 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 04/20] package/skeleton: make SKELETON_LIB_SYMLINK a macro

[Yann E. MORIN]

Thomas, All,

On 2017-07-22 15:50 +0200, Thomas Petazzoni spake thusly:
> On Tue, 18 Jul 2017 19:25:30 +0200, Yann E. MORIN wrote:
> > Rather than duplicate that code in all skeleton packages, make this a
> > macro, so that it is easier to use.
> 
> Is it really going to be used in *all* skeleton packages ? Because if
> it is, it means it's the exact same call that will be duplicated
> everywhere, which would mean it should be factored out.
> 
> But in practice, it seems it's not used in all skeleton packages, but
> just in skeleton-common.mk and skeleton-custom.mk, correct ?

Right.

Cordialement,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Arnout Vandecappelle]

 I think this patch should go in right away, even before the skeleton/systemd
fixes. Yes, that may make some tests fail, but that's fine.

On 18-07-17 19:25, Yann E. MORIN wrote:
> The "builtin" kernel does not boot a systemd-based system, so
> we resort to building the same one as currently used by our
> qemu_arm_vexpress_defconfig.
> 
> We test the 11 following combinations:
> 
>   - busybox, read-only, without network
>   - busybox, read-only, with network
>   - busybox, read-write, without network
>   - busybox, read-write, with network
> 
>   - basic systemd, read-only, network w/ ifupdown
>   - basic systemd, read-only, network w/ networkd
>   - basic systemd, read-write, network w/ ifupdown
>   - basic systemd, read-write, network w/ networkd
> 
>   - full systemd, read-only, network w/ networkd
>   - full systemd, read-write, network w/ networkd
> 
>   - no init system, read-only, without network

 Given that we're testing init here, it would be nice to also add tests with the
cpio filesystem, which runs a shell script before starting /sbin/init.

 It's probably sufficient to just do the maximal test with this scenario.

 Oh, and a test with sysvinit would also be nice...

> The tests just verify what the /sbin/init binary is, and that we were
> able to grab an IP address. More tests can be added later, for example
> to check each systemd features (journal, tmpfiles...)
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

 I have a bunch of comments, but nothing critical so

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

[snip]
> +    def startEmulator(self, fs_type, kernel=None, dtb=None, init=None):

 PEP8 says snake-case for functions. I'm not sure if the rest of the testing
infra satisfies this, but let's try to not make things worse.

> +        img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
> +        subprocess.call(["truncate", "-s", "%1M", img])

 It's a bit weird to do that here, but I see no better way either.

> +
> +        options = ["-drive",
> +                   "file={},if=sd,format=raw".format(img),
> +                   "-M", "vexpress-a9"]
> +
> +        if kernel is None:
> +            kernel = "builtin"
> +        else:
> +            kernel = os.path.join(self.builddir, "images", kernel)

 This condition should probably be refactored into emulator.boot itself -
although admittedly that doesn't have a reference to builddir at the moment.

> +            options.extend(["-dtb", os.path.join(self.builddir, "images",
> +                                             "{}.dtb".format(dtb))])

 This should only be done if dtb is not None.

> +
> +        kernel_cmdline = ["root=/dev/mmcblk0",
> +                          "rootfstype={}".format(fs_type),
> +                          "rootwait",
> +                          "ro",
> +                          "console=ttyAMA0"]
> +
> +        if not init is None:
> +            kernel_cmdline.extend(["init={}".format(init)])

 Just a single element, so append() instead of extend().

> +
> +        self.emulator.boot(arch="armv7",
> +                           kernel=kernel,
> +                           kernel_cmdline=kernel_cmdline,
> +                           options=options)
> +
> +        if init is None:
> +            self.emulator.login()

 This looks weird so warrants a comment:

 # The init argument is passed when there is no /sbin/init and we instead
 # directly run an executable. So when init is None, we have to log in.

> +
> +    def checkInit(self, path):
> +        cmd = "cmp /proc/1/exe {}".format(path)
> +        _, exit_code = self.emulator.run(cmd)
> +        self.assertEqual(exit_code, 0)
> +
> +    def checkNetwork(self, interface, exitCode=0):
> +        cmd = "ip addr show {} |grep inet".format(interface)
> +        _, exit_code = self.emulator.run(cmd)
> +        self.assertEqual(exit_code, exitCode)

 These you could also run automatically, and request the arguments from the
instance:

    def test_run(self):
        self.startEmulator(self.fs_type())
        self.checkInit(self.init_path())
        if self.have_network():
            self.checkNetwork("eth0")
        else:
            self.checkNetwork("eth0", 1)

 Ideally you'd then make this an ABC class and declare those callbacks as
@abstractmethod, but perhaps that's going too far for you :-)

[snip]
> +class TestInitSystemNone(InitSystemBase):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        BR2_INIT_NONE=y
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        BR2_TARGET_ROOTFS_SQUASHFS=y
> +        """
> +
> +    def test_run(self):
> +        self.startEmulator(fs_type="squashfs", init="/bin/sh")
> +        index = self.emulator.qemu.expect(["/bin/sh: can't access tty; job control turned off", pexpect.TIMEOUT], timeout=60)
> +        if index != 0:
> +            self.emulator.logfile.write("==> System does not boot")
> +            raise SystemError("System does not boot")

 Why do you need to raise an exception here? Why not

    if index != 0:
        self.fail(==> System does not boot")
        return


 Regards,
 Arnout

[snip]
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 03/20] package/skeleton: add macro to rsync skeleton directory

[Yann E. MORIN]

Arnout, All,

On 2017-07-22 15:02 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > Define a macro that skeleton packages can call to install their skeleton
> > directories. Since at least one occurence of it will want to install in
>                                   occurrence
> > staging, the macro must accept a destination path, rather than hard-code
> > target as the destination.
> 
>  By the way, the overlay copying uses almost the same rsync arguments, so they
> can be used there as well.

This series was only about the skeleton thingy, so I refrained from
doing cleanup works elsewhere. Or you'd have had a series with far more
than 20 patches. ;-)

But yeah, I see your point, and that can be added in the future.

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Yann E. MORIN]

Arnout, All,

On 2017-07-22 16:32 +0200, Arnout Vandecappelle spake thusly:
>  I think this patch should go in right away, even before the skeleton/systemd
> fixes. Yes, that may make some tests fail, but that's fine.

I am not so sure. We don;t know how long it will take this series to get
in in its entirety.

OTOH, ahving the tests in right now would be quite an incentive to get
the rest in. ;-]

> On 18-07-17 19:25, Yann E. MORIN wrote:
> > The "builtin" kernel does not boot a systemd-based system, so
> > we resort to building the same one as currently used by our
> > qemu_arm_vexpress_defconfig.
> > 
> > We test the 11 following combinations:
> > 
> >   - busybox, read-only, without network
> >   - busybox, read-only, with network
> >   - busybox, read-write, without network
> >   - busybox, read-write, with network
> > 
> >   - basic systemd, read-only, network w/ ifupdown
> >   - basic systemd, read-only, network w/ networkd
> >   - basic systemd, read-write, network w/ ifupdown
> >   - basic systemd, read-write, network w/ networkd
> > 
> >   - full systemd, read-only, network w/ networkd
> >   - full systemd, read-write, network w/ networkd
> > 
> >   - no init system, read-only, without network
> 
>  Given that we're testing init here, it would be nice to also add tests with the
> cpio filesystem, which runs a shell script before starting /sbin/init.

As I said, we can add more tests in the future. I already have a few
ideas of what we could/should add, but I'll refrain to do so for now,
to concentrate on doing the requested cleanups on this series. ;-)

>  It's probably sufficient to just do the maximal test with this scenario.
>  Oh, and a test with sysvinit would also be nice...
> 
> > The tests just verify what the /sbin/init binary is, and that we were
> > able to grab an IP address. More tests can be added later, for example
> > to check each systemd features (journal, tmpfiles...)
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> 
>  I have a bunch of comments, but nothing critical so
> 
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
> [snip]
> > +    def startEmulator(self, fs_type, kernel=None, dtb=None, init=None):
> 
>  PEP8 says snake-case for functions. I'm not sure if the rest of the testing
> infra satisfies this, but let's try to not make things worse.

We already have BRTest.setUp() and BRTest.tearDown(), so I mimicked
that. And I also applied Thomas suggestion to go camel-case with
functions (reviewed on IRC).

> > +        img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
> > +        subprocess.call(["truncate", "-s", "%1M", img])
> 
>  It's a bit weird to do that here, but I see no better way either.
> 
> > +
> > +        options = ["-drive",
> > +                   "file={},if=sd,format=raw".format(img),
> > +                   "-M", "vexpress-a9"]
> > +
> > +        if kernel is None:
> > +            kernel = "builtin"
> > +        else:
> > +            kernel = os.path.join(self.builddir, "images", kernel)
> 
>  This condition should probably be refactored into emulator.boot itself -
> although admittedly that doesn't have a reference to builddir at the moment.

I was planning on reworking the test infra as you suggested, because I
also found it did make sense, but again, that is out-of-scope for this
series.

> > +            options.extend(["-dtb", os.path.join(self.builddir, "images",
> > +                                             "{}.dtb".format(dtb))])
> 
>  This should only be done if dtb is not None.

Right.

> > +
> > +        kernel_cmdline = ["root=/dev/mmcblk0",
> > +                          "rootfstype={}".format(fs_type),
> > +                          "rootwait",
> > +                          "ro",
> > +                          "console=ttyAMA0"]
> > +
> > +        if not init is None:
> > +            kernel_cmdline.extend(["init={}".format(init)])
> 
>  Just a single element, so append() instead of extend().

Right.

> > +
> > +        self.emulator.boot(arch="armv7",
> > +                           kernel=kernel,
> > +                           kernel_cmdline=kernel_cmdline,
> > +                           options=options)
> > +
> > +        if init is None:
> > +            self.emulator.login()
> 
>  This looks weird so warrants a comment:
> 
>  # The init argument is passed when there is no /sbin/init and we instead
>  # directly run an executable. So when init is None, we have to log in.

Ack.

> > +
> > +    def checkInit(self, path):
> > +        cmd = "cmp /proc/1/exe {}".format(path)
> > +        _, exit_code = self.emulator.run(cmd)
> > +        self.assertEqual(exit_code, 0)
> > +
> > +    def checkNetwork(self, interface, exitCode=0):
> > +        cmd = "ip addr show {} |grep inet".format(interface)
> > +        _, exit_code = self.emulator.run(cmd)
> > +        self.assertEqual(exit_code, exitCode)
> 
>  These you could also run automatically, and request the arguments from the
> instance:
> 
>     def test_run(self):
>         self.startEmulator(self.fs_type())
>         self.checkInit(self.init_path())
>         if self.have_network():
>             self.checkNetwork("eth0")
>         else:
>             self.checkNetwork("eth0", 1)
> 
>  Ideally you'd then make this an ABC class and declare those callbacks as
> @abstractmethod, but perhaps that's going too far for you :-)

Is it really necessary that we go this route? It is much more complex,
for a dubious gain, if at all.

I'd prefer we keep with simple, plain code, so that everyone can
contribute more easily...

> [snip]
> > +class TestInitSystemNone(InitSystemBase):
> > +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> > +        """
> > +        BR2_INIT_NONE=y
> > +        # BR2_TARGET_ROOTFS_TAR is not set
> > +        BR2_TARGET_ROOTFS_SQUASHFS=y
> > +        """
> > +
> > +    def test_run(self):
> > +        self.startEmulator(fs_type="squashfs", init="/bin/sh")
> > +        index = self.emulator.qemu.expect(["/bin/sh: can't access tty; job control turned off", pexpect.TIMEOUT], timeout=60)
> > +        if index != 0:
> > +            self.emulator.logfile.write("==> System does not boot")
> > +            raise SystemError("System does not boot")
> 
>  Why do you need to raise an exception here? Why not
> 
>     if index != 0:
>         self.fail(==> System does not boot")
>         return

I did exactly as is currently done in Emulator.login():

    index = self.qemu.expect(["buildroot login:", pexpect.TIMEOUT],
                             timeout=60)
    if index != 0:
        self.logfile.write("==> System does not boot")
        raise SystemError("System does not boot")

Regards,
Yann E. MORIN.

> 
>  Regards,
>  Arnout
> 
> [snip]
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Thomas Petazzoni]

Hello,

On Sat, 22 Jul 2017 16:45:52 +0200, Yann E. MORIN wrote:

> >  PEP8 says snake-case for functions. I'm not sure if the rest of the testing
> > infra satisfies this, but let's try to not make things worse.  
> 
> We already have BRTest.setUp() and BRTest.tearDown(), so I mimicked
> that. And I also applied Thomas suggestion to go camel-case with
> functions (reviewed on IRC).

I don't think I said to use camel case for functions. What I said is to
not use ThisIsAFunction for functions/methods, because that's normally
used for classes.

> >     def test_run(self):
> >         self.startEmulator(self.fs_type())
> >         self.checkInit(self.init_path())
> >         if self.have_network():
> >             self.checkNetwork("eth0")
> >         else:
> >             self.checkNetwork("eth0", 1)
> > 
> >  Ideally you'd then make this an ABC class and declare those callbacks as
> > @abstractmethod, but perhaps that's going too far for you :-)  
> 
> Is it really necessary that we go this route? It is much more complex,
> for a dubious gain, if at all.
> 
> I'd prefer we keep with simple, plain code, so that everyone can
> contribute more easily...

Fully agreed with Yann. I know there are lots of funky but useful
constructs / features in Python, but I'd like to keep the testing
infrastructure to a usage of Python that can reasonably be understood
by the average C programmer. So, a good indicator is: do Yann and
Thomas understand what it does? If the answer is no, the construct
shouldn't be used :-)

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 04/20] package/skeleton: make SKELETON_LIB_SYMLINK a macro

[Thomas Petazzoni]

Hello,

On Tue, 18 Jul 2017 19:25:30 +0200, Yann E. MORIN wrote:
> Rather than duplicate that code in all skeleton packages, make this a
> macro, so that it is easier to use.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  package/skeleton/skeleton.mk | 30 ++++++++++++++++++------------
>  1 file changed, 18 insertions(+), 12 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 04/20] package/skeleton: make SKELETON_LIB_SYMLINK a macro

[Thomas Petazzoni]

Hello,

On Sat, 22 Jul 2017 14:56:19 +0200, Arnout Vandecappelle wrote:

> > +# Make a symlink lib32->lib or lib64->lib as appropriate.
> > +# MIPS64/n32 requires lib32 even though it's a 64-bit arch.
> > +# $(1): base dir (either staging or target)
> > +ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)  
> 
>  This makes me wonder: are we doing the right thing with NABI32? What is the
> semantics of BR2_ARCH_IS_64? I would say, it doesn't specify that you have a
> 32-bit or 64-bit capable CPU; instead, it says that you have a 32-bit or 64-bit
> userspace. I.e., that void* and long are 32 resp. 64 bits. Indeed, I don't think
> it's really relevant for Buildroot to know about the CPU capabilities, the
> important thing is the userpace ABI, i.e. how we compile packages.
> 
>  So, with that in mind, I think we should NOT select BR2_ARCH_IS_64 for the n32
> ABI. Similarly, when we add x32, it should also not be BR2_ARCH_IS_64.

I see your reasoning, but I believe it might be confusing to have
BR2_ARCH_IS_64 be false on a 64 bit architecture. However, we could
introduce something like BR2_UABI_IS_64, to define whether the
userspace ABI is 32 or 64 bits.

But doing that involves reviewing all the places where BR2_ARCH_IS_64
is used, and check if we're using it to say "the architecture is 64
bits" or "the userspace ABI is 32 bits".

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Arnout Vandecappelle]

On 22-07-17 16:45, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2017-07-22 16:32 +0200, Arnout Vandecappelle spake thusly:
>>  I think this patch should go in right away, even before the skeleton/systemd
>> fixes. Yes, that may make some tests fail, but that's fine.
> 
> I am not so sure. We don;t know how long it will take this series to get
> in in its entirety.
> 
> OTOH, ahving the tests in right now would be quite an incentive to get
> the rest in. ;-]

 Well, we already have two tests failing and nobody seems to fix them...

[snip]
>>> +    def startEmulator(self, fs_type, kernel=None, dtb=None, init=None):
>>
>>  PEP8 says snake-case for functions. I'm not sure if the rest of the testing
>> infra satisfies this, but let's try to not make things worse.
> 
> We already have BRTest.setUp() and BRTest.tearDown(), so I mimicked
> that.

 Well, that's because python unittest predates PEP8 and it already had these
names, so they couldn't be changed.


 There are two tools, pep8 and flake8, that help checking PEP8 compliance.

[snip]
>>  These you could also run automatically, and request the arguments from the
>> instance:
>>
>>     def test_run(self):
>>         self.startEmulator(self.fs_type())
>>         self.checkInit(self.init_path())
>>         if self.have_network():
>>             self.checkNetwork("eth0")
>>         else:
>>             self.checkNetwork("eth0", 1)
>>
>>  Ideally you'd then make this an ABC class and declare those callbacks as
>> @abstractmethod, but perhaps that's going too far for you :-)
> 
> Is it really necessary that we go this route? It is much more complex,
> for a dubious gain, if at all.

 That's why I said "ideally".

 The callbacks (or they could be class members as well, like we have with
config) are indeed useful I think.


> I'd prefer we keep with simple, plain code, so that everyone can
> contribute more easily...

 Fair enough.

 Regards,
 Arnout

[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 06/20] system: move setting getty to the corresponding init systems

[Thomas Petazzoni]

Hello,

On Sat, 22 Jul 2017 15:11:57 +0200, Arnout Vandecappelle wrote:
> [snip]
> > +define BUSYBOX_SET_GETTY
> > +	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(SYSTEM_GETTY_PORT)::respawn:/sbin/getty -L $(SYSTEM_GETTY_OPTIONS) $(SYSTEM_GETTY_PORT) $(SYSTEM_GETTY_BAUDRATE) $(SYSTEM_GETTY_TERM) #~' \
> > +		$(TARGET_DIR)/etc/inittab
> > +endef  
> [snip]
> > +define SYSVINIT_SET_GETTY
> > +	$(SED) '/# GENERIC_SERIAL$$/s~^.*#~$(shell echo $(SYSTEM_GETTY_PORT) | tail -c+4)::respawn:/sbin/getty -L $(SYSTEM_GETTY_OPTIONS) $(SYSTEM_GETTY_PORT) $(SYSTEM_GETTY_BAUDRATE) $(SYSTEM_GETTY_TERM) #~' \
> > +		$(TARGET_DIR)/etc/inittab
> > +endef  
> 
>  The only difference between these two is that for sysvinit we have to make sure
> it's only 4 characters. But it actually doesn't hurt to do the same for busybox
> as well. So perhaps we can factor the two in a single SYSTEM_SET_GETTY ?

I am not sure if that works, because the semantic of the first field is
not the same between sysvinit and Busybox init.

For sysvinit (http://www.manpages.info/linux/inittab.5.html)

       id     is a unique sequence of 1-4 characters which identifies an entry
	      in inittab (for versions of sysvinit compiled with  libraries  <
	      5.2.18 or a.out libraries the limit is 2 characters).

	      Note:  For  gettys or other login processes, the id field should
	      be the tty suffix of the corresponding tty,  e.g.	 1  for	 tty1.
	      Otherwise, the login accounting might not work correctly.

For Busybox init:

# <id>: WARNING: This field has a non-traditional meaning for BusyBox init!
#
#       The id field is used by BusyBox init to specify the controlling tty for
#       the specified process to run on.  The contents of this field are
#       appended to "/dev/" and used as-is.  There is no need for this field to
#       be unique, although if it isn't you may have strange results.  If this
#       field is left blank, then the init's stdin/out will be used.

Hence, with Busybox init, if you pass a truncated "id", the command
might fail to run as it will try to open a /dev/<something> that
doesn't exist. So, if you truncate ttyAMA0, it will be just ttyA,
Busybox will try to open /dev/ttyA and it will fail.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Thomas Petazzoni]

Hello,

On Sat, 22 Jul 2017 21:53:38 +0200, Arnout Vandecappelle wrote:

> > OTOH, ahving the tests in right now would be quite an incentive to get
> > the rest in. ;-]  
> 
>  Well, we already have two tests failing and nobody seems to fix them...

The patches fixing them are here:

  http://patchwork.ozlabs.org/patch/783135/
  http://patchwork.ozlabs.org/patch/783137/
  http://patchwork.ozlabs.org/patch/783136/

and nobody reviews them.

(Actually only the first patch of the series is needed to fix the
tests)

I made those on July 1st, pretty much the day we started running the
tests in Gitlab CI. So it's a bit hard to now read your complaint that
nobody cares about the failing tests...

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 06/20] system: move setting getty to the corresponding init systems

[Thomas Petazzoni]

Hello,

On Tue, 18 Jul 2017 19:25:32 +0200, Yann E. MORIN wrote:
> Currently, setting the getty is done:
>   - by the skeleton package when the init system is either busybox or
>     sysvinit;
>   - by the systemd package when the init system is systemd;
> both by registering a target-finalize hook.
> 
> This is not very consistent.
> 
> Move setting the getty out of the skeleton and into the package that
> provides the init system, by registering a per-package target-fialize
> hook.
> 
> This offloads yet a bit more out of the skeleton, so that it is easier
> to properly separate the skeletons for the various init systems.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  package/busybox/busybox.mk   | 12 +++++++++++-
>  package/skeleton/skeleton.mk | 22 ----------------------
>  package/sysvinit/sysvinit.mk |  8 ++++++++
>  system/system.mk             | 11 +++++++++++
>  4 files changed, 30 insertions(+), 23 deletions(-)

Applied to master, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 07/20] system: move remounting / to the corresponding init systems

[Thomas Petazzoni]

Hello,

On Tue, 18 Jul 2017 19:25:33 +0200, Yann E. MORIN wrote:
> Currently, remounting / read-write (or not) is done by the skeleton
> package when the init system is eitehr busybox or sysvinit, by
> registering a target-finalize hook; it is not done at all for systemd.
> 
> Move registering this target-finalize hook to both of busybox and
> sysvinit. Leave systemd alone, we'll take care of it later.
> 
> Rename the macro to a more meaningful name, and move it to system.mk
> with the other such macros.
> 
> Yet a little bit less init-system knowledge in the skeleton.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  package/busybox/busybox.mk   |  3 +++
>  package/skeleton/skeleton.mk | 15 ---------------
>  package/sysvinit/sysvinit.mk |  3 +++
>  system/system.mk             | 15 +++++++++++++++
>  4 files changed, 21 insertions(+), 15 deletions(-)

Applied to master, after dropping the two comments, as pointed by
Arnout.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] Is MIPS_NABI32 a 64-bit architecture? [was: [PATCH 04/20] package/skeleton: make SKELETON_LIB_SYMLINK a macro]

[Arnout Vandecappelle]

On 22-07-17 21:47, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat, 22 Jul 2017 14:56:19 +0200, Arnout Vandecappelle wrote:
> 
>>> +# Make a symlink lib32->lib or lib64->lib as appropriate.
>>> +# MIPS64/n32 requires lib32 even though it's a 64-bit arch.
>>> +# $(1): base dir (either staging or target)
>>> +ifeq ($(BR2_ARCH_IS_64)$(BR2_MIPS_NABI32),y)  
>>
>>  This makes me wonder: are we doing the right thing with NABI32? What is the
>> semantics of BR2_ARCH_IS_64? I would say, it doesn't specify that you have a
>> 32-bit or 64-bit capable CPU; instead, it says that you have a 32-bit or 64-bit
>> userspace. I.e., that void* and long are 32 resp. 64 bits. Indeed, I don't think
>> it's really relevant for Buildroot to know about the CPU capabilities, the
>> important thing is the userpace ABI, i.e. how we compile packages.
>>
>>  So, with that in mind, I think we should NOT select BR2_ARCH_IS_64 for the n32
>> ABI. Similarly, when we add x32, it should also not be BR2_ARCH_IS_64.
> 
> I see your reasoning, but I believe it might be confusing to have
> BR2_ARCH_IS_64 be false on a 64 bit architecture. However, we could
> introduce something like BR2_UABI_IS_64, to define whether the
> userspace ABI is 32 or 64 bits.
> 
> But doing that involves reviewing all the places where BR2_ARCH_IS_64
> is used, and check if we're using it to say "the architecture is 64
> bits" or "the userspace ABI is 32 bits".

 Well, first of all, there is currently only one arch in this situation, and
that's MIPS64/n32. We don't support x32 or ilp32 yet, and I don't think there's
something similar for PPC.

 Outside of arch/, BR2_ARCH_IS_64 is used in 15 places. 2 of them are
x86_64-specific (grub2, syslinux). 3 already treat MIPS_N32 specially. 6 look
like they should also treat MIPS_N32 specially, but since that ABI doesn't get
tested in the autobuilders (right?) we haven't noticed.

 There is one case that is interesting: BR2_TOOLCHAIN_HAS_SYNC_8 defaults y if
BR2_ARCH_IS_64. I guess that that is indeed true for n32, x32 and ilp32. If so,
it indeed makes sense to split off BR2_ARCH_IS_64 and BR2_UABI_IS_64. But I
think that that will be the only place where BR2_ARCH_IS_64 will be used.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 06/20] system: move setting getty to the corresponding init systems

[Arnout Vandecappelle]

On 22-07-17 21:57, Thomas Petazzoni wrote:
> Hello,
> 
> On Sat, 22 Jul 2017 15:11:57 +0200, Arnout Vandecappelle wrote:
[snip]
>>  The only difference between these two is that for sysvinit we have to make sure
>> it's only 4 characters. But it actually doesn't hurt to do the same for busybox
>> as well. So perhaps we can factor the two in a single SYSTEM_SET_GETTY ?
> 
> I am not sure if that works, because the semantic of the first field is
> not the same between sysvinit and Busybox init.

 Right! I thought busybox init just ignored it, I forgot about the controlling
terminal.

 Regards,
 Arnout

> 
> For sysvinit (http://www.manpages.info/linux/inittab.5.html)
> 
>        id     is a unique sequence of 1-4 characters which identifies an entry
> 	      in inittab (for versions of sysvinit compiled with  libraries  <
> 	      5.2.18 or a.out libraries the limit is 2 characters).
> 
> 	      Note:  For  gettys or other login processes, the id field should
> 	      be the tty suffix of the corresponding tty,  e.g.	 1  for	 tty1.
> 	      Otherwise, the login accounting might not work correctly.
> 
> For Busybox init:
> 
> # <id>: WARNING: This field has a non-traditional meaning for BusyBox init!
> #
> #       The id field is used by BusyBox init to specify the controlling tty for
> #       the specified process to run on.  The contents of this field are
> #       appended to "/dev/" and used as-is.  There is no need for this field to
> #       be unique, although if it isn't you may have strange results.  If this
> #       field is left blank, then the init's stdin/out will be used.
> 
> Hence, with Busybox init, if you pass a truncated "id", the command
> might fail to run as it will try to open a /dev/<something> that
> doesn't exist. So, if you truncate ttyAMA0, it will be just ttyA,
> Busybox will try to open /dev/ttyA and it will fail.
> 
> Best regards,
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 08/20] system: with no init system, only allow custom skeleton

[Arnout Vandecappelle]

On 22-07-17 15:53, Yann E. MORIN wrote:
> Ranout, All,
> 
> On 2017-07-22 15:28 +0200, Arnout Vandecappelle spake thusly:
>> On 18-07-17 19:25, Yann E. MORIN wrote:
>>> When there is no init system (i.e. a custom one), our bundled default
>>> skeleton is most probably un-fit for that (non-)init system.
>>>
>>> This will be the case when we introduce per init system skeletons. The
>>> systemd skeleton will be unfit except for a systemd init; the sysv
>>> skeleton will be unfit except for a sysv-like init system.
>>>
>>> In case they are using no init system (really, a custom one), the user
>>> should be responsible for providing their own, custom skeleton as well.
>>>
>>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>>
>>  NACK on this one. Our skeleton (as in, the contents of system/skeleton) is
>> already init-agnostic. If you use another init system (shell script, or no init
>> at all because you just use it as a chroot, or whatever), you will *still* need
>> that skelecton. Indeed, the skeleton contains things like the /sys and /dev
>> directories, a passwd file, ....
>>
>>  If there are things in that skeleton that you don't need for a custom init
>> system, then they probably shouldn't even be in the skeleton.
> 
> This patch is reverted later in the series, once all the types of
> skeleton are in place. I could not see how to do it sanely and cleanly
> otherwise.

 OK, in that case

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 I trust that you looked for a way to avoid this temporary workaround.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 09/20] package/skeleton: drop dependency on host-mkpasswd

[Arnout Vandecappelle]

 This makes no sense to me...

On 18-07-17 19:25, Yann E. MORIN wrote:
> Setting the root pasword is done in a target-finalize hook, so we do not
> need to enforce a dependency from the skeleton onto host-mkpasswd.
> 
> Dropping that dependency will simplify making skeleton a virtual
> package (in up-coming changes).

 How so? What's wrong with

SKELETON_COMMON_DEPENDENCIES += host-mkpasswd

in exactly the same place?

 You forgot to mention the other interesting bit: it is now selected, so we are
still sure it will get built before target-finalize.

> 
> This however introduces a slight change in behaviour: previously,
> host-mkpasswd would only be built if we needed to hash the root password
> from its plain-text value. Now, host-mkpasswd is always built as soon as
> the root password is non-empty, even if already pre-hashed.
> 
> Since host-mkpasswd is a really tiny weeny package bundled in Buidlroot,
                                                                Buildroot

> with only two C files, built as a single unit with a single gcc call,
> the overhead is really minimal. Compared to the simplifications this
> will allow in the skeleton packages (plural: common, sysv, systemd,
> custom) to come, this overhead is acceptable.
> 
> Yet another simplification, even if small, to ease providing multiple
> skeletons.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

 Although I don't think it's really *necessary* to remove the dependency, it is
certainly more correct. And we want to select host packages from Config.in
anyway on the long run, so doing it only for not-yet-hashed passwords won't be
possible anyway. Therefore:

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>


 Regards,
 Arnout

> ---
>  package/skeleton/skeleton.mk | 1 -
>  system/Config.in             | 1 +
>  2 files changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
> index 4a66f6ceca..4ec1ecbb51 100644
> --- a/package/skeleton/skeleton.mk
> +++ b/package/skeleton/skeleton.mk
> @@ -121,7 +121,6 @@ SKELETON_ROOT_PASSWORD =
>  else ifneq ($(filter $$1$$% $$5$$% $$6$$%,$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)),)
>  SKELETON_ROOT_PASSWORD = '$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)'
>  else
> -SKELETON_DEPENDENCIES += host-mkpasswd
>  # This variable will only be evaluated in the finalize stage, so we can
>  # be sure that host-mkpasswd will have already been built by that time.
>  SKELETON_ROOT_PASSWORD = "`$(MKPASSWD) -m "$(SKELETON_TARGET_GENERIC_PASSWD_METHOD)" "$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)"`"
> diff --git a/system/Config.in b/system/Config.in
> index ed539dcbe0..5cb921f108 100644
> --- a/system/Config.in
> +++ b/system/Config.in
> @@ -224,6 +224,7 @@ config BR2_ROOTFS_MERGED_USR
>  config BR2_TARGET_ENABLE_ROOT_LOGIN
>  	bool "Enable root login with password"
>  	default y
> +	select BR2_PACKAGE_HOST_MKPASSWD if BR2_TARGET_GENERIC_ROOT_PASSWD != ""
>  	help
>  	  Allow root to log in with a password.
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 10/20] package/skeleton: select it rather than default to y

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Our current skeleton is tailored to sysv-like init systems; it is not
> fit for systemd-based systems. So, in upcoming changes, we'll add
> another skeleton for systemd.
> 
> This means we can no longer have the current skeleton default to 'y', or
> it would be enabled also for systemd, which would be incorrect.
> 
> So, we remove the default to 'y' but have it selected by the default
> skeleton choice.
> 
> However, we do not yet have a way to directly build (really, install)
> the custom skeleton, it is built (really, installed) as a dependency of
> the default skeleton. So we must also forcibly select the default
> skeleton when using a custom one.
> 
> Until we have the mean to do only one or the other; i.e. when we have a
                    means

> virtual skeleton.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

 You're going about it in a pretty complicated way. I think it would have been
simpler to rename skeleton to skeleton-common and introducing the skeleton
virtual package in one commit, and then start splitting them up. However, you've
done it this way now, and it works, so

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout


> ---
>  package/skeleton/Config.in | 1 -
>  system/Config.in           | 2 ++
>  2 files changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/package/skeleton/Config.in b/package/skeleton/Config.in
> index d25147bd92..b22ac66b35 100644
> --- a/package/skeleton/Config.in
> +++ b/package/skeleton/Config.in
> @@ -1,5 +1,4 @@
>  config BR2_PACKAGE_SKELETON
>  	bool
> -	default y
>  	help
>  	  The basic skeleton for your rootfs.
> diff --git a/system/Config.in b/system/Config.in
> index 5cb921f108..9b42fbbbac 100644
> --- a/system/Config.in
> +++ b/system/Config.in
> @@ -8,6 +8,7 @@ choice
>  config BR2_ROOTFS_SKELETON_DEFAULT
>  	bool "default target skeleton"
>  	depends on !BR2_INIT_NONE
> +	select BR2_PACKAGE_SKELETON
>  	help
>  	  Use default target skeleton
>  
> @@ -16,6 +17,7 @@ comment "default target skeleton needs an init system"
>  
>  config BR2_ROOTFS_SKELETON_CUSTOM
>  	bool "custom target skeleton"
> +	select BR2_PACKAGE_SKELETON
>  	help
>  	  Use custom target skeleton.
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 08/20] system: with no init system, only allow custom skeleton

[Yann E. MORIN]

Arnout,  All,

On 2017-07-22 23:18 +0200, Arnout Vandecappelle spake thusly:
> On 22-07-17 15:53, Yann E. MORIN wrote:
> > Ranout, All,
> > 
> > On 2017-07-22 15:28 +0200, Arnout Vandecappelle spake thusly:
> >> On 18-07-17 19:25, Yann E. MORIN wrote:
> >>> When there is no init system (i.e. a custom one), our bundled default
> >>> skeleton is most probably un-fit for that (non-)init system.
> >>>
> >>> This will be the case when we introduce per init system skeletons. The
> >>> systemd skeleton will be unfit except for a systemd init; the sysv
> >>> skeleton will be unfit except for a sysv-like init system.
> >>>
> >>> In case they are using no init system (really, a custom one), the user
> >>> should be responsible for providing their own, custom skeleton as well.
> >>>
> >>> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> >>
> >>  NACK on this one. Our skeleton (as in, the contents of system/skeleton) is
> >> already init-agnostic. If you use another init system (shell script, or no init
> >> at all because you just use it as a chroot, or whatever), you will *still* need
> >> that skelecton. Indeed, the skeleton contains things like the /sys and /dev
> >> directories, a passwd file, ....
> >>
> >>  If there are things in that skeleton that you don't need for a custom init
> >> system, then they probably shouldn't even be in the skeleton.
> > 
> > This patch is reverted later in the series, once all the types of
> > skeleton are in place. I could not see how to do it sanely and cleanly
> > otherwise.
> 
>  OK, in that case
> 
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Thanks.

>  I trust that you looked for a way to avoid this temporary workaround.

I tried, but I could not see a simple, clean and quick solution...

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 11/20] package/skeleton: split out into skeleton-custom

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> For the custom skeleton, we practicaly do nothing, except ensure it
                              do practically

> contains the basic, required directories, and that those are properly
> setup wrt. merged /usr.
> 
> Furthermore, our current skeleton is not fit for systemd, and we'll
> have to split things out into various skeletons.
> 
> So, off-load the custom skeleton into its own package.
> 
> Thus, the existing skeleton package is now limited to:
> 
>   - when using our default skeleton, install and tweak it properly;
> 
>   - when using a custom skeleton, do nothing except for depending on
>     the skeleton-custom package.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

 Just spelling and comment fixes, so

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

[snip]
> +# Provided by the 'skeleton' package:
> +# - SKELETON_RSYNC
> +# - SKELETON_LIB_SYMLINK

 Like I wrote in a different patch: it's SYSTEM_*, not SKELETON_*, and the name
is sufficiently indicative of where it comes from, so the comment can be removed.

> +
> +# The target-dir-warning file is the only thing we customise in the
                                 and the lib{32,64} symlinks are the only things
> +# custom skeleton.
> +define SKELETON_CUSTOM_INSTALL_TARGET_CMDS
> +	$(call SYSTEM_RSYNC,$(SKELETON_CUSTOM_PATH),$(TARGET_DIR))
> +	$(call SYSTEM_LIB_SYMLINK,$(TARGET_DIR))
> +	$(INSTALL) -m 0644 support/misc/target-dir-warning.txt \
> +		$(TARGET_DIR_WARNING_FILE)
> +endef
> +
> +# For the staging dir, we don't really care what we install, but we
> +# need the /lib and /usr/lib appropriately setup. Since we ensure,
> +# above, that they are correct in the skeleton, we can simply copy the
> +# skeleton to staging.

 Great explanation!

 Regards,
 Arnout

> +define SKELETON_CUSTOM_INSTALL_STAGING_CMDS
> +	$(call SYSTEM_RSYNC,$(SKELETON_CUSTOM_PATH),$(STAGING_DIR))
> +	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
> +endef

[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 09/20] package/skeleton: drop dependency on host-mkpasswd

[Yann E. MORIN]

Arnout, All,

On 2017-07-22 23:24 +0200, Arnout Vandecappelle spake thusly:
>  This makes no sense to me...
> 
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > Setting the root pasword is done in a target-finalize hook, so we do not
> > need to enforce a dependency from the skeleton onto host-mkpasswd.
> > 
> > Dropping that dependency will simplify making skeleton a virtual
> > package (in up-coming changes).
> 
>  How so? What's wrong with
> 
> SKELETON_COMMON_DEPENDENCIES += host-mkpasswd
> 
> in exactly the same place?

Indeed, it does not simplify much. But at least, that was a little bit
less code I had to carry over to the new skeleton infra, see below [0].

And as you've seen, this series was no picnic...

>  You forgot to mention the other interesting bit: it is now selected, so we are
> still sure it will get built before target-finalize.



> > This however introduces a slight change in behaviour: previously,
> > host-mkpasswd would only be built if we needed to hash the root password
> > from its plain-text value. Now, host-mkpasswd is always built as soon as
> > the root password is non-empty, even if already pre-hashed.
> > 
> > Since host-mkpasswd is a really tiny weeny package bundled in Buidlroot,
>                                                                 Buildroot

I think I have unconciously associated Buildroot and Beetlejuice in my
head, as Buidlroot sounds too much like Beetlejuice, and I do this typo
all the time. Not that I never typoes anything else, for that matters...
;-]

> > with only two C files, built as a single unit with a single gcc call,
> > the overhead is really minimal. Compared to the simplifications this
> > will allow in the skeleton packages (plural: common, sysv, systemd,
> > custom) to come, this overhead is acceptable.
> > 
> > Yet another simplification, even if small, to ease providing multiple
> > skeletons.

[0[] here I said it was but a small simplification. ;-)

> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
>  Although I don't think it's really *necessary* to remove the dependency, it is
> certainly more correct. And we want to select host packages from Config.in
> anyway on the long run, so doing it only for not-yet-hashed passwords won't be
> possible anyway. Therefore:
> 
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

Thanks! :-)

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> > ---
> >  package/skeleton/skeleton.mk | 1 -
> >  system/Config.in             | 1 +
> >  2 files changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/package/skeleton/skeleton.mk b/package/skeleton/skeleton.mk
> > index 4a66f6ceca..4ec1ecbb51 100644
> > --- a/package/skeleton/skeleton.mk
> > +++ b/package/skeleton/skeleton.mk
> > @@ -121,7 +121,6 @@ SKELETON_ROOT_PASSWORD =
> >  else ifneq ($(filter $$1$$% $$5$$% $$6$$%,$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)),)
> >  SKELETON_ROOT_PASSWORD = '$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)'
> >  else
> > -SKELETON_DEPENDENCIES += host-mkpasswd
> >  # This variable will only be evaluated in the finalize stage, so we can
> >  # be sure that host-mkpasswd will have already been built by that time.
> >  SKELETON_ROOT_PASSWORD = "`$(MKPASSWD) -m "$(SKELETON_TARGET_GENERIC_PASSWD_METHOD)" "$(SKELETON_TARGET_GENERIC_ROOT_PASSWD)"`"
> > diff --git a/system/Config.in b/system/Config.in
> > index ed539dcbe0..5cb921f108 100644
> > --- a/system/Config.in
> > +++ b/system/Config.in
> > @@ -224,6 +224,7 @@ config BR2_ROOTFS_MERGED_USR
> >  config BR2_TARGET_ENABLE_ROOT_LOGIN
> >  	bool "Enable root login with password"
> >  	default y
> > +	select BR2_PACKAGE_HOST_MKPASSWD if BR2_TARGET_GENERIC_ROOT_PASSWD != ""
> >  	help
> >  	  Allow root to log in with a password.
> >  
> > 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 11/20] package/skeleton: split out into skeleton-custom

[Arnout Vandecappelle]

 On second thought...

On 23-07-17 00:31, Arnout Vandecappelle wrote:
>> +define SKELETON_CUSTOM_INSTALL_STAGING_CMDS
>> +	$(call SYSTEM_RSYNC,$(SKELETON_CUSTOM_PATH),$(STAGING_DIR))
>> +	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))

 In skeleton-common, we also have:

	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include

Shouldn't we do that here as well?

 Regards,
 Arnout

>> +endef

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Yann E. MORIN]

Thomas, All,

On 2017-07-22 22:10 +0200, Thomas Petazzoni spake thusly:
> On Sat, 22 Jul 2017 21:53:38 +0200, Arnout Vandecappelle wrote:
> 
> > > OTOH, ahving the tests in right now would be quite an incentive to get
> > > the rest in. ;-]  
> > 
> >  Well, we already have two tests failing and nobody seems to fix them...
> 
> The patches fixing them are here:
> 
>   http://patchwork.ozlabs.org/patch/783135/
>   http://patchwork.ozlabs.org/patch/783137/
>   http://patchwork.ozlabs.org/patch/783136/
> 
> and nobody reviews them.

I gave them a spin here, and they indeed fix the build, but the tests
are still broken because qemu can't boot the resulting system...

I'll investigate that tomorrow...

Regards,
Yann E. MORIN.

> (Actually only the first patch of the series is needed to fix the
> tests)
> 
> I made those on July 1st, pretty much the day we started running the
> tests in Gitlab CI. So it's a bit hard to now read your complaint that
> nobody cares about the failing tests...
> 
> Best regards,
> 
> Thomas
> -- 
> Thomas Petazzoni, CTO, Free Electrons
> Embedded Linux and Kernel engineering
> http://free-electrons.com

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 12/20] package/skeleton: split out into skeleton-common

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Move all the handling of the default skeleton into a new package,
> skeleton-common.
> 
> We don;t name it skeleton-default, because it will be further split
     don't

> later, into a skeleton for sysv and another for systemd, with some parts
> still common between the two. So just name it skeleton-common right now;
> this will save us a rename later.
> 
> We keep in the skeleton package all definitions of macros and variables
> that can be used by other skeleton packages and/or init systems, and
> just offload the actual feeding of the skeleton to the new package.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

[snip]
> +# The skeleton can't depend on the toolchain, since all packages depends on the
> +# skeleton and the toolchain is a target package, as is skeleton.
> +# Hence, skeleton would depends on the toolchain and the toolchain would depend
> +# on skeleton.

 I don't think this comment is very useful - it's pretty obvious. Actually, it
would make sense for SKELETON_COMMON_ADD_SKELETON_DEPENDENCY=NO to imply
SKELETON_COMMON_ADD_TOOLCHAIN_DEPENDENCY=NO, but really we don't need to
complicate the infra for that...

 You can't have too many comments though, so let's not block the patch for that.

> +SKELETON_COMMON_ADD_TOOLCHAIN_DEPENDENCY = NO
> +SKELETON_COMMON_ADD_SKELETON_DEPENDENCY = NO
> +
> +# The skeleton also handles the merged /usr case in the sysroot

 This comment, on the other hand, *is* very useful.

> +SKELETON_COMMON_INSTALL_STAGING = YES
> +
> +SKELETON_COMMON_PATH = system/skeleton

 Not for this patch of course, but I think it would make sense to also move that
here. But you probably thought of that already :-)


 Regards,
 Arnout

[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 13/20] package/skeleton: make it a virtual package

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> We now have two packages that can act as a skeleton, skeleton-common,
> also known as our default skeleton, and skeleton-custom.
> 
> This means that the skeleton package can be a standard virtual package
> now.
> 
> Well, except that, besides being a virtual package, it also provides
> variables and macros that can be used by the other skeletons and/or the
> init systems.

 I guess this is a left-over from when you hadn't moved everything to system.mk
yet...

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

 On this patch, I do have kind of big comments.

[snip]
> diff --git a/package/skeleton-custom/Config.in b/package/skeleton-custom/Config.in
> index b12bd8f73c..601c3b247e 100644
> --- a/package/skeleton-custom/Config.in
> +++ b/package/skeleton-custom/Config.in
> @@ -1,3 +1,6 @@
>  config BR2_PACKAGE_SKELETON_CUSTOM
>  	bool
> -	select BR2_PACKAGE_SKELETON
> +	select BR2_PACKAGE_HAS_SKELETON
> +
> +config BR2_PACKAGE_PROVIDES_SKELETON
> +	default "skeleton-custom" if BR2_PACKAGE_SKELETON_CUSTOM

 In almost all virtual package providers, we do it like:

if BR2_PACKAGE_SKELETON_CUSTOM
config BR2_PACKAGE_PROVIDES_SKELETON
	default "skeleton-custom"
endif

[snip]
>  config BR2_PACKAGE_SKELETON
>  	bool
> -	help
> -	  The basic skeleton for your rootfs.
> +	default y

 For most virtual packages, we don't define the Kconfig symbol (e.g. there is no
BR2_PACKAGE_EGL). Do we need it for skeleton?

> +
> +config BR2_PACKAGE_HAS_SKELETON
> +	bool

 We also don't really need this. It is used by packages that need the virtual
package to check if there is a provider for it. But no package 'depends on'
skeleton (or rather, all of them do), so we don't need this.

> +
> +config BR2_PACKAGE_PROVIDES_SKELETON
> +	string

 This one we do need :-)

[snip]
> diff --git a/system/Config.in b/system/Config.in
> index c27b013784..75b6a8edac 100644
> --- a/system/Config.in
> +++ b/system/Config.in
> @@ -1,5 +1,9 @@
>  menu "System configuration"
>  
> +# Note: usually, it is not possible to select a provider of a virtual
> +# package.

 This is not true cryptodev, jpeg and mysql do exactly that. libressl/openssl
will also do it that way. In fact, a choice is in a way the better way to do it,
because then you're guaranteed not to have conflicting providers.

 So this entire comment is wrong and can be removed. IMO.

 Regards,
 Arnout

> But here we have an exception: there are only two providers
> +# and they only get selected each by separate entries in this choice.
> +# So this is a safe situation.
>  choice
>  	prompt "Root FS skeleton"
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 14/20] package/skeleton-common: simplify staging install

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> We don't really care what's going in staging, as long as it is properly
> setup for merged/non-merged /usr, especially for the lib/ directory.
> 
> So we can just copy the skeleton as-is.
> 
> This simplify the maintenance, should we ever need to tweak the layout:
> we'd just have to do it once in the skeleton directory to have it
> propagated to both target and staging.

 Yes, very nice, I like it!

> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 Regards,
 Arnout

> ---
>  package/skeleton-common/skeleton-common.mk | 14 +++++---------
>  1 file changed, 5 insertions(+), 9 deletions(-)
> 
> diff --git a/package/skeleton-common/skeleton-common.mk b/package/skeleton-common/skeleton-common.mk
> index ee44eaf810..475c07e32d 100644
> --- a/package/skeleton-common/skeleton-common.mk
> +++ b/package/skeleton-common/skeleton-common.mk
> @@ -26,18 +26,14 @@ define SKELETON_COMMON_INSTALL_TARGET_CMDS
>  		$(TARGET_DIR_WARNING_FILE)
>  endef
>  
> -# For the staging dir, we don't really care about /bin and /sbin.
> -# But for consistency with the target dir, and to simplify the code,
> -# we still handle them for the merged or non-merged /usr cases.
> -# Since the toolchain is not yet available, the staging is not yet
> -# populated, so we need to create the directories in /usr
> +# We don't care much about what goes in staging, as long as it is
> +# correctly setup for merged/non-merged /usr. The simplest is to
> +# fill it in with the content of the skeleton.
>  define SKELETON_COMMON_INSTALL_STAGING_CMDS
> -	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/lib
> -	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/bin
> -	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/sbin
> -	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
> +	$(call SYSTEM_RSYNC,$(SKELETON_COMMON_PATH),$(STAGING_DIR))
>  	$(call SYSTEM_USR_SYMLINKS_OR_DIRS,$(STAGING_DIR))
>  	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
> +	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
>  endef
>  
>  SKELETON_COMMON_HOSTNAME = $(call qstrip,$(BR2_TARGET_GENERIC_HOSTNAME))
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 15/20] package/skeleton: introduce sysv- and systemd-specific skeletons

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Currently, we use the same skeleton for sysv-like init systems and
> systemd, even though systemd has some peculiarities that makes out
                                                                 our
> default skeleton unfit.
> 
> So, we'll need to provide different skeletons (really, only part of it)
> for sysv-like and systemd.
> 
> Introduce two new skeleton packages, aptly named skeleton-sysv and
> skeleton-systemd, one for each, that are both providers of the skeleton
> package, in lieue of the skeleton-common, which is now a simple
> dependency of both the two new skeletons.
> 
> Those packages are empty for now. In followup changes:
>   - sysv-specific stuff will be moved out of skeleton-common and into
>     skeleton-sysv;
>   - systemd-specific stuff will be added to skeleton-systemd.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>

Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

 But...

[snip]
> +config BR2_PACKAGE_SKELETON_SYSTEMD
> +	bool
> +	select BR2_PACKAGE_HAS_SKELETON
> +	select BR2_PACKAGE_SKELETON_COMMON
> +
> +config BR2_PACKAGE_PROVIDES_SKELETON
> +	default "skeleton-systemd" if BR2_PACKAGE_SKELETON_SYSTEMD

 Again, we usually do it with if...config...endif. Same for sysv, obviously.

[snip]
> @@ -12,7 +12,9 @@ choice
>  config BR2_ROOTFS_SKELETON_DEFAULT
>  	bool "default target skeleton"
>  	depends on !BR2_INIT_NONE
> -	select BR2_PACKAGE_SKELETON_COMMON
> +	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_SYSV
> +	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_BUSYBOX
> +	select BR2_PACKAGE_SKELETON_SYSTEMD if BR2_INIT_SYSTEMD

 This works, but I'm thinking of an alternative, let's see if it flies:

 Instead of choosing the different skeletons here, we could do it in the select
of BR2_INIT_*. Indeed, you have a skeleton for each init option. Of course, all
the selects there would have to look like

	select BR2_PACKAGE_SKELETON_SYSV if BR2_ROOTFS_SKELETON_DEFAULT


 I'm not entirely sure if it's an improvement, but I have the feeling it's
easier to follow if the different skeleton options are put together with the
different init options.

 Regards,
 Arnout


>  	help
>  	  Use default target skeleton
>  
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> systems does not like that the content of /var be symlinks to /tmp,
  systemd

> especially journald that starts before /tmp is mounted, and thus the
> journal files are hidden from view, which causes wuite a bit if fuss...
                                                   quite
> 
> Instead, move the current /var to a sysv-only skeleton.
> 
> systemd at install time will create the /var content it needs, so we

 I guess systemd will also create a tmpfs on /var?

> just create an empty /var for systemd. Yet, systemd really requires a
> bunch of extra directories to exist, so we also create those.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>

 Commit message needs to be improved (see also below), but with that
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>

> 
> ---
> Note: /home and /srv will be created at runtime if missing, but it is
> better to create them right now, to simplify supporting systemd on a
> RO filesystem in the (near) future.
> ---
>  package/skeleton-systemd/skeleton-systemd.mk              | 7 +++++++
>  package/skeleton-sysv/skeleton-sysv.mk                    | 4 ++++
>  {system => package/skeleton-sysv}/skeleton/dev/log        | 0
>  {system => package/skeleton-sysv}/skeleton/dev/pts/.empty | 0
>  {system => package/skeleton-sysv}/skeleton/dev/shm/.empty | 0

 I guess indeed the /dev stuff is not needed for systemd because there is anyway
a devtmpfs mounted over it. Which makes me wonder: shouldn't these be moved to
the device-table? Only the static devices case needs these to be present... Of
course, such a change is not for this patch, but I'd say that we can just leave
this /dev stuff in the common part - systemd has no less use of it than sysv.

>  {system => package/skeleton-sysv}/skeleton/etc/fstab      | 0

 Why? systemd does use fstab, so it seems weird not to have an fstab in the
systemd case. Well, I guess our skeleton fstab doesn't contain anything relevant
for systemd because all of that is handled already by tmpfiles.
But the commit log should at least explain this.


 Regards,
 Arnout


[snip]


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Arnout Vandecappelle]

On 23-07-17 02:08, Arnout Vandecappelle wrote:
> 
> On 18-07-17 19:25, Yann E. MORIN wrote:
>> systems does not like that the content of /var be symlinks to /tmp,
>   systemd
> 
>> especially journald that starts before /tmp is mounted, and thus the
>> journal files are hidden from view, which causes wuite a bit if fuss...
>                                                    quite
>> Instead, move the current /var to a sysv-only skeleton.
>>
>> systemd at install time will create the /var content it needs, so we
>  I guess systemd will also create a tmpfs on /var?

 Seems the following sentence got dropped:

What happens when a package installs something in /var?


 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 17/20] sytem: no-init systems may use our default, common skeleton

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
> Requiring that no-init system indeed provide their own custom skeletons
                         systems

> is a bit too-high a burden, especially for quick one-off tests when
> testing our packages.
> 
> Allow no-init systems to use our default skeleton, but only use the
> common part, for the specific sysv or systemd ones are unfit.

 For people who use BR2_INIT_NONE to create a chroot-able rootfs, this entails
an important change: the /var stuff (e.g. /var/run symlink) is no longer there.
This could have implications for some use cases. So I think a comment about this
should be added to the release notes.

> 
> Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  package/skeleton-common/Config.in | 29 +++++++++++++++++++++++++++++
>  system/Config.in                  |  5 +----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/package/skeleton-common/Config.in b/package/skeleton-common/Config.in
> index 9abed4292a..ba0abe9d35 100644
> --- a/package/skeleton-common/Config.in
> +++ b/package/skeleton-common/Config.in
> @@ -1,2 +1,31 @@
> +# Normally, skeleton-common is a dependency of either the sysv or
> +# the systemd ones, so it should not declare itself as a provider
> +# for the skeleton virtual package.
> +#
> +# However, when one uses a no-init system (BR2_INIT_NONE), neither
> +# the sysv not the systemd skeleton are enabled, so skeleton-common
> +# is the one and only skeleton available, so it must declare itself
> +# as a provider.
> +#
> +# We can safely select _HAS_SKELETON, because either we are the only
> +# provider (in which case we must select it) or we were selected by
> +# another skeleton which would have selected it by itself (in which
> +# case us selecting _HAS_SKELETON is harmless).
> +#
>  config BR2_PACKAGE_SKELETON_COMMON
>  	bool
> +	select BR2_PACKAGE_HAS_SKELETON

 Instead of putting it here...

> +
> +if BR2_PACKAGE_SKELETON_COMMON
> +
> +config BR2_PACKAGE_SKELETON_COMMON_ONLY
> +	bool
> +	default y
> +	depends on !BR2_PACKAGE_SKELETON_SYSV
> +	depends on !BR2_PACKAGE_SKELETON_SYSTEMD
> +	depends on !BR2_PACKAGE_SKELETON_CUSTOM

 ... put it here. Then you don't need the comment above, there will always be
just one symbol that selects skeleton.

 But as I wrote in an earlier reply, we don't even need to select
BR2_PACKAGE_HAS_SKELETON, so the point is moot.


 Regards,
 Arnout

[snip]
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Thomas Petazzoni]

Hello,

On Sun, 23 Jul 2017 00:50:16 +0200, Yann E. MORIN wrote:

> > The patches fixing them are here:
> > 
> >   http://patchwork.ozlabs.org/patch/783135/
> >   http://patchwork.ozlabs.org/patch/783137/
> >   http://patchwork.ozlabs.org/patch/783136/
> > 
> > and nobody reviews them.  
> 
> I gave them a spin here, and they indeed fix the build, but the tests
> are still broken because qemu can't boot the resulting system...
> 
> I'll investigate that tomorrow...

Hum, weird, because I'm pretty sure I tested the test cases with those
patches. I'll test again, maybe I missed something.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 02/20] core/pkg-generic: add variable to skip skeleton dependency

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

 >> Instead of checking for all the skeleton-providing packages in the core
 >> infra, add a new package options so that packages can express they do
 >> not need the dependency on the skeleton, like we have an option to avoid
 >> the depednency on the toolchain. The only packages that will use that
 >> option are probably the skeletons, so we need not document this
 >> variable, like we did not document the option to exclude the dependency
 >> on the toolchain.
 >> 
 >> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
 >> Reviewed-by: Romain Naour <romain.naour@gmail.com>

 >  I absolutely hate adding additional variables that are used for only a few
 > packages, but I have no better ideas, so

An alternative could be to enforce a naming convention for these
packages, E.G. skeleton-*, and then use $(filter skeleton%,$(1)) in the
check, but I'm not sure that really is much nicer.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 02/20] core/pkg-generic: add variable to skip skeleton dependency

[Yann E. MORIN]

Peter, All,

On 2017-07-23 11:12 +0200, Peter Korsgaard spake thusly:
> >>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:
> 
>  >> Instead of checking for all the skeleton-providing packages in the core
>  >> infra, add a new package options so that packages can express they do
>  >> not need the dependency on the skeleton, like we have an option to avoid
>  >> the depednency on the toolchain. The only packages that will use that
>  >> option are probably the skeletons, so we need not document this
>  >> variable, like we did not document the option to exclude the dependency
>  >> on the toolchain.
>  >> 
>  >> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
>  >> Reviewed-by: Romain Naour <romain.naour@gmail.com>
> 
>  >  I absolutely hate adding additional variables that are used for only a few
>  > packages, but I have no better ideas, so
> 
> An alternative could be to enforce a naming convention for these
> packages, E.G. skeleton-*, and then use $(filter skeleton%,$(1)) in the
> check, but I'm not sure that really is much nicer.

We also talked about that possibility on IRC, and came to the conclusion
that it was not nice to hard-code magic values in the infra.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Yann E. MORIN]

Thomas, All,

On 2017-07-23 09:58 +0200, Thomas Petazzoni spake thusly:
> On Sun, 23 Jul 2017 00:50:16 +0200, Yann E. MORIN wrote:
> > > The patches fixing them are here:
> > > 
> > >   http://patchwork.ozlabs.org/patch/783135/
> > >   http://patchwork.ozlabs.org/patch/783137/
> > >   http://patchwork.ozlabs.org/patch/783136/
> > > 
> > > and nobody reviews them.  
> > 
> > I gave them a spin here, and they indeed fix the build, but the tests
> > are still broken because qemu can't boot the resulting system...
> > 
> > I'll investigate that tomorrow...
> 
> Hum, weird, because I'm pretty sure I tested the test cases with those
> patches. I'll test again, maybe I missed something.

So, to rule out any sleep-deprivation issue while testign yesterday
evening, I re-spawn the tests here this morning, and they still do not
run.

Maybe it is my qemu version?

I'll try and bisect them between July the 1st and now.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 11/20] package/skeleton: split out into skeleton-custom

[Yann E. MORIN]

On 2017-07-23 00:38 +0200, Arnout Vandecappelle spake thusly:
>  On second thought...
> 
> On 23-07-17 00:31, Arnout Vandecappelle wrote:
> >> +define SKELETON_CUSTOM_INSTALL_STAGING_CMDS
> >> +	$(call SYSTEM_RSYNC,$(SKELETON_CUSTOM_PATH),$(STAGING_DIR))
> >> +	$(call SYSTEM_LIB_SYMLINK,$(STAGING_DIR))
> 
>  In skeleton-common, we also have:
> 
> 	$(INSTALL) -d -m 0755 $(STAGING_DIR)/usr/include
> 
> Shouldn't we do that here as well?

In fact, I argue that we should not have this mkdir at all:
    https://patchwork.ozlabs.org/patch/710672/

I propagated it here because the patch is just about moving stuff, not
cleaning up.

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> >> +endef
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 12/20] package/skeleton: split out into skeleton-common

[Yann E. MORIN]

Arnout, All,

On 2017-07-23 01:06 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > Move all the handling of the default skeleton into a new package,
> > skeleton-common.
> > 
> > We don;t name it skeleton-default, because it will be further split
>      don't
> 
> > later, into a skeleton for sysv and another for systemd, with some parts
> > still common between the two. So just name it skeleton-common right now;
> > this will save us a rename later.
> > 
> > We keep in the skeleton package all definitions of macros and variables
> > that can be used by other skeleton packages and/or init systems, and
> > just offload the actual feeding of the skeleton to the new package.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
> [snip]
> > +# The skeleton can't depend on the toolchain, since all packages depends on the
> > +# skeleton and the toolchain is a target package, as is skeleton.
> > +# Hence, skeleton would depends on the toolchain and the toolchain would depend
> > +# on skeleton.
> 
>  I don't think this comment is very useful - it's pretty obvious. Actually, it
> would make sense for SKELETON_COMMON_ADD_SKELETON_DEPENDENCY=NO to imply
> SKELETON_COMMON_ADD_TOOLCHAIN_DEPENDENCY=NO, but really we don't need to
> complicate the infra for that...

Yeah, it is obvious. But since there was a comment about the toolchain
already in the skeleton, I stubornly duplicated it for the skeleton as
well, and then stubornly propagated to all skeleton packages...

I am entirely fine with dropping both, though.

>  You can't have too many comments though, so let's not block the patch for that.
> 
> > +SKELETON_COMMON_ADD_TOOLCHAIN_DEPENDENCY = NO
> > +SKELETON_COMMON_ADD_SKELETON_DEPENDENCY = NO
> > +
> > +# The skeleton also handles the merged /usr case in the sysroot
> 
>  This comment, on the other hand, *is* very useful.
> 
> > +SKELETON_COMMON_INSTALL_STAGING = YES
> > +
> > +SKELETON_COMMON_PATH = system/skeleton
> 
>  Not for this patch of course, but I think it would make sense to also move that
> here. But you probably thought of that already :-)

Yes, that's is what I did in the first iteration, but Thomas was against
moving it, so I left it alone to stay there.

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> [snip]
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 13/20] package/skeleton: make it a virtual package

[Yann E. MORIN]

On 2017-07-23 01:36 +0200, Arnout Vandecappelle spake thusly:
> 
> 
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > We now have two packages that can act as a skeleton, skeleton-common,
> > also known as our default skeleton, and skeleton-custom.
> > 
> > This means that the skeleton package can be a standard virtual package
> > now.
> > 
> > Well, except that, besides being a virtual package, it also provides
> > variables and macros that can be used by the other skeletons and/or the
> > init systems.
> 
>  I guess this is a left-over from when you hadn't moved everything to system.mk
> yet...

Arg, yes...

> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
>  On this patch, I do have kind of big comments.
> 
> [snip]
> > diff --git a/package/skeleton-custom/Config.in b/package/skeleton-custom/Config.in
> > index b12bd8f73c..601c3b247e 100644
> > --- a/package/skeleton-custom/Config.in
> > +++ b/package/skeleton-custom/Config.in
> > @@ -1,3 +1,6 @@
> >  config BR2_PACKAGE_SKELETON_CUSTOM
> >  	bool
> > -	select BR2_PACKAGE_SKELETON
> > +	select BR2_PACKAGE_HAS_SKELETON
> > +
> > +config BR2_PACKAGE_PROVIDES_SKELETON
> > +	default "skeleton-custom" if BR2_PACKAGE_SKELETON_CUSTOM
> 
>  In almost all virtual package providers, we do it like:
> 
> if BR2_PACKAGE_SKELETON_CUSTOM
> config BR2_PACKAGE_PROVIDES_SKELETON
> 	default "skeleton-custom"
> endif

> [snip]
> >  config BR2_PACKAGE_SKELETON
> >  	bool
> > -	help
> > -	  The basic skeleton for your rootfs.
> > +	default y
> 
>  For most virtual packages, we don't define the Kconfig symbol (e.g. there is no
> BR2_PACKAGE_EGL). Do we need it for skeleton?

We have it already for toolchain, another 'speical' package.

Having a package options ensures that:
  - the package is always enabled, even if none selects it,
  - the package appears in the graph-depends.

> > +
> > +config BR2_PACKAGE_HAS_SKELETON
> > +	bool
> 
>  We also don't really need this. It is used by packages that need the virtual
> package to check if there is a provider for it. But no package 'depends on'
> skeleton (or rather, all of them do), so we don't need this.

Technically, we do not need it. But I want to avoid scratching my head
if/when I have to revisit the virtual package infra, and the skeleton
stuff suddenly breaks for no reason.

This is a virtual package, let's make it a real virtual package. Let's
try not to be subtle.

> > +
> > +config BR2_PACKAGE_PROVIDES_SKELETON
> > +	string
> 
>  This one we do need :-)
> 
> [snip]
> > diff --git a/system/Config.in b/system/Config.in
> > index c27b013784..75b6a8edac 100644
> > --- a/system/Config.in
> > +++ b/system/Config.in
> > @@ -1,5 +1,9 @@
> >  menu "System configuration"
> >  
> > +# Note: usually, it is not possible to select a provider of a virtual
> > +# package.
> 
>  This is not true cryptodev, jpeg and mysql do exactly that. libressl/openssl
> will also do it that way. In fact, a choice is in a way the better way to do it,
> because then you're guaranteed not to have conflicting providers.

But then you disallow implementations from a br2-external tree, for
example for jpeg, a provider that has a hardware-accelerated libjpeg.

>  So this entire comment is wrong and can be removed. IMO.

No, the comment is true, when you look at a virtual package definition
in the manual. The way we do it for jpeg, cryptodev and mysql is that
we have a real package and a virtual package both in one, which have a
choice, and thus lift the limitation.

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> > But here we have an exception: there are only two providers
> > +# and they only get selected each by separate entries in this choice.
> > +# So this is a safe situation.
> >  choice
> >  	prompt "Root FS skeleton"
> >  
> > 
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 15/20] package/skeleton: introduce sysv- and systemd-specific skeletons

[Yann E. MORIN]

Arnout, All,

On 2017-07-23 01:49 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > Currently, we use the same skeleton for sysv-like init systems and
> > systemd, even though systemd has some peculiarities that makes out
>                                                                  our
> > default skeleton unfit.
> > 
> > So, we'll need to provide different skeletons (really, only part of it)
> > for sysv-like and systemd.
> > 
> > Introduce two new skeleton packages, aptly named skeleton-sysv and
> > skeleton-systemd, one for each, that are both providers of the skeleton
> > package, in lieue of the skeleton-common, which is now a simple
> > dependency of both the two new skeletons.
> > 
> > Those packages are empty for now. In followup changes:
> >   - sysv-specific stuff will be moved out of skeleton-common and into
> >     skeleton-sysv;
> >   - systemd-specific stuff will be added to skeleton-systemd.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> 
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
>  But...
> 
> [snip]
> > +config BR2_PACKAGE_SKELETON_SYSTEMD
> > +	bool
> > +	select BR2_PACKAGE_HAS_SKELETON
> > +	select BR2_PACKAGE_SKELETON_COMMON
> > +
> > +config BR2_PACKAGE_PROVIDES_SKELETON
> > +	default "skeleton-systemd" if BR2_PACKAGE_SKELETON_SYSTEMD
> 
>  Again, we usually do it with if...config...endif. Same for sysv, obviously.
> 
> [snip]
> > @@ -12,7 +12,9 @@ choice
> >  config BR2_ROOTFS_SKELETON_DEFAULT
> >  	bool "default target skeleton"
> >  	depends on !BR2_INIT_NONE
> > -	select BR2_PACKAGE_SKELETON_COMMON
> > +	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_SYSV
> > +	select BR2_PACKAGE_SKELETON_SYSV if BR2_INIT_BUSYBOX
> > +	select BR2_PACKAGE_SKELETON_SYSTEMD if BR2_INIT_SYSTEMD
> 
>  This works, but I'm thinking of an alternative, let's see if it flies:
> 
>  Instead of choosing the different skeletons here, we could do it in the select
> of BR2_INIT_*. Indeed, you have a skeleton for each init option. Of course, all
> the selects there would have to look like
> 
> 	select BR2_PACKAGE_SKELETON_SYSV if BR2_ROOTFS_SKELETON_DEFAULT
> 
> 
>  I'm not entirely sure if it's an improvement, but I have the feeling it's
> easier to follow if the different skeleton options are put together with the
> different init options.

I also tried that, but I found it uglier, because it is split away in
various options, while here it is fully concentrated in a single one, so
it is (arguably) easier to maintain...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Yann E. MORIN]

Arnout, All,

On 2017-07-23 02:08 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > systems does not like that the content of /var be symlinks to /tmp,
>   systemd
> 
> > especially journald that starts before /tmp is mounted, and thus the
> > journal files are hidden from view, which causes wuite a bit if fuss...
>                                                    quite
> > 
> > Instead, move the current /var to a sysv-only skeleton.
> > 
> > systemd at install time will create the /var content it needs, so we
> 
>  I guess systemd will also create a tmpfs on /var?

No it does not. But so far we do not support systemd on a read-only
root filesystem anyway (see the depends on for systemd).

In a later patch, we add support for a read-only rootfs, and then we add
an explicit tmpfs mont on /var, which systemd does not do on its own,
which is the reason for 886f3109bc (system: systemd only really supports
a R/W rootfs)

> > just create an empty /var for systemd. Yet, systemd really requires a
> > bunch of extra directories to exist, so we also create those.
> > 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Cc: Maxime Hadjinlian <maxime.hadjinlian@gmail.com>
> 
>  Commit message needs to be improved (see also below), but with that
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> 
> > 
> > ---
> > Note: /home and /srv will be created at runtime if missing, but it is
> > better to create them right now, to simplify supporting systemd on a
> > RO filesystem in the (near) future.
> > ---
> >  package/skeleton-systemd/skeleton-systemd.mk              | 7 +++++++
> >  package/skeleton-sysv/skeleton-sysv.mk                    | 4 ++++
> >  {system => package/skeleton-sysv}/skeleton/dev/log        | 0
> >  {system => package/skeleton-sysv}/skeleton/dev/pts/.empty | 0
> >  {system => package/skeleton-sysv}/skeleton/dev/shm/.empty | 0
> 
>  I guess indeed the /dev stuff is not needed for systemd because there is anyway
> a devtmpfs mounted over it. Which makes me wonder: shouldn't these be moved to
> the device-table? Only the static devices case needs these to be present... Of
> course, such a change is not for this patch, but I'd say that we can just leave
> this /dev stuff in the common part - systemd has no less use of it than sysv.
> 
> >  {system => package/skeleton-sysv}/skeleton/etc/fstab      | 0
> 
>  Why? systemd does use fstab, so it seems weird not to have an fstab in the
> systemd case.

We do have one, it is created in the skeleton-systemd.mk file:

    define SKELETON_SYSTEMD_INSTALL_TARGET_CMDS
            mkdir -p $(TARGET_DIR)/home
            mkdir -p $(TARGET_DIR)/srv
            mkdir -p $(TARGET_DIR)/var
            echo "/dev/root / auto rw 0 1" >$(TARGET_DIR)/etc/fstab
    endef

> Well, I guess our skeleton fstab doesn't contain anything relevant
> for systemd because all of that is handled already by tmpfiles.

Exactly.

> But the commit log should at least explain this.

Like so, maybe?

    For systemd, we only ever need an entry for /, because all the rest
    is totally irrelevant for systemd (it automatically mounts what it
    needs where it needs it).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Yann E. MORIN]

On 2017-07-23 02:13 +0200, Arnout Vandecappelle spake thusly:
> 
> 
> On 23-07-17 02:08, Arnout Vandecappelle wrote:
> > 
> > On 18-07-17 19:25, Yann E. MORIN wrote:
> >> systems does not like that the content of /var be symlinks to /tmp,
> >   systemd
> > 
> >> especially journald that starts before /tmp is mounted, and thus the
> >> journal files are hidden from view, which causes wuite a bit if fuss...
> >                                                    quite
> >> Instead, move the current /var to a sysv-only skeleton.
> >>
> >> systemd at install time will create the /var content it needs, so we
> >  I guess systemd will also create a tmpfs on /var?
> 
>  Seems the following sentence got dropped:
> 
> What happens when a package installs something in /var?

Since we do not (yet) support a read-only root filesystem for systemd,
and that systemd does _not_ automatically mount anything on /var, we
always have /var that is writable, and thus whatever a package installs
in /var at build time is in /var at runtime.

Now, see the followup patches, that add support for a read-only rootfs
with systemd:

    fs: add pre- and post-command hooks
    system: make systemd work on a read-only rootfs

Even though the code for each is relatively simple, there is a *lot* of
explanations in there about /var handling on a read-only rootfs with
systemd.

But again, so far, we only ever supported systemd on a read-write rootfs.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 17/20] sytem: no-init systems may use our default, common skeleton

[Yann E. MORIN]

Arnout, All,

On 2017-07-23 02:18 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > Requiring that no-init system indeed provide their own custom skeletons
>                          systems
> 
> > is a bit too-high a burden, especially for quick one-off tests when
> > testing our packages.
> > 
> > Allow no-init systems to use our default skeleton, but only use the
> > common part, for the specific sysv or systemd ones are unfit.
> 
>  For people who use BR2_INIT_NONE to create a chroot-able rootfs, this entails
> an important change: the /var stuff (e.g. /var/run symlink) is no longer there.
> This could have implications for some use cases. So I think a comment about this
> should be added to the release notes.
> 
> > 
> > Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > ---
> >  package/skeleton-common/Config.in | 29 +++++++++++++++++++++++++++++
> >  system/Config.in                  |  5 +----
> >  2 files changed, 30 insertions(+), 4 deletions(-)
> > 
> > diff --git a/package/skeleton-common/Config.in b/package/skeleton-common/Config.in
> > index 9abed4292a..ba0abe9d35 100644
> > --- a/package/skeleton-common/Config.in
> > +++ b/package/skeleton-common/Config.in
> > @@ -1,2 +1,31 @@
> > +# Normally, skeleton-common is a dependency of either the sysv or
> > +# the systemd ones, so it should not declare itself as a provider
> > +# for the skeleton virtual package.
> > +#
> > +# However, when one uses a no-init system (BR2_INIT_NONE), neither
> > +# the sysv not the systemd skeleton are enabled, so skeleton-common
> > +# is the one and only skeleton available, so it must declare itself
> > +# as a provider.
> > +#
> > +# We can safely select _HAS_SKELETON, because either we are the only
> > +# provider (in which case we must select it) or we were selected by
> > +# another skeleton which would have selected it by itself (in which
> > +# case us selecting _HAS_SKELETON is harmless).
> > +#
> >  config BR2_PACKAGE_SKELETON_COMMON
> >  	bool
> > +	select BR2_PACKAGE_HAS_SKELETON
> 
>  Instead of putting it here...
> 
> > +
> > +if BR2_PACKAGE_SKELETON_COMMON
> > +
> > +config BR2_PACKAGE_SKELETON_COMMON_ONLY
> > +	bool
> > +	default y
> > +	depends on !BR2_PACKAGE_SKELETON_SYSV
> > +	depends on !BR2_PACKAGE_SKELETON_SYSTEMD
> > +	depends on !BR2_PACKAGE_SKELETON_CUSTOM
> 
>  ... put it here. Then you don't need the comment above, there will always be
> just one symbol that selects skeleton.

But, but. but... My nice comment... It took me so much time to write
that big, nice, explication... I would be sad to let it go...

Yes, you're right, it would be much cleaner and more obvious.

>  But as I wrote in an earlier reply, we don't even need to select
> BR2_PACKAGE_HAS_SKELETON, so the point is moot.

And I stand by my previous reply, that we should not try to be smart.
Well, we can try to be smart, but not _too_ smart, becasue it _will_
cause grievance later on. ;-]

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 02/20] core/pkg-generic: add variable to skip skeleton dependency

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 >> An alternative could be to enforce a naming convention for these
 >> packages, E.G. skeleton-*, and then use $(filter skeleton%,$(1)) in the
 >> check, but I'm not sure that really is much nicer.

 > We also talked about that possibility on IRC, and came to the conclusion
 > that it was not nice to hard-code magic values in the infra.

Ok, We aldready did have skeleton hardcoded - But ok, the variable
solution is also fine by me.

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Arnout Vandecappelle]

On 23-07-17 12:24, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2017-07-23 02:08 +0200, Arnout Vandecappelle spake thusly:

[snip]
>> But the commit log should at least explain this.
> 
> Like so, maybe?
> 
>     For systemd, we only ever need an entry for /, because all the rest
>     is totally irrelevant for systemd (it automatically mounts what it
>     needs where it needs it).

 But... skeleton-common installs a lot more than just /, no? It installs
/etc/stuff, and a bunch of top-level directories...

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Arnout Vandecappelle]

On 23-07-17 15:32, Arnout Vandecappelle wrote:
> 
> 
> On 23-07-17 12:24, Yann E. MORIN wrote:
>> Arnout, All,
>>
>> On 2017-07-23 02:08 +0200, Arnout Vandecappelle spake thusly:
> 
> [snip]
>>> But the commit log should at least explain this.
>>
>> Like so, maybe?
>>
>>     For systemd, we only ever need an entry for /, because all the rest
>>     is totally irrelevant for systemd (it automatically mounts what it
>>     needs where it needs it).
> 
>  But... skeleton-common installs a lot more than just /, no? It installs
> /etc/stuff, and a bunch of top-level directories...

 Sorry, I missed that this was about fstab, I thought it was about the skeleton
itself.

 fstab is not yet mentioned at all in the commit message, so this should be
rephrased a little. But content-wise this is indeed what I wanted.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 16/20] system: separate sysv and systemd parts of the skeleton

[Yann E. MORIN]

Arnout, All,

On 2017-07-23 15:39 +0200, Arnout Vandecappelle spake thusly:
> On 23-07-17 15:32, Arnout Vandecappelle wrote:
> > On 23-07-17 12:24, Yann E. MORIN wrote:
> >> Arnout, All,
> >>
> >> On 2017-07-23 02:08 +0200, Arnout Vandecappelle spake thusly:
> > 
> > [snip]
> >>> But the commit log should at least explain this.
> >>
> >> Like so, maybe?
> >>
> >>     For systemd, we only ever need an entry for /, because all the rest
> >>     is totally irrelevant for systemd (it automatically mounts what it
> >>     needs where it needs it).
> > 
> >  But... skeleton-common installs a lot more than just /, no? It installs
> > /etc/stuff, and a bunch of top-level directories...
> 
>  Sorry, I missed that this was about fstab, I thought it was about the skeleton
> itself.
> 
>  fstab is not yet mentioned at all in the commit message, so this should be
> rephrased a little. But content-wise this is indeed what I wanted.

Ok, good. Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 18/20] fs: add pre- and post-command hooks

[Arnout Vandecappelle]

 Hi Yann,

On 18-07-17 19:25, Yann E. MORIN wrote:
> In some cases, the directory structure we want in the filesystem is not
> exactly what we have in target/
> 
> For example, when systemd is used on a read-only rootfs, /var must be a
> tmpfs. However, we may have packages that install stuff in there, and
> set important rights (via the permission-table). So, at build time, we
> need /var to be a symlink to the remanent location (/usr/share/factory)
> while at runtime we need /var to be a directory.
> 
> One option would have seen to have /var as a real directory even during
> build time, and in a target-finalize hook, move everything out of there
> and into the "factory" location. However, that's not possible because
> it's too early: some packages may want to set ownership and/or acces
> rights on directories or files in /var, and this is only done in the
> fakeroot script, which is called only later during the assembling of the
> filesystem images.
> 
> Also, there would have been no way to undo the tweak (i.e. we need to
> restore the /var symlink so that subsequent builds continue to work) if
> it were done as a target-finalize hook.
> 
> The only solution is to allow packages to register pre- and post-hooks
> that are called right before and right after the rootfs commands are
> executed, and inside in the fakeroot script.
> 
> We can however not re-use the BR2_ROOTFS_POST_FAKEROOT_SCRIPT feature
> either because it is done before the filesystem command, but there is
> nothing that is done after. Also, we don't want to add to, and modify a
> user-supplied variable.

 I completely agree with this approach.

> 
> So, we introduce two new variables that packages can set to add the
> commands they need to run to tweak the filesystem right at the last
> moment.

 But not with this.


> Those hooks are not documented on-purpose; they are probably going to
> only ever be used by systemd.

 If systemd is the only one ever to use it, then the approach with a list of
hooks is just too much over the top. Why not hardcode it? Like so:


> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Reviewed-by: Romain Naour <romain.naour@gmail.com>
> 
> ---
> Changes v2 -> v3:
>   - add space between >> and $$   (Romain)
> ---
>  fs/common.mk           | 4 ++++
>  package/pkg-generic.mk | 4 ++++
>  2 files changed, 8 insertions(+)
> 
> diff --git a/fs/common.mk b/fs/common.mk
> index 14e0a6b868..9a7758ff49 100644
> --- a/fs/common.mk
> +++ b/fs/common.mk
> @@ -95,10 +95,14 @@ endif
>  	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
>  		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
>  		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
> +	$$(foreach hook,$$(ROOTFS_PRE_CMD_HOOKS),\
> +		$$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT))

	$$(call PRINTF,$$(ROOTFS_SKELETON_PRE_CMD_HOOK)) >> $$(FAKEROOT_SCRIPT)

 You may want to change the variable name, obviously.

 Regards,
 Arnout


[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 18/20] fs: add pre- and post-command hooks

[Yann E. MORIN]

Arnout, All,

On 2017-07-23 15:42 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> > In some cases, the directory structure we want in the filesystem is not
> > exactly what we have in target/
> > 
> > For example, when systemd is used on a read-only rootfs, /var must be a
> > tmpfs. However, we may have packages that install stuff in there, and
> > set important rights (via the permission-table). So, at build time, we
> > need /var to be a symlink to the remanent location (/usr/share/factory)
> > while at runtime we need /var to be a directory.
> > 
> > One option would have seen to have /var as a real directory even during
> > build time, and in a target-finalize hook, move everything out of there
> > and into the "factory" location. However, that's not possible because
> > it's too early: some packages may want to set ownership and/or acces
> > rights on directories or files in /var, and this is only done in the
> > fakeroot script, which is called only later during the assembling of the
> > filesystem images.
> > 
> > Also, there would have been no way to undo the tweak (i.e. we need to
> > restore the /var symlink so that subsequent builds continue to work) if
> > it were done as a target-finalize hook.
> > 
> > The only solution is to allow packages to register pre- and post-hooks
> > that are called right before and right after the rootfs commands are
> > executed, and inside in the fakeroot script.
> > 
> > We can however not re-use the BR2_ROOTFS_POST_FAKEROOT_SCRIPT feature
> > either because it is done before the filesystem command, but there is
> > nothing that is done after. Also, we don't want to add to, and modify a
> > user-supplied variable.
> 
>  I completely agree with this approach.
> 
> > 
> > So, we introduce two new variables that packages can set to add the
> > commands they need to run to tweak the filesystem right at the last
> > moment.
> 
>  But not with this.
> 
> > Those hooks are not documented on-purpose; they are probably going to
> > only ever be used by systemd.
> 
>  If systemd is the only one ever to use it, then the approach with a list of
> hooks is just too much over the top. Why not hardcode it? Like so:
> 
> > Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> > Reviewed-by: Romain Naour <romain.naour@gmail.com>
> > 
> > ---
> > Changes v2 -> v3:
> >   - add space between >> and $$   (Romain)
> > ---
> >  fs/common.mk           | 4 ++++
> >  package/pkg-generic.mk | 4 ++++
> >  2 files changed, 8 insertions(+)
> > 
> > diff --git a/fs/common.mk b/fs/common.mk
> > index 14e0a6b868..9a7758ff49 100644
> > --- a/fs/common.mk
> > +++ b/fs/common.mk
> > @@ -95,10 +95,14 @@ endif
> >  	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
> >  		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
> >  		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
> > +	$$(foreach hook,$$(ROOTFS_PRE_CMD_HOOKS),\
> > +		$$(call PRINTF,$$($$(hook))) >> $$(FAKEROOT_SCRIPT))
> 
> 	$$(call PRINTF,$$(ROOTFS_SKELETON_PRE_CMD_HOOK)) >> $$(FAKEROOT_SCRIPT)
> 
>  You may want to change the variable name, obviously.

And it also means that the declaration of ROOTFS_SKELETON_PRE_CMD_HOOK
be conditional in the skeleton-systemd package, like:

    ifeq ($(BR2_PACKAGE_SKELETON_SYSTEMD),y)
    define ROOTFS_SKELETON_PRE_CMD_HOOK
        blabla
    endef
    endif

But I do not like these kind of constructs in the packages themselves,
because they are too easy to miss...

Alternatively, I would rather move that logic into the filesystem code
itself, because after all that is its responsibility, and guard its use
with something like (with a proper variable name, of course):

    ifeq ($(ROOTFS_VAR_FACTORY_BLABLA),YES)
        blabla...
    endif

And then have the packages that need it just set:
    ROOTFS_VAR_FACTORY_BLABLA = YES

All that, because Thomas already suggested (last year in Toulouse) that
we could re-use the same factory mechanism for the other init systems as
well (on read-only rootfs). In this case, the code would then be guarded
by an test against an empty $(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),
because then there would be no relation to an init system at all.

But the tmpfiles format is very specific to systemd; we'd have to write
our own parser for that (not too difficult, I think, but it'd run on
the target so it'd have to be really dead-simple).

So in the end, I'd remove the hooks and move the code to the fs
handling, protected by a conditional that so far only systemd would set.

OK with this new approach?

Still, there is yet ine thing that I dislike in this solution, whether
it be with hooks or the variable, and that noone has seen so far: it is
not atomic and the target/ directory will be broken if the filesystem
command fails right in the middle, before the var symlink was restored.

I have seen no solution to this issue, except doing a copy of target
first, right before assembling the image(s), becasue in that case, we
wouldn't care about restoring the previous status, so we would no longer
need atomicity.

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> 
> [snip]
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] TestIso9660GrubExternal and TestIso9660GrubInternal failing

[Ricardo Martincoski]

Hello,

I renamed the e-mail thread from:
[PATCH 20/20] support/testing: add runtime testing for init systems

On Sun, Jul 23, 2017 at 06:26 AM, Yann E. MORIN wrote:

> On 2017-07-23 09:58 +0200, Thomas Petazzoni spake thusly:
>> On Sun, 23 Jul 2017 00:50:16 +0200, Yann E. MORIN wrote:
>> > > The patches fixing them are here:
>> > > 
>> > >   http://patchwork.ozlabs.org/patch/783135/
>> > >   http://patchwork.ozlabs.org/patch/783137/
>> > >   http://patchwork.ozlabs.org/patch/783136/
>> > > 
>> > > and nobody reviews them.  
>> > 
>> > I gave them a spin here, and they indeed fix the build, but the tests
>> > are still broken because qemu can't boot the resulting system...
>> > 
>> > I'll investigate that tomorrow...
>> 
>> Hum, weird, because I'm pretty sure I tested the test cases with those
>> patches. I'll test again, maybe I missed something.
> 
> So, to rule out any sleep-deprivation issue while testign yesterday
> evening, I re-spawn the tests here this morning, and they still do not
> run.
> 
> Maybe it is my qemu version?
> 
> I'll try and bisect them between July the 1st and now.

Since you are investigating this, let me provide some data I hope is useful.

My local computer has:
binutils 2.26.1
qemu 2.5.0

Using my local computer the tests TestIso9660GrubExternal and
TestIso9660GrubInternal always pass, in current master and also with the 3 grub
patches applied.

Using the image buildroot/base I can reproduce the problems:
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23825707
GRUB requires a working absolute objcopy; upgrade your binutils
https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23831270
==> System does not boot

If you need me to check versions for other tools in my local computer (Ubuntu
16.04.2, not plain vanilla), feel free to ask.

Regards,
Ricardo

[Buildroot] TestIso9660GrubExternal and TestIso9660GrubInternal failing

[Yann E. MORIN]

Ricardo, All,

On 2017-07-23 15:13 -0300, Ricardo Martincoski spake thusly:
> On Sun, Jul 23, 2017 at 06:26 AM, Yann E. MORIN wrote:
> > On 2017-07-23 09:58 +0200, Thomas Petazzoni spake thusly:
> >> On Sun, 23 Jul 2017 00:50:16 +0200, Yann E. MORIN wrote:
> >> > > The patches fixing them are here:
> >> > > 
> >> > >   http://patchwork.ozlabs.org/patch/783135/
> >> > >   http://patchwork.ozlabs.org/patch/783137/
> >> > >   http://patchwork.ozlabs.org/patch/783136/
> >> > > 
> >> > > and nobody reviews them.  
> >> > 
> >> > I gave them a spin here, and they indeed fix the build, but the tests
> >> > are still broken because qemu can't boot the resulting system...
> >> > 
> >> > I'll investigate that tomorrow...
> >> 
> >> Hum, weird, because I'm pretty sure I tested the test cases with those
> >> patches. I'll test again, maybe I missed something.
> > 
> > So, to rule out any sleep-deprivation issue while testign yesterday
> > evening, I re-spawn the tests here this morning, and they still do not
> > run.
> > 
> > Maybe it is my qemu version?
> > 
> > I'll try and bisect them between July the 1st and now.
> 
> Since you are investigating this, let me provide some data I hope is useful.

Thanks for the feedback! :-)

So I was not even capable of running it on a snashot of July (at the
time the patches were made). It already failed to boot.

> My local computer has:
> binutils 2.26.1
> qemu 2.5.0

I have Ubuntu 17.04:
    GNU ar (GNU Binutils for Ubuntu) 2.28
    QEMU emulator version 2.8.0(Debian 1:2.8+dfsg-3ubuntu2.3)

> Using my local computer the tests TestIso9660GrubExternal and
> TestIso9660GrubInternal always pass, in current master and also with the 3 grub
> patches applied.

It is ex[ected that the build suceeds without the patches on your
machine, because it is binutils 2.28 that broke the build.

However, that you were ablt to run is interesting. I'll spawn that on my
server, which is a 16.04 as well.

> Using the image buildroot/base I can reproduce the problems:
> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23825707
> GRUB requires a working absolute objcopy; upgrade your binutils
> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23831270
> ==> System does not boot

So, it would look like it is either:
  - a qemu issue,
  - a binutils regression we have not identified (maybe another Debian
    patch is required?),
  - a grub regression.

> If you need me to check versions for other tools in my local computer (Ubuntu
> 16.04.2, not plain vanilla), feel free to ask.

I plan on doign those tests;

  - build on 16.04, run on 16.04
  - build on 16.04, run on 17.04
  - build on 17.04, run on 16.04

This should identify if qemu is the culprit (the second would not work),
or if binutils is the culprit (the third would not work).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] TestIso9660GrubExternal and TestIso9660GrubInternal failing

[Yann E. MORIN]

Ricardo, All,

On 2017-07-23 22:02 +0200, Yann E. MORIN spake thusly:
> On 2017-07-23 15:13 -0300, Ricardo Martincoski spake thusly:
> > On Sun, Jul 23, 2017 at 06:26 AM, Yann E. MORIN wrote:
> > > On 2017-07-23 09:58 +0200, Thomas Petazzoni spake thusly:
> > >> On Sun, 23 Jul 2017 00:50:16 +0200, Yann E. MORIN wrote:
> > >> > > The patches fixing them are here:
> > >> > > 
> > >> > >   http://patchwork.ozlabs.org/patch/783135/
> > >> > >   http://patchwork.ozlabs.org/patch/783137/
> > >> > >   http://patchwork.ozlabs.org/patch/783136/
> > >> > > 
> > >> > > and nobody reviews them.  
> > >> > 
> > >> > I gave them a spin here, and they indeed fix the build, but the tests
> > >> > are still broken because qemu can't boot the resulting system...
> > >> > 
> > >> > I'll investigate that tomorrow...
> > >> 
> > >> Hum, weird, because I'm pretty sure I tested the test cases with those
> > >> patches. I'll test again, maybe I missed something.
> > > 
> > > So, to rule out any sleep-deprivation issue while testign yesterday
> > > evening, I re-spawn the tests here this morning, and they still do not
> > > run.
> > > 
> > > Maybe it is my qemu version?
> > > 
> > > I'll try and bisect them between July the 1st and now.
> > 
> > Since you are investigating this, let me provide some data I hope is useful.
[--SNIP--]
> I plan on doign those tests;
> 
>   - build on 16.04, run on 16.04
>   - build on 16.04, run on 17.04
>   - build on 17.04, run on 16.04

So, even before the tests are all done, I already noticed an issue here:
on my machine, the iso images are about 1GiB large, while on my server,
they are only a bit less than 5MiB.

So, it looks like this is a binutils issue, with objdump being the
culprit (it very much look like older similar issues).
I'll continue investigating...

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 12/20] package/skeleton: split out into skeleton-common

[Arnout Vandecappelle]

 First of all: I agree/accept all the feedback you gave to my review comments.

On 23-07-17 11:48, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2017-07-23 01:06 +0200, Arnout Vandecappelle spake thusly:
[snip]
>>> +SKELETON_COMMON_PATH = system/skeleton
>>
>>  Not for this patch of course, but I think it would make sense to also move that
>> here. But you probably thought of that already :-)
> 
> Yes, that's is what I did in the first iteration, but Thomas was against
> moving it, so I left it alone to stay there.

 Thomas, do you remember why you were against moving the skeleton to
package/skeleton-common?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] TestIso9660GrubExternal and TestIso9660GrubInternal failing

[Yann E. MORIN]

Ricardo, All,

On 2017-07-23 22:02 +0200, Yann E. MORIN spake thusly:
[--SNIP--]
> I plan on doign those tests;

  - build on 16.04, run on 16.04
    ==> builds and runs OK

  - build on 16.04, run on 17.04
    ==> builds and runs OK

  - build on 17.04, run on 16.04
    ==> builds OK, does not run.

So, I think this is realyl a regression with newer binutils:

    $ ls -l fail/build/grub-0.97/stage1/stage1
    -rwxr-xr-x 1 ymorin ymorin 129M juil. 23 22:10 stage1

    $ ls -l works/build/grub-0.97/stage1/stage1
    -rwxr-xr-x 1 ymorin ymorin  512 juil. 23 22:09 stage1

Similarly with stage2 and above, with file reaching over 256MiB...

So, definitely something fishy with binutils-2.28...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 13/20] package/skeleton: make it a virtual package

[Arnout Vandecappelle]

On 23-07-17 12:13, Yann E. MORIN wrote:
> On 2017-07-23 01:36 +0200, Arnout Vandecappelle spake thusly:
>>
>>
>> On 18-07-17 19:25, Yann E. MORIN wrote:
[snip]
>>>  config BR2_PACKAGE_SKELETON
>>>  	bool
>>> -	help
>>> -	  The basic skeleton for your rootfs.
>>> +	default y
>>
>>  For most virtual packages, we don't define the Kconfig symbol (e.g. there is no
>> BR2_PACKAGE_EGL). Do we need it for skeleton?
> 
> We have it already for toolchain, another 'speical' package.

 However, why don't we have it for other virtual packages?

 Or maybe I should ask: why do we have a separate BR2_PACKAGE_SKELETON and
BR2_PACKAGE_HAS_SKELETON?

 Indeed for consistency it's better to add all these Kconfig symbols for
skeleton now.

[snip]
>>> +# Note: usually, it is not possible to select a provider of a virtual
>>> +# package.
>>
>>  This is not true cryptodev, jpeg and mysql do exactly that. libressl/openssl
>> will also do it that way. In fact, a choice is in a way the better way to do it,
>> because then you're guaranteed not to have conflicting providers.
> 
> But then you disallow implementations from a br2-external tree, for
> example for jpeg, a provider that has a hardware-accelerated libjpeg.

 There are pros and cons to the choice.

> 
>>  So this entire comment is wrong and can be removed. IMO.
> 
> No, the comment is true, when you look at a virtual package definition
> in the manual. The way we do it for jpeg, cryptodev and mysql is that
> we have a real package and a virtual package both in one, which have a
> choice, and thus lift the limitation.

 Well, IMO skeleton is similar to jpeg, cryptodev and mysql in that respect.

 But let's not discuss about adding or removing a comment :-)

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 18/20] fs: add pre- and post-command hooks

[Arnout Vandecappelle]

On 23-07-17 16:17, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2017-07-23 15:42 +0200, Arnout Vandecappelle spake thusly:
[snip]
>> 	$$(call PRINTF,$$(ROOTFS_SKELETON_PRE_CMD_HOOK)) >> $$(FAKEROOT_SCRIPT)
>>
>>  You may want to change the variable name, obviously.
> 
> And it also means that the declaration of ROOTFS_SKELETON_PRE_CMD_HOOK
> be conditional in the skeleton-systemd package, like:
> 
>     ifeq ($(BR2_PACKAGE_SKELETON_SYSTEMD),y)
>     define ROOTFS_SKELETON_PRE_CMD_HOOK
>         blabla
>     endef
>     endif
> 
> But I do not like these kind of constructs in the packages themselves,
> because they are too easy to miss...

 I'm not sure if it's that hard to miss. You just grep for
ROOTFS_SKELETON_PRE_CMD_HOOK to get the context.


> Alternatively, I would rather move that logic into the filesystem code
> itself, because after all that is its responsibility, and guard its use
> with something like (with a proper variable name, of course):
> 
>     ifeq ($(ROOTFS_VAR_FACTORY_BLABLA),YES)
>         blabla...
>     endif
> 
> And then have the packages that need it just set:
>     ROOTFS_VAR_FACTORY_BLABLA = YES

 I'm not so sure I like that better...


> All that, because Thomas already suggested (last year in Toulouse) that
> we could re-use the same factory mechanism for the other init systems as
> well (on read-only rootfs). In this case, the code would then be guarded
> by an test against an empty $(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),
> because then there would be no relation to an init system at all.

 You completely lost me here.

> But the tmpfiles format is very specific to systemd; we'd have to write
> our own parser for that (not too difficult, I think, but it'd run on
> the target so it'd have to be really dead-simple).
> 
> So in the end, I'd remove the hooks and move the code to the fs
> handling, protected by a conditional that so far only systemd would set.
> 
> OK with this new approach?

 I'm not so convinced. I mean, if you're putting systemd-specific code into
ROOTFS_TARGET_INTERNAL, you might just as well make that bit depend on
BR2_INIT_SYSTEMD directly. But then it becomes really too intertwined for my
liking. I think my ROOTFS_SKELETON_PRE_CMD_HOOK proposal is a better middle
ground between complexity and abstraction. If you really don't like that, I
prefer the original ROOTFS_PRE_CMD_HOOKS approach. I just think that giving the
possibility of defining more than one hook is useless. Particularly since the
systemd hook hinges on being the very last thing before rootfs creation.


> Still, there is yet ine thing that I dislike in this solution, whether
> it be with hooks or the variable, and that noone has seen so far: it is
> not atomic and the target/ directory will be broken if the filesystem
> command fails right in the middle, before the var symlink was restored.

 Can we create the symlink early in the build process, every time? I.e. run
SKELETON_SYSTEMD_POST_ROOTFS_VAR somehwere at the start of the build as well?
Only, I'm not sure where we could put it.


 Regards,
 Arnout


> I have seen no solution to this issue, except doing a copy of target
> first, right before assembling the image(s), becasue in that case, we
> wouldn't care about restoring the previous status, so we would no longer
> need atomicity.

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 19/20] system: make systemd work on a read-only rootfs

[Arnout Vandecappelle]

On 18-07-17 19:25, Yann E. MORIN wrote:
[snip]
> With a line like:
> 
>     C /var/something - - - -
> 
> it will look for /usr/share/factory/something and copy it (recursively
> if it is a directory) to /var/something, but only if it does not already
> exist there.

 So we could just have a line

C /var - - - -

and be done with it?

[snip]
> Note that we treat symlinks a little bit specially, by creating symlinks
> to the factory defaults rather than copying them.

 You forgot to explain why. I guess because a relative symlink that was created
based on the /usr/share/factory location wouldn't work any more.

 However, it is possible that a package creates a relative symlink based on the
/var location. Most symlinks that we create in Buildroot stupidly assume that
all the path components are not symlinks. So a package could for example do

ln -sf ../srv/www $DESTDIR/var/www

This will now create a broken symlink. It's fine as long as it is copied as-is
to /var, but not with your approach...

[snip]
> +define SKELETON_SYSTEMD_ROOT_RO_OR_RW
> +	mkdir -p $(TARGET_DIR)/etc/systemd/tmpfiles.d
> +	mkdir -p $(TARGET_DIR)/usr/share/factory
> +	ln -s usr/share/factory $(TARGET_DIR)/var

 Since other stuff may also want to put things in /usr/share/factory, I think it
should be
	mkdir -p $(TARGET_DIR)/usr/share/factory/var
	ln -s usr/share/factory/var $(TARGET_DIR)/var

 Regards,
 Arnout

[snip]

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] TestIso9660GrubExternal and TestIso9660GrubInternal failing

[Thomas Petazzoni]

Hello,

On Sun, 23 Jul 2017 15:13:41 -0300, Ricardo Martincoski wrote:

> My local computer has:
> binutils 2.26.1
> qemu 2.5.0
> 
> Using my local computer the tests TestIso9660GrubExternal and
> TestIso9660GrubInternal always pass, in current master and also with the 3 grub
> patches applied.
> 
> Using the image buildroot/base I can reproduce the problems:
> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23825707
> GRUB requires a working absolute objcopy; upgrade your binutils
> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23831270
> ==> System does not boot  

And my patches did not fix this problem ?

Best regards,

Thomas Petazoni
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 12/20] package/skeleton: split out into skeleton-common

[Thomas Petazzoni]

Hello,

On Sun, 23 Jul 2017 22:32:24 +0200, Arnout Vandecappelle wrote:

> >>  Not for this patch of course, but I think it would make sense to also move that
> >> here. But you probably thought of that already :-)  
> > 
> > Yes, that's is what I did in the first iteration, but Thomas was against
> > moving it, so I left it alone to stay there.  
> 
>  Thomas, do you remember why you were against moving the skeleton to
> package/skeleton-common?

I honestly don't remember. I believe it might be that I thought that
system/skeleton/ was a more "obvious" place to look for the skeleton,
rather than buried in package/skeleton-common/. But this argument feels
a bit fishy, so I'm not sure if that was that or something else.

I don't currently see a problem with having the base skeleton moved to
package/skeleton-common/.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] TestIso9660GrubExternal and TestIso9660GrubInternal failing

[Ricardo Martincoski]

Hello,

----- Original Message -----
> From: "Thomas Petazzoni"
> Sent: Monday, July 24, 2017 4:13:21 AM

> On Sun, 23 Jul 2017 15:13:41 -0300, Ricardo Martincoski wrote:
> 
>> My local computer has:
>> binutils 2.26.1
>> qemu 2.5.0
>> 
>> Using my local computer the tests TestIso9660GrubExternal and
>> TestIso9660GrubInternal always pass, in current master and also with the 3 grub
>> patches applied.
>> 
>> Using the image buildroot/base I can reproduce the problems:
>> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23825707
>> GRUB requires a working absolute objcopy; upgrade your binutils
>> https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23831270
>> ==> System does not boot
> 
> And my patches did not fix this problem ?

Your patches fix the build problem with newer binutils.

But the runtime problem remains, according to Yann's analysis related to
binutils-2.28 installed in the host machine.
The second url above (ending in 270) are the tests running with your 3 patches
applied.

Regards,
--
Ricardo Martincoski
DATACOM 
Ethernet Switches 
Rua Am?rica, 1000 - Eldorado do Sul, RS - 92990-000 - Brasil
+55 51 3933 3000 - Ramal 3307
ricardo.martincoski at datacom.ind.br
www.datacom.ind.br

[Buildroot] TestIso9660GrubExternal and TestIso9660GrubInternal failing

[Yann E. MORIN]

Thomas, All,

On 2017-07-24 09:13 +0200, Thomas Petazzoni spake thusly:
> On Sun, 23 Jul 2017 15:13:41 -0300, Ricardo Martincoski wrote:
> 
> > My local computer has:
> > binutils 2.26.1
> > qemu 2.5.0
> > 
> > Using my local computer the tests TestIso9660GrubExternal and
> > TestIso9660GrubInternal always pass, in current master and also with the 3 grub
> > patches applied.
> > 
> > Using the image buildroot/base I can reproduce the problems:
> > https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23825707
> > GRUB requires a working absolute objcopy; upgrade your binutils
> > https://gitlab.com/RicardoMartincoski/buildroot/-/jobs/23831270
> > ==> System does not boot  
> 
> And my patches did not fix this problem ?

Not really, no.

Without your patches, the build breaks, because objdump fails, and we
catch a build failure.

With your patches, the build seems to succeed, because it does not fail.
However, the generated iso images are about 1GiB in size (instead of
less than 5MiB), and they do not boot. See the summary I've written in:
    http://lists.busybox.net/pipermail/buildroot/2017-July/198940.html

So no, your patches are not sufficient to solve the run-time test.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 12/20] package/skeleton: split out into skeleton-common

[Yann E. MORIN]

Thomas, All,

On 2017-07-24 09:19 +0200, Thomas Petazzoni spake thusly:
> On Sun, 23 Jul 2017 22:32:24 +0200, Arnout Vandecappelle wrote:
> 
> > >>  Not for this patch of course, but I think it would make sense to also move that
> > >> here. But you probably thought of that already :-)  
> > > 
> > > Yes, that's is what I did in the first iteration, but Thomas was against
> > > moving it, so I left it alone to stay there.  
> > 
> >  Thomas, do you remember why you were against moving the skeleton to
> > package/skeleton-common?
> 
> I honestly don't remember. I believe it might be that I thought that
> system/skeleton/ was a more "obvious" place to look for the skeleton,
> rather than buried in package/skeleton-common/. But this argument feels
> a bit fishy, so I'm not sure if that was that or something else.
> 
> I don't currently see a problem with having the base skeleton moved to
> package/skeleton-common/.

OK, I'll move that when I respin later tonight.

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Andrey Smirnov]

On Tue, Jul 18, 2017 at 10:25 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> The "builtin" kernel does not boot a systemd-based system, so
> we resort to building the same one as currently used by our
> qemu_arm_vexpress_defconfig.
>
> We test the 11 following combinations:
>
>   - busybox, read-only, without network
>   - busybox, read-only, with network
>   - busybox, read-write, without network
>   - busybox, read-write, with network
>
>   - basic systemd, read-only, network w/ ifupdown
>   - basic systemd, read-only, network w/ networkd
>   - basic systemd, read-write, network w/ ifupdown
>   - basic systemd, read-write, network w/ networkd
>
>   - full systemd, read-only, network w/ networkd
>   - full systemd, read-write, network w/ networkd
>
>   - no init system, read-only, without network
>
> The tests just verify what the /sbin/init binary is, and that we were
> able to grab an IP address. More tests can be added later, for example
> to check each systemd features (journal, tmpfiles...)
>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---
>  support/testing/tests/init/__init__.py     |   0
>  support/testing/tests/init/base.py         |  47 ++++++++++
>  support/testing/tests/init/test_busybox.py |  67 +++++++++++++
>  support/testing/tests/init/test_none.py    |  32 +++++++
>  support/testing/tests/init/test_systemd.py | 145 +++++++++++++++++++++++++++++
>  5 files changed, 291 insertions(+)
>  create mode 100644 support/testing/tests/init/__init__.py
>  create mode 100644 support/testing/tests/init/base.py
>  create mode 100644 support/testing/tests/init/test_busybox.py
>  create mode 100644 support/testing/tests/init/test_none.py
>  create mode 100644 support/testing/tests/init/test_systemd.py
>
> diff --git a/support/testing/tests/init/__init__.py b/support/testing/tests/init/__init__.py
> new file mode 100644
> index 0000000000..e69de29bb2
> diff --git a/support/testing/tests/init/base.py b/support/testing/tests/init/base.py
> new file mode 100644
> index 0000000000..a261d7dd46
> --- /dev/null
> +++ b/support/testing/tests/init/base.py
> @@ -0,0 +1,47 @@
> +import os
> +import subprocess
> +import infra.basetest
> +
> +class InitSystemBase(infra.basetest.BRTest):
> +
> +    def startEmulator(self, fs_type, kernel=None, dtb=None, init=None):
> +        img = os.path.join(self.builddir, "images", "rootfs.{}".format(fs_type))
> +        subprocess.call(["truncate", "-s", "%1M", img])
> +
> +        options = ["-drive",
> +                   "file={},if=sd,format=raw".format(img),
> +                   "-M", "vexpress-a9"]
> +
> +        if kernel is None:
> +            kernel = "builtin"
> +        else:
> +            kernel = os.path.join(self.builddir, "images", kernel)
> +            options.extend(["-dtb", os.path.join(self.builddir, "images",
> +                                             "{}.dtb".format(dtb))])
> +
> +        kernel_cmdline = ["root=/dev/mmcblk0",
> +                          "rootfstype={}".format(fs_type),
> +                          "rootwait",
> +                          "ro",
> +                          "console=ttyAMA0"]
> +
> +        if not init is None:
> +            kernel_cmdline.extend(["init={}".format(init)])
> +
> +        self.emulator.boot(arch="armv7",
> +                           kernel=kernel,
> +                           kernel_cmdline=kernel_cmdline,
> +                           options=options)
> +
> +        if init is None:
> +            self.emulator.login()
> +
> +    def checkInit(self, path):
> +        cmd = "cmp /proc/1/exe {}".format(path)
> +        _, exit_code = self.emulator.run(cmd)
> +        self.assertEqual(exit_code, 0)
> +
> +    def checkNetwork(self, interface, exitCode=0):
> +        cmd = "ip addr show {} |grep inet".format(interface)
> +        _, exit_code = self.emulator.run(cmd)
> +        self.assertEqual(exit_code, exitCode)
> diff --git a/support/testing/tests/init/test_busybox.py b/support/testing/tests/init/test_busybox.py
> new file mode 100644
> index 0000000000..c3e425bf5d
> --- /dev/null
> +++ b/support/testing/tests/init/test_busybox.py
> @@ -0,0 +1,67 @@
> +import infra.basetest
> +from tests.init.base import InitSystemBase as InitSystemBase
> +
> +class InitSystemBusyboxBase(InitSystemBase):
> +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> +        """
> +        # BR2_TARGET_ROOTFS_TAR is not set
> +        """
> +
> +    def checkInit(self):
> +        super(InitSystemBusyboxBase, self).checkInit("/bin/busybox")
> +

Just as a suggestion, you can probably get away without needing to
override that function by defining a class variable "init_path" that
SystemBusyboxBase and SystemSystemdBase will override to some
appropriate value. See, for example, the code I wrote for Python and
IPython related tests.

https://git.buildroot.net/buildroot/tree/support/testing/tests/package/test_python.py#n11
https://git.buildroot.net/buildroot/tree/support/testing/tests/package/test_ipython.py#n18

Thanks,
Andrey Smirnov

[Buildroot] [PATCH 19/20] system: make systemd work on a read-only rootfs

[Yann E. MORIN]

Arnout, All,

On 2017-07-24 00:18 +0200, Arnout Vandecappelle spake thusly:
> On 18-07-17 19:25, Yann E. MORIN wrote:
> [snip]
> > With a line like:
> > 
> >     C /var/something - - - -
> > 
> > it will look for /usr/share/factory/something and copy it (recursively
> > if it is a directory) to /var/something, but only if it does not already
> > exist there.
> 
>  So we could just have a line
> 
> C /var - - - -
> 
> and be done with it?

No, because systemd-tmpfiles will only act if the entry does not exist.
But our case, /var does exist, and it is a mount point. We must have
/var to exist, because / is read-only.

If we were to use your construct, then systemd-tmpfiles would never do
anything at all.

From TMPFILES.D(5):

  C
    Recursively copy a file or directory, if the destination files
    or directories do not exist yet. Note that this command will
    not descend into subdirectories if the destination directory
    already exists. Instead, the entire copy operation is skipped.
    If the argument is omitted, files from the source directory
    /usr/share/factory/ with the same name are copied. Does not
    follow symlinks.

So we must have one tmpfile entry for each entry we need in /var/.

> [snip]
> > Note that we treat symlinks a little bit specially, by creating symlinks
> > to the factory defaults rather than copying them.
> 
>  You forgot to explain why. I guess because a relative symlink that was created
> based on the /usr/share/factory location wouldn't work any more.
> 
>  However, it is possible that a package creates a relative symlink based on the
> /var location. Most symlinks that we create in Buildroot stupidly assume that
> all the path components are not symlinks. So a package could for example do
> 
> ln -sf ../srv/www $DESTDIR/var/www
> 
> This will now create a broken symlink. It's fine as long as it is copied as-is
> to /var, but not with your approach...

Hmm... I'll have to think about that a bit more, even though I think I
have taken that into account...

At least I did see no breakage so far, but the set of packages I built
was arguably rather limited...

> [snip]
> > +define SKELETON_SYSTEMD_ROOT_RO_OR_RW
> > +	mkdir -p $(TARGET_DIR)/etc/systemd/tmpfiles.d
> > +	mkdir -p $(TARGET_DIR)/usr/share/factory
> > +	ln -s usr/share/factory $(TARGET_DIR)/var
> 
>  Since other stuff may also want to put things in /usr/share/factory, I think it
> should be
> 	mkdir -p $(TARGET_DIR)/usr/share/factory/var
> 	ln -s usr/share/factory/var $(TARGET_DIR)/var

Not really, because systemd-tmpfile will look in exactly the factory
path to find the source of the copy/link. See the exerpt I pasted above.

So, given this entry, for example:

    C /var/foo - - - -

systemd-tmpfiles will see if it can find /usr/share/factory/foo not
/usr/share/factory/var/foo

And if other packages install things in factory, they would most
probably install their own tmpfile description to copy it in place on
boot, becasue factory *is* for factory settings.

Thanks!

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> [snip]
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 20/20] support/testing: add runtime testing for init systems

[Yann E. MORIN]

Andrey, All,

On 2017-07-24 08:20 -0700, Andrey Smirnov spake thusly:
> On Tue, Jul 18, 2017 at 10:25 AM, Yann E. MORIN <yann.morin.1998@free.fr> wrote:
> > The "builtin" kernel does not boot a systemd-based system, so
> > we resort to building the same one as currently used by our
> > qemu_arm_vexpress_defconfig.
[--SNIP--]
> > diff --git a/support/testing/tests/init/test_busybox.py b/support/testing/tests/init/test_busybox.py
> > new file mode 100644
> > index 0000000000..c3e425bf5d
> > --- /dev/null
> > +++ b/support/testing/tests/init/test_busybox.py
> > @@ -0,0 +1,67 @@
> > +import infra.basetest
> > +from tests.init.base import InitSystemBase as InitSystemBase
> > +
> > +class InitSystemBusyboxBase(InitSystemBase):
> > +    config = infra.basetest.BASIC_TOOLCHAIN_CONFIG + \
> > +        """
> > +        # BR2_TARGET_ROOTFS_TAR is not set
> > +        """
> > +
> > +    def checkInit(self):
> > +        super(InitSystemBusyboxBase, self).checkInit("/bin/busybox")
> > +
> 
> Just as a suggestion, you can probably get away without needing to
> override that function by defining a class variable "init_path" that
> SystemBusyboxBase and SystemSystemdBase will override to some
> appropriate value. See, for example, the code I wrote for Python and
> IPython related tests.
> 
> https://git.buildroot.net/buildroot/tree/support/testing/tests/package/test_python.py#n11
> https://git.buildroot.net/buildroot/tree/support/testing/tests/package/test_ipython.py#n18

Hmm, I see what you mean. I'll see if that helps with the code.

OTOH, calling super() is not too complex either. ;-)

Thanks for the suggestion! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 18/20] fs: add pre- and post-command hooks

[Yann E. MORIN]

On 2017-07-23 23:51 +0200, Arnout Vandecappelle spake thusly:
> 
> 
> On 23-07-17 16:17, Yann E. MORIN wrote:
> > Arnout, All,
> > 
> > On 2017-07-23 15:42 +0200, Arnout Vandecappelle spake thusly:
> [snip]
> >> 	$$(call PRINTF,$$(ROOTFS_SKELETON_PRE_CMD_HOOK)) >> $$(FAKEROOT_SCRIPT)
> >>
> >>  You may want to change the variable name, obviously.
> > 
> > And it also means that the declaration of ROOTFS_SKELETON_PRE_CMD_HOOK
> > be conditional in the skeleton-systemd package, like:
> > 
> >     ifeq ($(BR2_PACKAGE_SKELETON_SYSTEMD),y)
> >     define ROOTFS_SKELETON_PRE_CMD_HOOK
> >         blabla
> >     endef
> >     endif
> > 
> > But I do not like these kind of constructs in the packages themselves,
> > because they are too easy to miss...
> 
>  I'm not sure if it's that hard to miss. You just grep for
> ROOTFS_SKELETON_PRE_CMD_HOOK to get the context.
> 
> 
> > Alternatively, I would rather move that logic into the filesystem code
> > itself, because after all that is its responsibility, and guard its use
> > with something like (with a proper variable name, of course):
> > 
> >     ifeq ($(ROOTFS_VAR_FACTORY_BLABLA),YES)
> >         blabla...
> >     endif
> > 
> > And then have the packages that need it just set:
> >     ROOTFS_VAR_FACTORY_BLABLA = YES
> 
>  I'm not so sure I like that better...

I'm not sure either, but I'll further explain below, so you get a better
picture than the blurred one I posted yesterday...

> > All that, because Thomas already suggested (last year in Toulouse) that
> > we could re-use the same factory mechanism for the other init systems as
> > well (on read-only rootfs). In this case, the code would then be guarded
> > by an test against an empty $(BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW),
> > because then there would be no relation to an init system at all.
> 
>  You completely lost me here.

Ah...

So, the factory is something that is indeed specific to systemd.

However, it allows for a systemd-based rootfs to start with a known
first-boot state, that is put in place early, before
applications/services need them.

IWith this patch, it means that every boot if a first-boot, but at least
the content of /var is what packages isntalled, and they know it is
writable.

But with the other init systems (busybox, syysvinit) we do not have a
factory. Instead /var contains only symlinks to /tmp, which is clean on
boot. So packages won't find what they need in /var, either:
  - it is redirected to /tmp, in which case it is empty, or
  - it is not redirected to /tmp, and it is read-only.

So Thomas suggested that we apply the factory feature to all init
systems. With systemd, it would be built-in (by mean of tmpfiles),
while for the others we'd need some extra startup scripts to mount a
tmpfs on /var and populate it.

> > But the tmpfiles format is very specific to systemd; we'd have to write
> > our own parser for that (not too difficult, I think, but it'd run on
> > the target so it'd have to be really dead-simple).
> > 
> > So in the end, I'd remove the hooks and move the code to the fs
> > handling, protected by a conditional that so far only systemd would set.
> > 
> > OK with this new approach?
> 
>  I'm not so convinced. I mean, if you're putting systemd-specific code into
> ROOTFS_TARGET_INTERNAL, you might just as well make that bit depend on
> BR2_INIT_SYSTEMD directly.

But then it is no longer systemd-specific, see above.

> But then it becomes really too intertwined for my
> liking. I think my ROOTFS_SKELETON_PRE_CMD_HOOK proposal is a better middle
> ground between complexity and abstraction. If you really don't like that, I
> prefer the original ROOTFS_PRE_CMD_HOOKS approach. I just think that giving the
> possibility of defining more than one hook is useless. Particularly since the
> systemd hook hinges on being the very last thing before rootfs creation.
> 
> 
> > Still, there is yet ine thing that I dislike in this solution, whether
> > it be with hooks or the variable, and that noone has seen so far: it is
> > not atomic and the target/ directory will be broken if the filesystem
> > command fails right in the middle, before the var symlink was restored.
> 
>  Can we create the symlink early in the build process, every time? I.e. run
> SKELETON_SYSTEMD_POST_ROOTFS_VAR somehwere at the start of the build as well?
> Only, I'm not sure where we could put it.

Hmm... Good point. I'll try to find a place. But I'm not sure that would
be possible.

Maybe with the step-hooks, that are called very early for all steps, and
then we'd use that to create the symlink if missing. But I do not like
that too much...

I'll investigate.

In the meantime, I'll drop the read-only patches from the series, to
avoid too much noise for now, and so we concentrate on the skeleton
stuff for this release.

Thanks!

Regards,
Yann E. MORIN.

>  Regards,
>  Arnout
> 
> 
> > I have seen no solution to this issue, except doing a copy of target
> > first, right before assembling the image(s), becasue in that case, we
> > wouldn't care about restoring the previous status, so we would no longer
> > need atomicity.
> 
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 18/20] fs: add pre- and post-command hooks

[Arnout Vandecappelle]

On 24-07-17 18:01, Yann E. MORIN wrote:
> On 2017-07-23 23:51 +0200, Arnout Vandecappelle spake thusly:

[snip]
> So Thomas suggested that we apply the factory feature to all init
> systems. With systemd, it would be built-in (by mean of tmpfiles),
> while for the others we'd need some extra startup scripts to mount a
> tmpfs on /var and populate it.

 Hum-hum, I thought that the Buildroot rule was: we don't add features. If
someone needs the factory feature, they should use systemd (or s6 I believe also
has some kind of factory feature).

[snip]
>>  Can we create the symlink early in the build process, every time? I.e. run
>> SKELETON_SYSTEMD_POST_ROOTFS_VAR somehwere at the start of the build as well?
>> Only, I'm not sure where we could put it.
> 
> Hmm... Good point. I'll try to find a place. But I'm not sure that would
> be possible.

 "dependencies" is an option, but that's *really* dirty.

 Perhaps we can add a stamp file to TARGET_DIR/usr/share/factory, and then add
TARGET_DIR/var/stampfile as a dependency of dirs. If TARGET_DIR/var is an
(empty) directory, then TARGET_DIR/var/stampfile will not exist so the rule will
trigger.

 But there are a few problems with that:

- dirs is also "called" for legal-info, but legal-info doesn't otherwise create
a target directory;

- it would be natural for this stampfile to depend on skeleton-systemd (which
can then create the stampfile), but that would give a circular dependency.


> Maybe with the step-hooks, that are called very early for all steps, and
> then we'd use that to create the symlink if missing. But I do not like
> that too much...

 Meh, no, that would repeat the check for *every* package...


> I'll investigate.
> 
> In the meantime, I'll drop the read-only patches from the series, to
> avoid too much noise for now, and so we concentrate on the skeleton
> stuff for this release.

 I think it's better if you still include them:

- it doesn't hurt if they are part of the series but don't get applied;
- it gives some context about the direction in which we are going;
- it gives an opportunity to continue discussion on these patches.

 Even if you re-post with some changes NOT applied, just say so in the commit
message. Perhaps remove your Sob to make sure it doesn't get applied by
accident. Or remove it from patchwork.


 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 19/20] system: make systemd work on a read-only rootfs

[Arnout Vandecappelle]

On 24-07-17 17:45, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2017-07-24 00:18 +0200, Arnout Vandecappelle spake thusly:
>> On 18-07-17 19:25, Yann E. MORIN wrote:
>> [snip]
>>> With a line like:
>>>
>>>     C /var/something - - - -
>>>
>>> it will look for /usr/share/factory/something and copy it (recursively
>>> if it is a directory) to /var/something, but only if it does not already
>>> exist there.
>>
>>  So we could just have a line
>>
>> C /var - - - -
>>
>> and be done with it?
> 
> No, because systemd-tmpfiles will only act if the entry does not exist.
> But our case, /var does exist, and it is a mount point. We must have
> /var to exist, because / is read-only.

 Right! I knew that, actually... You even wrote it there :-)

 Thinking out of the box: can we make /var a symlink to /tmp/var, and avoid
creating a tmpfs on /var at all? Since /tmp/var doesn't exist, "C! /tmp/var"
should work, no? Or did that cause journald to bail out?


[snip]
>>  However, it is possible that a package creates a relative symlink based on the
>> /var location. Most symlinks that we create in Buildroot stupidly assume that
>> all the path components are not symlinks. So a package could for example do
>>
>> ln -sf ../srv/www $DESTDIR/var/www
>>
>> This will now create a broken symlink. It's fine as long as it is copied as-is
>> to /var, but not with your approach...

 Note that this is *NOT* fine anymore if /var is a symlink to /tmp/var ...


> Hmm... I'll have to think about that a bit more, even though I think I
> have taken that into account...

 There is unfortunately no way to fix this. If the package is smart, it does
calculates the relative path based on the real path of the source and
destination. If the package is stupid, it just assumes ../something. Whatever we
do, one of them will break... The only safe approach would be to put stuff in
/factory instead of /usr/share/factory.


> At least I did see no breakage so far, but the set of packages I built
> was arguably rather limited...
> 
>> [snip]
>>> +define SKELETON_SYSTEMD_ROOT_RO_OR_RW
>>> +	mkdir -p $(TARGET_DIR)/etc/systemd/tmpfiles.d
>>> +	mkdir -p $(TARGET_DIR)/usr/share/factory
>>> +	ln -s usr/share/factory $(TARGET_DIR)/var
>>
>>  Since other stuff may also want to put things in /usr/share/factory, I think it
>> should be
>> 	mkdir -p $(TARGET_DIR)/usr/share/factory/var
>> 	ln -s usr/share/factory/var $(TARGET_DIR)/var
> 
> Not really, because systemd-tmpfile will look in exactly the factory
> path to find the source of the copy/link. See the exerpt I pasted above.
> 
> So, given this entry, for example:
> 
>     C /var/foo - - - -
> 
> systemd-tmpfiles will see if it can find /usr/share/factory/foo not
> /usr/share/factory/var/foo

 Weird, the way I read both the man page and the source code seem to suggest
that "C /var/something" will copy from /usr/share/factory/var/something... But I
guess you tested that so I must be misreading the source code.


> And if other packages install things in factory, they would most
> probably install their own tmpfile description to copy it in place on
> boot, becasue factory *is* for factory settings.

 Yes but you could have a name collision.


 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 19/20] system: make systemd work on a read-only rootfs

[Yann E. MORIN]

Arnout, All,

On 2017-07-25 00:44 +0200, Arnout Vandecappelle spake thusly:
> On 24-07-17 17:45, Yann E. MORIN wrote:
> > On 2017-07-24 00:18 +0200, Arnout Vandecappelle spake thusly:
> >> On 18-07-17 19:25, Yann E. MORIN wrote:
[--SNIP--]
>  Thinking out of the box: can we make /var a symlink to /tmp/var, and avoid
> creating a tmpfs on /var at all? Since /tmp/var doesn't exist, "C! /tmp/var"
> should work, no? Or did that cause journald to bail out?

Worth a try, of course. But still we'd need hoks of some kind to allow
package to install in /var at build time, and move that to factory at
the end of the build, while still preserving the ability to be able to
rebuild stuff without requiring a 'make clean'.

[--SNIP symlinks in /var--]
>  There is unfortunately no way to fix this. If the package is smart, it does
> calculates the relative path based on the real path of the source and
> destination. If the package is stupid, it just assumes ../something. Whatever we
> do, one of them will break... The only safe approach would be to put stuff in
> /factory instead of /usr/share/factory.

But that is not where systemd-tmpfiles will look at...

Unless you meant we have /factory during the build, then move it at the
moment we aggregaate the images, and move it back (moving the problem
from handling /var to handling /factory instead) ?

[--SNIP--]
> > Not really, because systemd-tmpfile will look in exactly the factory
> > path to find the source of the copy/link. See the exerpt I pasted above.
> > 
> > So, given this entry, for example:
> > 
> >     C /var/foo - - - -
> > 
> > systemd-tmpfiles will see if it can find /usr/share/factory/foo not
> > /usr/share/factory/var/foo
> 
>  Weird, the way I read both the man page and the source code seem to suggest
> that "C /var/something" will copy from /usr/share/factory/var/something... But I
> guess you tested that so I must be misreading the source code.

And of course you are right.

It just happened that my testing was borked because systemd (or one of
its minions like journald) would create the required directory structure
and files they need, so I did not see mine were not copied over. Now I
have another test that does not use those "standard" files, and I'll add
it to the testing infra.

So yes, it should be /usr/share/factory/var/foo to populate /var/foo.

Which led to another bug, which is that tmpfiles should be in
/etc/tmpfiles.d/ and not in /etc/systemd/tmpfiles.d/

I've now fixed that.

However (thinking aloud now), I have another tentative to do, which
instead of creating a symlink for /var at runtime, would just creat it
as a direcotry as expected.

Then the hooks would simply "mv /var /usr/share/factory/var; mkdir /var",
create the tmpfiles entries, aggregate the image(s), the rm -rf /var; mv
/usr/share/factory/var /var".

This is probably much simpler than moving the entries one by one.

The only drawback in this case would be a package that already creates a
factory/var/foo entry. It would end up in /var/foo afterwards... :-/
So I think the /var symlink is better, in fact, so I'll keep that.

Thanks! :-)

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 19/20] system: make systemd work on a read-only rootfs

[Arnout Vandecappelle]

On 25-07-17 18:07, Yann E. MORIN wrote:
> Arnout, All,
> 
> On 2017-07-25 00:44 +0200, Arnout Vandecappelle spake thusly:
>> On 24-07-17 17:45, Yann E. MORIN wrote:
>>> On 2017-07-24 00:18 +0200, Arnout Vandecappelle spake thusly:
>>>> On 18-07-17 19:25, Yann E. MORIN wrote:
> [--SNIP--]
>>  Thinking out of the box: can we make /var a symlink to /tmp/var, and avoid
>> creating a tmpfs on /var at all? Since /tmp/var doesn't exist, "C! /tmp/var"
>> should work, no? Or did that cause journald to bail out?
> 
> Worth a try, of course. But still we'd need hoks of some kind to allow
> package to install in /var at build time, and move that to factory at
> the end of the build, while still preserving the ability to be able to
> rebuild stuff without requiring a 'make clean'.

 I forgot to add: and create a symlink from /tmp/var to
/usr/share/factory/tmp/var. That way stuff will still get installed directly
into /usr/share/factory/tmp/var. We don't need to remove any of the symlinks,
however: when the tmpfs is mounted over /tmp, the symlink magically disappears.

 But that still doesn't fix:

> [--SNIP symlinks in /var--]
>>  There is unfortunately no way to fix this. If the package is smart, it does
>> calculates the relative path based on the real path of the source and
>> destination. If the package is stupid, it just assumes ../something. Whatever we
>> do, one of them will break... The only safe approach would be to put stuff in
>> /factory instead of /usr/share/factory.
> 
> But that is not where systemd-tmpfiles will look at...

 Yes, then we'd still have to copy/move /factory to /usr/share/factory.

 BTW, systemd-tmpfiles will look at anywhere you tell it to look:

C! /var - - - - /factory/var


> Unless you meant we have /factory during the build, then move it at the
> moment we aggregaate the images, and move it back (moving the problem
> from handling /var to handling /factory instead) ?

 It is what I meant. This was not to solve the interrupted rootfs creation
problem, this was to solve the broken symlink problem.


[snip]
> However (thinking aloud now), I have another tentative to do, which
> instead of creating a symlink for /var at runtime, would just creat it
> as a direcotry as expected.
> 
> Then the hooks would simply "mv /var /usr/share/factory/var; mkdir /var",
> create the tmpfiles entries, aggregate the image(s), the rm -rf /var; mv
> /usr/share/factory/var /var".

 You still have the interrupted rootfs creation problem. But the symlinks are fixed.


> This is probably much simpler than moving the entries one by one.
> 
> The only drawback in this case would be a package that already creates a
> factory/var/foo entry. It would end up in /var/foo afterwards... :-/
> So I think the /var symlink is better, in fact, so I'll keep that.

 Except that it may break symlinks...

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Marcus Hoffmann]

Hey,

On 22.07.2017 00:15, Marcus Hoffmann wrote:
> Hey,
> 
> On 20.07.2017 09:32, Thomas Petazzoni wrote:
>> Hello,
>>
>> On Wed, 19 Jul 2017 15:17:21 -0700, Andrey Yurovsky wrote:
>>
>>>> Thanks a lot for testing, very useful to get some feedback!
>>>>
>>>> Can you confirm that it fixes the journald issue reported in bug
>>>> https://bugs.busybox.net/show_bug.cgi?id=7892 ?  
>>>
>>> It does.
>>
>> Awesome, thanks.
>>
>>>> Also, it would be useful if someone interested in systemd could look at
>>>> bugs https://bugs.busybox.net/show_bug.cgi?id=9881 and  
>>>
>>> I did an ARMv7 build with networkd+resolvd and couldn't reproduce the
>>> described problem with these patches applied.
>>
>> Can you reproduce the problem *without* the patches ?
> 
> We are experiencing this problem. I'll try to test it with this patch
> series on Monday.


I tested this with Yanns latest series on top of master (well, a few
days back, where it applied cleanly.) and I still see this problem.

/etc/resolv.conf -> ../run/systemd/resolve/resolv.conf

And there is no /run/systemd/resolve because systemd-resolved is not
selected.

I guess in packages/systemd/systemd.mk:292

define SYSTEMD_INSTALL_RESOLVCONF_HOOK
	ln -sf ../run/systemd/resolve/resolv.conf \
		$(TARGET_DIR)/etc/resolv.conf
endef

This should be in the BR2_PACKAGE_SYSTEMD_RESOLVED and not in the
BR2_PACKAGE_SYSTEMD_NETWORKD branch? Will test what happens in that case.


> 
>>
>>> /etc/resolv.conf points
>>> to ../tmp/resolv.conf which in turn is properly populated and names do
>>> resolve. systemd-hostnamed is acting like it does on Fedora (for what
>>> it's worth), I'm still learning about how to test it properly (that
>>> is, I just have a static hostname at the moment).
>>>
>>>> https://bugs.busybox.net/show_bug.cgi?id=9926.  
>>>
>>> Ok, I'll do a clean rebuild without networkd and with NetworkManager.
>>
>> Thanks a lot!
>>
>> BTW, for these bugs, if there is something easy we can add to the
>> testing infrastructure to make sure it keeps working, it would be nice!

I'm not yet familiar with the the testing infrastructure. Would that be
something like providing a certain set of defconfigs, booting into the
generated filesystems and testing if dns works?

>>
>> Best regards,
>>
>> Thomas
>>
> 

Best wishes,
Marcus

[Buildroot] [PATCH 00/20] system: properly handle systemd as init system

[Thomas Petazzoni]

Hello,

On Fri, 28 Jul 2017 22:41:07 +0200, Marcus Hoffmann wrote:

> >> BTW, for these bugs, if there is something easy we can add to the
> >> testing infrastructure to make sure it keeps working, it would be nice!  
> 
> I'm not yet familiar with the the testing infrastructure. Would that be
> something like providing a certain set of defconfigs, booting into the
> generated filesystems and testing if dns works?

Exactly that. The testing infrastructure makes it very easy to provide
a defconfig fragment, build it, boot it under Qemu, and run arbitrary
commands.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com