[PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Alex Kiernan]

This is largely Jonas Bonn <jonas@norrbonn.se> patchset to move towards a
"stateless" configuration:

  These patches make some modifications to systemd with the long-term goal
  of being able to run OE in systemd's "stateless" configuration.
  "Stateless" boils down to building an image with empty /etc and /var
  directories so that volatile (tmpfs) filesystems can be mounted there;
  this requires that the system subsequently be able to populate these
  directories dynamically, which systemd mostly takes care of if things are
  done right.

  In these patches:
  i)   Don't include machine-id in writable images so that systemd can run
       its first-boot machinery
  ii)  Move systemd configuration files out of /etc
  iii) Allow systemd to dynamically enable services and populate
       /etc/systemd/system via the presets mechanism

  There's a long way to go to get to a working "stateless" configuration.
  Getting to a "volatile" system (just empty /var) should be easier and I'll
  post patches moving things in that direction shortly.

As a result of the systemd 242 upgrade, which includes 01d2041e41f4
("meson: stop creating enablement symlinks in /etc during
installation"), services such as systemd-networkd are no longer enabled
in images.

This patch set fixes this problem in addition to satisfying the goal of
moving towards "stateless" configurations.

The issue with respect to image testing during CI was caused by
systemd-time-wait-sync.service being enabled due to the lack of a
default preset policy:

https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto

We include a default policy of "enable nothing" which matches that used
by (at least) Arch and Fedora.

Changes in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"

Changes in v4:
- add patch to make systemd-firstboot a non-default option to systemd to
  prevent unexpected prompts at runtime

Changes in v3:
- parameterize preset-file installation location (as suggested in patch
  review)

Changes in v2:
- address patchtest warnings:
  - include sign-offs on patches
  - include License-Update tag on patch that makes licensing change
- dropped patch implementing brute-force removal of unit files installed
  by systemd into /etc
- reworked the systemd-conf patch to restore the installation of the
  base systemd configuration files

Jonas Bonn (6):
  systemd: don't build firstboot by default
  systemd: do not create machine-id
  systemd-conf: simplify creation of machine-specific configuration
  systemd: create preset files instead of installing in image
  systemd-systemctl-native: simplify and support preset-all
  rootfs-postcommands: call preset-all for read-only-rootfs

 meta/classes/rootfs-postcommands.bbclass      |  8 +++
 meta/classes/systemd.bbclass                  | 33 ++++++------
 meta/recipes-core/systemd/systemd-conf_242.bb | 53 ++++++++-----------
 .../systemd/systemd-systemctl/systemctl       | 36 +++----------
 .../systemd/systemd/99-default.preset         |  1 +
 meta/recipes-core/systemd/systemd_242.bb      | 23 ++++----
 6 files changed, 68 insertions(+), 86 deletions(-)
 create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset

-- 
2.17.1

[PATCH v5 1/6] systemd: don't build firstboot by default

[Alex Kiernan]

From: Jonas Bonn <jonas@norrbonn.se>

The firstboot service prompts the user for information about the host at
first boot.  Systemd determines whether or not a boot is a "first boot"
by the existence of the file /etc/machine-id.  Since oe-core always
includes this file (it is part of the systemd package), the firstboot
service never runs so this service is being built but never run.

A follow-up patch to this one will remove the machine-id from the
systemd build and allow it to be created automatically by systemd at
"first boot".  With that patch, we don't want the firstboot service to
suddenly start being invoked and presenting a prompt to the user.

With this patch, the firstboot service becomes a PACKAGECONFIG option
that the user must actively select.

Signed-off-by: Jonas Bonn <jonas@norrbonn.se>
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

Changes in v5:
- rebased for systemd 242

 meta/recipes-core/systemd/systemd_242.bb | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/recipes-core/systemd/systemd_242.bb b/meta/recipes-core/systemd/systemd_242.bb
index 33d3e0b000d9..9f194f2b6ca1 100644
--- a/meta/recipes-core/systemd/systemd_242.bb
+++ b/meta/recipes-core/systemd/systemd_242.bb
@@ -66,7 +66,6 @@ PACKAGECONFIG ??= " \
     acl \
     backlight \
     binfmt \
-    firstboot \
     gshadow \
     hibernate \
     hostnamed \
-- 
2.17.1

[PATCH v5 2/6] systemd: do not create machine-id

[Alex Kiernan]

From: Jonas Bonn <jonas@norrbonn.se>

There is no reason to have an emtpy machine-id as part of the systemd
package.  Either:

i)  the filesystem is writable and the file will be created
automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file
should be created as part of the read-only-rootfs tweaks.

Note we need a default distribution preset policy of "enable nothing" in
order to avoid all units being activated as part of `systemctl
preset-all`.

Signed-off-by: Jonas Bonn <jonas@norrbonn.se>
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

Changes in v5:
- install default preset distribution policy of "enable nothing"
- rebased for systemd 242

 meta/classes/rootfs-postcommands.bbclass            | 6 ++++++
 meta/recipes-core/systemd/systemd-conf_242.bb       | 9 ++-------
 meta/recipes-core/systemd/systemd/99-default.preset | 1 +
 meta/recipes-core/systemd/systemd_242.bb            | 6 +++++-
 4 files changed, 14 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset

diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index bde58ad6cd36..89f8efd32363 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -126,6 +126,12 @@ read_only_rootfs_hook () {
 			${IMAGE_ROOTFS}/etc/init.d/populate-volatile.sh
 		fi
 	fi
+
+	if ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true", "false", d)}; then
+	# Create machine-id
+	# 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
+		touch ${IMAGE_ROOTFS}${sysconfdir}/machine-id
+	fi
 }
 
 #
diff --git a/meta/recipes-core/systemd/systemd-conf_242.bb b/meta/recipes-core/systemd/systemd-conf_242.bb
index 9bb27fd96d9c..7fe2e1105bef 100644
--- a/meta/recipes-core/systemd/systemd-conf_242.bb
+++ b/meta/recipes-core/systemd/systemd-conf_242.bb
@@ -7,14 +7,13 @@ DefaultTimeoutStartSec setting."
 
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 
-CONFFILES_${PN} = "${sysconfdir}/machine-id \
-${sysconfdir}/systemd/coredump.conf \
+CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \
 ${sysconfdir}/systemd/journald.conf \
 ${sysconfdir}/systemd/logind.conf \
 ${sysconfdir}/systemd/system.conf \
 ${sysconfdir}/systemd/user.conf"
 
-FILES_${PN} = "${sysconfdir}/machine-id ${sysconfdir}/systemd"
+FILES_${PN} = "${sysconfdir}/systemd"
 
 do_configure[noexec] = '1'
 do_compile[noexec] = '1'
@@ -23,10 +22,6 @@ do_install() {
 	rm -rf ${D}/${sysconfdir}/systemd
 	install -d ${D}/${sysconfdir}/systemd
 
-	# Create machine-id
-	# 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
-	touch ${D}${sysconfdir}/machine-id
-
 	install -m 0644 ${S}/src/coredump/coredump.conf ${D}${sysconfdir}/systemd/coredump.conf
 
 	install -m 0644 ${S}/src/journal/journald.conf ${D}${sysconfdir}/systemd/journald.conf
diff --git a/meta/recipes-core/systemd/systemd/99-default.preset b/meta/recipes-core/systemd/systemd/99-default.preset
new file mode 100644
index 000000000000..1f29b50597f0
--- /dev/null
+++ b/meta/recipes-core/systemd/systemd/99-default.preset
@@ -0,0 +1 @@
+disable *
diff --git a/meta/recipes-core/systemd/systemd_242.bb b/meta/recipes-core/systemd/systemd_242.bb
index 9f194f2b6ca1..7d1b0ec13a44 100644
--- a/meta/recipes-core/systemd/systemd_242.bb
+++ b/meta/recipes-core/systemd/systemd_242.bb
@@ -22,6 +22,7 @@ SRC_URI += "file://touchscreen.rules \
            file://0003-implment-systemd-sysv-install-for-OE.patch \
            file://0004-rules-whitelist-hd-devices.patch \
            file://0005-rules-watch-metadata-changes-in-ide-devices.patch \
+           file://99-default.preset \
            "
 
 # patches needed by musl
@@ -277,7 +278,6 @@ do_install() {
 	fi
 
 	# conf files are handled by systemd-conf
-	rm -f ${D}${sysconfdir}/machine-id
 	rm -f ${D}${sysconfdir}/systemd/coredump.conf
 	rm -f ${D}${sysconfdir}/systemd/journald.conf
 	rm -f ${D}${sysconfdir}/systemd/logind.conf
@@ -287,6 +287,10 @@ do_install() {
 	# duplicate udevadm for postinst script
 	install -d ${D}${libexecdir}
 	ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm
+
+	# install default policy for presets
+	# https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
+	install -Dm 0644 ${WORKDIR}/99-default.preset ${D}${systemd_unitdir}/system-preset/99-default.preset
 }
 
 
-- 
2.17.1

[PATCH v5 3/6] systemd-conf: simplify creation of machine-specific configuration

[Alex Kiernan]

From: Jonas Bonn <jonas@norrbonn.se>

The configuration files that systemd installs are just skeletons
detailing the available options and their default values.  The
recommended means of changing the configuration is to provide snippets
in configuration directories.  For example, journald.conf settings are
best set in /usr/lib/journald.conf.d/ and can be overridden by the user
by providing overriding snippets in /etc/systemd/journald.conf.d/.

The systemd-conf package is just providing machine-specific overrides
for some systemd defaults.

This patch restores the installation of config files by systemd and
reduces systemd-conf to just providing the config snippets in
/usr/lib/*.conf.d.  This simplfies the systemd-conf recipe considerably
since it now just sets up a couple of text files and doesn't even need
access to the systemd source anymore.

Given, however, that the base configuration files in /etc/systemd have
the lowest priority, they will always be overridden by any snippets.  As
it may confuse a user who is used to modifying these base files
directly, this patch puts symlinks in the overriding
/etc/systemd/*.conf.d/ directories back to the top-level files, thereby
restoring them to their top-level glory.  The default config files
installed by systemd are just lists of (commented out) defaults so this
should be safe.  The symlinks also provide an indication to the user
that there is more to systemd configuration than just the top-level file
and that checking the man page may be in order...

With this patch, systemd gets the right configuration irregardless of
whether /etc is populated at boot or not (i.e. in a "stateless"
system).

License-Update: configuration snippets licensing is independent of
systemd licensing

Signed-off-by: Jonas Bonn <jonas@norrbonn.se>
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

Changes in v5:
- rebased for systemd 242

 meta/recipes-core/systemd/systemd-conf_242.bb | 48 +++++++++----------
 meta/recipes-core/systemd/systemd_242.bb      | 16 ++++---
 2 files changed, 31 insertions(+), 33 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd-conf_242.bb b/meta/recipes-core/systemd/systemd-conf_242.bb
index 7fe2e1105bef..8a9f01c48a93 100644
--- a/meta/recipes-core/systemd/systemd-conf_242.bb
+++ b/meta/recipes-core/systemd/systemd-conf_242.bb
@@ -1,48 +1,44 @@
-require systemd.inc
-
 SUMMARY = "Systemd system configuration"
 DESCRIPTION = "Systemd may require slightly different configuration for \
 different machines.  For example, qemu machines require a longer \
 DefaultTimeoutStartSec setting."
+LICENSE = "MIT"
 
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 
-CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \
-${sysconfdir}/systemd/journald.conf \
-${sysconfdir}/systemd/logind.conf \
-${sysconfdir}/systemd/system.conf \
-${sysconfdir}/systemd/user.conf"
-
-FILES_${PN} = "${sysconfdir}/systemd"
-
-do_configure[noexec] = '1'
-do_compile[noexec] = '1'
+FILES_${PN} = "/usr/lib/journald.conf.d/* \
+	/usr/lib/logind.conf.d/* \
+	/usr/lib/system.conf.d/* \
+	/etc/systemd/* \
+"
 
 do_install() {
-	rm -rf ${D}/${sysconfdir}/systemd
-	install -d ${D}/${sysconfdir}/systemd
-
-	install -m 0644 ${S}/src/coredump/coredump.conf ${D}${sysconfdir}/systemd/coredump.conf
-
-	install -m 0644 ${S}/src/journal/journald.conf ${D}${sysconfdir}/systemd/journald.conf
+	install -d ${D}/usr/lib/journald.conf.d
 	# Enable journal to forward message to syslog daemon
-	sed -i -e 's/.*ForwardToSyslog.*/ForwardToSyslog=yes/' ${D}${sysconfdir}/systemd/journald.conf
+	echo "ForwardToSyslog=yes" >> ${D}/usr/lib/journald.conf.d/00-${PN}.conf
 	# Set the maximium size of runtime journal to 64M as default
-	sed -i -e 's/.*RuntimeMaxUse.*/RuntimeMaxUse=64M/' ${D}${sysconfdir}/systemd/journald.conf
+	echo "RuntimeMaxUse=64M" >> ${D}/usr/lib/journald.conf.d/00-${PN}.conf
+
+	install -d ${D}${sysconfdir}/systemd/journald.conf.d
+	ln -s ../journald.conf ${D}${sysconfdir}/systemd/journald.conf.d/00-${PN}.conf
 
-	install -m 0644 ${S}/src/login/logind.conf.in ${D}${sysconfdir}/systemd/logind.conf
+	install -d ${D}/usr/lib/logind.conf.d
 	# Set KILL_USER_PROCESSES to yes
-	sed -i -e 's/@KILL_USER_PROCESSES@/yes/' ${D}${sysconfdir}/systemd/logind.conf
+	echo "KillUserProcesses=yes" >> ${D}/usr/lib/logind.conf.d/00-${PN}.conf
+
+	install -d ${D}${sysconfdir}/systemd/logind.conf.d
+	ln -s ../logind.conf ${D}${sysconfdir}/systemd/logind.conf.d/00-${PN}.conf
 
-	install -m 0644 ${S}/src/core/system.conf.in ${D}${sysconfdir}/systemd/system.conf
+	install -d ${D}/usr/lib/system.conf.d
 	# Set MEMORY_ACCOUNTING_DEFAULT to yes
-	sed -i -e 's/@MEMORY_ACCOUNTING_DEFAULT@/yes/' ${D}${sysconfdir}/systemd/system.conf
+	echo "DefaultMemoryAccounting=yes" >> ${D}/usr/lib/system.conf.d/00-${PN}.conf
 
-	install -m 0644 ${S}/src/core/user.conf ${D}${sysconfdir}/systemd/user.conf
+	install -d ${D}${sysconfdir}/systemd/system.conf.d
+	ln -s ../system.conf ${D}${sysconfdir}/systemd/system.conf.d/00-${PN}.conf
 }
 
 # Based on change from YP bug 8141, OE commit 5196d7bacaef1076c361adaa2867be31759c1b52
 do_install_append_qemuall() {
 	# Change DefaultTimeoutStartSec from 90s to 240s
-	echo "DefaultTimeoutStartSec = 240s" >> ${D}${sysconfdir}/systemd/system.conf
+	echo "DefaultTimeoutStartSec = 240s" >> ${D}/usr/lib/system.conf.d/00-${PN}.conf
 }
diff --git a/meta/recipes-core/systemd/systemd_242.bb b/meta/recipes-core/systemd/systemd_242.bb
index 7d1b0ec13a44..4f36dd3c8b4d 100644
--- a/meta/recipes-core/systemd/systemd_242.bb
+++ b/meta/recipes-core/systemd/systemd_242.bb
@@ -277,13 +277,6 @@ do_install() {
 		fi
 	fi
 
-	# conf files are handled by systemd-conf
-	rm -f ${D}${sysconfdir}/systemd/coredump.conf
-	rm -f ${D}${sysconfdir}/systemd/journald.conf
-	rm -f ${D}${sysconfdir}/systemd/logind.conf
-	rm -f ${D}${sysconfdir}/systemd/system.conf
-	rm -f ${D}${sysconfdir}/systemd/user.conf
-
 	# duplicate udevadm for postinst script
 	install -d ${D}${libexecdir}
 	ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm
@@ -486,6 +479,15 @@ FILES_${PN}-extra-utils = "\
                         ${rootlibexecdir}/systemd/systemd-cgroups-agent \
 "
 
+CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \
+	${sysconfdir}/systemd/journald.conf \
+	${sysconfdir}/systemd/logind.conf \
+	${sysconfdir}/systemd/system.conf \
+	${sysconfdir}/systemd/user.conf \
+	${sysconfdir}/systemd/resolved.conf \
+	${sysconfdir}/systemd/timesyncd.conf \
+"
+
 FILES_${PN} = " ${base_bindir}/* \
                 ${base_sbindir}/shutdown \
                 ${base_sbindir}/halt \
-- 
2.17.1

[PATCH v5 4/6] systemd: create preset files instead of installing in image

[Alex Kiernan]

From: Jonas Bonn <jonas@norrbonn.se>

At first boot, systemd will create the /etc/systemd/system directory
from service preset files.  As such, for a normal, writable /etc
(writable rootfs), there is no need to set up this directory at image
creation time.

This patch changes the systemd machinery to create preset files and to
rely on systemd to do the service enablement.

This breaks the read-only-rootfs case; there's a fix for this in a
follow-up patch.

Signed-off-by: Jonas Bonn <jonas@norrbonn.se>
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

 meta/classes/systemd.bbclass | 33 ++++++++++++++++-----------------
 1 file changed, 16 insertions(+), 17 deletions(-)

diff --git a/meta/classes/systemd.bbclass b/meta/classes/systemd.bbclass
index c8f4fdec88dc..403d2108df6d 100644
--- a/meta/classes/systemd.bbclass
+++ b/meta/classes/systemd.bbclass
@@ -16,43 +16,32 @@ python __anonymous() {
     # from doing any work so that pure-systemd images don't have redundant init
     # files.
     if bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):
-        d.appendVar("DEPENDS", " systemd-systemctl-native")
-        d.appendVar("PACKAGE_WRITE_DEPS", " systemd-systemctl-native")
         if not bb.utils.contains('DISTRO_FEATURES', 'sysvinit', True, False, d):
             d.setVar("INHIBIT_UPDATERCD_BBCLASS", "1")
 }
 
 systemd_postinst() {
-OPTS=""
-
 if [ -n "$D" ]; then
-    OPTS="--root=$D"
+	exit 0
 fi
 
 if type systemctl >/dev/null 2>/dev/null; then
-	if [ -z "$D" ]; then
-		systemctl daemon-reload
-	fi
-
-	systemctl $OPTS ${SYSTEMD_AUTO_ENABLE} ${SYSTEMD_SERVICE_ESCAPED}
+	systemctl daemon-reload
+	systemctl preset ${SYSTEMD_SERVICE_ESCAPED}
 
-	if [ -z "$D" -a "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then
+	if [ "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then
 		systemctl --no-block restart ${SYSTEMD_SERVICE_ESCAPED}
 	fi
 fi
 }
 
 systemd_prerm() {
-OPTS=""
-
 if [ -n "$D" ]; then
-    OPTS="--root=$D"
+	exit 0
 fi
 
 if type systemctl >/dev/null 2>/dev/null; then
-	if [ -z "$D" ]; then
-		systemctl stop ${SYSTEMD_SERVICE_ESCAPED}
-	fi
+	systemctl stop ${SYSTEMD_SERVICE_ESCAPED}
 
 	systemctl $OPTS disable ${SYSTEMD_SERVICE_ESCAPED}
 fi
@@ -177,12 +166,22 @@ python systemd_populate_packages() {
                 else:
                     bb.fatal("SYSTEMD_SERVICE_%s value %s does not exist" % (pkg_systemd, service))
 
+    def systemd_create_presets(pkg):
+        action = get_package_var(d, 'SYSTEMD_AUTO_ENABLE', pkg)
+        presetf = oe.path.join(d.getVar("PKGD"), d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg)
+        bb.utils.mkdirhier(os.path.dirname(presetf))
+        with open(presetf, 'a') as fd:
+            for service in d.getVar('SYSTEMD_SERVICE_%s' % pkg).split():
+                fd.write("%s %s\n" % (action,service))
+        d.appendVar("FILES_%s" % pkg, ' ' + oe.path.join(d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg))
+
     # Run all modifications once when creating package
     if os.path.exists(d.getVar("D")):
         for pkg in d.getVar('SYSTEMD_PACKAGES').split():
             systemd_check_package(pkg)
             if d.getVar('SYSTEMD_SERVICE_' + pkg):
                 systemd_generate_package_scripts(pkg)
+                systemd_create_presets(pkg)
         systemd_check_services()
 }
 
-- 
2.17.1

[PATCH v5 5/6] systemd-systemctl-native: simplify and support preset-all

[Alex Kiernan]

From: Jonas Bonn <jonas@norrbonn.se>

Now that the systemd class sets up service presets instead of actively
enabling services, the 'enable' and 'disable' subcommands for systemctl
are not actually used anywhere.  As such, we can remove these to make
sure that nobody inadvertently introduces new uses of them.

We do, however, one case where the enable/disable machinery is still
required; that is for the read-only-rootfs case where the 'preset-all'
command can not be called at runtime but needs to be called when creatng
the image.  For this case, we implement 'preset-all' here.

There was also a previous implement of 'preset'... not sure that this
ever worked as the implementation looks bogus and there aren't any users
anyway.  This patch removes the 'preset' subcommand, as well.

Signed-off-by: Jonas Bonn <jonas@norrbonn.se>
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

 .../systemd/systemd-systemctl/systemctl       | 36 ++++---------------
 1 file changed, 7 insertions(+), 29 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd-systemctl/systemctl b/meta/recipes-core/systemd/systemd-systemctl/systemctl
index 2bc648961761..ce50352018c0 100755
--- a/meta/recipes-core/systemd/systemd-systemctl/systemctl
+++ b/meta/recipes-core/systemd/systemd-systemctl/systemctl
@@ -9,22 +9,6 @@ while [ $# != 0 ]; do
 	opt="$1"
 
 	case "$opt" in
-		enable)
-			shift
-
-			action="$opt"
-			services="$1"
-			cmd_args="1"
-			shift
-			;;
-		disable)
-			shift
-
-			action="$opt"
-			services="$1"
-			cmd_args="1"
-			shift
-			;;
 		mask)
 			shift
 
@@ -33,13 +17,11 @@ while [ $# != 0 ]; do
 			cmd_args="1"
 			shift
 			;;
-		preset)
+		preset-all)
 			shift
 
 			action="$opt"
-			services="$1"
-			cmd_args="1"
-			shift
+			cmd_args="0"
 			;;
 		--root=*)
 			ROOT=${opt##--root=}
@@ -57,16 +39,12 @@ while [ $# != 0 ]; do
 			;;
 	esac
 done
-if [ "$action" = "preset" -a "$service_file" = "" ]; then
+
+if [ "$action" = "preset-all" ]; then
 	services=$(for f in `find $ROOT/etc/systemd/system $ROOT/lib/systemd/system $ROOT/usr/lib/systemd/system -type f 2>1`; do basename $f; done)
-	services="$services $opt"
-	presetall=1
 fi
 
 for service in $services; do
-	if [ "$presetall" = "1" ]; then
-		action="preset"
-	fi
 	if [ "$action" = "mask" ]; then
 		if [ ! -d $ROOT/etc/systemd/system/ ]; then
 			mkdir -p $ROOT/etc/systemd/system/
@@ -105,10 +83,10 @@ for service in $services; do
 	# If any new unit types are added to systemd they should be added
 	# to this regular expression.
 	unit_types_re='\.\(service\|socket\|device\|mount\|automount\|swap\|target\|target\.wants\|path\|timer\|snapshot\)\s*$'
-	if [ "$action" = "preset" ]; then
-		action=`egrep -sh  $service $ROOT/etc/systemd/user-preset/*.preset | cut -f1 -d' '`
+	if [ "$action" = "preset-all" ]; then
+		action=`egrep -sh  $service $ROOT/usr/lib/systemd/system-preset/*.preset | cut -f1 -d' '`
 		if [ -z "$action" ]; then
-			globalpreset=`egrep -sh  '\*'  $ROOT/etc/systemd/user-preset/*.preset | cut -f1 -d' '`
+			globalpreset=`egrep -sh  '\*'  $ROOT/usr/lib/systemd/system-preset/*.preset | cut -f1 -d' '`
 			if [ -n "$globalpreset" ]; then
 				action="$globalpreset"
 			else
-- 
2.17.1

[PATCH v5 6/6] rootfs-postcommands: call preset-all for read-only-rootfs

[Alex Kiernan]

From: Jonas Bonn <jonas@norrbonn.se>

When the rootfs is read-only, we cannot rely on systemd's default
invocation of preset-all at runtime in order to enable services.  As
such, we need to do it at image creation time.

Signed-off-by: Jonas Bonn <jonas@norrbonn.se>
Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>
---

 meta/classes/rootfs-postcommands.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index 89f8efd32363..5fcd53c4ca7a 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -16,6 +16,7 @@ ROOTFS_POSTPROCESS_COMMAND += "rootfs_update_timestamp ; "
 
 # Tweak the mount options for rootfs in /etc/fstab if read-only-rootfs is enabled
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "read_only_rootfs_hook; ", "",d)}'
+IMAGE_EXTRADEPENDS += '${@bb.utils.contains("IMAGE_FEATURES", "read-only-rootfs", "systemd-systemctl-native", "", d)}'
 
 # We also need to do the same for the kernel boot parameters,
 # otherwise kernel or initramfs end up mounting the rootfs read/write
@@ -131,6 +132,7 @@ read_only_rootfs_hook () {
 	# Create machine-id
 	# 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
 		touch ${IMAGE_ROOTFS}${sysconfdir}/machine-id
+		systemctl --root=${IMAGE_ROOTFS} preset-all
 	fi
 }
 
-- 
2.17.1

Re: [PATCH v5 5/6] systemd-systemctl-native: simplify and support preset-all

[Alex Kiernan]

On Mon, Apr 29, 2019 at 3:43 PM Alex Kiernan <alex.kiernan@gmail.com> wrote:
>
> From: Jonas Bonn <jonas@norrbonn.se>
>
> Now that the systemd class sets up service presets instead of actively
> enabling services, the 'enable' and 'disable' subcommands for systemctl
> are not actually used anywhere.  As such, we can remove these to make
> sure that nobody inadvertently introduces new uses of them.
>
> We do, however, one case where the enable/disable machinery is still
> required; that is for the read-only-rootfs case where the 'preset-all'
> command can not be called at runtime but needs to be called when creatng
> the image.  For this case, we implement 'preset-all' here.
>
> There was also a previous implement of 'preset'... not sure that this
> ever worked as the implementation looks bogus and there aren't any users
> anyway.  This patch removes the 'preset' subcommand, as well.
>
> Signed-off-by: Jonas Bonn <jonas@norrbonn.se>
> Signed-off-by: Alex Kiernan <alex.kiernan@gmail.com>

So it turns out we need something more sophisticated for the read-only
rootfs case... assuming this series passes CI (and if it got applied),
it's still better than where we were, or I can split it in two, one
which addresses the machine-id changes and another for the read-only
root case.

I'm just rewriting systemctl-native in python and adding in handling
of preset files... the shell's gotten way too knarly!

-- 
Alex Kiernan

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[richard.purdie]

On Mon, 2019-04-29 at 15:43 +0100, Alex Kiernan wrote:
> This is largely Jonas Bonn <jonas@norrbonn.se> patchset to move
> towards a
> "stateless" configuration:
> 
>   These patches make some modifications to systemd with the long-term 
> goal
>   of being able to run OE in systemd's "stateless" configuration.
>   "Stateless" boils down to building an image with empty /etc and
> /var
>   directories so that volatile (tmpfs) filesystems can be mounted
> there;
>   this requires that the system subsequently be able to populate
> these
>   directories dynamically, which systemd mostly takes care of if
> things are
>   done right.
> 
>   In these patches:
>   i)   Don't include machine-id in writable images so that systemd
> can run
>        its first-boot machinery
>   ii)  Move systemd configuration files out of /etc
>   iii) Allow systemd to dynamically enable services and populate
>        /etc/systemd/system via the presets mechanism
> 
>   There's a long way to go to get to a working "stateless"
> configuration.
>   Getting to a "volatile" system (just empty /var) should be easier
> and I'll
>   post patches moving things in that direction shortly.
> 
> As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> ("meson: stop creating enablement symlinks in /etc during
> installation"), services such as systemd-networkd are no longer
> enabled
> in images.
> 
> This patch set fixes this problem in addition to satisfying the goal
> of
> moving towards "stateless" configurations.
> 
> The issue with respect to image testing during CI was caused by
> systemd-time-wait-sync.service being enabled due to the lack of a
> default preset policy:
> 
> https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
> 
> We include a default policy of "enable nothing" which matches that
> used
> by (at least) Arch and Fedora.
> 
> Changes in v5:
> - rebased for systemd 242
> - install default preset distribution policy of "enable nothing"
> 
> Changes in v4:
> - add patch to make systemd-firstboot a non-default option to systemd
> to
>   prevent unexpected prompts at runtime
> 
> Changes in v3:
> - parameterize preset-file installation location (as suggested in
> patch
>   review)
> 
> Changes in v2:
> - address patchtest warnings:
>   - include sign-offs on patches
>   - include License-Update tag on patch that makes licensing change
> - dropped patch implementing brute-force removal of unit files
> installed
>   by systemd into /etc
> - reworked the systemd-conf patch to restore the installation of the
>   base systemd configuration files
> 
> Jonas Bonn (6):
>   systemd: don't build firstboot by default
>   systemd: do not create machine-id
>   systemd-conf: simplify creation of machine-specific configuration
>   systemd: create preset files instead of installing in image
>   systemd-systemctl-native: simplify and support preset-all
>   rootfs-postcommands: call preset-all for read-only-rootfs

I think this is close but in one of our test configurations it appears
the ssh server isn't starting:

https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/548

(the stdio log shows the configuration its building for step 6 and
step6c shows it failing to ssh into the image)

I reran it and the same error happened twice so it is consistent.

Cheers,

Richard

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Alex Kiernan]

On Tue, Apr 30, 2019 at 11:59 AM <richard.purdie@linuxfoundation.org> wrote:
>
> On Mon, 2019-04-29 at 15:43 +0100, Alex Kiernan wrote:
> > This is largely Jonas Bonn <jonas@norrbonn.se> patchset to move
> > towards a
> > "stateless" configuration:
> >
> >   These patches make some modifications to systemd with the long-term
> > goal
> >   of being able to run OE in systemd's "stateless" configuration.
> >   "Stateless" boils down to building an image with empty /etc and
> > /var
> >   directories so that volatile (tmpfs) filesystems can be mounted
> > there;
> >   this requires that the system subsequently be able to populate
> > these
> >   directories dynamically, which systemd mostly takes care of if
> > things are
> >   done right.
> >
> >   In these patches:
> >   i)   Don't include machine-id in writable images so that systemd
> > can run
> >        its first-boot machinery
> >   ii)  Move systemd configuration files out of /etc
> >   iii) Allow systemd to dynamically enable services and populate
> >        /etc/systemd/system via the presets mechanism
> >
> >   There's a long way to go to get to a working "stateless"
> > configuration.
> >   Getting to a "volatile" system (just empty /var) should be easier
> > and I'll
> >   post patches moving things in that direction shortly.
> >
> > As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> > ("meson: stop creating enablement symlinks in /etc during
> > installation"), services such as systemd-networkd are no longer
> > enabled
> > in images.
> >
> > This patch set fixes this problem in addition to satisfying the goal
> > of
> > moving towards "stateless" configurations.
> >
> > The issue with respect to image testing during CI was caused by
> > systemd-time-wait-sync.service being enabled due to the lack of a
> > default preset policy:
> >
> > https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
> >
> > We include a default policy of "enable nothing" which matches that
> > used
> > by (at least) Arch and Fedora.
> >
> > Changes in v5:
> > - rebased for systemd 242
> > - install default preset distribution policy of "enable nothing"
> >
> > Changes in v4:
> > - add patch to make systemd-firstboot a non-default option to systemd
> > to
> >   prevent unexpected prompts at runtime
> >
> > Changes in v3:
> > - parameterize preset-file installation location (as suggested in
> > patch
> >   review)
> >
> > Changes in v2:
> > - address patchtest warnings:
> >   - include sign-offs on patches
> >   - include License-Update tag on patch that makes licensing change
> > - dropped patch implementing brute-force removal of unit files
> > installed
> >   by systemd into /etc
> > - reworked the systemd-conf patch to restore the installation of the
> >   base systemd configuration files
> >
> > Jonas Bonn (6):
> >   systemd: don't build firstboot by default
> >   systemd: do not create machine-id
> >   systemd-conf: simplify creation of machine-specific configuration
> >   systemd: create preset files instead of installing in image
> >   systemd-systemctl-native: simplify and support preset-all
> >   rootfs-postcommands: call preset-all for read-only-rootfs
>
> I think this is close but in one of our test configurations it appears
> the ssh server isn't starting:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/548
>
> (the stdio log shows the configuration its building for step 6 and
> step6c shows it failing to ssh into the image)
>
> I reran it and the same error happened twice so it is consistent.
>

I'll have a look - thanks for throwing it through!

-- 
Alex Kiernan

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Alex Kiernan]

On Tue, Apr 30, 2019 at 12:45 PM Alex Kiernan <alex.kiernan@gmail.com> wrote:
>
> On Tue, Apr 30, 2019 at 11:59 AM <richard.purdie@linuxfoundation.org> wrote:
> >
> > On Mon, 2019-04-29 at 15:43 +0100, Alex Kiernan wrote:
> > > This is largely Jonas Bonn <jonas@norrbonn.se> patchset to move
> > > towards a
> > > "stateless" configuration:
> > >
> > >   These patches make some modifications to systemd with the long-term
> > > goal
> > >   of being able to run OE in systemd's "stateless" configuration.
> > >   "Stateless" boils down to building an image with empty /etc and
> > > /var
> > >   directories so that volatile (tmpfs) filesystems can be mounted
> > > there;
> > >   this requires that the system subsequently be able to populate
> > > these
> > >   directories dynamically, which systemd mostly takes care of if
> > > things are
> > >   done right.
> > >
> > >   In these patches:
> > >   i)   Don't include machine-id in writable images so that systemd
> > > can run
> > >        its first-boot machinery
> > >   ii)  Move systemd configuration files out of /etc
> > >   iii) Allow systemd to dynamically enable services and populate
> > >        /etc/systemd/system via the presets mechanism
> > >
> > >   There's a long way to go to get to a working "stateless"
> > > configuration.
> > >   Getting to a "volatile" system (just empty /var) should be easier
> > > and I'll
> > >   post patches moving things in that direction shortly.
> > >
> > > As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> > > ("meson: stop creating enablement symlinks in /etc during
> > > installation"), services such as systemd-networkd are no longer
> > > enabled
> > > in images.
> > >
> > > This patch set fixes this problem in addition to satisfying the goal
> > > of
> > > moving towards "stateless" configurations.
> > >
> > > The issue with respect to image testing during CI was caused by
> > > systemd-time-wait-sync.service being enabled due to the lack of a
> > > default preset policy:
> > >
> > > https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
> > >
> > > We include a default policy of "enable nothing" which matches that
> > > used
> > > by (at least) Arch and Fedora.
> > >
> > > Changes in v5:
> > > - rebased for systemd 242
> > > - install default preset distribution policy of "enable nothing"
> > >
> > > Changes in v4:
> > > - add patch to make systemd-firstboot a non-default option to systemd
> > > to
> > >   prevent unexpected prompts at runtime
> > >
> > > Changes in v3:
> > > - parameterize preset-file installation location (as suggested in
> > > patch
> > >   review)
> > >
> > > Changes in v2:
> > > - address patchtest warnings:
> > >   - include sign-offs on patches
> > >   - include License-Update tag on patch that makes licensing change
> > > - dropped patch implementing brute-force removal of unit files
> > > installed
> > >   by systemd into /etc
> > > - reworked the systemd-conf patch to restore the installation of the
> > >   base systemd configuration files
> > >
> > > Jonas Bonn (6):
> > >   systemd: don't build firstboot by default
> > >   systemd: do not create machine-id
> > >   systemd-conf: simplify creation of machine-specific configuration
> > >   systemd: create preset files instead of installing in image
> > >   systemd-systemctl-native: simplify and support preset-all
> > >   rootfs-postcommands: call preset-all for read-only-rootfs
> >
> > I think this is close but in one of our test configurations it appears
> > the ssh server isn't starting:
> >
> > https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/548
> >
> > (the stdio log shows the configuration its building for step 6 and
> > step6c shows it failing to ssh into the image)
> >
> > I reran it and the same error happened twice so it is consistent.
> >
>
> I'll have a look - thanks for throwing it through!
>

Bother... I've seen that before:

  DISTRO_FEATURES_append = ' systemd'
  VIRTUAL-RUNTIME_init_manager = 'sysvinit'

I fell over it myself when I was first getting the image tests
working, then completely forgot about it...

Unrelated to this error (but relevant for the patch set) - do we have
a test that runs for read-only rootfs?

-- 
Alex Kiernan

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Paul Barker]

On Mon, 29 Apr 2019, at 15:44, Alex Kiernan wrote:
> As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> ("meson: stop creating enablement symlinks in /etc during
> installation"), services such as systemd-networkd are no longer enabled
> in images.
> 

As an aside - is this systemd upgrade why on raspberrypi3 with master branches I recently have no console on tty1 and `/etc/systemd/system/getty.target.wants/getty@tty1.service` is missing?

Thanks,

-- 
Paul Barker
Managing Director & Principal Engineer
Beta Five Ltd

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Alex Kiernan]

On Tue, Apr 30, 2019 at 1:08 PM Paul Barker <paul@betafive.co.uk> wrote:
>
> On Mon, 29 Apr 2019, at 15:44, Alex Kiernan wrote:
> > As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> > ("meson: stop creating enablement symlinks in /etc during
> > installation"), services such as systemd-networkd are no longer enabled
> > in images.
> >
>
> As an aside - is this systemd upgrade why on raspberrypi3 with master branches I recently have no console on tty1 and `/etc/systemd/system/getty.target.wants/getty@tty1.service` is missing?
>

I suspect so - certainly getty@ is one of the things that's was
touched in the upstream commit.

-- 
Alex Kiernan

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Paul Barker]

On Tue, 30 Apr 2019, at 13:14, Alex Kiernan wrote:
> On Tue, Apr 30, 2019 at 1:08 PM Paul Barker <paul@betafive.co.uk> wrote:
> >
> > On Mon, 29 Apr 2019, at 15:44, Alex Kiernan wrote:
> > > As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> > > ("meson: stop creating enablement symlinks in /etc during
> > > installation"), services such as systemd-networkd are no longer enabled
> > > in images.
> > >
> >
> > As an aside - is this systemd upgrade why on raspberrypi3 with master branches I recently have no console on tty1 and `/etc/systemd/system/getty.target.wants/getty@tty1.service` is missing?
> >
> 
> I suspect so - certainly getty@ is one of the things that's was
> touched in the upstream commit.
> 

Doing some reading now.

The discussion on https://github.com/systemd/systemd/issues/12345 suggests we should be running `systemctl preset-all` or at least `systemctl enable getty@tty1` on first boot. Maybe this should be handled at the BSP level so we can decide whether to enable getty@tty1 or not on a per-machine basis.

However, I can't see an easy way to resolve this at BSP level right now. I know how to add a systemd service file in a package and enable it but in this case we're looking to enable a service which is already installed. Reading through systemd.bbclass and the relevant docs doesn't give any suggestions. Is there any straightforward way to enable an already installed service?

Thanks,

-- 
Paul Barker
Managing Director & Principal Engineer
Beta Five Ltd

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Alex Kiernan]

On Tue, Apr 30, 2019 at 1:24 PM Paul Barker <paul@betafive.co.uk> wrote:
>
> On Tue, 30 Apr 2019, at 13:14, Alex Kiernan wrote:
> > On Tue, Apr 30, 2019 at 1:08 PM Paul Barker <paul@betafive.co.uk> wrote:
> > >
> > > On Mon, 29 Apr 2019, at 15:44, Alex Kiernan wrote:
> > > > As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> > > > ("meson: stop creating enablement symlinks in /etc during
> > > > installation"), services such as systemd-networkd are no longer enabled
> > > > in images.
> > > >
> > >
> > > As an aside - is this systemd upgrade why on raspberrypi3 with master branches I recently have no console on tty1 and `/etc/systemd/system/getty.target.wants/getty@tty1.service` is missing?
> > >
> >
> > I suspect so - certainly getty@ is one of the things that's was
> > touched in the upstream commit.
> >
>
> Doing some reading now.
>
> The discussion on https://github.com/systemd/systemd/issues/12345 suggests we should be running `systemctl preset-all` or at least `systemctl enable getty@tty1` on first boot.

That's what this patchset does (`systemctl preset-all`), or rather it
builds images without /etc/machine-id which triggers systemd to do it.

> Maybe this should be handled at the BSP level so we can decide whether to enable getty@tty1 or not on a per-machine basis.
>

Once I'm done you should be able to install preset files which then
determine what gets enabled/disabled, the only real question I've got
is if we should always do this at image build time, or leave it for
first boot and let systemd do it (on read-write images at least).

> However, I can't see an easy way to resolve this at BSP level right now. I know how to add a systemd service file in a package and enable it but in this case we're looking to enable a service which is already installed. Reading through systemd.bbclass and the relevant docs doesn't give any suggestions. Is there any straightforward way to enable an already installed service?
>

Personally, if I needed it fixing right now, I'd probably add a hack
during rootfs build to put the symlinks in the right place.

-- 
Alex Kiernan

Re: [PATCH v5 0/6] Fix systemd service enables/move towards "stateless" configuration

[Paul Barker]

On Tue, 30 Apr 2019, at 13:33, Alex Kiernan wrote:
> On Tue, Apr 30, 2019 at 1:24 PM Paul Barker <paul@betafive.co.uk> wrote:
> >
> > On Tue, 30 Apr 2019, at 13:14, Alex Kiernan wrote:
> > > On Tue, Apr 30, 2019 at 1:08 PM Paul Barker <paul@betafive.co.uk> wrote:
> > > >
> > > > On Mon, 29 Apr 2019, at 15:44, Alex Kiernan wrote:
> > > > > As a result of the systemd 242 upgrade, which includes 01d2041e41f4
> > > > > ("meson: stop creating enablement symlinks in /etc during
> > > > > installation"), services such as systemd-networkd are no longer enabled
> > > > > in images.
> > > > >
> > > >
> > > > As an aside - is this systemd upgrade why on raspberrypi3 with master branches I recently have no console on tty1 and `/etc/systemd/system/getty.target.wants/getty@tty1.service` is missing?
> > > >
> > >
> > > I suspect so - certainly getty@ is one of the things that's was
> > > touched in the upstream commit.
> > >
> >
> > Doing some reading now.
> >
> > The discussion on https://github.com/systemd/systemd/issues/12345 suggests we should be running `systemctl preset-all` or at least `systemctl enable getty@tty1` on first boot.
> 
> That's what this patchset does (`systemctl preset-all`), or rather it
> builds images without /etc/machine-id which triggers systemd to do it.
> 
> > Maybe this should be handled at the BSP level so we can decide whether to enable getty@tty1 or not on a per-machine basis.
> >
> 
> Once I'm done you should be able to install preset files which then
> determine what gets enabled/disabled, the only real question I've got
> is if we should always do this at image build time, or leave it for
> first boot and let systemd do it (on read-write images at least).
> 
> > However, I can't see an easy way to resolve this at BSP level right now. I know how to add a systemd service file in a package and enable it but in this case we're looking to enable a service which is already installed. Reading through systemd.bbclass and the relevant docs doesn't give any suggestions. Is there any straightforward way to enable an already installed service?
> >
> 
> Personally, if I needed it fixing right now, I'd probably add a hack
> during rootfs build to put the symlinks in the right place.
> 

Thanks for the advice, I understand much better now. I think for now I'll pin oe-core to the last commit before the systemd upgrade (ac64c3b96b...) so we can carry on our development. I'll watch out for this patch series going into master and then test it out again.

Thanks,

-- 
Paul Barker
Managing Director & Principal Engineer
Beta Five Ltd