[OE-core][PATCH 0/3] systemd: split timesync and networkd to packages

[OE-core][PATCH 0/3] systemd: split timesync and networkd to packages

[Peter Marko]

Motivation for this patch series is sstate-cache reuse.

Basically we have dozen of products with several
different systemd configurations.
Problem is that there are many recipes which depend on systemd.
(or their packaging depends on it)
Especially qtwebengine which takes hours to build and created
sstate objects have several hundereds of gigabytes.

So instead of:
  PACKAGECONFIG:remove:pn-systemd = " timesyncd"
we would like to do just:
  BAD_RECOMMENDATIONS += "systemd-timesyncd"

That allows reuse of sstate-cache for such recipes because
it will build identically, just install to rootfs differently.

One alternative we considered was via systemd-conf, but binaries still
occupy space in rootfs which is not desirable for some devices:
  ln -sf /dev/null ${D}${sysconfdir}/systemd/system/systemd-timesyncd.service

Please comment if you think that we should take a different
approach which is more in line with Yocto project best practices.

Peter Marko (3):
  systemd: split timesyncd to its own package
  systemd.bbclass: add non-recursive service packaging
  systemd: split networkd to its own package

 documentation/ref-manual/variables.rst     | 10 ++++
 meta/classes-recipe/systemd.bbclass        | 15 ++---
 meta/recipes-core/systemd/systemd_252.4.bb | 64 ++++++++++++++++++++--
 3 files changed, 78 insertions(+), 11 deletions(-)

-- 
2.30.2

[OE-core][PATCH 1/3] systemd: split timesyncd to its own package

[Peter Marko]

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 meta/recipes-core/systemd/systemd_252.4.bb | 27 ++++++++++++++++++++--
 1 file changed, 25 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_252.4.bb b/meta/recipes-core/systemd/systemd_252.4.bb
index e6c873f53b..de3ca93df8 100644
--- a/meta/recipes-core/systemd/systemd_252.4.bb
+++ b/meta/recipes-core/systemd/systemd_252.4.bb
@@ -376,6 +376,7 @@ PACKAGE_BEFORE_PN = "\
     ${PN}-journal-remote \
     ${PN}-extra-utils \
     ${PN}-udev-rules \
+    ${PN}-timesyncd \
     libsystemd-shared \
     udev \
     udev-hwdb \
@@ -393,12 +394,16 @@ DESCRIPTION:${PN}-journal-upload = "systemd-journal-upload uploads journal entri
 SUMMARY:${PN}-journal-remote = "Receive journal messages over the network"
 DESCRIPTION:${PN}-journal-remote = "systemd-journal-remote is a command to receive serialized journal events and store them to journal files."
 
+SUMMARY:${PN}-timesyncd = "Synchronize local system clock with a remote NTP server"
+DESCRIPTION:${PN}-timesyncd = "systemd-timesyncd is a system service that may be used to synchronize the local system clock with a remote Network Time Protocol (NTP) server."
+
 SUMMARY:libsystemd-shared = "Systemd shared library"
 
 SYSTEMD_PACKAGES = "${@bb.utils.contains('PACKAGECONFIG', 'binfmt', '${PN}-binfmt', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'microhttpd', '${PN}-journal-gatewayd', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'microhttpd', '${PN}-journal-remote', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'journal-upload', '${PN}-journal-upload', '', d)} \
+                    ${@bb.utils.contains('PACKAGECONFIG', 'timesyncd', '${PN}-timesyncd', '', d)} \
 "
 SYSTEMD_SERVICE:${PN}-binfmt = "systemd-binfmt.service"
 
@@ -415,7 +420,7 @@ USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'coredump', '--sys
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'networkd', '--system -d / -M --shell /sbin/nologin systemd-network;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'polkit', '--system --no-create-home --user-group --home-dir ${sysconfdir}/polkit-1 polkitd;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'resolved', '--system -d / -M --shell /sbin/nologin systemd-resolve;', '', d)}"
-USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'timesyncd', '--system -d / -M --shell /sbin/nologin systemd-timesync;', '', d)}"
+USERADD_PARAM:${PN}-timesyncd = "--system -d / -M --shell /sbin/nologin systemd-timesync"
 USERADD_PARAM:${PN}-extra-utils = "--system -d / -M --shell /sbin/nologin systemd-bus-proxy"
 USERADD_PARAM:${PN}-journal-gatewayd = "--system -d / -M --shell /sbin/nologin systemd-journal-gateway"
 USERADD_PARAM:${PN}-journal-remote = "--system -d / -M --shell /sbin/nologin systemd-journal-remote"
@@ -473,6 +478,23 @@ FILES:${PN}-journal-remote = "${rootlibexecdir}/systemd/systemd-journal-remote \
                              "
 SYSTEMD_SERVICE:${PN}-journal-remote = "systemd-journal-remote.socket"
 
+FILES:${PN}-timesyncd = "${sysconfdir}/systemd/timesyncd.conf \
+                         ${rootlibexecdir}/systemd/ntp-units.d/80-systemd-timesync.list \
+                         ${systemd_system_unitdir}/systemd-timesyncd.service \
+                         ${systemd_system_unitdir}/systemd-time-wait-sync.service \
+                         ${rootlibexecdir}/systemd/systemd-timesyncd \
+                         ${rootlibexecdir}/systemd/systemd-time-wait-sync \
+                         ${exec_prefix}/lib/sysusers.d/systemd-timesync.conf \
+                         ${datadir}/dbus-1/system.d/org.freedesktop.timesync1.conf \
+                         ${datadir}/dbus-1/system-services/org.freedesktop.timesync1.service \
+                         ${datadir}/polkit-1/actions/org.freedesktop.timesync1.policy \
+                        "
+
+SYSTEMD_SERVICE:${PN}-timesyncd = "systemd-timesyncd.service \
+                                   systemd-time-wait-sync.service \
+                                  "
+
+RDEPENDS:${PN}-timesyncd = "${PN}"
 
 FILES:${PN}-container = "${sysconfdir}/dbus-1/system.d/org.freedesktop.import1.conf \
                          ${sysconfdir}/dbus-1/system.d/org.freedesktop.machine1.conf \
@@ -588,9 +610,9 @@ CONFFILES:${PN} = "${sysconfdir}/systemd/coredump.conf \
 	${sysconfdir}/systemd/resolved.conf \
 	${sysconfdir}/systemd/sleep.conf \
 	${sysconfdir}/systemd/system.conf \
-	${sysconfdir}/systemd/timesyncd.conf \
 	${sysconfdir}/systemd/user.conf \
 "
+CONFFILES:${PN}-timesyncd = "${sysconfdir}/systemd/timesyncd.conf"
 
 FILES:${PN} = " ${base_bindir}/* \
                 ${base_sbindir}/shutdown \
@@ -669,6 +691,7 @@ RRECOMMENDS:${PN} += "systemd-extra-utils \
                       kernel-module-autofs4 kernel-module-unix kernel-module-ipv6 kernel-module-sch-fq-codel \
                       os-release \
                       systemd-conf \
+                      ${@bb.utils.contains('PACKAGECONFIG', 'timesyncd', '${PN}-timesyncd', '', d)} \
 "
 
 INSANE_SKIP:${PN} += "dev-so libdir"
-- 
2.30.2

[OE-core][PATCH 2/3] systemd.bbclass: add non-recursive service packaging

[Peter Marko]

When service is split to separate package, it will take
all services it depends on. It does not matter is the dependency
is strong or week or if there is rdepends/rrecommends which would
be the proper way to pull it.

New variable SYSTEMD_PACKAGES_DONT_RECURSE allows to
skip this recursion for packages which are extracted to a package.
It is mostly useful for catch-all main package and splitting
additional packages with PACKAGE_BEFORE_PN.

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 documentation/ref-manual/variables.rst | 10 ++++++++++
 meta/classes-recipe/systemd.bbclass    | 15 ++++++++-------
 2 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 725f5c54cc..910b99aed2 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -8271,6 +8271,16 @@ system and gives an overview of their function and contents.
       :term:`SYSTEMD_PACKAGES`. Overrides not included in :term:`SYSTEMD_PACKAGES`
       will be silently ignored.
 
+   :term:`SYSTEMD_PACKAGES_DONT_RECURSE`
+      By default service files declared in :term:`SYSTEMD_SERVICE` are scanned
+      and all related service files are added to parsed package recursively.
+
+      It allows more readable and future-proof recipes, however it does not work well
+      when services are split to separate packages. This variable prevents this behavior.
+      Here is an example from systemd recipe::
+
+         SYSTEMD_PACKAGES_DONT_RECURSE:${PN}-networkd = "1"
+
    :term:`SYSVINIT_ENABLED_GETTYS`
       When using
       :ref:`SysVinit <dev-manual/new-recipe:enabling system services>`,
diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-recipe/systemd.bbclass
index f9c92e6c2a..c8cee482fe 100644
--- a/meta/classes-recipe/systemd.bbclass
+++ b/meta/classes-recipe/systemd.bbclass
@@ -124,19 +124,19 @@ python systemd_populate_packages() {
         return appended
 
     # Add systemd files to FILES:*-systemd, parse for Also= and follow recursive
-    def systemd_add_files_and_parse(pkg_systemd, path, service, keys):
+    def systemd_add_files_and_parse(pkg_systemd, path, service, keys, recurse):
         # avoid infinite recursion
-        if systemd_append_file(pkg_systemd, oe.path.join(path, service)):
+        if systemd_append_file(pkg_systemd, oe.path.join(path, service)) and recurse:
             fullpath = oe.path.join(d.getVar("D"), path, service)
             if service.find('.service') != -1:
                 # for *.service add *@.service
                 service_base = service.replace('.service', '')
-                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys)
+                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse)
             if service.find('.socket') != -1:
                 # for *.socket add *.service and *@.service
                 service_base = service.replace('.socket', '')
-                systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys)
-                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys)
+                systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys, recurse)
+                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse)
             for key in keys.split():
                 # recurse all dependencies found in keys ('Also';'Conflicts';..) and add to files
                 cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" % (key, shlex.quote(fullpath), key)
@@ -144,7 +144,7 @@ python systemd_populate_packages() {
                 line = pipe.readline()
                 while line:
                     line = line.replace('\n', '')
-                    systemd_add_files_and_parse(pkg_systemd, path, line, keys)
+                    systemd_add_files_and_parse(pkg_systemd, path, line, keys, recurse)
                     line = pipe.readline()
                 pipe.close()
 
@@ -157,6 +157,7 @@ python systemd_populate_packages() {
         keys = 'Also'
         # scan for all in SYSTEMD_SERVICE[]
         for pkg_systemd in systemd_packages.split():
+            recurse = False if d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd) else True
             for service in get_package_var(d, 'SYSTEMD_SERVICE', pkg_systemd).split():
                 path_found = ''
 
@@ -178,7 +179,7 @@ python systemd_populate_packages() {
                             break
 
                 if path_found != '':
-                    systemd_add_files_and_parse(pkg_systemd, path_found, service, keys)
+                    systemd_add_files_and_parse(pkg_systemd, path_found, service, keys, recurse)
                 else:
                     bb.fatal("Didn't find service unit '{0}', specified in SYSTEMD_SERVICE:{1}. {2}".format(
                         service, pkg_systemd, "Also looked for service unit '{0}'.".format(base) if base is not None else ""))
-- 
2.30.2

[OE-core][PATCH 3/3] systemd: split networkd to its own package

[Peter Marko]

Signed-off-by: Peter Marko <peter.marko@siemens.com>
---
 meta/recipes-core/systemd/systemd_252.4.bb | 37 ++++++++++++++++++++--
 1 file changed, 35 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_252.4.bb b/meta/recipes-core/systemd/systemd_252.4.bb
index de3ca93df8..180b95d6de 100644
--- a/meta/recipes-core/systemd/systemd_252.4.bb
+++ b/meta/recipes-core/systemd/systemd_252.4.bb
@@ -361,6 +361,7 @@ python populate_packages:prepend (){
 }
 PACKAGES_DYNAMIC += "^lib(udev|systemd|nss).*"
 
+# networkd needs to be after udev
 PACKAGE_BEFORE_PN = "\
     ${PN}-gui \
     ${PN}-vconsole-setup \
@@ -380,6 +381,7 @@ PACKAGE_BEFORE_PN = "\
     libsystemd-shared \
     udev \
     udev-hwdb \
+    ${PN}-networkd \
 "
 
 SUMMARY:${PN}-container = "Tools for containers and VMs"
@@ -394,6 +396,9 @@ DESCRIPTION:${PN}-journal-upload = "systemd-journal-upload uploads journal entri
 SUMMARY:${PN}-journal-remote = "Receive journal messages over the network"
 DESCRIPTION:${PN}-journal-remote = "systemd-journal-remote is a command to receive serialized journal events and store them to journal files."
 
+SUMMARY:${PN}-networkd = "System daemon that manages network configurations"
+DESCRIPTION:${PN}-networkd = "systemd-networkd daemon detects and configures network devices as they appear; it can also create virtual network devices"
+
 SUMMARY:${PN}-timesyncd = "Synchronize local system clock with a remote NTP server"
 DESCRIPTION:${PN}-timesyncd = "systemd-timesyncd is a system service that may be used to synchronize the local system clock with a remote Network Time Protocol (NTP) server."
 
@@ -403,6 +408,7 @@ SYSTEMD_PACKAGES = "${@bb.utils.contains('PACKAGECONFIG', 'binfmt', '${PN}-binfm
                     ${@bb.utils.contains('PACKAGECONFIG', 'microhttpd', '${PN}-journal-gatewayd', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'microhttpd', '${PN}-journal-remote', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'journal-upload', '${PN}-journal-upload', '', d)} \
+                    ${@bb.utils.contains('PACKAGECONFIG', 'networkd', '${PN}-networkd', '', d)} \
                     ${@bb.utils.contains('PACKAGECONFIG', 'timesyncd', '${PN}-timesyncd', '', d)} \
 "
 SYSTEMD_SERVICE:${PN}-binfmt = "systemd-binfmt.service"
@@ -417,9 +423,9 @@ GROUPADD_PARAM:${PN} = "-r systemd-journal;"
 GROUPADD_PARAM:udev = "-r render"
 GROUPADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'polkit_hostnamed_fallback', '-r systemd-hostname;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'coredump', '--system -d / -M --shell /sbin/nologin systemd-coredump;', '', d)}"
-USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'networkd', '--system -d / -M --shell /sbin/nologin systemd-network;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'polkit', '--system --no-create-home --user-group --home-dir ${sysconfdir}/polkit-1 polkitd;', '', d)}"
 USERADD_PARAM:${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'resolved', '--system -d / -M --shell /sbin/nologin systemd-resolve;', '', d)}"
+USERADD_PARAM:${PN}-networkd = "--system -d / -M --shell /sbin/nologin systemd-network"
 USERADD_PARAM:${PN}-timesyncd = "--system -d / -M --shell /sbin/nologin systemd-timesync"
 USERADD_PARAM:${PN}-extra-utils = "--system -d / -M --shell /sbin/nologin systemd-bus-proxy"
 USERADD_PARAM:${PN}-journal-gatewayd = "--system -d / -M --shell /sbin/nologin systemd-journal-gateway"
@@ -478,6 +484,32 @@ FILES:${PN}-journal-remote = "${rootlibexecdir}/systemd/systemd-journal-remote \
                              "
 SYSTEMD_SERVICE:${PN}-journal-remote = "systemd-journal-remote.socket"
 
+FILES:${PN}-networkd = "${base_bindir}/networkctl \
+                        ${sysconfdir}/systemd/network \
+                        ${sysconfdir}/systemd/networkd.conf \
+                        ${rootlibexecdir}/systemd/network/ \
+                        ${systemd_system_unitdir}/systemd-networkd.service \
+                        ${systemd_system_unitdir}/systemd-networkd.socket \
+                        ${systemd_system_unitdir}/systemd-networkd-wait-online.service \
+                        ${systemd_system_unitdir}/systemd-networkd-wait-online@.service \
+                        ${rootlibexecdir}/systemd/systemd-networkd \
+                        ${rootlibexecdir}/systemd/systemd-networkd-wait-online \
+                        ${exec_prefix}/lib/sysusers.d/systemd-network.conf \
+                        ${exec_prefix}/lib/tmpfiles.d/systemd-network.conf \
+                        ${datadir}/dbus-1/system.d/org.freedesktop.network1.conf \
+                        ${datadir}/dbus-1/system-services/org.freedesktop.network1.service \
+                        ${datadir}/polkit-1/actions/org.freedesktop.network1.policy \
+                       "
+
+SYSTEMD_SERVICE:${PN}-networkd = "systemd-networkd.service \
+                                  systemd-networkd.socket \
+                                  systemd-networkd-wait-online.service \
+                                  systemd-networkd-wait-online@.service \
+                                 "
+SYSTEMD_PACKAGES_DONT_RECURSE:${PN}-networkd = "1"
+
+RDEPENDS:${PN}-networkd = "${PN}"
+
 FILES:${PN}-timesyncd = "${sysconfdir}/systemd/timesyncd.conf \
                          ${rootlibexecdir}/systemd/ntp-units.d/80-systemd-timesync.list \
                          ${systemd_system_unitdir}/systemd-timesyncd.service \
@@ -605,13 +637,13 @@ FILES:${PN}-udev-rules = "\
 CONFFILES:${PN} = "${sysconfdir}/systemd/coredump.conf \
 	${sysconfdir}/systemd/journald.conf \
 	${sysconfdir}/systemd/logind.conf \
-	${sysconfdir}/systemd/networkd.conf \
 	${sysconfdir}/systemd/pstore.conf \
 	${sysconfdir}/systemd/resolved.conf \
 	${sysconfdir}/systemd/sleep.conf \
 	${sysconfdir}/systemd/system.conf \
 	${sysconfdir}/systemd/user.conf \
 "
+CONFFILES:${PN}-networkd = "${sysconfdir}/systemd/networkd.conf"
 CONFFILES:${PN}-timesyncd = "${sysconfdir}/systemd/timesyncd.conf"
 
 FILES:${PN} = " ${base_bindir}/* \
@@ -691,6 +723,7 @@ RRECOMMENDS:${PN} += "systemd-extra-utils \
                       kernel-module-autofs4 kernel-module-unix kernel-module-ipv6 kernel-module-sch-fq-codel \
                       os-release \
                       systemd-conf \
+                      ${@bb.utils.contains('PACKAGECONFIG', 'networkd', '${PN}-networkd', '', d)} \
                       ${@bb.utils.contains('PACKAGECONFIG', 'timesyncd', '${PN}-timesyncd', '', d)} \
 "
 
-- 
2.30.2

Re: [OE-core][PATCH 0/3] systemd: split timesync and networkd to packages

[Alexander Kanavin]

On Wed, 8 Feb 2023 at 08:13, Peter Marko <peter.marko@siemens.com> wrote:
> One alternative we considered was via systemd-conf, but binaries still
> occupy space in rootfs which is not desirable for some devices:
>   ln -sf /dev/null ${D}${sysconfdir}/systemd/system/systemd-timesyncd.service

On the other hand it's only a single line of code, instead of doubling
the ways systemd services can be packaged, chosen via a global,
publicly documented variable. When you are writing new code paths,
consider the ever-growing maintenance burden, always, please.

I'm leaning towards installing the services, but not starting them approach.

Alex

Re: [OE-core][PATCH 0/3] systemd: split timesync and networkd to packages

[ChenQi]

I think using systemd-conf to solve your original problem is more 
reasonable.
After all, the extra rootfs space is very small.

Also, BAD_RECOMMENDATIONS is not supported well for deb package backend.
poky/meta/classes-recipe/rootfs_deb.bbclass: bb.warn("Debian package 
install does not support BAD_RECOMMENDATIONS")

Regards,
Qi

On 2/8/23 15:38, Alexander Kanavin wrote:
> On Wed, 8 Feb 2023 at 08:13, Peter Marko <peter.marko@siemens.com> wrote:
>> One alternative we considered was via systemd-conf, but binaries still
>> occupy space in rootfs which is not desirable for some devices:
>>    ln -sf /dev/null ${D}${sysconfdir}/systemd/system/systemd-timesyncd.service
> On the other hand it's only a single line of code, instead of doubling
> the ways systemd services can be packaged, chosen via a global,
> publicly documented variable. When you are writing new code paths,
> consider the ever-growing maintenance burden, always, please.
>
> I'm leaning towards installing the services, but not starting them approach.
>
> Alex
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#176893): https://lists.openembedded.org/g/openembedded-core/message/176893
> Mute This Topic: https://lists.openembedded.org/mt/96825676/7304865
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Qi.Chen@eng.windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

RE: [OE-core][PATCH 2/3] systemd.bbclass: add non-recursive service packaging

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Peter Marko
> Sent: den 8 februari 2023 08:13
> To: openembedded-core@lists.openembedded.org
> Cc: Peter Marko <peter.marko@siemens.com>
> Subject: [OE-core][PATCH 2/3] systemd.bbclass: add non-recursive service packaging
> 
> When service is split to separate package, it will take
> all services it depends on. It does not matter is the dependency

Change "is" to "if".

> is strong or week or if there is rdepends/rrecommends which would

Change "week" to "weak".

> be the proper way to pull it.
> 
> New variable SYSTEMD_PACKAGES_DONT_RECURSE allows to
> skip this recursion for packages which are extracted to a package.
> It is mostly useful for catch-all main package and splitting

Change "catch-all main package" to either "a catch-all main package" 
or "catch-all main packages".

> additional packages with PACKAGE_BEFORE_PN.
> 
> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> ---
>  documentation/ref-manual/variables.rst | 10 ++++++++++
>  meta/classes-recipe/systemd.bbclass    | 15 ++++++++-------
>  2 files changed, 18 insertions(+), 7 deletions(-)
> 
> diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
> index 725f5c54cc..910b99aed2 100644
> --- a/documentation/ref-manual/variables.rst
> +++ b/documentation/ref-manual/variables.rst
> @@ -8271,6 +8271,16 @@ system and gives an overview of their function and contents.
>        :term:`SYSTEMD_PACKAGES`. Overrides not included in :term:`SYSTEMD_PACKAGES`
>        will be silently ignored.
> 
> +   :term:`SYSTEMD_PACKAGES_DONT_RECURSE`

Negative variable names should be avoided when possible. I suggest 
changing this to "SYSTEMD_PACKAGES_RECURSE". See below for how the 
implementation should be changed.

> +      By default service files declared in :term:`SYSTEMD_SERVICE` are scanned
> +      and all related service files are added to parsed package recursively.

Change "parsed" to "the parsed".

> +
> +      It allows more readable and future-proof recipes, however it does not work well
> +      when services are split to separate packages. This variable prevents this behavior.

Change "prevents" to "can be used to prevent".

> +      Here is an example from systemd recipe::
> +
> +         SYSTEMD_PACKAGES_DONT_RECURSE:${PN}-networkd = "1"
> +
>     :term:`SYSVINIT_ENABLED_GETTYS`
>        When using
>        :ref:`SysVinit <dev-manual/new-recipe:enabling system services>`,
> diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-recipe/systemd.bbclass
> index f9c92e6c2a..c8cee482fe 100644
> --- a/meta/classes-recipe/systemd.bbclass
> +++ b/meta/classes-recipe/systemd.bbclass
> @@ -124,19 +124,19 @@ python systemd_populate_packages() {
>          return appended
> 
>      # Add systemd files to FILES:*-systemd, parse for Also= and follow recursive
> -    def systemd_add_files_and_parse(pkg_systemd, path, service, keys):
> +    def systemd_add_files_and_parse(pkg_systemd, path, service, keys, recurse):
>          # avoid infinite recursion
> -        if systemd_append_file(pkg_systemd, oe.path.join(path, service)):
> +        if systemd_append_file(pkg_systemd, oe.path.join(path, service)) and recurse:
>              fullpath = oe.path.join(d.getVar("D"), path, service)
>              if service.find('.service') != -1:
>                  # for *.service add *@.service
>                  service_base = service.replace('.service', '')
> -                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys)
> +                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse)
>              if service.find('.socket') != -1:
>                  # for *.socket add *.service and *@.service
>                  service_base = service.replace('.socket', '')
> -                systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys)
> -                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys)
> +                systemd_add_files_and_parse(pkg_systemd, path, service_base + '.service', keys, recurse)
> +                systemd_add_files_and_parse(pkg_systemd, path, service_base + '@.service', keys, recurse)
>              for key in keys.split():
>                  # recurse all dependencies found in keys ('Also';'Conflicts';..) and add to files
>                  cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" % (key, shlex.quote(fullpath), key)
> @@ -144,7 +144,7 @@ python systemd_populate_packages() {
>                  line = pipe.readline()
>                  while line:
>                      line = line.replace('\n', '')
> -                    systemd_add_files_and_parse(pkg_systemd, path, line, keys)
> +                    systemd_add_files_and_parse(pkg_systemd, path, line, keys, recurse)
>                      line = pipe.readline()
>                  pipe.close()
> 
> @@ -157,6 +157,7 @@ python systemd_populate_packages() {
>          keys = 'Also'
>          # scan for all in SYSTEMD_SERVICE[]
>          for pkg_systemd in systemd_packages.split():
> +            recurse = False if d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd) else True

Change to:

            recurse = not bb.utils.to_boolean(d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd))

Or preferably, to avoid the negative variable name, change to:

            recurse = bb.utils.to_boolean(d.getVar('SYSTEMD_PACKAGES_RECURSE:' + pkg_systemd), True)

You should probably also change "d.getVar('SYSTEMD_PACKAGES_RECURSE:' + pkg_systemd)" 
to "get_package_var(d, 'SYSTEMD_PACKAGES_RECURSE', pkg_systemd)", which 
would allow setting SYSTEMD_PACKAGES_RECURSE = "0" in the recipe to disable 
recursing for all packages. In that case, also adapt the documentation 
accordingly.

>              for service in get_package_var(d, 'SYSTEMD_SERVICE', pkg_systemd).split():
>                  path_found = ''
> 
> @@ -178,7 +179,7 @@ python systemd_populate_packages() {
>                              break
> 
>                  if path_found != '':
> -                    systemd_add_files_and_parse(pkg_systemd, path_found, service, keys)
> +                    systemd_add_files_and_parse(pkg_systemd, path_found, service, keys, recurse)
>                  else:
>                      bb.fatal("Didn't find service unit '{0}', specified in SYSTEMD_SERVICE:{1}. {2}".format(
>                          service, pkg_systemd, "Also looked for service unit '{0}'.".format(base) if base is not None else ""))
> --
> 2.30.2

//Peter

Re: [OE-core][PATCH 2/3] systemd.bbclass: add non-recursive service packaging

[adrian.freihofer]

On Wed, 2023-02-08 at 08:12 +0100, Peter Marko wrote:
> When service is split to separate package, it will take
> all services it depends on. It does not matter is the dependency
> is strong or week or if there is rdepends/rrecommends which would
> be the proper way to pull it.
> 
> New variable SYSTEMD_PACKAGES_DONT_RECURSE allows to
> skip this recursion for packages which are extracted to a package.
> It is mostly useful for catch-all main package and splitting
> additional packages with PACKAGE_BEFORE_PN.
> 
> Signed-off-by: Peter Marko <peter.marko@siemens.com>
> ---
>  documentation/ref-manual/variables.rst | 10 ++++++++++
>  meta/classes-recipe/systemd.bbclass    | 15 ++++++++-------
>  2 files changed, 18 insertions(+), 7 deletions(-)
> 
> diff --git a/documentation/ref-manual/variables.rst
> b/documentation/ref-manual/variables.rst
> index 725f5c54cc..910b99aed2 100644
> --- a/documentation/ref-manual/variables.rst
> +++ b/documentation/ref-manual/variables.rst
> @@ -8271,6 +8271,16 @@ system and gives an overview of their function
> and contents.
>        :term:`SYSTEMD_PACKAGES`. Overrides not included in
> :term:`SYSTEMD_PACKAGES`
>        will be silently ignored.
>  
> +   :term:`SYSTEMD_PACKAGES_DONT_RECURSE`
> +      By default service files declared in :term:`SYSTEMD_SERVICE`
> are scanned
> +      and all related service files are added to parsed package
> recursively.
> +
> +      It allows more readable and future-proof recipes, however it
> does not work well
> +      when services are split to separate packages. This variable
> prevents this behavior.
> +      Here is an example from systemd recipe::
> +
> +         SYSTEMD_PACKAGES_DONT_RECURSE:${PN}-networkd = "1"
> +
>     :term:`SYSVINIT_ENABLED_GETTYS`
>        When using
>        :ref:`SysVinit <dev-manual/new-recipe:enabling system
> services>`,
> diff --git a/meta/classes-recipe/systemd.bbclass b/meta/classes-
> recipe/systemd.bbclass
> index f9c92e6c2a..c8cee482fe 100644
> --- a/meta/classes-recipe/systemd.bbclass
> +++ b/meta/classes-recipe/systemd.bbclass
> @@ -124,19 +124,19 @@ python systemd_populate_packages() {
>          return appended
>  
>      # Add systemd files to FILES:*-systemd, parse for Also= and
> follow recursive
> -    def systemd_add_files_and_parse(pkg_systemd, path, service,
> keys):
> +    def systemd_add_files_and_parse(pkg_systemd, path, service,
> keys, recurse):
>          # avoid infinite recursion
> -        if systemd_append_file(pkg_systemd, oe.path.join(path,
> service)):
> +        if systemd_append_file(pkg_systemd, oe.path.join(path,
> service)) and recurse:
>              fullpath = oe.path.join(d.getVar("D"), path, service)
>              if service.find('.service') != -1:
>                  # for *.service add *@.service
>                  service_base = service.replace('.service', '')
> -                systemd_add_files_and_parse(pkg_systemd, path,
> service_base + '@.service', keys)
> +                systemd_add_files_and_parse(pkg_systemd, path,
> service_base + '@.service', keys, recurse)
>              if service.find('.socket') != -1:
>                  # for *.socket add *.service and *@.service
>                  service_base = service.replace('.socket', '')
> -                systemd_add_files_and_parse(pkg_systemd, path,
> service_base + '.service', keys)
> -                systemd_add_files_and_parse(pkg_systemd, path,
> service_base + '@.service', keys)
> +                systemd_add_files_and_parse(pkg_systemd, path,
> service_base + '.service', keys, recurse)
> +                systemd_add_files_and_parse(pkg_systemd, path,
> service_base + '@.service', keys, recurse)
>              for key in keys.split():
>                  # recurse all dependencies found in keys
> ('Also';'Conflicts';..) and add to files
>                  cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" %
> (key, shlex.quote(fullpath), key)
> @@ -144,7 +144,7 @@ python systemd_populate_packages() {
>                  line = pipe.readline()
>                  while line:
>                      line = line.replace('\n', '')
> -                    systemd_add_files_and_parse(pkg_systemd, path,
> line, keys)
> +                    systemd_add_files_and_parse(pkg_systemd, path,
> line, keys, recurse)
>                      line = pipe.readline()
>                  pipe.close()
>  
> @@ -157,6 +157,7 @@ python systemd_populate_packages() {
>          keys = 'Also'

There would probably be another solution without introducing a new
SYSTEMD_PACKAGES_DONT_RECURSE variable. But that would require a fix
for the systemd.bbclass, which is a bit hard to test.

To me, it looks like the issue is that systemd.bbclass looks for
service files with "Also" in them. Depending on that, it changes the
FILES variable of packages. However, according to
https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Also=
"Also" means that a package with such a service file depends on another
package. This means that systemd.bbclass should change the RDEPENDS
variable but not the FILES variable. This bug potentially causes
service files to end up in the wrong package and that should be fixed.

My first thought was to fix the class as it was probably originally
intended. Instead of altering the FILES variable it should
automatically add packages to the RDEPENDS variable which are referred
by service files in this package with "Also".

However, since the "Also" could potentially also refer from a service
file provided by one recipe to a service file provided by another
recipe this saems not to be a complete solution.

Maybe the best would be:
1. Fix the invalid packaging by deleting the code from systemd.bbclass
which handles the "Also" service files:

            for key in keys.split():
                # recurse all dependencies found in keys
('Also';'Conflicts';..) and add to files
                cmd = "grep %s %s | sed 's,%s=,,g' | tr ',' '\\n'" %
(key, shlex.quote(fullpath), key)
                pipe = os.popen(cmd, 'r')
                line = pipe.readline()
                while line:
                    line = line.replace('\n', '')
                    systemd_add_files_and_parse(pkg_systemd, path,
line, keys)
                    bb.warn("After: %s : %s" % (pkg_systemd, path))
                    line = pipe.readline()
                pipe.close()

2. This might results in build failures because of not packaged service
files. That's not only bad because the automatic packaging was
potentially wrong anyway. The build failures need to be fixed manually
by adding the service files to the FILES variables of the broken
recipes.
3. To prevent the risk of building images with service files refering
to packages which are not in the image, a QA check on image level could
be added. I'm thinkg about grepping for "After" in all systemd units an
verify the referred file is in the image.

I'm going to analyze how many recipes in poky would be affected by such
a fix. Does that make sense?

Regards,
Adrian


>          # scan for all in SYSTEMD_SERVICE[]
>          for pkg_systemd in systemd_packages.split():
> +            recurse = False if
> d.getVar('SYSTEMD_PACKAGES_DONT_RECURSE:' + pkg_systemd) else True
>              for service in get_package_var(d, 'SYSTEMD_SERVICE',
> pkg_systemd).split():
>                  path_found = ''
>  
> @@ -178,7 +179,7 @@ python systemd_populate_packages() {
>                              break
>  
>                  if path_found != '':
> -                    systemd_add_files_and_parse(pkg_systemd,
> path_found, service, keys)
> +                    systemd_add_files_and_parse(pkg_systemd,
> path_found, service, keys, recurse)
>                  else:
>                      bb.fatal("Didn't find service unit '{0}',
> specified in SYSTEMD_SERVICE:{1}. {2}".format(
>                          service, pkg_systemd, "Also looked for
> service unit '{0}'.".format(base) if base is not None else ""))
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#176838):
> https://lists.openembedded.org/g/openembedded-core/message/176838
> Mute This Topic: https://lists.openembedded.org/mt/96825675/4454582
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub [
> adrian.freihofer@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>