[PATCH V2 0/1] systemd: add back alternatives for init utitilies

[PATCH V2 0/1] systemd: add back alternatives for init utitilies

[Chen Qi]

Changes in V2:
* Remove 'init' from alternatives.

The following changes since commit 8e4c03744cfa8cc559682ebf7074643216dd2b9d:

  oe-selftest: devtool: Support meta being a symbolic link (2018-10-21 21:30:35 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib ChenQi/systemd-alternatives
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/systemd-alternatives

Chen Qi (1):
  systemd: add back alternatives for init utitilies

 meta/recipes-core/systemd/systemd_239.bb | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

-- 
1.9.1

[PATCH 1/1] systemd: add back alternatives for init utitilies

[Chen Qi]

Add back alternatives for init utilities to avoid regression.

These alternatives were removed when upgradeing systemd to 239.
They were removed out of the logic that init utitilies should be
bound to init manager. However, it turned out that two use cases
were not covered.

1) initramfs using commands like 'reboot' from busybox.
2) Users use customized busybox defconfig which enables init utilities.

The first use case caused a regression bug in yocto.
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=12914
Patches were sent to fix the reboot problem.

But this is not enough. As we may have the second use case. In such
situation, users will find themselves having regression error when
using 'busybox + systemd' (and busybox is installed after systemd,
overriding the systemd symlinks).

So in order to avoid regression, add back these alternatives.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/systemd/systemd_239.bb | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_239.bb b/meta/recipes-core/systemd/systemd_239.bb
index bc16407..61159ad 100644
--- a/meta/recipes-core/systemd/systemd_239.bb
+++ b/meta/recipes-core/systemd/systemd_239.bb
@@ -438,7 +438,7 @@ FILES_${PN} = " ${base_bindir}/* \
                 ${base_sbindir}/runlevel \
                 ${base_sbindir}/telinit \
                 ${base_sbindir}/resolvconf \
-                ${base_sbindir}/reboot.systemd \
+                ${base_sbindir}/reboot \
                 ${base_sbindir}/init \
                 ${datadir}/dbus-1/services \
                 ${datadir}/dbus-1/system-services \
@@ -546,14 +546,31 @@ python __anonymous() {
         d.setVar("INHIBIT_UPDATERCD_BBCLASS", "1")
 }
 
-ALTERNATIVE_${PN} = "resolv-conf reboot"
+ALTERNATIVE_${PN} = "halt reboot shutdown poweroff runlevel resolv-conf"
 
 ALTERNATIVE_TARGET[resolv-conf] = "${sysconfdir}/resolv-conf.systemd"
 ALTERNATIVE_LINK_NAME[resolv-conf] = "${sysconfdir}/resolv.conf"
 ALTERNATIVE_PRIORITY[resolv-conf] ?= "50"
 
+ALTERNATIVE_TARGET[halt] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[halt] = "${base_sbindir}/halt"
+ALTERNATIVE_PRIORITY[halt] ?= "300"
+
+ALTERNATIVE_TARGET[reboot] = "${base_bindir}/systemctl"
 ALTERNATIVE_LINK_NAME[reboot] = "${base_sbindir}/reboot"
-ALTERNATIVE_PRIORITY[reboot] = "100"
+ALTERNATIVE_PRIORITY[reboot] ?= "300"
+
+ALTERNATIVE_TARGET[shutdown] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[shutdown] = "${base_sbindir}/shutdown"
+ALTERNATIVE_PRIORITY[shutdown] ?= "300"
+
+ALTERNATIVE_TARGET[poweroff] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[poweroff] = "${base_sbindir}/poweroff"
+ALTERNATIVE_PRIORITY[poweroff] ?= "300"
+
+ALTERNATIVE_TARGET[runlevel] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[runlevel] = "${base_sbindir}/runlevel"
+ALTERNATIVE_PRIORITY[runlevel] ?= "300"
 
 pkg_postinst_${PN} () {
 	sed -e '/^hosts:/s/\s*\<myhostname\>//' \
-- 
1.9.1

Re: [PATCH 1/1] systemd: add back alternatives for init utitilies

[Richard Purdie]

On Mon, 2018-10-22 at 15:03 +0800, Chen Qi wrote:
> Add back alternatives for init utilities to avoid regression.
> 
> These alternatives were removed when upgradeing systemd to 239.
> They were removed out of the logic that init utitilies should be
> bound to init manager. However, it turned out that two use cases
> were not covered.
> 
> 1) initramfs using commands like 'reboot' from busybox.
> 2) Users use customized busybox defconfig which enables init
> utilities.
> 
> The first use case caused a regression bug in yocto.
>   https://bugzilla.yoctoproject.org/show_bug.cgi?id=12914
> Patches were sent to fix the reboot problem.
> 
> But this is not enough. As we may have the second use case. In such
> situation, users will find themselves having regression error when
> using 'busybox + systemd' (and busybox is installed after systemd,
> overriding the systemd symlinks).
> 
> So in order to avoid regression, add back these alternatives.
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>

Sorry but this is still failing:


https://autobuilder.yoctoproject.org/typhoon/#/builders/35/builds/107/steps/7/logs/step7c

Cheers,

Richard

Re: [PATCH 1/1] systemd: add back alternatives for init utitilies

[ChenQi]

On 10/24/2018 05:48 AM, Richard Purdie wrote:
> On Mon, 2018-10-22 at 15:03 +0800, Chen Qi wrote:
>> Add back alternatives for init utilities to avoid regression.
>>
>> These alternatives were removed when upgradeing systemd to 239.
>> They were removed out of the logic that init utitilies should be
>> bound to init manager. However, it turned out that two use cases
>> were not covered.
>>
>> 1) initramfs using commands like 'reboot' from busybox.
>> 2) Users use customized busybox defconfig which enables init
>> utilities.
>>
>> The first use case caused a regression bug in yocto.
>>    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12914
>> Patches were sent to fix the reboot problem.
>>
>> But this is not enough. As we may have the second use case. In such
>> situation, users will find themselves having regression error when
>> using 'busybox + systemd' (and busybox is installed after systemd,
>> overriding the systemd symlinks).
>>
>> So in order to avoid regression, add back these alternatives.
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> Sorry but this is still failing:
>
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/35/builds/107/steps/7/logs/step7c
>
> Cheers,
>
> Richard
>
>
Hi Richard,

The failure is revealed by Kevin's patches regarding udev-extraconf. 
More particularly, it's the following patch that reveals the problem.
"udev-extraconf: Use the canonical file name of systemd"

I've sent out a patch to remove udev-extraconf from 
packagegroup-core-lsb/-x11-sato to fix this failure.
I tested 'testimage + core-image-sato/lsb' with the following 5 patches 
(3 from Kevin which are now on master-next, 2 from me) with master 
branch, and the tests passed.
  packagegroup-core-lsb/-x11-sato: no udev-extraconf in case of systemd
  systemd: add back alternatives for init utitilies
  udev-extraconf: Skip the entry in /etc/fstab when using the systemd-mount
  udev-extraconf: Fix the recursively dependency for the systemd-mount
  udev-extraconf: Use the canonical file name of systemd

P.S.
I chose to remove udev-extraconf from these two packagegroups because:
1) udev-extraconf is needed in live image, so the automount rule needs 
to be there in the final package, regardless of the init manager of the 
real rootfs.
2) It's not clear whether users need the automount feature in case of 
systemd. So I didn't choose to modify the mount.sh script to exit 
directly if init manager is systemd.
3) I think it's not easy to make mount.sh reliable in systemd. Kevin's 
patches are good and helpful, but still not solve all problems. e.g. The 
mount.sh still doesn't take into consideration of .mount and .automount 
units; and it does not consider this failure case, i.e. no medium found 
on /dev/hdc.

Best Regards,
Chen Qi

Best Regards,
Chen Qi

Re: [PATCH 1/1] systemd: add back alternatives for init utitilies

[richard.purdie]

On Wed, 2018-10-24 at 14:16 +0800, ChenQi wrote:
> The failure is revealed by Kevin's patches regarding udev-extraconf. 
> More particularly, it's the following patch that reveals the problem.
> "udev-extraconf: Use the canonical file name of systemd"
> 
> I've sent out a patch to remove udev-extraconf from 
> packagegroup-core-lsb/-x11-sato to fix this failure.
> I tested 'testimage + core-image-sato/lsb' with the following 5
> patches 
> (3 from Kevin which are now on master-next, 2 from me) with master 
> branch, and the tests passed.
>   packagegroup-core-lsb/-x11-sato: no udev-extraconf in case of
> systemd
>   systemd: add back alternatives for init utitilies
>   udev-extraconf: Skip the entry in /etc/fstab when using the
> systemd-mount
>   udev-extraconf: Fix the recursively dependency for the systemd-
> mount
>   udev-extraconf: Use the canonical file name of systemd
> 
> P.S.
> I chose to remove udev-extraconf from these two packagegroups
> because:
> 1) udev-extraconf is needed in live image, so the automount rule
> needs 
> to be there in the final package, regardless of the init manager of
> the 
> real rootfs.
> 2) It's not clear whether users need the automount feature in case
> of 
> systemd. So I didn't choose to modify the mount.sh script to exit 
> directly if init manager is systemd.
> 3) I think it's not easy to make mount.sh reliable in systemd.
> Kevin's 
> patches are good and helpful, but still not solve all problems. e.g.
> The 
> mount.sh still doesn't take into consideration of .mount and
> .automount 
> units; and it does not consider this failure case, i.e. no medium
> found 
> on /dev/hdc.

Thanks for this, with travelling for ELC-E its been a week of
distractions so I appreciate someone looking into and figuring this
out!

Cheers,

Richard

[PATCH 0/1] systemd: add back alternatives for init utitilies

[Chen Qi]

*** BLURB HERE ***
The following changes since commit 52f1041e1a99d391d954f138589cef497a143cb1:

  qemux86/gstreamer: Move kernel module recommendation to the machine configuration (2018-10-18 23:26:35 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib ChenQi/systemd-alternatives
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=ChenQi/systemd-alternatives

Chen Qi (1):
  systemd: add back alternatives for init utitilies

 meta/recipes-core/systemd/systemd_239.bb | 27 ++++++++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

-- 
1.9.1

[PATCH 1/1] systemd: add back alternatives for init utitilies

[Chen Qi]

Add back alternatives for init utilities to avoid regression.

These alternatives were removed when upgradeing systemd to 239.
They were removed out of the logic that init utitilies should be
bound to init manager. However, it turned out that two use cases
were not covered.

1) initramfs using commands like 'reboot' from busybox.
2) Users use customized busybox defconfig which enables init utilities.

The first use case caused a regression bug in yocto.
  https://bugzilla.yoctoproject.org/show_bug.cgi?id=12914
Patches were sent to fix the reboot problem.

But this is not enough. As we may have the second use case. In such
situation, users will find themselves having regression error when
using 'busybox + systemd' (and busybox is installed after systemd,
overriding the systemd symlinks).

So in order to avoid regression, add back these alternatives.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/recipes-core/systemd/systemd_239.bb | 27 ++++++++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/systemd/systemd_239.bb b/meta/recipes-core/systemd/systemd_239.bb
index 7ed9321..0e84531 100644
--- a/meta/recipes-core/systemd/systemd_239.bb
+++ b/meta/recipes-core/systemd/systemd_239.bb
@@ -447,7 +447,7 @@ FILES_${PN} = " ${base_bindir}/* \
                 ${base_sbindir}/runlevel \
                 ${base_sbindir}/telinit \
                 ${base_sbindir}/resolvconf \
-                ${base_sbindir}/reboot.systemd \
+                ${base_sbindir}/reboot \
                 ${base_sbindir}/init \
                 ${datadir}/dbus-1/services \
                 ${datadir}/dbus-1/system-services \
@@ -556,14 +556,35 @@ python __anonymous() {
         d.setVar("INHIBIT_UPDATERCD_BBCLASS", "1")
 }
 
-ALTERNATIVE_${PN} = "resolv-conf reboot"
+ALTERNATIVE_${PN} = "init halt reboot shutdown poweroff runlevel resolv-conf"
 
 ALTERNATIVE_TARGET[resolv-conf] = "${sysconfdir}/resolv-conf.systemd"
 ALTERNATIVE_LINK_NAME[resolv-conf] = "${sysconfdir}/resolv.conf"
 ALTERNATIVE_PRIORITY[resolv-conf] ?= "50"
 
+ALTERNATIVE_TARGET[init] = "${rootlibexecdir}/systemd/systemd"
+ALTERNATIVE_LINK_NAME[init] = "${base_sbindir}/init"
+ALTERNATIVE_PRIORITY[init] ?= "300"
+
+ALTERNATIVE_TARGET[halt] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[halt] = "${base_sbindir}/halt"
+ALTERNATIVE_PRIORITY[halt] ?= "300"
+
+ALTERNATIVE_TARGET[reboot] = "${base_bindir}/systemctl"
 ALTERNATIVE_LINK_NAME[reboot] = "${base_sbindir}/reboot"
-ALTERNATIVE_PRIORITY[reboot] = "100"
+ALTERNATIVE_PRIORITY[reboot] ?= "300"
+
+ALTERNATIVE_TARGET[shutdown] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[shutdown] = "${base_sbindir}/shutdown"
+ALTERNATIVE_PRIORITY[shutdown] ?= "300"
+
+ALTERNATIVE_TARGET[poweroff] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[poweroff] = "${base_sbindir}/poweroff"
+ALTERNATIVE_PRIORITY[poweroff] ?= "300"
+
+ALTERNATIVE_TARGET[runlevel] = "${base_bindir}/systemctl"
+ALTERNATIVE_LINK_NAME[runlevel] = "${base_sbindir}/runlevel"
+ALTERNATIVE_PRIORITY[runlevel] ?= "300"
 
 pkg_postinst_${PN} () {
 	sed -e '/^hosts:/s/\s*\<myhostname\>//' \
-- 
1.9.1

Re: [PATCH 1/1] systemd: add back alternatives for init utitilies

[Richard Purdie]

On Fri, 2018-10-19 at 13:19 +0800, Chen Qi wrote:
> Add back alternatives for init utilities to avoid regression.
> 
> These alternatives were removed when upgradeing systemd to 239.
> They were removed out of the logic that init utitilies should be
> bound to init manager. However, it turned out that two use cases
> were not covered.
> 
> 1) initramfs using commands like 'reboot' from busybox.
> 2) Users use customized busybox defconfig which enables init
> utilities.
> 
> The first use case caused a regression bug in yocto.
>   https://bugzilla.yoctoproject.org/show_bug.cgi?id=12914
> Patches were sent to fix the reboot problem.
> 
> But this is not enough. As we may have the second use case. In such
> situation, users will find themselves having regression error when
> using 'busybox + systemd' (and busybox is installed after systemd,
> overriding the systemd symlinks).
> 
> So in order to avoid regression, add back these alternatives.
> 
> Signed-off-by

There is something odd going on which this change since it triggers:

https://autobuilder.yoctoproject.org/typhoon/#/builders/35/builds/95/steps/7/logs/step7c

I've isolated it to this change having initially thought it was Mark's
systemd change...

Cheers,

Richard

Re: [PATCH 1/1] systemd: add back alternatives for init utitilies

[ChenQi]

On 10/21/2018 05:37 AM, Richard Purdie wrote:
> On Fri, 2018-10-19 at 13:19 +0800, Chen Qi wrote:
>> Add back alternatives for init utilities to avoid regression.
>>
>> These alternatives were removed when upgradeing systemd to 239.
>> They were removed out of the logic that init utitilies should be
>> bound to init manager. However, it turned out that two use cases
>> were not covered.
>>
>> 1) initramfs using commands like 'reboot' from busybox.
>> 2) Users use customized busybox defconfig which enables init
>> utilities.
>>
>> The first use case caused a regression bug in yocto.
>>    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12914
>> Patches were sent to fix the reboot problem.
>>
>> But this is not enough. As we may have the second use case. In such
>> situation, users will find themselves having regression error when
>> using 'busybox + systemd' (and busybox is installed after systemd,
>> overriding the systemd symlinks).
>>
>> So in order to avoid regression, add back these alternatives.
>>
>> Signed-off-by
> There is something odd going on which this change since it triggers:
>
> https://autobuilder.yoctoproject.org/typhoon/#/builders/35/builds/95/steps/7/logs/step7c
>
> I've isolated it to this change having initially thought it was Mark's
> systemd change...
>
> Cheers,
>
> Richard
>
>
Hi Richard,

I saw all those reverts and tests about master-next on oe mailing list 
and I really feel sorry for causing autobuilder failures.
Before I sent out the patch, I did do testimage test with 
'core-image-minimal + systemd + ssh + package-management' and things 
were working well.

Back to this failure, after some investigation, I finally found the root 
cause. It's about udev-extraconf.
Before my patch, udev-extraconf actually does not work as originally 
designed. Check the following codes in mount.sh from udev-extraconf.
BASE_INIT="`readlink "/sbin/init"`"
INIT_SYSTEMD="/lib/systemd/systemd"

if [ "x$BASE_INIT" = "x$INIT_SYSTEMD" ];then
     # systemd as init uses systemd-mount to mount block devices
     MOUNT="/usr/bin/systemd-mount"
     UMOUNT="/usr/bin/systemd-umount"
[snip]

In our system, what we have is:
root@qemux86-64:~# ls -l /sbin/init
lrwxrwxrwx    1 root     root            22 Oct 22 04:00 /sbin/init -> 
../lib/systemd/systemd

As '../lib/systemd/systemd' does not equal to '/lib/systemd/systemd', 
the mount.sh is not using systemd-mount.

When checking links, we should use `readlnk -f' instead of just 
'readlink'. In other words, things happen to succeed for core-image-sato 
because of some error in the mount.sh script from udev-extraconf.

My patch links /sbin/init to /lib/systemd/systemd and that makes the 
mount.sh start to work, thus revealing the error.
In fact, besides the dev-vda.mount problem, I also got 
media-run-hdc.mount failure on core-image-sato. They are both caused by 
the udev-extraconf.

I'm going to remove the 'init' from alternatives and send out V2. But 
udev-extraconf also needs to be fixed.
(CC Hongzhi who did the change.)

Richard, what do you think we should do about the automount udev rule 
from udev-extraconf?
I'd suggest that we do not install the automount udev rule in case of 
systemd. I think the mount.sh script is likely to cause conflicts and 
failures unless it's constructed very carefully in case of systemd. 
Unfortunately, I don't think that script could be easily made reliable, 
considering all the possible .mount and .automount units that users may 
add as custom configuration.
Hongzhi, what's your opinion?

Best Regards,
Chen Qi

Re: [PATCH 1/1] systemd: add back alternatives for init utitilies

[Hongzhi, Song]

On 10/22/2018 03:01 PM, ChenQi wrote:
> On 10/21/2018 05:37 AM, Richard Purdie wrote:
>> On Fri, 2018-10-19 at 13:19 +0800, Chen Qi wrote:
>>> Add back alternatives for init utilities to avoid regression.
>>>
>>> These alternatives were removed when upgradeing systemd to 239.
>>> They were removed out of the logic that init utitilies should be
>>> bound to init manager. However, it turned out that two use cases
>>> were not covered.
>>>
>>> 1) initramfs using commands like 'reboot' from busybox.
>>> 2) Users use customized busybox defconfig which enables init
>>> utilities.
>>>
>>> The first use case caused a regression bug in yocto.
>>>    https://bugzilla.yoctoproject.org/show_bug.cgi?id=12914
>>> Patches were sent to fix the reboot problem.
>>>
>>> But this is not enough. As we may have the second use case. In such
>>> situation, users will find themselves having regression error when
>>> using 'busybox + systemd' (and busybox is installed after systemd,
>>> overriding the systemd symlinks).
>>>
>>> So in order to avoid regression, add back these alternatives.
>>>
>>> Signed-off-by
>> There is something odd going on which this change since it triggers:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/35/builds/95/steps/7/logs/step7c 
>>
>>
>> I've isolated it to this change having initially thought it was Mark's
>> systemd change...
>>
>> Cheers,
>>
>> Richard
>>
>>
> Hi Richard,
>
> I saw all those reverts and tests about master-next on oe mailing list 
> and I really feel sorry for causing autobuilder failures.
> Before I sent out the patch, I did do testimage test with 
> 'core-image-minimal + systemd + ssh + package-management' and things 
> were working well.
>
> Back to this failure, after some investigation, I finally found the 
> root cause. It's about udev-extraconf.
> Before my patch, udev-extraconf actually does not work as originally 
> designed. Check the following codes in mount.sh from udev-extraconf.
> BASE_INIT="`readlink "/sbin/init"`"
> INIT_SYSTEMD="/lib/systemd/systemd"
>
> if [ "x$BASE_INIT" = "x$INIT_SYSTEMD" ];then
>     # systemd as init uses systemd-mount to mount block devices
>     MOUNT="/usr/bin/systemd-mount"
>     UMOUNT="/usr/bin/systemd-umount"
> [snip]
>
> In our system, what we have is:
> root@qemux86-64:~# ls -l /sbin/init
> lrwxrwxrwx    1 root     root            22 Oct 22 04:00 /sbin/init -> 
> ../lib/systemd/systemd
>
> As '../lib/systemd/systemd' does not equal to '/lib/systemd/systemd', 
> the mount.sh is not using systemd-mount.
>
> When checking links, we should use `readlnk -f' instead of just 
> 'readlink'. In other words, things happen to succeed for 
> core-image-sato because of some error in the mount.sh script from 
> udev-extraconf.
>
> My patch links /sbin/init to /lib/systemd/systemd and that makes the 
> mount.sh start to work, thus revealing the error.
> In fact, besides the dev-vda.mount problem, I also got 
> media-run-hdc.mount failure on core-image-sato. They are both caused 
> by the udev-extraconf.
>
> I'm going to remove the 'init' from alternatives and send out V2. But 
> udev-extraconf also needs to be fixed.
> (CC Hongzhi who did the change.)
>
> Richard, what do you think we should do about the automount udev rule 
> from udev-extraconf?
> I'd suggest that we do not install the automount udev rule in case of 
> systemd. I think the mount.sh script is likely to cause conflicts and 
> failures unless it's constructed very carefully in case of systemd. 
> Unfortunately, I don't think that script could be easily made 
> reliable, considering all the possible .mount and .automount units 
> that users may add as custom configuration.
> Hongzhi, what's your opinion?

Hi all,

I am so sorry for causing failures. I will send patch to fix the issue 
as soon as possible.

I agree with Chen Qi's suggestions. The "mount.sh" is out-of-date.

--Hongzhi

>
> Best Regards,
> Chen Qi
>