State of systemd from an integrator point of view

State of systemd from an integrator point of view

[Holger Hans Peter Freyther]

Hi,

this is my second attempt to use systemd with Poky for our sysmoBTS
product (mostly to replace some respawn shell scripts and not to use
the daemontools). I aborted the first trial because of journald and
now that systemd is being merged into OE-Core I started again and I
would like to share some of my issues.


* systemd as a watchdog. For our BTS I have a init script for the
watchdog package and a configuration file in our BSP layer. For systemd
I want the watchdog functionality provided by systemd itself. So I
have a system.conf in my BSP layer an this line in my machine config:

MACHINE_ESSENTIAL_EXTRA_RDEPENDS = "\
	...
	${@['watchdog', ''][d.getVar('DISTRO_FEATURES_INITMAN', True) == 'systemd']} \
	kernel-module-rtfifo "

IMAGE_FSTYPES ?= "tar.bz2 cpio.gz ubifs ubi jffs2"

MACHINE_EXTRA_RDEPENDS = "\
	...
	${@['watchdog', ''][d.getVar('DISTRO_FEATURES_INITMAN', True) == 'systemd']} \
	"

this is obviously not a piece of beauty.. and it would be nicer if
systemd itself could build a watchdog package or such...


* systemd and svsv start scripts. I am using the busybox ifplugd to
configure the network on plug/un-plug of the cable. My start script ended
with .sh in the name. systemd recognized the script but failed to start
it. Renaming it to not end with .sh fixed the issue for me. This is more
like a systemd bug than OE but I wonder if we could change update-rc.d
bbclass to check the for the .sh in the start script and then complain
about it?


* systemd's journald is not suitable for small devices. I think the default
should be to disable the journald and do not disable the busybox log
application in the systemd-compat units package. journald is dominated by
memory allocations and reading procfs over and over again.


regards
	holger

Re: State of systemd from an integrator point of view

[Burton, Ross]

Hi Holger,

On 18 February 2013 10:24, Holger Hans Peter Freyther
<holger@moiji-mobile.com> wrote:
> this is my second attempt to use systemd with Poky for our sysmoBTS
> product (mostly to replace some respawn shell scripts and not to use
> the daemontools). I aborted the first trial because of journald and
> now that systemd is being merged into OE-Core I started again and I
> would like to share some of my issues.

Great feedback, thanks Holger.

> * systemd as a watchdog. For our BTS I have a init script for the
> watchdog package and a configuration file in our BSP layer. For systemd
> I want the watchdog functionality provided by systemd itself. So I
> have a system.conf in my BSP layer an this line in my machine config:
>
> MACHINE_ESSENTIAL_EXTRA_RDEPENDS = "\
>         ...
>         ${@['watchdog', ''][d.getVar('DISTRO_FEATURES_INITMAN', True) == 'systemd']} \
>         kernel-module-rtfifo "
>
> IMAGE_FSTYPES ?= "tar.bz2 cpio.gz ubifs ubi jffs2"
>
> MACHINE_EXTRA_RDEPENDS = "\
>         ...
>         ${@['watchdog', ''][d.getVar('DISTRO_FEATURES_INITMAN', True) == 'systemd']} \
>         "
>
> this is obviously not a piece of beauty.. and it would be nicer if
> systemd itself could build a watchdog package or such...

The INITMAN variable is injected into DISTRO_FEATURES, so I do find
the more idiomatic base_contains more readable:

${@base_contains("DISTRO_FEATURES", "systemd", "", "watchdog", d)}

systemd could RPROVIDE="watchdog" but as the means to configure it are
different that might not be ideal, and if you ever switched entirely
to systemd with these conditionals it's obvious where your redundant
dependencies are.

Personally, I'd stick with conditional dependencies.

> * systemd and svsv start scripts. I am using the busybox ifplugd to
> configure the network on plug/un-plug of the cable. My start script ended
> with .sh in the name. systemd recognized the script but failed to start
> it. Renaming it to not end with .sh fixed the issue for me. This is more
> like a systemd bug than OE but I wonder if we could change update-rc.d
> bbclass to check the for the .sh in the start script and then complain
> about it?

Oh, interesting.  Without any thinking I can't think of reasons why
that would fail.    Filed #3900.

> * systemd's journald is not suitable for small devices. I think the default
> should be to disable the journald and do not disable the busybox log
> application in the systemd-compat units package. journald is dominated by
> memory allocations and reading procfs over and over again.

Agreed, although it would be good if the journal could be tuned.  IMHO
it's technically great, just young.

I wonder if we can split out the bulk of the journal binaries into a
separate package, so by default it's disabled and then installing
systemd-journal would enable it.

Filed #3901.

Cheers,
Ross

Re: State of systemd from an integrator point of view

[Burton, Ross]

On 18 February 2013 13:10, Burton, Ross <ross.burton@intel.com> wrote:
>> * systemd's journald is not suitable for small devices. I think the default
>> should be to disable the journald and do not disable the busybox log
>> application in the systemd-compat units package. journald is dominated by
>> memory allocations and reading procfs over and over again.
>
> Agreed, although it would be good if the journal could be tuned.  IMHO
> it's technically great, just young.
>
> I wonder if we can split out the bulk of the journal binaries into a
> separate package, so by default it's disabled and then installing
> systemd-journal would enable it.

Just noticed that the disabling of syslog isn't working for me, it
disables syslog-busybox but there's a syslog symlink to it, which is
being started. :)

Ross

Re: State of systemd from an integrator point of view

[Holger Hans Peter Freyther]

On Mon, Feb 18, 2013 at 01:14:47PM +0000, Burton, Ross wrote:

> Just noticed that the disabling of syslog isn't working for me, it
> disables syslog-busybox but there's a syslog symlink to it, which is
> being started. :)

I haven't gotten to the part to get it started again. The symlink
probably exists to deal with a real syslog package being installed.
I am not sure how to deal with that. The way forward is probably to
have syslog package install some kind of /etc/default/* that inhibits
starting the busybox one.

regards
	holger

Re: State of systemd from an integrator point of view

[Holger Hans Peter Freyther]

On Mon, Feb 18, 2013 at 01:10:27PM +0000, Burton, Ross wrote:

> The INITMAN variable is injected into DISTRO_FEATURES, so I do find
> the more idiomatic base_contains more readable:
> 
> ${@base_contains("DISTRO_FEATURES", "systemd", "", "watchdog", d)}

thanks. I will update my BSP.

> 
> systemd could RPROVIDE="watchdog" but as the means to configure it are
> different that might not be ideal, and if you ever switched entirely
> to systemd with these conditionals it's obvious where your redundant
> dependencies are.
> 
> Personally, I'd stick with conditional dependencies.

true, the watchdog package can potentially do a lot more as well. I also
have one watchdog related issue in my kernel/systemd. The TI Davinci wdt
doesn't allow to set the interval via IOCTL.


> Oh, interesting.  Without any thinking I can't think of reasons why
> that would fail.    Filed #3900.

I will try to find some time toward the end to re-produce and provide
feedback. I think it is a genuine bug in systemd. I am going to update
the ticket then.


> Agreed, although it would be good if the journal could be tuned.  IMHO
> it's technically great, just young.

journald is doing a 'lot' of work for each syslog message. E.g. if one
takes a look at journald-server.c:dispatch_message_real. I don't know how
the messages are flowing but from what I see the only way it could be
made usable is:

a.) API changes to there helpers to avoid dynamic allocations in many
of the cases. E.g. re-use one dynamically allocated string.

   r = get_process_cmdline(ucred->pid, 0, false, &t);
   if (r >= 0) {
        cmdline = strappend("_CMDLINE=", t);
        free(t);

                        if (cmdline)
        IOVEC_SET_STRING(iovec[n++], cmdline);
   }

in here there are at least:
   1.) a malloc for /proc/%lu/cmdline
   2.) a malloc for the FILE* pointer by glibc
   (3.) a mmap for ANOM memory in the fgets callby glibc)
     this example uses fgetc so it might not apply
   3.) re-allocs for every character read from the file
   4.) the strappend does another allocation and memcpy.


b.) Reducing the work by having the systemd send all this info with
each message (probably it doesn't work because of the interface that
is attempted to be used between systemd and the 'syslog'?

c.) Disabling/caching most of this information. E.g. the audit handling
should probably be behind a #ifdef if AUDIT is disabled. I will probably
experiment with putting a #ifdef to most of the additional information
it wants to query.

Upstream will probably end up doing nothing about it.

regards

holger