systemd timeouts started daemons

systemd timeouts started daemons

[Jukka Rissanen]

Hi,

I have a weird problem with systemd. It starts a daemon then kills it 
after a 30 second timeout.

Mar 12 08:50:19 eca connmand[970]: Connection Manager version 1.12
...
Mar 12 08:51:49 eca systemd[1]: connman.service operation timed out. 
Terminating.
Mar 12 08:51:49 eca connmand[970]: Terminating

I am seeing this behaviour at least with connman, bluez and ofono.
I can start the daemons from command line (as a root) and they work just 
fine. I can connect to connman via dbus and it can properly connect to 
ethernet etc.

When connman tries to activate bluez, following is printed

Mar 12 08:50:19 eca dbus[941]: [system] Activating service 
name='org.bluez' (using servicehelper)
Mar 12 08:50:19 eca dbus[941]: [system] Activated service 'org.bluez' 
failed: Launch helper exited with unknown return code 1

Usually also connman-vpnd cannot be started, but in this run it was 
activated successfully

Mar 12 08:50:19 eca dbus[941]: [system] Activating service 
name='net.connman.vpn' (using servicehelper)
Mar 12 08:50:19 eca dbus[941]: [system] Successfully activated service 
'net.connman.vpn'

So it looks there is something fishy with dbus.

I am pretty sure this was working something like a month ago, but cannot 
really point out what version I had at that time.

Currently I am using latest yocto + oe from git.


Any pointers / ideas what is wrong here?


Cheers,
Jukka

Re: systemd timeouts started daemons

[Jukka Rissanen]

Hi all,

some update to the problem I am seeing with systemd.

So the symptom of the problem is that after systemd has started a daemon 
like connmand, it kills it after 30 secs.

The reason for this was that systemd did not receive NameOwnerChanged 
dbus signal telling that the daemon managed to register its name to the 
system bus.

I checked with dbus-monitor that these NameOwnerChanged signals are 
indeed sent by dbus-daemon but they are not seen in debug prints from 
systemd.

The systemd version I have is 197, dbus version is 1.6.8. I tried with 
kernel 3.4 and 3.8, the same result. The distro I am using is very 
minimal one without X11 or any other graphics, so it might be that I 
have accidentally removed some vital component. The distro is using 
packagegroup-base + some additional packages, polkit is not installed.

What is even weirder that in qemu image, I can start connman 
successfully via systemd but not ofono or bluez. In Intel NUC device, I 
am not able to start either connman, bluez or ofono from systemd. All 
these daemons work just fine when started manually from command line.

This paste contains qemu log where connman starts successfully but ofono 
and bluez do not. http://pastebin.com/UTUVBjqu

I am using latest yocto + oe from git.

Any ideas what is wrong here?


Cheers,
Jukka

Re: systemd timeouts started daemons

[Burton, Ross]

On 20 March 2013 08:27, Jukka Rissanen <jukka.rissanen@linux.intel.com> wrote:
> The reason for this was that systemd did not receive NameOwnerChanged dbus
> signal telling that the daemon managed to register its name to the system
> bus.
>
> I checked with dbus-monitor that these NameOwnerChanged signals are indeed
> sent by dbus-daemon but they are not seen in debug prints from systemd.

Andreas sent a patch earlier that masks the dbus init script, because
the names don't match so you'll end up with two system buses
attempting to run.  I'm not sure how this will fail, but it could mean
that systemd isn't listening to the bus that connman is starting on,
or something.

The easy hack to test this is to delete /etc/init.d/dbus-1 and the
symlinks to it in /etc/rc*.

Ross

Re: systemd timeouts started daemons

[Jukka Rissanen]

Hi Ross,

On 21.03.2013 14:36, Burton, Ross wrote:
> On 20 March 2013 08:27, Jukka Rissanen <jukka.rissanen@linux.intel.com> wrote:
>> The reason for this was that systemd did not receive NameOwnerChanged dbus
>> signal telling that the daemon managed to register its name to the system
>> bus.
>>
>> I checked with dbus-monitor that these NameOwnerChanged signals are indeed
>> sent by dbus-daemon but they are not seen in debug prints from systemd.
>
> Andreas sent a patch earlier that masks the dbus init script, because
> the names don't match so you'll end up with two system buses
> attempting to run.  I'm not sure how this will fail, but it could mean
> that systemd isn't listening to the bus that connman is starting on,
> or something.
>
> The easy hack to test this is to delete /etc/init.d/dbus-1 and the
> symlinks to it in /etc/rc*.
>

Removed the dbus from init.d and rc* and everything started to work.
This was really good news, thanks a lot for your help!

Cheers,
Jukka

Re: systemd timeouts started daemons

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1180 bytes --]

Does this resolve also update-rc.d postinst trying to start /etc/init.d/dbus-1
and failing to find it?


On Thu, Mar 21, 2013 at 1:36 PM, Burton, Ross <ross.burton@intel.com> wrote:

> On 20 March 2013 08:27, Jukka Rissanen <jukka.rissanen@linux.intel.com>
> wrote:
> > The reason for this was that systemd did not receive NameOwnerChanged
> dbus
> > signal telling that the daemon managed to register its name to the system
> > bus.
> >
> > I checked with dbus-monitor that these NameOwnerChanged signals are
> indeed
> > sent by dbus-daemon but they are not seen in debug prints from systemd.
>
> Andreas sent a patch earlier that masks the dbus init script, because
> the names don't match so you'll end up with two system buses
> attempting to run.  I'm not sure how this will fail, but it could mean
> that systemd isn't listening to the bus that connman is starting on,
> or something.
>
> The easy hack to test this is to delete /etc/init.d/dbus-1 and the
> symlinks to it in /etc/rc*.
>
> Ross
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

[-- Attachment #2: Type: text/html, Size: 1934 bytes --]

Re: systemd timeouts started daemons

[Burton, Ross]

On 21 March 2013 14:10, Martin Jansa <martin.jansa@gmail.com> wrote:
> Does this resolve also update-rc.d postinst trying to start
> /etc/init.d/dbus-1 and failing to find it?

That's something else to be fixed.

Ross