pseudo 1.8.1 doesn't work with docker & dumb-init

pseudo 1.8.1 doesn't work with docker & dumb-init

[wenzong fan]

Hi Experts,

While I trying to build Yocto in Docker Container which using dumb-init 
as init system, I found the build always be stopped at some point and 
the container was terminated as well with below errors:

     Child process timeout after 2 seconds.
     Child process exit status 4: lock_held

Sometimes there's not any obvious error message.

After some `git bisect` testing, I believe the issue was started since 
commit:

----------------------
9df3cdf42d8c1216682f497f0b166a43ef9f4184 is the first bad commit
commit 9df3cdf42d8c1216682f497f0b166a43ef9f4184
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Tue Jul 5 13:18:31 2016 +0100

     pseudo: Upgrade to 1.8.1

     * Drop patches where the changes exist upstream
     * Fetch from git as no tarball is available for 1.8.1
     * Move common code to pseudo.inc
     * Update patchset in git recipe

     (From OE-Core rev: 0c36984d4c501d12fa91cf7371511641585cc256)
-----------------------

Finally I narrowed it down to pseudo commit:

------------------------
commit 77ee254a6c974aad9bcab2c58c9ee9e0880c9718
Author: Peter Seebach <peter.seebach@windriver.com>
Date: Tue Mar 1 16:21:15 2016 -0600

     Server launch reworking.

     This is the big overhaul to have the server provide meaningful exit 
status
     to clients.

     In the process, I discovered that the server was running with 
signals blocked
     if launched by a client, which is not a good thing, and prevented 
this from
     working as intended.

     Still looking to see why more than one server spawn seems to happen.
------------------------

I also created a testcase for reproducing the issue at:

     https://github.com/WenzongFan/docker-build-yocto

For dumb-init please refer to:

     https://github.com/Yelp/dumb-init

Could anyone help to fix the signal handling in pseudo?


Thanks
Wenzong

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[Joshua Lock]

On Wed, 2016-08-31 at 17:21 +0800, wenzong fan wrote:
> Hi Experts,
> 
> While I trying to build Yocto in Docker Container which using dumb-
> init 
> as init system, I found the build always be stopped at some point
> and 
> the container was terminated as well with below errors:
> 
>      Child process timeout after 2 seconds.
>      Child process exit status 4: lock_held
> 
> Sometimes there's not any obvious error message.
> 
> After some `git bisect` testing, I believe the issue was started
> since 
> commit:
> 
> ----------------------
> 9df3cdf42d8c1216682f497f0b166a43ef9f4184 is the first bad commit
> commit 9df3cdf42d8c1216682f497f0b166a43ef9f4184
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date: Tue Jul 5 13:18:31 2016 +0100
> 
>      pseudo: Upgrade to 1.8.1
> 
>      * Drop patches where the changes exist upstream
>      * Fetch from git as no tarball is available for 1.8.1
>      * Move common code to pseudo.inc
>      * Update patchset in git recipe
> 
>      (From OE-Core rev: 0c36984d4c501d12fa91cf7371511641585cc256)
> -----------------------
> 
> Finally I narrowed it down to pseudo commit:
> 
> ------------------------
> commit 77ee254a6c974aad9bcab2c58c9ee9e0880c9718
> Author: Peter Seebach <peter.seebach@windriver.com>
> Date: Tue Mar 1 16:21:15 2016 -0600
> 
>      Server launch reworking.
> 
>      This is the big overhaul to have the server provide meaningful
> exit 
> status
>      to clients.
> 
>      In the process, I discovered that the server was running with 
> signals blocked
>      if launched by a client, which is not a good thing, and
> prevented 
> this from
>      working as intended.
> 
>      Still looking to see why more than one server spawn seems to
> happen.
> ------------------------
> 
> I also created a testcase for reproducing the issue at:
> 
>      https://github.com/WenzongFan/docker-build-yocto

Thanks for providing a detailed reproducer. I'm trying to configure a
container behind my proxy here.

> 
> For dumb-init please refer to:
> 
>      https://github.com/Yelp/dumb-init
> 
> Could anyone help to fix the signal handling in pseudo?

It may not actually be pseudo at fault here. I've only skimmed the
dumb-init README but it looks like there might be a strange interaction
between the newly fixed signal handling in pseudo and dumb-init's
signal handling.

Should dumb-init be running in single-child/non-setsid mode so that
signals are only forwarded to the direct child rather than all child
processes in the dumb-init session? Is this a scenario you've tested?

Regards,

Joshua

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[Seebs]

On 31 Aug 2016, at 4:21, wenzong fan wrote:

> Finally I narrowed it down to pseudo commit:

Yes, that makes sense, we expect that there'd be potential issues, but I 
didn't have a reproducer for any. Thanks! I'll see whether it reproduces 
for me now. Any specific version of docker I might need?

-s

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[wenzong fan]

On 08/31/2016 11:11 PM, Joshua Lock wrote:
> On Wed, 2016-08-31 at 17:21 +0800, wenzong fan wrote:
>> Hi Experts,
>>
>> While I trying to build Yocto in Docker Container which using dumb-
>> init
>> as init system, I found the build always be stopped at some point
>> and
>> the container was terminated as well with below errors:
>>
>>      Child process timeout after 2 seconds.
>>      Child process exit status 4: lock_held
>>
>> Sometimes there's not any obvious error message.
>>
>> After some `git bisect` testing, I believe the issue was started
>> since
>> commit:
>>
>> ----------------------
>> 9df3cdf42d8c1216682f497f0b166a43ef9f4184 is the first bad commit
>> commit 9df3cdf42d8c1216682f497f0b166a43ef9f4184
>> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
>> Date: Tue Jul 5 13:18:31 2016 +0100
>>
>>      pseudo: Upgrade to 1.8.1
>>
>>      * Drop patches where the changes exist upstream
>>      * Fetch from git as no tarball is available for 1.8.1
>>      * Move common code to pseudo.inc
>>      * Update patchset in git recipe
>>
>>      (From OE-Core rev: 0c36984d4c501d12fa91cf7371511641585cc256)
>> -----------------------
>>
>> Finally I narrowed it down to pseudo commit:
>>
>> ------------------------
>> commit 77ee254a6c974aad9bcab2c58c9ee9e0880c9718
>> Author: Peter Seebach <peter.seebach@windriver.com>
>> Date: Tue Mar 1 16:21:15 2016 -0600
>>
>>      Server launch reworking.
>>
>>      This is the big overhaul to have the server provide meaningful
>> exit
>> status
>>      to clients.
>>
>>      In the process, I discovered that the server was running with
>> signals blocked
>>      if launched by a client, which is not a good thing, and
>> prevented
>> this from
>>      working as intended.
>>
>>      Still looking to see why more than one server spawn seems to
>> happen.
>> ------------------------
>>
>> I also created a testcase for reproducing the issue at:
>>
>>      https://github.com/WenzongFan/docker-build-yocto
>
> Thanks for providing a detailed reproducer. I'm trying to configure a
> container behind my proxy here.
>
>>
>> For dumb-init please refer to:
>>
>>      https://github.com/Yelp/dumb-init
>>
>> Could anyone help to fix the signal handling in pseudo?
>
> It may not actually be pseudo at fault here. I've only skimmed the
> dumb-init README but it looks like there might be a strange interaction
> between the newly fixed signal handling in pseudo and dumb-init's
> signal handling.
>
> Should dumb-init be running in single-child/non-setsid mode so that
> signals are only forwarded to the direct child rather than all child
> processes in the dumb-init session? Is this a scenario you've tested?

Yes, I had try below options, but all of them don't work:

1) Run dumb-init with the -c flag: 
https://github.com/Yelp/dumb-init/issues/51 - single-child/non-setsid mode
2) Update dumb-init to latest version v1.1.3 (the release notes mention 
fixes for race conditions)
3) Switch to tini which an alterative to dumb-init: 
https://github.com/krallin/tini

Thanks
Wenzong

>
> Regards,
>
> Joshua
>
>

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[wenzong fan]

On 08/31/2016 11:48 PM, Seebs wrote:
> On 31 Aug 2016, at 4:21, wenzong fan wrote:
>
>> Finally I narrowed it down to pseudo commit:
>
> Yes, that makes sense, we expect that there'd be potential issues, but I
> didn't have a reproducer for any. Thanks! I'll see whether it reproduces
> for me now. Any specific version of docker I might need?

No, I didn't think it's related to any specific docker version.

I tested it on "Docker version 1.7.1, build 786b29d" & "Docker version 
1.11.2, build b9f10c9".

BTW, I also tested the docker build w/o dumb-init, and the build works ...

Thanks
Wenzong

>
> -s
>

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[Seebs]

On 1 Sep 2016, at 20:33, wenzong fan wrote:

> No, I didn't think it's related to any specific docker version.
>
> I tested it on "Docker version 1.7.1, build 786b29d" & "Docker version 
> 1.11.2, build b9f10c9".
>
> BTW, I also tested the docker build w/o dumb-init, and the build works 
> ...

Yeah, it's definitely specific in some way to docker.

However, it doesn't appear to be 100% reproducible; I just tried a build 
with your reproducer and it completed without problems. (Unless the 
problems are more subtle, and don't prevent a build.) So this one's 
gonna be really fun to track down.

-s

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[wenzong fan]

On 09/02/2016 10:10 AM, Seebs wrote:
> On 1 Sep 2016, at 20:33, wenzong fan wrote:
>
>> No, I didn't think it's related to any specific docker version.
>>
>> I tested it on "Docker version 1.7.1, build 786b29d" & "Docker version
>> 1.11.2, build b9f10c9".
>>
>> BTW, I also tested the docker build w/o dumb-init, and the build works
>> ...
>
> Yeah, it's definitely specific in some way to docker.
>
> However, it doesn't appear to be 100% reproducible; I just tried a build
> with your reproducer and it completed without problems. (Unless the
> problems are more subtle, and don't prevent a build.) So this one's
> gonna be really fun to track down.

Yes, I believe it's not a 100 reproducible issue. Maybe you could run it 
with other builds in parallel and try it 3 times or more.

It keeps high probability on my work host which a server that shared by 
several persons, I can always get the error from 1 ~ 3 times build.

Thanks
Wenzong

>
> -s

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[Seebs]

On 7 Sep 2016, at 1:32, wenzong fan wrote:

> Yes, I believe it's not a 100 reproducible issue. Maybe you could run 
> it with other builds in parallel and try it 3 times or more.

I can try, but that might need bigger hardware than I have to hand at 
the moment.

-s

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[Bystricky, Juro]

I am pretty sure I glimpsed the messages: 
	Child process timeout after 2 seconds.
    	Child process exit status 4: lock_held
on several occasions recently, just before my Xserver was restarted and I was kicked back to the login prompt.  
I typically ran several parallel bitbake builds. Ubuntu 16.04, not using container. The last message in the syslog (first error message) was always:
Fatal IO error 11 (Resource temporarily unavailable) on X server :0

Possibly not related to this problem, nevertheless worth mentioning.

Thanks

Juro


> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Seebs
> Sent: Tuesday, September 6, 2016 11:40 PM
> To: Fan, Wenzong (Wind River) <wenzong.fan@windriver.com>
> Cc: Patches and discussions about the oe-core layer <openembedded-
> core@lists.openembedded.org>
> Subject: Re: [OE-core] pseudo 1.8.1 doesn't work with docker & dumb-init
> 
> On 7 Sep 2016, at 1:32, wenzong fan wrote:
> 
> > Yes, I believe it's not a 100 reproducible issue. Maybe you could run
> > it with other builds in parallel and try it 3 times or more.
> 
> I can try, but that might need bigger hardware than I have to hand at
> the moment.
> 
> -s
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[Randy MacLeod]

On 2016-09-14 04:46 PM, Bystricky, Juro wrote:
> I am pretty sure I glimpsed the messages:
> 	Child process timeout after 2 seconds.
>     	Child process exit status 4: lock_held
> on several occasions recently, just before my Xserver was restarted and I was kicked back to the login prompt.
> I typically ran several parallel bitbake builds. Ubuntu 16.04, not using container. The last message in the syslog (first error message) was always:
> Fatal IO error 11 (Resource temporarily unavailable) on X server :0
>
> Possibly not related to this problem, nevertheless worth mentioning.

Yes, it may be. Thanks for reporting it.

Two  weeks ago, I was building a qemuarm64 image on my laptop
(i7, 16 GB, SSD running Ubuntu-16.04)
and I saw a similarity bizarre result from running a build
in that chrome and then the X server were both killed.
I wasn't in front of the system when this happened so
I can't say exactly what was going on.

I did collect some of the logs from my IRC client and chrome:


[423679.028437] konversation[23416]: segfault at 7f72d2c33ce0 ip 
00007f72eca4e818 sp 00007ffc7f450ae0 error 4 in 
libQt5Gui.so.5.5.1[7f72ec8da000+527000]
[423679.325315] chrome[28083]: segfault at 968 ip 00007f63f7615643 sp 
00007ffd26c25af0 error 4 in libX11.so.6.3.0[7f63f75ed000+135000]


and then from the X server:

Aug 29 16:11:59 laptop org.a11y.atspi.Registry[4763]: XIO:  fatal IO 
error 11 (Resource temporarily unavailable) on X server ":0"
Aug 29 16:11:59 laptop org.a11y.atspi.Registry[4763]:       after 67649 
requests (67649 known processed) with 0 events remaining.
Aug 29 16:11:59 laptop gnome-session[4748]: (diodon:4925): Gdk-WARNING 
**: diodon: Fatal IO error 11 (Resource temporarily unavailable) on X 
server :0
.
...
Aug 29 16:11:59 laptop systemd[1]: Started Process Core Dump (PID 
28084/UID 0).
Aug 29 16:11:59 laptop gnome-session[4748]: Failed to connect to Mir: 
Failed to connect to server socket: No such file or directory
Aug 29 16:11:59 laptop kernel: [423679.325315] chrome[28083]: segfault 
at 968 ip 00007f63f7615643 sp 00007ffd26c25af0 error 4 in 
libX11.so.6.3.0[7f63f75ed000+135000]



In my case, I had added meta-oe to oe-core and was building:
    MACHINE=qemuarm64 bitbake imagemagick

I reproduced it once in X then did NOT see it happen when
I built with an X session running but building on the console.
i.e. the build of imagemagick for qemuarm64 succeeded.

I've removed the build logs it seems but I'll see if I can reproduce
the failure overnight.

../Randy

>
> Thanks
>
> Juro
>
>
>> -----Original Message-----
>> From: openembedded-core-bounces@lists.openembedded.org
>> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
>> Seebs
>> Sent: Tuesday, September 6, 2016 11:40 PM
>> To: Fan, Wenzong (Wind River) <wenzong.fan@windriver.com>
>> Cc: Patches and discussions about the oe-core layer <openembedded-
>> core@lists.openembedded.org>
>> Subject: Re: [OE-core] pseudo 1.8.1 doesn't work with docker & dumb-init
>>
>> On 7 Sep 2016, at 1:32, wenzong fan wrote:
>>
>>> Yes, I believe it's not a 100 reproducible issue. Maybe you could run
>>> it with other builds in parallel and try it 3 times or more.
>>
>> I can try, but that might need bigger hardware than I have to hand at
>> the moment.
>>
>> -s
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core


-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
Canada, K2K 2W5

Re: pseudo 1.8.1 doesn't work with docker & dumb-init

[Randy MacLeod]

On 2016-09-14 10:24 PM, Randy MacLeod wrote:
> I'll see if I can reproduce
> the failure overnight.

The laptop build worked without error. I may try again tonight.

-- 
# Randy MacLeod. SMTS, Linux, Wind River
Direct: 613.963.1350 | 350 Terry Fox Drive, Suite 200, Ottawa, ON, 
Canada, K2K 2W5