Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

[srikanth krishnakar]

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

Hello,

Environment: Ubuntu-14.04.5/16.04.1 (64-bit)
Yocto build: qemuarm
Target image: core-image-sato
Error nature: The lightdm restarts on its own and logs out by killing
processes running and brings up a login UI.


We have been observing yocto-2.2 build failures on Ubuntu 14.04.5 and
Ubuntu-16.04.1 hosts due to restart of "lightdm" (Light Desktop Manager)
 that is triggered by crash of "unity-settings-daemon", we couldn't figure
out any workaround so far to overcome the issue. The build goes fine in the
beginning but eventually the unity desktop logs out and kills all the
processes running in the user session (including bitbake) and lands us into
"Login screen" and when we login its a new session where as the bitbake
running in previous session is killed and we need to continue the build.
This is happening consistently when the user logs into desktop via lightdm
and triggers a build.

Another interesting thing to notice is the build goes fine if we connect to
host via. SSH session and invoke a bitbake build. We are suspecting on
unity-settings-daemon that is crashing consistently and is evident from the
".xsession-errors" log file as shown below:

builder@ubuntu:~$ cat .xsession-errors.old
Script for ibus started at run_im.
Script for auto started at run_im.
Script for default started at run_im.
init: Disconnected from notified D-Bus bus
init:"unity-settings-daemon main process (2076) terminated with status 1"
init: indicator-bluetooth main process (2212) killed by TERM signal
init: indicator-power main process (2214) killed by TERM signal
init: indicator-datetime main process (2215) killed by TERM signal
init: indicator-printers main process (2222) killed by TERM signal
init: indicator-session main process (2245) killed by TERM signal
init: indicator-application main process (2270) killed by TERM signal


Corresponding unity-settings-daemon bug reported in Launchpad:

https://bugs.launchpad.net/ubuntu/+source/unity-settings-daemon/+bug/1546641

Would anyone kindly confirm the logout behaviour during bitbake process on
Ubuntu-14.04.5 & Ubuntu-16.04.1 ? Since Yocto 2.2 mentions both
Ubuntu-14.04 & 16.04 as supported distributions this blocker issue must be
resolved at earliest.

Appreciate your Inputs !

Thanks in Advance.
-Srikant

[-- Attachment #2: Type: text/html, Size: 4498 bytes --]

Re: Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

[Richard Purdie]

On Mon, 2016-12-26 at 15:11 +0530, srikanth krishnakar wrote:
> Environment: Ubuntu-14.04.5/16.04.1 (64-bit)
> Yocto build: qemuarm
> Target image: core-image-sato
> Error nature: The lightdm restarts on its own and logs out by killing
> processes running and brings up a login UI. 
> 
> 
> We have been observing yocto-2.2 build failures on Ubuntu 14.04.5 and
> Ubuntu-16.04.1 hosts due to restart of "lightdm" (Light Desktop
> Manager)  that is triggered by crash of "unity-settings-daemon", we
> couldn't figure out any workaround so far to overcome the issue. The
> build goes fine in the beginning but eventually the unity desktop
> logs out and kills all the processes running in the user session
> (including bitbake) and lands us into "Login screen" and when we
> login its a new session where as the bitbake running in previous
> session is killed and we need to continue the build. This is
> happening consistently when the user logs into desktop via lightdm
> and triggers a build. 
> 
> Another interesting thing to notice is the build goes fine if we
> connect to host via. SSH session and invoke a bitbake build. We are
> suspecting on unity-settings-daemon that is crashing consistently and
> is evident from the ".xsession-errors" log file as shown below:
> 
> builder@ubuntu:~$ cat .xsession-errors.old
> Script for ibus started at run_im.
> Script for auto started at run_im.
> Script for default started at run_im.
> init: Disconnected from notified D-Bus bus
> init:"unity-settings-daemon main process (2076) terminated with
> status 1"
> init: indicator-bluetooth main process (2212) killed by TERM signal
> init: indicator-power main process (2214) killed by TERM signal
> init: indicator-datetime main process (2215) killed by TERM signal
> init: indicator-printers main process (2222) killed by TERM signal
> init: indicator-session main process (2245) killed by TERM signal
> init: indicator-application main process (2270) killed by TERM signal
> 
> Corresponding unity-settings-daemon bug reported in Launchpad:
> 
> https://bugs.launchpad.net/ubuntu/+source/unity-settings-daemon/+bug/
> 1546641
> 
> Would anyone kindly confirm the logout behaviour during bitbake
> process on Ubuntu-14.04.5 & Ubuntu-16.04.1 ? Since Yocto 2.2 mentions
> both Ubuntu-14.04 & 16.04 as supported distributions this blocker
> issue must be resolved at earliest. 
> 
> Appreciate your Inputs !

What would really help is knowing what recipe is being built when this
happens. There are a few ways you might be able to figure this out:

a) Watching the screen to see which recipes were building when it
happens
b) Perhaps using some kind of screen recorder to automate this, you'd
be able to review the recording to see when it breaks
c) Looking at the bitbake/build logs to see what it was doing when the
build broke (assuming bitbake stops when the session dies).

"bitbake | cat" can provide better output for this since it will
produce a scrolling log rather than the more interactive UI.

Once you find the problematic recipe, a "bitbake XXX -c clean; bitbake
XXX" should hopefully reproduce this more easily.

Cheers,

Richard

Re: Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

[srikanth krishnakar]

[-- Attachment #1: Type: text/plain, Size: 3961 bytes --]

Hello Richard,

Sure, watching is best thing we can do, but it happens somewhere after 2900
tasks i.e just after linux-yocto.
Screen recorders usually ask for file name and destination folder for
saving the file, will try to see if it works as you said.
"bitbake | cat" is the best method that we can get info from.

The issue is noticeable only in ubuntu (unity) desktop's and not in Kubuntu
(KDE), Ubuntu-gnome (Gnome-3) or Xubuntu (XFCE) desktops.

Will fetch more details as per your suggestion.

Thank you !

On Mon, Dec 26, 2016 at 4:06 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Mon, 2016-12-26 at 15:11 +0530, srikanth krishnakar wrote:
> > Environment: Ubuntu-14.04.5/16.04.1 (64-bit)
> > Yocto build: qemuarm
> > Target image: core-image-sato
> > Error nature: The lightdm restarts on its own and logs out by killing
> > processes running and brings up a login UI.
> >
> >
> > We have been observing yocto-2.2 build failures on Ubuntu 14.04.5 and
> > Ubuntu-16.04.1 hosts due to restart of "lightdm" (Light Desktop
> > Manager)  that is triggered by crash of "unity-settings-daemon", we
> > couldn't figure out any workaround so far to overcome the issue. The
> > build goes fine in the beginning but eventually the unity desktop
> > logs out and kills all the processes running in the user session
> > (including bitbake) and lands us into "Login screen" and when we
> > login its a new session where as the bitbake running in previous
> > session is killed and we need to continue the build. This is
> > happening consistently when the user logs into desktop via lightdm
> > and triggers a build.
> >
> > Another interesting thing to notice is the build goes fine if we
> > connect to host via. SSH session and invoke a bitbake build. We are
> > suspecting on unity-settings-daemon that is crashing consistently and
> > is evident from the ".xsession-errors" log file as shown below:
> >
> > builder@ubuntu:~$ cat .xsession-errors.old
> > Script for ibus started at run_im.
> > Script for auto started at run_im.
> > Script for default started at run_im.
> > init: Disconnected from notified D-Bus bus
> > init:"unity-settings-daemon main process (2076) terminated with
> > status 1"
> > init: indicator-bluetooth main process (2212) killed by TERM signal
> > init: indicator-power main process (2214) killed by TERM signal
> > init: indicator-datetime main process (2215) killed by TERM signal
> > init: indicator-printers main process (2222) killed by TERM signal
> > init: indicator-session main process (2245) killed by TERM signal
> > init: indicator-application main process (2270) killed by TERM signal
> >
> > Corresponding unity-settings-daemon bug reported in Launchpad:
> >
> > https://bugs.launchpad.net/ubuntu/+source/unity-settings-daemon/+bug/
> > 1546641
> >
> > Would anyone kindly confirm the logout behaviour during bitbake
> > process on Ubuntu-14.04.5 & Ubuntu-16.04.1 ? Since Yocto 2.2 mentions
> > both Ubuntu-14.04 & 16.04 as supported distributions this blocker
> > issue must be resolved at earliest.
> >
> > Appreciate your Inputs !
>
> What would really help is knowing what recipe is being built when this
> happens. There are a few ways you might be able to figure this out:
>
> a) Watching the screen to see which recipes were building when it
> happens
> b) Perhaps using some kind of screen recorder to automate this, you'd
> be able to review the recording to see when it breaks
> c) Looking at the bitbake/build logs to see what it was doing when the
> build broke (assuming bitbake stops when the session dies).
>
> "bitbake | cat" can provide better output for this since it will
> produce a scrolling log rather than the more interactive UI.
>
> Once you find the problematic recipe, a "bitbake XXX -c clean; bitbake
> XXX" should hopefully reproduce this more easily.
>
> Cheers,
>
> Richard
>

[-- Attachment #2: Type: text/html, Size: 5401 bytes --]

Re: Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

[srikanth krishnakar]

[-- Attachment #1: Type: text/plain, Size: 9413 bytes --]

Hello Richard,

bitbake | cat shows that when this logout happened during build, log file
captured below packages in process:

--------------------------------------------------------------------------------------------
NOTE: Running task 1886 of 4212
(/home/builder/poky/meta/recipes-graphics/xorg-proto/kbproto_1.0.7.bb:
do_packagedata)
NOTE: recipe kbproto-1_1.0.7-r0: task do_packagedata: Started
NOTE: recipe kbproto-1_1.0.7-r0: task do_packagedata: Succeeded
NOTE: Running task 1887 of 4212
(/home/builder/poky/meta/recipes-support/libffi/libffi_3.2.1.bb:do_package)
NOTE: recipe ncurses-6.0+20160625-r0: task do_package: Started
NOTE: recipe libffi-3.2.1-r0: task do_package: Started
NOTE: recipe libffi-3.2.1-r0: task do_package: Succeeded
NOTE: Running task 1888 of 4212
(/home/builder/poky/meta/recipes-support/libffi/libffi_3.2.1.bb:
do_packagedata)
NOTE: recipe libffi-3.2.1-r0: task do_packagedata: Started
NOTE: recipe libffi-3.2.1-r0: task do_packagedata: Succeeded
NOTE: Running task 1889 of 4212
(/home/builder/poky/meta/recipes-extended/bzip2/bzip2_1.0.6.bb:do_package)
NOTE: recipe bzip2-1.0.6-r5: task do_package: Started
-----------------------------------------------------------------------------------------------

Also there was a crash file generated in "/var/crash" folder:
/var/crash/_usr_lib_accountsservice_accounts-daemon.0.crash

That has following information in it:
-------------------------------------------------------------------------------------------------
ProblemType: Crash
Architecture: amd64
Date: Mon Dec 26 18:23:22 2016
DistroRelease: Ubuntu 14.04
ExecutablePath: /usr/lib/accountsservice/accounts-daemon
ExecutableTimestamp: 1478200365
ProcCmdline: /usr/lib/accountsservice/accounts-daemon
ProcCwd: /
ProcEnviron:
ProcMaps:
 00400000-00426000 r-xp 00000000 08:01 2626520
/usr/lib/accountsservice/accounts-daemon
 00625000-00628000 r--p 00025000 08:01 2626520
/usr/lib/accountsservice/accounts-daemon
 00628000-00629000 rw-p 00028000 08:01 2626520
/usr/lib/accountsservice/accounts-daemon
 024aa000-0252a000 rw-p 00000000 00:00 0
[heap]
 7fe460000000-7fe460022000 rw-p 00000000 00:00 0
 7fe460022000-7fe464000000 ---p 00000000 00:00 0
 7fe464000000-7fe464021000 rw-p 00000000 00:00 0
 7fe464021000-7fe468000000 ---p 00000000 00:00 0
 7fe469f56000-7fe469f60000 r-xp 00000000 08:01 1839849
/lib/x86_64-linux-gnu/libnss_files-2.19.so
 7fe469f60000-7fe46a15f000 ---p 0000a000 08:01 1839849
/lib/x86_64-linux-gnu/libnss_files-2.19.so
 7fe46a15f000-7fe46a160000 r--p 00009000 08:01 1839849
/lib/x86_64-linux-gnu/libnss_files-2.19.so
...
...
 7fe474b3e000-7fe474b40000 rw-p 00000000 00:00 0
 7fe474b40000-7fe474b41000 r--p 00022000 08:01 1839750
/lib/x86_64-linux-gnu/ld-2.19.so
 7fe474b41000-7fe474b42000 rw-p 00023000 08:01 1839750
/lib/x86_64-linux-gnu/ld-2.19.so
 7fe474b42000-7fe474b43000 rw-p 00000000 00:00 0
 7ffcaca64000-7ffcaca85000 rw-p 00000000 00:00 0
[stack]
 7ffcacaf5000-7ffcacaf7000 r--p 00000000 00:00 0
[vvar]
 7ffcacaf7000-7ffcacaf9000 r-xp 00000000 00:00 0
[vdso]
 ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
ProcStatus:
 Name:  accounts-daemon
 State: D (disk sleep)
 Tgid:  1248
 Ngid:  0
 Pid:   1248
 PPid:  1
 TracerPid:     0
 Uid:   0       0       0       0
 Gid:   0       0       0       0
 FDSize:        64
 Groups:        0
 NStgid:        1248
 NSpid: 1248
 NSpgid:        795
 NSsid: 795
 VmPeak:          302404 kB
 VmSize:          302368 kB
 VmPin:        0 kB
 VmHWM:    10128 kB
 VmRSS:     4464 kB
 VmData:          221944 kB
 VmStk:      136 kB
 VmExe:      152 kB
 VmLib:    10140 kB
 VmPTE:      200 kB
 VmPMD:       12 kB
 VmSwap:            2976 kB
 HugetlbPages:         0 kB
 Threads:       3
 SigQ:  0/31670
 SigPnd:        0000000000000000
 ShdPnd:        0000000000000000
 SigBlk:        0000000000000000
 SigIgn:        0000000000001000
 SigCgt:        0000000180004002
 CapInh:        0000000000000000
 CapPrm:        0000003fffffffff
 CapEff:        0000003fffffffff
 CapBnd:        0000003fffffffff
 CapAmb:        0000000000000000
 Seccomp:       0
 Cpus_allowed:  ff
 Cpus_allowed_list:     0-7
 Mems_allowed:  00000000,00000001
 Mems_allowed_list:     0
 voluntary_ctxt_switches:       152
 nonvoluntary_ctxt_switches:    2
Signal: 7
Uname: Linux 4.4.0-57-generic x86_64
UserGroups:
CoreDump: base64
-------------------------------------------------------------------------------------------------

My machine is:

*Quad-Core:* (0-8)
---------------------------------------
$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 42
model name    : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
-----
$ cat /proc/meminfo
MemTotal:        8132472 kB
----------------------------------------

When I checked free memory just before the logout was about "125MB" out of
8GB RAM.

NOTE: The Ubuntu-14.04.4 that had older graphics stack seem to work
normally without any issue. The "ubuntu-14.04.5" & "16.04" came with latest
graphics stack & kernel with removal of non-opensource fglrx drivers.

Thanks for your patience!

--
Regards,
Srikant


On Mon, Dec 26, 2016 at 4:28 PM, srikanth krishnakar <skrishnakar@gmail.com>
wrote:

> Hello Richard,
>
> Sure, watching is best thing we can do, but it happens somewhere after
> 2900 tasks i.e just after linux-yocto.
> Screen recorders usually ask for file name and destination folder for
> saving the file, will try to see if it works as you said.
> "bitbake | cat" is the best method that we can get info from.
>
> The issue is noticeable only in ubuntu (unity) desktop's and not in
> Kubuntu (KDE), Ubuntu-gnome (Gnome-3) or Xubuntu (XFCE) desktops.
>
> Will fetch more details as per your suggestion.
>
> Thank you !
>
> On Mon, Dec 26, 2016 at 4:06 PM, Richard Purdie <richard.purdie@
> linuxfoundation.org> wrote:
>
>> On Mon, 2016-12-26 at 15:11 +0530, srikanth krishnakar wrote:
>> > Environment: Ubuntu-14.04.5/16.04.1 (64-bit)
>> > Yocto build: qemuarm
>> > Target image: core-image-sato
>> > Error nature: The lightdm restarts on its own and logs out by killing
>> > processes running and brings up a login UI.
>> >
>> >
>> > We have been observing yocto-2.2 build failures on Ubuntu 14.04.5 and
>> > Ubuntu-16.04.1 hosts due to restart of "lightdm" (Light Desktop
>> > Manager)  that is triggered by crash of "unity-settings-daemon", we
>> > couldn't figure out any workaround so far to overcome the issue. The
>> > build goes fine in the beginning but eventually the unity desktop
>> > logs out and kills all the processes running in the user session
>> > (including bitbake) and lands us into "Login screen" and when we
>> > login its a new session where as the bitbake running in previous
>> > session is killed and we need to continue the build. This is
>> > happening consistently when the user logs into desktop via lightdm
>> > and triggers a build.
>> >
>> > Another interesting thing to notice is the build goes fine if we
>> > connect to host via. SSH session and invoke a bitbake build. We are
>> > suspecting on unity-settings-daemon that is crashing consistently and
>> > is evident from the ".xsession-errors" log file as shown below:
>> >
>> > builder@ubuntu:~$ cat .xsession-errors.old
>> > Script for ibus started at run_im.
>> > Script for auto started at run_im.
>> > Script for default started at run_im.
>> > init: Disconnected from notified D-Bus bus
>> > init:"unity-settings-daemon main process (2076) terminated with
>> > status 1"
>> > init: indicator-bluetooth main process (2212) killed by TERM signal
>> > init: indicator-power main process (2214) killed by TERM signal
>> > init: indicator-datetime main process (2215) killed by TERM signal
>> > init: indicator-printers main process (2222) killed by TERM signal
>> > init: indicator-session main process (2245) killed by TERM signal
>> > init: indicator-application main process (2270) killed by TERM signal
>> >
>> > Corresponding unity-settings-daemon bug reported in Launchpad:
>> >
>> > https://bugs.launchpad.net/ubuntu/+source/unity-settings-daemon/+bug/
>> > 1546641
>> >
>> > Would anyone kindly confirm the logout behaviour during bitbake
>> > process on Ubuntu-14.04.5 & Ubuntu-16.04.1 ? Since Yocto 2.2 mentions
>> > both Ubuntu-14.04 & 16.04 as supported distributions this blocker
>> > issue must be resolved at earliest.
>> >
>> > Appreciate your Inputs !
>>
>> What would really help is knowing what recipe is being built when this
>> happens. There are a few ways you might be able to figure this out:
>>
>> a) Watching the screen to see which recipes were building when it
>> happens
>> b) Perhaps using some kind of screen recorder to automate this, you'd
>> be able to review the recording to see when it breaks
>> c) Looking at the bitbake/build logs to see what it was doing when the
>> build broke (assuming bitbake stops when the session dies).
>>
>> "bitbake | cat" can provide better output for this since it will
>> produce a scrolling log rather than the more interactive UI.
>>
>> Once you find the problematic recipe, a "bitbake XXX -c clean; bitbake
>> XXX" should hopefully reproduce this more easily.
>>
>> Cheers,
>>
>> Richard
>>
>
>

[-- Attachment #2: Type: text/html, Size: 13360 bytes --]

Re: Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

[srikanth krishnakar]

[-- Attachment #1: Type: text/plain, Size: 12807 bytes --]

Hello Richard,

I was under impression that installing Ubuntu-14.04.4 (+lastest updates)
would give me stable build environment, unfortunately that is not the case
! I could see the logout issue on this installation too. So I believe that
issue should be reproducible on any 14.04.x installation including 16.04.x
with all latest updates installed. The second attempt of build on
"Ubuntu-14.04.4" shows the status of bitbake as halted at libvorbis when
logout happened:

---------------------------------------------------------
NOTE: Running task 2460 of 4212
(/home/poky/meta/recipes-core/ncurses/ncurses_6.0+20160625.bb:
do_packagedata)
NOTE: recipe ncurses-6.0+20160625-r0: task do_packagedata: Started
NOTE: recipe ncurses-6.0+20160625-r0: task do_packagedata: Succeeded
NOTE: Running task 2461 of 4212
(/home/poky/meta/recipes-core/readline/readline_6.3.bb:do_package)
NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_package: Succeeded
NOTE: Running task 2462 of 4212
(/home/poky/meta/recipes-graphics/jpeg/libjpeg-turbo_1.5.0.bb:
do_packagedata)
NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_packagedata: Started
NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_packagedata: Succeeded
NOTE: Running task 2463 of 4212
(/home/poky/meta/recipes-multimedia/libvorbis/libvorbis_1.3.5.bb:do_package)
NOTE: recipe readline-6.3-r0: task do_package: Started
NOTE: recipe libvorbis-1.3.5-r0: task do_package: Started
---------------------------------------------------------

The "$HOME/.xsession-errors.old" file shows the message:
---------------------------------------------------------
Script for ibus started at run_im.
Script for auto started at run_im.
Script for default started at run_im.
init: at-spi2-registryd main process ended, respawning
init: at-spi2-registryd main process ended, respawning
init: at-spi2-registryd main process ended, respawning
init: at-spi2-registryd respawning too fast, stopped
init: indicator-messages main process (17542) killed by SEGV signal
init: indicator-messages main process ended, respawning
init: update-notifier-crash
(/var/crash/_usr_lib_x86_64-linux-gnu_indicator-messages_indicator-messages-service.1000.crash)
main process (18792) terminated with status 1
init: update-notifier-crash
(/var/crash/_usr_share_apport_apport-gtk.1000.crash) main process (21151)
terminated with status 1
init: unity-panel-service-lockscreen main process (29904) killed by KILL
signal
---------------------------------------------------------

It appears that lightdm is unable the handle the SIGNAL and restarting when
it receives them. Some of the most anticipated bugfixes patches for lightdm
yet to arrive in ubuntu-14.04.6 which are not released yet can be found
here:

https://launchpad.net/lightdm/+milestone/1.10.7

Current version of lightdm in ubuntu-14.04.4/5 -> 1.10.6

Thanks,
Srikant

On Mon, Dec 26, 2016 at 6:59 PM, srikanth krishnakar <skrishnakar@gmail.com>
wrote:

> Hello Richard,
>
> bitbake | cat shows that when this logout happened during build, log file
> captured below packages in process:
>
> ------------------------------------------------------------
> --------------------------------
> NOTE: Running task 1886 of 4212 (/home/builder/poky/meta/
> recipes-graphics/xorg-proto/kbproto_1.0.7.bb:do_packagedata)
> NOTE: recipe kbproto-1_1.0.7-r0: task do_packagedata: Started
> NOTE: recipe kbproto-1_1.0.7-r0: task do_packagedata: Succeeded
> NOTE: Running task 1887 of 4212 (/home/builder/poky/meta/
> recipes-support/libffi/libffi_3.2.1.bb:do_package)
> NOTE: recipe ncurses-6.0+20160625-r0: task do_package: Started
> NOTE: recipe libffi-3.2.1-r0: task do_package: Started
> NOTE: recipe libffi-3.2.1-r0: task do_package: Succeeded
> NOTE: Running task 1888 of 4212 (/home/builder/poky/meta/
> recipes-support/libffi/libffi_3.2.1.bb:do_packagedata)
> NOTE: recipe libffi-3.2.1-r0: task do_packagedata: Started
> NOTE: recipe libffi-3.2.1-r0: task do_packagedata: Succeeded
> NOTE: Running task 1889 of 4212 (/home/builder/poky/meta/
> recipes-extended/bzip2/bzip2_1.0.6.bb:do_package)
> NOTE: recipe bzip2-1.0.6-r5: task do_package: Started
> ------------------------------------------------------------
> -----------------------------------
>
> Also there was a crash file generated in "/var/crash" folder:
> /var/crash/_usr_lib_accountsservice_accounts-daemon.0.crash
>
> That has following information in it:
> ------------------------------------------------------------
> -------------------------------------
> ProblemType: Crash
> Architecture: amd64
> Date: Mon Dec 26 18:23:22 2016
> DistroRelease: Ubuntu 14.04
> ExecutablePath: /usr/lib/accountsservice/accounts-daemon
> ExecutableTimestamp: 1478200365
> ProcCmdline: /usr/lib/accountsservice/accounts-daemon
> ProcCwd: /
> ProcEnviron:
> ProcMaps:
>  00400000-00426000 r-xp 00000000 08:01 2626520
> /usr/lib/accountsservice/accounts-daemon
>  00625000-00628000 r--p 00025000 08:01 2626520
> /usr/lib/accountsservice/accounts-daemon
>  00628000-00629000 rw-p 00028000 08:01 2626520
> /usr/lib/accountsservice/accounts-daemon
>  024aa000-0252a000 rw-p 00000000 00:00 0
> [heap]
>  7fe460000000-7fe460022000 rw-p 00000000 00:00 0
>  7fe460022000-7fe464000000 ---p 00000000 00:00 0
>  7fe464000000-7fe464021000 rw-p 00000000 00:00 0
>  7fe464021000-7fe468000000 ---p 00000000 00:00 0
>  7fe469f56000-7fe469f60000 r-xp 00000000 08:01 1839849
> /lib/x86_64-linux-gnu/libnss_files-2.19.so
>  7fe469f60000-7fe46a15f000 ---p 0000a000 08:01 1839849
> /lib/x86_64-linux-gnu/libnss_files-2.19.so
>  7fe46a15f000-7fe46a160000 r--p 00009000 08:01 1839849
> /lib/x86_64-linux-gnu/libnss_files-2.19.so
> ...
> ...
>  7fe474b3e000-7fe474b40000 rw-p 00000000 00:00 0
>  7fe474b40000-7fe474b41000 r--p 00022000 08:01 1839750
> /lib/x86_64-linux-gnu/ld-2.19.so
>  7fe474b41000-7fe474b42000 rw-p 00023000 08:01 1839750
> /lib/x86_64-linux-gnu/ld-2.19.so
>  7fe474b42000-7fe474b43000 rw-p 00000000 00:00 0
>  7ffcaca64000-7ffcaca85000 rw-p 00000000 00:00 0
> [stack]
>  7ffcacaf5000-7ffcacaf7000 r--p 00000000 00:00 0
> [vvar]
>  7ffcacaf7000-7ffcacaf9000 r-xp 00000000 00:00 0
> [vdso]
>  ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
> [vsyscall]
> ProcStatus:
>  Name:  accounts-daemon
>  State: D (disk sleep)
>  Tgid:  1248
>  Ngid:  0
>  Pid:   1248
>  PPid:  1
>  TracerPid:     0
>  Uid:   0       0       0       0
>  Gid:   0       0       0       0
>  FDSize:        64
>  Groups:        0
>  NStgid:        1248
>  NSpid: 1248
>  NSpgid:        795
>  NSsid: 795
>  VmPeak:          302404 kB
>  VmSize:          302368 kB
>  VmPin:        0 kB
>  VmHWM:    10128 kB
>  VmRSS:     4464 kB
>  VmData:          221944 kB
>  VmStk:      136 kB
>  VmExe:      152 kB
>  VmLib:    10140 kB
>  VmPTE:      200 kB
>  VmPMD:       12 kB
>  VmSwap:            2976 kB
>  HugetlbPages:         0 kB
>  Threads:       3
>  SigQ:  0/31670
>  SigPnd:        0000000000000000
>  ShdPnd:        0000000000000000
>  SigBlk:        0000000000000000
>  SigIgn:        0000000000001000
>  SigCgt:        0000000180004002
>  CapInh:        0000000000000000
>  CapPrm:        0000003fffffffff
>  CapEff:        0000003fffffffff
>  CapBnd:        0000003fffffffff
>  CapAmb:        0000000000000000
>  Seccomp:       0
>  Cpus_allowed:  ff
>  Cpus_allowed_list:     0-7
>  Mems_allowed:  00000000,00000001
>  Mems_allowed_list:     0
>  voluntary_ctxt_switches:       152
>  nonvoluntary_ctxt_switches:    2
> Signal: 7
> Uname: Linux 4.4.0-57-generic x86_64
> UserGroups:
> CoreDump: base64
> ------------------------------------------------------------
> -------------------------------------
>
> My machine is:
>
> *Quad-Core:* (0-8)
> ---------------------------------------
> $ cat /proc/cpuinfo
> processor    : 0
> vendor_id    : GenuineIntel
> cpu family    : 6
> model        : 42
> model name    : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
> -----
> $ cat /proc/meminfo
> MemTotal:        8132472 kB
> ----------------------------------------
>
> When I checked free memory just before the logout was about "125MB" out of
> 8GB RAM.
>
> NOTE: The Ubuntu-14.04.4 that had older graphics stack seem to work
> normally without any issue. The "ubuntu-14.04.5" & "16.04" came with latest
> graphics stack & kernel with removal of non-opensource fglrx drivers.
>
> Thanks for your patience!
>
> --
> Regards,
> Srikant
>
>
> On Mon, Dec 26, 2016 at 4:28 PM, srikanth krishnakar <
> skrishnakar@gmail.com> wrote:
>
>> Hello Richard,
>>
>> Sure, watching is best thing we can do, but it happens somewhere after
>> 2900 tasks i.e just after linux-yocto.
>> Screen recorders usually ask for file name and destination folder for
>> saving the file, will try to see if it works as you said.
>> "bitbake | cat" is the best method that we can get info from.
>>
>> The issue is noticeable only in ubuntu (unity) desktop's and not in
>> Kubuntu (KDE), Ubuntu-gnome (Gnome-3) or Xubuntu (XFCE) desktops.
>>
>> Will fetch more details as per your suggestion.
>>
>> Thank you !
>>
>> On Mon, Dec 26, 2016 at 4:06 PM, Richard Purdie <
>> richard.purdie@linuxfoundation.org> wrote:
>>
>>> On Mon, 2016-12-26 at 15:11 +0530, srikanth krishnakar wrote:
>>> > Environment: Ubuntu-14.04.5/16.04.1 (64-bit)
>>> > Yocto build: qemuarm
>>> > Target image: core-image-sato
>>> > Error nature: The lightdm restarts on its own and logs out by killing
>>> > processes running and brings up a login UI.
>>> >
>>> >
>>> > We have been observing yocto-2.2 build failures on Ubuntu 14.04.5 and
>>> > Ubuntu-16.04.1 hosts due to restart of "lightdm" (Light Desktop
>>> > Manager)  that is triggered by crash of "unity-settings-daemon", we
>>> > couldn't figure out any workaround so far to overcome the issue. The
>>> > build goes fine in the beginning but eventually the unity desktop
>>> > logs out and kills all the processes running in the user session
>>> > (including bitbake) and lands us into "Login screen" and when we
>>> > login its a new session where as the bitbake running in previous
>>> > session is killed and we need to continue the build. This is
>>> > happening consistently when the user logs into desktop via lightdm
>>> > and triggers a build.
>>> >
>>> > Another interesting thing to notice is the build goes fine if we
>>> > connect to host via. SSH session and invoke a bitbake build. We are
>>> > suspecting on unity-settings-daemon that is crashing consistently and
>>> > is evident from the ".xsession-errors" log file as shown below:
>>> >
>>> > builder@ubuntu:~$ cat .xsession-errors.old
>>> > Script for ibus started at run_im.
>>> > Script for auto started at run_im.
>>> > Script for default started at run_im.
>>> > init: Disconnected from notified D-Bus bus
>>> > init:"unity-settings-daemon main process (2076) terminated with
>>> > status 1"
>>> > init: indicator-bluetooth main process (2212) killed by TERM signal
>>> > init: indicator-power main process (2214) killed by TERM signal
>>> > init: indicator-datetime main process (2215) killed by TERM signal
>>> > init: indicator-printers main process (2222) killed by TERM signal
>>> > init: indicator-session main process (2245) killed by TERM signal
>>> > init: indicator-application main process (2270) killed by TERM signal
>>> >
>>> > Corresponding unity-settings-daemon bug reported in Launchpad:
>>> >
>>> > https://bugs.launchpad.net/ubuntu/+source/unity-settings-daemon/+bug/
>>> > 1546641
>>> >
>>> > Would anyone kindly confirm the logout behaviour during bitbake
>>> > process on Ubuntu-14.04.5 & Ubuntu-16.04.1 ? Since Yocto 2.2 mentions
>>> > both Ubuntu-14.04 & 16.04 as supported distributions this blocker
>>> > issue must be resolved at earliest.
>>> >
>>> > Appreciate your Inputs !
>>>
>>> What would really help is knowing what recipe is being built when this
>>> happens. There are a few ways you might be able to figure this out:
>>>
>>> a) Watching the screen to see which recipes were building when it
>>> happens
>>> b) Perhaps using some kind of screen recorder to automate this, you'd
>>> be able to review the recording to see when it breaks
>>> c) Looking at the bitbake/build logs to see what it was doing when the
>>> build broke (assuming bitbake stops when the session dies).
>>>
>>> "bitbake | cat" can provide better output for this since it will
>>> produce a scrolling log rather than the more interactive UI.
>>>
>>> Once you find the problematic recipe, a "bitbake XXX -c clean; bitbake
>>> XXX" should hopefully reproduce this more easily.
>>>
>>> Cheers,
>>>
>>> Richard
>>>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 17689 bytes --]

Re: Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

[Richard Purdie]

On Tue, 2016-12-27 at 15:18 +0530, srikanth krishnakar wrote:
> Hello Richard,
> 
> I was under impression that installing Ubuntu-14.04.4 (+lastest
> updates) would give me stable build environment, unfortunately that
> is not the case ! I could see the logout issue on this installation
> too. So I believe that issue should be reproducible on any 14.04.x
> installation including 16.04.x with all latest updates installed. The
> second attempt of build on "Ubuntu-14.04.4" shows the status of
> bitbake as halted at libvorbis when logout happened:
> 
> ---------------------------------------------------------
> NOTE: Running task 2460 of 4212 (/home/poky/meta/recipes-
> core/ncurses/ncurses_6.0+20160625.bb:do_packagedata)
> NOTE: recipe ncurses-6.0+20160625-r0: task do_packagedata: Started
> NOTE: recipe ncurses-6.0+20160625-r0: task do_packagedata: Succeeded
> NOTE: Running task 2461 of 4212 (/home/poky/meta/recipes-
> core/readline/readline_6.3.bb:do_package)
> NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_package: Succeeded
> NOTE: Running task 2462 of 4212 (/home/poky/meta/recipes-
> graphics/jpeg/libjpeg-turbo_1.5.0.bb:do_packagedata)
> NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_packagedata: Started
> NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_packagedata: Succeeded
> NOTE: Running task 2463 of 4212 (/home/poky/meta/recipes-
> multimedia/libvorbis/libvorbis_1.3.5.bb:do_package)
> NOTE: recipe readline-6.3-r0: task do_package: Started
> NOTE: recipe libvorbis-1.3.5-r0: task do_package: Started


None of these look particularly suspicios for causing your X session to
die.

Some further questions:

a) Is anything showing in the dmesg logs? Is the OOM killer the thing
which is killing processes?
b) How are you measuring free memory? You mentioned it was low, it
would depend on the disk cache size and swap status too.
c) Have you tried a lower BB_NUMBER_THREADS and lower PARALLEL_MAKE
values to rule out resource problems and perhaps identify more
specifically which task is causing your problem?

As I mentioned previously, this problem should reduce to a specific
recipe+task causing the problem, unless its a general resource issue in
which case the threads/parallel make will stop the problem occurring.

We have autobuilders running various versions of Ubuntu and we don't
notice problems with processes dying, admittedly we don't run X
desktops on them in general though.

Cheers,

Richard

Re: Yocto-2.2 build failures on Ubuntu 14.04.5 & 16.04.x hosts due to crash of unity-settings-daemon followed by restart of lightdm

[Randy MacLeod]

On 12/27/2016 11:49 AM, Richard Purdie wrote:
> On Tue, 2016-12-27 at 15:18 +0530, srikanth krishnakar wrote:
>> Hello Richard,
>>
>> I was under impression that installing Ubuntu-14.04.4 (+lastest
>> updates) would give me stable build environment, unfortunately that
>> is not the case ! I could see the logout issue on this installation
>> too. So I believe that issue should be reproducible on any 14.04.x
>> installation including 16.04.x with all latest updates installed. The
>> second attempt of build on "Ubuntu-14.04.4" shows the status of
>> bitbake as halted at libvorbis when logout happened:
>>
>> ---------------------------------------------------------
>> NOTE: Running task 2460 of 4212 (/home/poky/meta/recipes-
>> core/ncurses/ncurses_6.0+20160625.bb:do_packagedata)
>> NOTE: recipe ncurses-6.0+20160625-r0: task do_packagedata: Started
>> NOTE: recipe ncurses-6.0+20160625-r0: task do_packagedata: Succeeded
>> NOTE: Running task 2461 of 4212 (/home/poky/meta/recipes-
>> core/readline/readline_6.3.bb:do_package)
>> NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_package: Succeeded
>> NOTE: Running task 2462 of 4212 (/home/poky/meta/recipes-
>> graphics/jpeg/libjpeg-turbo_1.5.0.bb:do_packagedata)
>> NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_packagedata: Started
>> NOTE: recipe libjpeg-turbo-1_1.5.0-r0: task do_packagedata: Succeeded
>> NOTE: Running task 2463 of 4212 (/home/poky/meta/recipes-
>> multimedia/libvorbis/libvorbis_1.3.5.bb:do_package)
>> NOTE: recipe readline-6.3-r0: task do_package: Started
>> NOTE: recipe libvorbis-1.3.5-r0: task do_package: Started
> 
> 
> None of these look particularly suspicios for causing your X session to
> die.
> 
> Some further questions:
> 
> a) Is anything showing in the dmesg logs? Is the OOM killer the thing
> which is killing processes?
> b) How are you measuring free memory? You mentioned it was low, it
> would depend on the disk cache size and swap status too.
> c) Have you tried a lower BB_NUMBER_THREADS and lower PARALLEL_MAKE
> values to rule out resource problems and perhaps identify more
> specifically which task is causing your problem?
> 
> As I mentioned previously, this problem should reduce to a specific
> recipe+task causing the problem, unless its a general resource issue in
> which case the threads/parallel make will stop the problem occurring.
> 
> We have autobuilders running various versions of Ubuntu and we don't
> notice problems with processes dying, admittedly we don't run X
> desktops on them in general though.

Google found this old thread...

A co-worker is encountering this problem now as well on oe-core-2.4.


Srikanth,

Did you ever manage to identify the cause or a work-around?

Of course if you ssh to localhost and then do the build
that will work-around the issue.

More when we've had a chance to narrow down
the issue now that we have a system where we are
able to reproduce the problem.

../Randy

> 
> Cheers,
> 
> Richard
> 
> 


-- 
# Randy MacLeod
# Wind River Linux