[Buildroot] Segmentation faults after gcc update

[Buildroot] Segmentation faults after gcc update

[Sven Neumann]

Hi,

we are using buildroot successfully for quite a while now and after I've
recently managed to bring our buildroot environment up-to-date with
current git, I've changed the gcc version from 4.3.6 to 4.6.1. That was
the only change to the config. The resulting image boots and some
services (syslogd, klogd, udevd, sshd) are up and running. I can log
into the system and the busybox tools I tried to far seem to work fine.
However lots of applications that used to run fine now immidiately die
with a segmentation fault:

# dbus-daemon 
Segmentation fault

# uuidgen
Segmentation fault

# sqlite3
Segmentation fault

# iperf
Segmentation fault

What could possibly be wrong here? Is there anything else I need to
change when changing the version of gcc?


Sven

[Buildroot] Segmentation faults after gcc update

[Thomas Petazzoni]

Le Tue, 27 Sep 2011 10:04:12 +0200,
Sven Neumann <s.neumann@raumfeld.com> a ?crit :

> we are using buildroot successfully for quite a while now and after
> I've recently managed to bring our buildroot environment up-to-date
> with current git, I've changed the gcc version from 4.3.6 to 4.6.1.
> That was the only change to the config. The resulting image boots and
> some services (syslogd, klogd, udevd, sshd) are up and running. I can
> log into the system and the busybox tools I tried to far seem to work
> fine. However lots of applications that used to run fine now
> immidiately die with a segmentation fault:
> 
> # dbus-daemon 
> Segmentation fault
> 
> # uuidgen
> Segmentation fault
> 
> # sqlite3
> Segmentation fault
> 
> # iperf
> Segmentation fault
> 
> What could possibly be wrong here? Is there anything else I need to
> change when changing the version of gcc?

Hum, odd. Some questions:

 * Do you have the same version of uClibc in both cases ?

 * Is your Busybox (which works) statically linked or dynamically
   linked ? If statically linked, is the problem that all dynamically
   linked binaries are non working (which would indicate a bug in the
   dynamic linker) ? If not, is there anything common between all the
   non-working binaries ?

 * What does a gdb session says for the execution of one of those
   binaries ?

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Segmentation faults after gcc update

[Thomas Petazzoni]

Le Tue, 27 Sep 2011 10:04:12 +0200,
Sven Neumann <s.neumann@raumfeld.com> a ?crit :

> What could possibly be wrong here? Is there anything else I need to
> change when changing the version of gcc?

Other questions:

 * Is your CPU-specific compiler flags properly set ? Maybe they were
   slightly wrong but working with gcc 4.3, and they no longer generate
   appropriate code for your CPU with gcc 4.6. What is your CPU ? What
   CPU-specific compiler flags are you using ?

 * Did you try to do 'ulimit -c unlimited' before running one of those
   binaries, and then inspect the resulting core file ?

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Segmentation faults after gcc update

[Sven Neumann]

On Tue, 2011-09-27 at 23:35 +0200, Thomas Petazzoni wrote:
> Le Tue, 27 Sep 2011 10:04:12 +0200,
> Sven Neumann <s.neumann@raumfeld.com> a ?crit :
> 
> > we are using buildroot successfully for quite a while now and after
> > I've recently managed to bring our buildroot environment up-to-date
> > with current git, I've changed the gcc version from 4.3.6 to 4.6.1.
> > That was the only change to the config. The resulting image boots and
> > some services (syslogd, klogd, udevd, sshd) are up and running. I can
> > log into the system and the busybox tools I tried to far seem to work
> > fine. However lots of applications that used to run fine now
> > immidiately die with a segmentation fault:
> > 
> > # dbus-daemon 
> > Segmentation fault
> > 
> > # uuidgen
> > Segmentation fault
> > 
> > # sqlite3
> > Segmentation fault
> > 
> > # iperf
> > Segmentation fault
> > 
> > What could possibly be wrong here? Is there anything else I need to
> > change when changing the version of gcc?
> 
> Hum, odd. Some questions:
> 
>  * Do you have the same version of uClibc in both cases ?

Yes. Both use 0.9.32 with NPTL as thread library implementation. As I
said already, the gcc version was the only thing that changed.

>  * Is your Busybox (which works) statically linked or dynamically
>    linked ? If statically linked, is the problem that all dynamically
>    linked binaries are non working (which would indicate a bug in the
>    dynamic linker) ? If not, is there anything common between all the
>    non-working binaries ?

busybox is linked dynamically. As far as I can tell all the problematic
binaries make use of threads, but I haven't completely verified this
yet.


Thanks,
Sven

[Buildroot] Segmentation faults after gcc update

[Sven Neumann]

On Tue, 2011-09-27 at 23:36 +0200, Thomas Petazzoni wrote:
> Le Tue, 27 Sep 2011 10:04:12 +0200,
> Sven Neumann <s.neumann@raumfeld.com> a ?crit :
> 
> > What could possibly be wrong here? Is there anything else I need to
> > change when changing the version of gcc?
> 
> Other questions:
> 
>  * Is your CPU-specific compiler flags properly set ? Maybe they were
>    slightly wrong but working with gcc 4.3, and they no longer generate
>    appropriate code for your CPU with gcc 4.6. What is your CPU ? What
>    CPU-specific compiler flags are you using ?

This is a Marvell PXA300:

Processor	: XScale-V3 based processor rev 1 (v5l)
BogoMIPS	: 206.64
Features	: swp half thumb fastmult edsp iwmmxt 
CPU implementer	: 0x69
CPU architecture: 5TE
CPU variant	: 0x0
CPU part	: 0x688
CPU revision	: 1

And the relevant parts of the buildroot config are:

BR2_iwmmxt=y
BR2_ARM_TYPE="ARM_IWMMXT"
BR2_ARM_EABI=y
BR2_ARCH="arm"
BR2_ENDIAN="LITTLE"
BR2_GCC_TARGET_TUNE="iwmmxt"
BR2_GCC_TARGET_ARCH="iwmmxt"
BR2_GCC_TARGET_ABI="aapcs-linux"


Regards,
Sven

[Buildroot] Segmentation faults after gcc update

[Sven Neumann]

Hi,

On Tue, 2011-09-27 at 23:35 +0200, Thomas Petazzoni wrote:

>  * What does a gdb session says for the execution of one of those
>    binaries ?

OK, so I had to compile a gdb for the target first. Here's the result:

# gdb dbus-daemon 
GNU gdb (GDB) 7.1
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-unknown-linux-uclibcgnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/dbus-daemon...(no debugging symbols found)...done.
(gdb) r
Starting program: /usr/bin/dbus-daemon 
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.

Program received signal SIGSEGV, Segmentation fault.
0x40086254 in _dl_get_tls_static_info () from /lib/ld-uClibc.so.0
(gdb) bt
#0  0x40086254 in _dl_get_tls_static_info () from /lib/ld-uClibc.so.0
#1  0x40087064 in _dl_update_slotinfo () from /lib/ld-uClibc.so.0
#2  0x401584a0 in sched_get_priority_max () from /lib/libc.so.0
#3  0x4015f040 in hasmntopt () from /lib/libc.so.0
#4  0x4015f040 in hasmntopt () from /lib/libc.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) 


I don't quite understand why libthread_db is not found as I did enable
BR2_PTHREAD_DEBUG for this build and recompiled everything. Anyway, I
tried gdb on a different (crashing) binary and it crashes in
_dl_get_tls_static_info() as well.


Regards,
Sven

[Buildroot] Segmentation faults after gcc update

[Sven Neumann]

Hi,

On Wed, 2011-09-28 at 16:44 +0200, Sven Neumann wrote:

> On Tue, 2011-09-27 at 23:35 +0200, Thomas Petazzoni wrote:
> 
> >  * What does a gdb session says for the execution of one of those
> >    binaries ?
> 
> OK, so I had to compile a gdb for the target first. Here's the result:
> 
> # gdb dbus-daemon 
> GNU gdb (GDB) 7.1
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "arm-unknown-linux-uclibcgnueabi".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/bin/dbus-daemon...(no debugging symbols found)...done.
> (gdb) r
> Starting program: /usr/bin/dbus-daemon 
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x40086254 in _dl_get_tls_static_info () from /lib/ld-uClibc.so.0
> (gdb) bt
> #0  0x40086254 in _dl_get_tls_static_info () from /lib/ld-uClibc.so.0
> #1  0x40087064 in _dl_update_slotinfo () from /lib/ld-uClibc.so.0
> #2  0x401584a0 in sched_get_priority_max () from /lib/libc.so.0
> #3  0x4015f040 in hasmntopt () from /lib/libc.so.0
> #4  0x4015f040 in hasmntopt () from /lib/libc.so.0
> Backtrace stopped: previous frame identical to this frame (corrupt stack?)
> (gdb) 
> 
> 
> I don't quite understand why libthread_db is not found as I did enable
> BR2_PTHREAD_DEBUG for this build and recompiled everything. Anyway, I
> tried gdb on a different (crashing) binary and it crashes in
> _dl_get_tls_static_info() as well.

I have now added some debug output to _dl_get_tls_static_info() and
related functions and from that I can tell that gdb is wrong here. The
code does definitely not crash in _dl_get_tls_static_info() and not in
_dl_update_slotinfo() either.

Now I am somewhat stuck. Any further ideas on how I could tackle this
problem?


Thanks,
Sven

[Buildroot] Segmentation faults after gcc update

[Thomas Petazzoni]

Le Fri, 30 Sep 2011 11:06:46 +0200,
Sven Neumann <s.neumann@raumfeld.com> a ?crit :

> > I don't quite understand why libthread_db is not found as I did enable
> > BR2_PTHREAD_DEBUG for this build and recompiled everything. Anyway, I
> > tried gdb on a different (crashing) binary and it crashes in
> > _dl_get_tls_static_info() as well.
> 
> I have now added some debug output to _dl_get_tls_static_info() and
> related functions and from that I can tell that gdb is wrong here. The
> code does definitely not crash in _dl_get_tls_static_info() and not in
> _dl_update_slotinfo() either.
> 
> Now I am somewhat stuck. Any further ideas on how I could tackle this
> problem?

I guess the whole stack trace is stupid, because I don't see how
hasmntopt() can end up calling sched_get_priority_max():

char *hasmntopt(const struct mntent *mnt, const char *opt)
{
        return strstr(mnt->mnt_opts, opt);
}

Maybe you should raise the issue on the uClibc mailing list?

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Segmentation faults after gcc update

[Thomas Petazzoni]

Le Fri, 30 Sep 2011 21:59:15 +0200,
Thomas Petazzoni <thomas.petazzoni@free-electrons.com> a ?crit :

> Maybe you should raise the issue on the uClibc mailing list?

A quick data point: I tested a build with gcc 4.6.1, uClibc 0.9.32,
NPTL, and both iperf and dbus-daemon start fine for me:

# iperf 
Usage: iperf [-s|-c host] [options]
Try `iperf --help' for more information.
# dbus-daemon 
No configuration file specified.
dbus-daemon [--version] [--session] [--system] [--config-file=FILE] [--print-address[=DESCRIPTOR]] [--print-pid[=DESCRIPTOR]] [--fork] [--nofork] [--introspect] [--address=ADDRESS] [--systemd-activation]
# uname -a
Linux buildroot 2.6.38.8 #1 Fri Sep 30 20:24:38 UTC 2011 armv5tejl
GNU/Linux

My target optimization options is:

BR2_GCC_TARGET_TUNE="arm926ej-s"
BR2_GCC_TARGET_ARCH="armv5te"
BR2_GCC_TARGET_ABI="aapcs-linux"

Therefore, I guess that the problem might be iwmmxt related. Can you
try with "lower" optimization options ?

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] Segmentation faults after gcc update

[Sven Neumann]

On Fri, 2011-09-30 at 21:59 +0200, Thomas Petazzoni wrote:
> Le Fri, 30 Sep 2011 11:06:46 +0200,
> Sven Neumann <s.neumann@raumfeld.com> a ?crit :
> 
> > > I don't quite understand why libthread_db is not found as I did enable
> > > BR2_PTHREAD_DEBUG for this build and recompiled everything. Anyway, I
> > > tried gdb on a different (crashing) binary and it crashes in
> > > _dl_get_tls_static_info() as well.
> > 
> > I have now added some debug output to _dl_get_tls_static_info() and
> > related functions and from that I can tell that gdb is wrong here. The
> > code does definitely not crash in _dl_get_tls_static_info() and not in
> > _dl_update_slotinfo() either.
> > 
> > Now I am somewhat stuck. Any further ideas on how I could tackle this
> > problem?
> 
> I guess the whole stack trace is stupid, because I don't see how
> hasmntopt() can end up calling sched_get_priority_max():
> 
> char *hasmntopt(const struct mntent *mnt, const char *opt)
> {
>         return strstr(mnt->mnt_opts, opt);
> }
> 
> Maybe you should raise the issue on the uClibc mailing list?

I've opened https://bugs.busybox.net/show_bug.cgi?id=4291 for it and
will also ask on the mailing list in case that the bug report won't get
any attention.


Thanks,
Sven

[Buildroot] Segmentation faults after gcc update

[Sven Neumann]

Hi,

I am picking up this old thread to let you know that we've finally found
a solution to this. As a reminder, the problem I reported was that
processes started to segfault on startup after an update of gcc from
version 4.3 to newer versions.

On Fri, 2011-09-30 at 22:35 +0200, Thomas Petazzoni wrote:

> Therefore, I guess that the problem might be iwmmxt related. Can you
> try with "lower" optimization options ?

It turned out that Thomas guessed right. It seems that the iwmmxt
optimizations are broken in gcc newer than 4.3. Changing the target to
xscale fixed the problem. And the tests I have done so far seem to imply
that the performance of code compiled with gcc 4.6 optimized for xscale
is slightly better than the same code compiled with gcc 4.3 optimized
for iwmmxt.

Not sure if this should somehow be handled in buildroot. But I thought
I'd at least let you know on the mailing-list so that others can avoid
running into the same problem.


Greetings,
Sven

[Buildroot] Segmentation faults after gcc update

[Thomas Petazzoni]

Le Mon, 20 Feb 2012 11:37:00 +0100,
Sven Neumann <s.neumann@raumfeld.com> a ?crit :

> It turned out that Thomas guessed right. It seems that the iwmmxt
> optimizations are broken in gcc newer than 4.3. Changing the target to
> xscale fixed the problem. And the tests I have done so far seem to
> imply that the performance of code compiled with gcc 4.6 optimized
> for xscale is slightly better than the same code compiled with gcc
> 4.3 optimized for iwmmxt.

Ok.

> Not sure if this should somehow be handled in buildroot. But I thought
> I'd at least let you know on the mailing-list so that others can avoid
> running into the same problem.

If the problem is still there in the latest versions of gcc, it should
be reported as a bug to the gcc project.

Thanks for the investigation!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com