[Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Paul Wilkinson]

I have a legacy multithreaded C++ app that requires one thread to run RT
and
communicate using EtherCAT via RTnet, I'm running Debian 8 + Xenomai 3.0.5
+
ipipe-core-4.9.24-x86-2.patch.

By extracting the functionality of that RT thread from the application and
creating a standalone POC (1RT + 2NRT threads) compiled and linked under
Xenomai Cobalt I've proved the RT & EtherCAT requirements can be met.

In addition to the RT requirement, a number of threads forming this
application
need to communicate with clients using TCP/IP via non-RTnet NIC.

To achieve this I prefixed all socket related calls in the class handling
TCP/IP with __real_ and compiled and linked the app using the xeno-config
output.

The app starts OK (I haven't rolled in the RT EtherCAT thread yet) with a
NRT thread  listening on a TCP/IP port, when a client connects over TCP/IP
the message is received and a NRT thread is created, immediately followed
by the following message .... and the app aborts.

terminate called after throwing an instance of 'std::system_error'
  what():  Operation not permitted

The application currently uses std::thread & std::mutex and I'm mindful
that
Xenomai/Cobalt may be reducing thread stack sizes, and that std::thread
doesn't seem to provide a mechanism to increase.

My questions :-

- is the approach I'm taking to allow threads of the class requiring TCP/IP
  comms reasonable, namely using __real_ on all socket related calls ?

- I'm assuming, but don't know for certain, that Zenomai/Cobalt has
  intercepted the pthread calls underlying std::thread ?
  'nm -a' executable shows _wrap_pthread*

- Does anyone have any insight into what could be causing this problem
and/or
  how I might further diagnose it ?

Any and all help very much appreciated.

Paul.

compile and link parameters :-

xeno-config --posix --cflags --skin=cobalt

-I/usr/xenomai/include/cobalt -I/usr/xenomai/include -D_GNU_SOURCE
-D_REENTRANT -D__COBALT__ -D__COBALT_WRAP__

xeno-config --posix --ldflags --skin=cobalt

-Wl,--no-as-needed -Wl,@/usr/xenomai/lib/cobalt.wrappers
-Wl,@/usr/xenomai/lib/modechk.wrappers 
/usr/xenomai/lib/xenomai/bootstrap.o -Wl,--wrap=main
-Wl,--dynamic-list=/usr/xenomai/lib/dynlist.ld -L/usr/xenomai/lib -lcobalt
-lmodechk -lpthread -lrt

actual gcc arguments :-

-std=c++14 -D_linux -DCONFIG_XENO_DEBUG -D__XENO__ -I/usr/xenomai/include
-I/usr/xenomai/include/cobalt -O0 -g3 -Wall -c -fmessage-length=0
-D_GNU_SOURCE -D_REENTRANT -D__COBALT__ -D__COBALT_WRAP_

-L/usr/xenomai/lib -Wl,--no-as-needed -Wl,@/usr/xenomai/lib/cobalt.wrappers
-Wl,@/usr/xenomai/lib/modechk.wrappers 
/usr/xenomai/lib/xenomai/bootstrap.o -Wl,--wrap=main
-Wl,--dynamic-list=/usr/xenomai/lib/dynlist.ld -L/usr/xenomai/lib -lcobalt
-lmodechk -lpthread -lrt -pthread

The gdb session output.

# gdb ./XenomaiDemo
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./XenomaiDemo...done.
(gdb) r
Starting program: /home/paul/Xenomai/server/XenomaiDemo/Debug/XenomaiDemo
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff69c6700 (LWP 3487)]
   0"000.005| WARNING: [main] Xenomai compiled with partial debug
enabled,
                              high latencies
expected [--enable-debug=partial]
Starting System...
[New Thread 0x7ffff61c5700 (LWP 3488)]
XenomaiDemo Server Started
Running in Dynamic Mode
[New Thread 0x7ffff59c4700 (LWP 3489)]
[New Thread 0x7ffff51c3700 (LWP 3490)]
[New Thread 0x7ffff49c2700 (LWP 3491)]
[New Thread 0x7fffeffff700 (LWP 3492)]
XenomaiDemo Server listening for clients on port 26000
[New Thread 0x7fffef7fe700 (LWP 3493)]
terminate called after throwing an instance of 'std::system_error'
  what():  Operation not permitted

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffef7fe700 (LWP 3493)]
0x00007ffff69fc067 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or
directory.
(gdb) bt
#0  0x00007ffff69fc067 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff69fd448 in __GI_abort () at abort.c:89
#2  0x00007ffff72e9b3d in __gnu_cxx::__verbose_terminate_handler() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff72e7bb6 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff72e7c01 in std::terminate() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff733fa01 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff77a4064 in start_thread (arg=0x7fffef7fe700) at
pthread_create.c:309
#7  0x00007ffff6aaf62d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Jan Kiszka]

On 2017-09-07 17:36, Paul Wilkinson wrote:
> I have a legacy multithreaded C++ app that requires one thread to run RT
> and
> communicate using EtherCAT via RTnet, I'm running Debian 8 + Xenomai 3.0.5
> +
> ipipe-core-4.9.24-x86-2.patch.
> 
> By extracting the functionality of that RT thread from the application and
> creating a standalone POC (1RT + 2NRT threads) compiled and linked under
> Xenomai Cobalt I've proved the RT & EtherCAT requirements can be met.
> 
> In addition to the RT requirement, a number of threads forming this
> application
> need to communicate with clients using TCP/IP via non-RTnet NIC.
> 
> To achieve this I prefixed all socket related calls in the class handling
> TCP/IP with __real_ and compiled and linked the app using the xeno-config
> output.
> 
> The app starts OK (I haven't rolled in the RT EtherCAT thread yet) with a
> NRT thread  listening on a TCP/IP port, when a client connects over TCP/IP
> the message is received and a NRT thread is created, immediately followed
> by the following message .... and the app aborts.
> 
> terminate called after throwing an instance of 'std::system_error'
>   what():  Operation not permitted
> 
> The application currently uses std::thread & std::mutex and I'm mindful
> that
> Xenomai/Cobalt may be reducing thread stack sizes, and that std::thread
> doesn't seem to provide a mechanism to increase.
> 
> My questions :-
> 
> - is the approach I'm taking to allow threads of the class requiring TCP/IP
>   comms reasonable, namely using __real_ on all socket related calls ?
> 
> - I'm assuming, but don't know for certain, that Zenomai/Cobalt has
>   intercepted the pthread calls underlying std::thread ?
>   'nm -a' executable shows _wrap_pthread*
> 
> - Does anyone have any insight into what could be causing this problem
> and/or
>   how I might further diagnose it ?

Are you running the application as normal user instead of root, or is
something else taking away permissions (CAP_SYS_NICE) from it?

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Paul Wilkinson]

Quoting Jan Kiszka <jan.kiszka@siemens.com>:

>
> Are you running the application as normal user instead of root, or is
> something else taking away permissions (CAP_SYS_NICE) from it?
>
> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA ITP SES-DE
> Corporate Competence Center Embedded Linux


Ordinarily I run the app as a normal user which is a member of the
group saved in /sys/module/xenomai/parameters/allowed_group

However I have tried running it as root, with same failure mode .... as
reflected in the gdb session provided.

Paul.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Philippe Gerum]

On 09/07/2017 05:36 PM, Paul Wilkinson wrote:
> XenomaiDemo Server listening for clients on port 26000
> [New Thread 0x7fffef7fe700 (LWP 3493)]
> terminate called after throwing an instance of 'std::system_error'
>   what():  Operation not permitted
> 
> Program received signal SIGABRT, Aborted.
> [Switching to Thread 0x7fffef7fe700 (LWP 3493)]
> 0x00007ffff69fc067 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> 56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or
> directory.
> (gdb) bt
> #0  0x00007ffff69fc067 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x00007ffff69fd448 in __GI_abort () at abort.c:89
> #2  0x00007ffff72e9b3d in __gnu_cxx::__verbose_terminate_handler() () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #3  0x00007ffff72e7bb6 in ?? () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #4  0x00007ffff72e7c01 in std::terminate() () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #5  0x00007ffff733fa01 in ?? () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6

Do you have the libstdc++ debug package installed? It would be useful to
know which source location issues the exception. GDB is not able to
resolve the symbols in the libstdc++ DSO it seems.

-- 
Philippe.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Paul Wilkinson]

Quoting Philippe Gerum <rpm@xenomai.org>:


>
> Do you have the libstdc++ debug package installed? It would be useful to
> know which source location issues the exception. GDB is not able to
> resolve the symbols in the libstdc++ DSO it seems.
>


Replacing the std::thread calls with pthread* calls and increasing the thread
stack sizes gets around this and a couple of subsequent SIGSEGV failures.

Although not quite running correctly yet, this enables the application to get
significantly further.

I have installed the libstdc++ debug package now, though.

Thanks.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Philippe Gerum]

On 09/11/2017 02:18 PM, Paul Wilkinson wrote:
> 
> Quoting Philippe Gerum <rpm@xenomai.org>:
> 
> 
>>
>> Do you have the libstdc++ debug package installed? It would be useful to
>> know which source location issues the exception. GDB is not able to
>> resolve the symbols in the libstdc++ DSO it seems.
>>
> 
> 
> Replacing the std::thread calls with pthread* calls and increasing the
> thread
> stack sizes gets around this and a couple of subsequent SIGSEGV failures.
> 
> Although not quite running correctly yet, this enables the application
> to get
> significantly further.
> 
> I have installed the libstdc++ debug package now, though.
> 

libcobalt uses the PTHREAD_STACK_DEFAULT constant as the default stack
size, which is defined in include/boilerplate/libc.h. This value is
currently 64k. STL is prone to massive stack usage; I'm inclined to add
a tunable parameter for this default value.

-- 
Philippe.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Paul Wilkinson]

Quoting Philippe Gerum <rpm@xenomai.org>:


>> Replacing the std::thread calls with pthread* calls and increasing the
>> thread
>> stack sizes gets around this and a couple of subsequent SIGSEGV failures.
>>
>> Although not quite running correctly yet, this enables the application
>> to get
>> significantly further.
>>
>> I have installed the libstdc++ debug package now, though.
>>
>
> libcobalt uses the PTHREAD_STACK_DEFAULT constant as the default stack
> size, which is defined in include/boilerplate/libc.h. This value is
> currently 64k. STL is prone to massive stack usage; I'm inclined to add
> a tunable parameter for this default value.


Thanks Philippe.

A tunable parameter would be very useful.

Although knowing where to change the default may save me replacing  
std::thread with pthread*.


Paul.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Paul Wilkinson]

Quoting Paul Wilkinson <paul.w@mrmoco.com>:

> Quoting Philippe Gerum <rpm@xenomai.org>:

>>> I have installed the libstdc++ debug package now, though.
>>>
>>
>> libcobalt uses the PTHREAD_STACK_DEFAULT constant as the default stack
>> size, which is defined in include/boilerplate/libc.h. This value is
>> currently 64k. STL is prone to massive stack usage; I'm inclined to add
>> a tunable parameter for this default value.
>
>
> Thanks Philippe.
>
> A tunable parameter would be very useful.
>
> Although knowing where to change the default may save me replacing  
> std::thread with pthread*.


I changed PTHREAD_STACK_DEFAULT to ~8M and reintroduced std:thread in one case
where thread creation failed previously.

The SIGABRT failure mode returns, gdb stack trace below.

Stack frame 6 shows std:thread is using pthreads under the hood, but  
the SIGABRT
seems to imply that my increasing PTHREAD_STACK_DEFAULT hasn't had any  
effect, I
now doubt Xenomai Cobalt is wrapping the pthread calls and applying the new
PTHREAD_STACK_DEFAULT to the std::thread calls.

I'd appreciate any insight into when and when not calls such as  
pthread* will be
intercepted by Xenomai Cobalt.

Thank you.



terminate called after throwing an instance of 'std::system_error'
   what():  Operation not permitted

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff49c2700 (LWP 8373)]
0x00007ffff69fc067 in __GI_raise (sig=sig@entry=6)
     at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff69fc067 in __GI_raise (sig=sig@entry=6)
     at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff69fd448 in __GI_abort () at abort.c:89
#2  0x00007ffff72e9b3d in __gnu_cxx::__verbose_terminate_handler ()
     at ../../../../src/libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007ffff72e7bb6 in __cxxabiv1::__terminate (handler=<optimized out>)
     at ../../../../src/libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007ffff72e7c01 in std::terminate ()
     at ../../../../src/libstdc++-v3/libsupc++/eh_terminate.cc:57
#5  0x00007ffff733fa01 in std::(anonymous  
namespace)::execute_native_thread_routine (__p=<optimized out>)
     at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:92
#6  0x00007ffff77a4064 in start_thread (arg=0x7ffff49c2700)
     at pthread_create.c:309
#7  0x00007ffff6aaf62d in clone ()
     at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Paul Wilkinson]

Quoting Paul Wilkinson <paul.w@mrmoco.com>:


>
> I changed PTHREAD_STACK_DEFAULT to ~8M and reintroduced std:thread  
> in one case
> where thread creation failed previously.
>
> The SIGABRT failure mode returns, gdb stack trace below.
>
> Stack frame 6 shows std:thread is using pthreads under the hood, but  
> the SIGABRT
> seems to imply that my increasing PTHREAD_STACK_DEFAULT hasn't had  
> any effect, I
> now doubt Xenomai Cobalt is wrapping the pthread calls and applying the new
> PTHREAD_STACK_DEFAULT to the std::thread calls.
>
> I'd appreciate any insight into when and when not calls such as  
> pthread* will be
> intercepted by Xenomai Cobalt.
>


I should add that the new PTHREAD_STACK_DEFAULT of ~8M that I introduced
to ./include/boilerplate/libc.h is reflected in the initial stacksize, as
returned by pthread_attr_getstacksize(), for the remaining threads created
using pthread_create().


Paul.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Philippe Gerum]

On 09/12/2017 03:26 PM, Paul Wilkinson wrote:
> 
> Quoting Paul Wilkinson <paul.w@mrmoco.com>:
> 
>> Quoting Philippe Gerum <rpm@xenomai.org>:
> 
>>>> I have installed the libstdc++ debug package now, though.
>>>>
>>>
>>> libcobalt uses the PTHREAD_STACK_DEFAULT constant as the default stack
>>> size, which is defined in include/boilerplate/libc.h. This value is
>>> currently 64k. STL is prone to massive stack usage; I'm inclined to add
>>> a tunable parameter for this default value.
>>
>>
>> Thanks Philippe.
>>
>> A tunable parameter would be very useful.
>>
>> Although knowing where to change the default may save me replacing
>> std::thread with pthread*.
> 
> 
> I changed PTHREAD_STACK_DEFAULT to ~8M and reintroduced std:thread in
> one case
> where thread creation failed previously.
> 
> The SIGABRT failure mode returns, gdb stack trace below.
> 
> Stack frame 6 shows std:thread is using pthreads under the hood, but the
> SIGABRT
> seems to imply that my increasing PTHREAD_STACK_DEFAULT hasn't had any
> effect, I
> now doubt Xenomai Cobalt is wrapping the pthread calls and applying the new
> PTHREAD_STACK_DEFAULT to the std::thread calls.
> 

Most likely, as the dynamic linker won't do the symbol wrapping we need
from pthread_create -> __wrap_pthread_create when loading a DSO. A not
so elegant way to force this would be to link against the static version
of libstdc++, but, you would risk unwanted wrapping of other POSIX calls
throughout the library in the same move.

This post may help:
https://xenomai.org/pipermail/xenomai/2014-August/031475.html

> I'd appreciate any insight into when and when not calls such as pthread*
> will be
> intercepted by Xenomai Cobalt.
> 

Easiest way: use gdb, setting a breakpoint in the __wrap_pthread_create
routine. This one is provided by libcobalt, and should not be hit when
instantiating the std::thread class from the libstdc++ DSO.

-- 
Philippe.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Paul Wilkinson]

Quoting Philippe Gerum <rpm@xenomai.org>:


>> Stack frame 6 shows std:thread is using pthreads under the hood, but the
>> SIGABRT
>> seems to imply that my increasing PTHREAD_STACK_DEFAULT hasn't had any
>> effect, I
>> now doubt Xenomai Cobalt is wrapping the pthread calls and applying the new
>> PTHREAD_STACK_DEFAULT to the std::thread calls.
>>
>
> Most likely, as the dynamic linker won't do the symbol wrapping we need
> from pthread_create -> __wrap_pthread_create when loading a DSO. A not
> so elegant way to force this would be to link against the static version
> of libstdc++, but, you would risk unwanted wrapping of other POSIX calls
> throughout the library in the same move.
>
> This post may help:
> https://xenomai.org/pipermail/xenomai/2014-August/031475.html

Very interesting and insightful.

>
>> I'd appreciate any insight into when and when not calls such as pthread*
>> will be
>> intercepted by Xenomai Cobalt.
>>
>
> Easiest way: use gdb, setting a breakpoint in the __wrap_pthread_create
> routine. This one is provided by libcobalt, and should not be hit when
> instantiating the std::thread class from the libstdc++ DSO.

Thank you.

As you suggested I set a breakpoint in __wrap_pthread_create, it is never
called following a std::thread thread creation.

Which makes me wonder why replacing the std::thread thread creation with
pthread_create and bumping up the thread stack sizes made the SIGABRT and
SIGSEGV's disappear.

Paul.

Re: [Xenomai] Questions around porting app to Xenomai/Cobalt resulting in SIGABRT ?

[Philippe Gerum]

On 09/15/2017 05:58 PM, Paul Wilkinson wrote:
> 
> Quoting Philippe Gerum <rpm@xenomai.org>:
> 
> 
>>> Stack frame 6 shows std:thread is using pthreads under the hood, but the
>>> SIGABRT
>>> seems to imply that my increasing PTHREAD_STACK_DEFAULT hasn't had any
>>> effect, I
>>> now doubt Xenomai Cobalt is wrapping the pthread calls and applying
>>> the new
>>> PTHREAD_STACK_DEFAULT to the std::thread calls.
>>>
>>
>> Most likely, as the dynamic linker won't do the symbol wrapping we need
>> from pthread_create -> __wrap_pthread_create when loading a DSO. A not
>> so elegant way to force this would be to link against the static version
>> of libstdc++, but, you would risk unwanted wrapping of other POSIX calls
>> throughout the library in the same move.
>>
>> This post may help:
>> https://xenomai.org/pipermail/xenomai/2014-August/031475.html
> 
> Very interesting and insightful.
> 
>>
>>> I'd appreciate any insight into when and when not calls such as pthread*
>>> will be
>>> intercepted by Xenomai Cobalt.
>>>
>>
>> Easiest way: use gdb, setting a breakpoint in the __wrap_pthread_create
>> routine. This one is provided by libcobalt, and should not be hit when
>> instantiating the std::thread class from the libstdc++ DSO.
> 
> Thank you.
> 
> As you suggested I set a breakpoint in __wrap_pthread_create, it is never
> called following a std::thread thread creation.
> 
> Which makes me wonder why replacing the std::thread thread creation with
> pthread_create and bumping up the thread stack sizes made the SIGABRT and
> SIGSEGV's disappear.
> 

Considering the opposite, i.e. not having libcobalt limiting the stack
size by setting the proper attribute before calling pthread_create(), we
may have std::thread rely on the default size setting. If the stack size
is unspecified in the attribute block, the NPTL may use RLIMIT_STACK if
set as a default value for pthread_create(), which is commonly 8Mb
(ulimit -s).

Xenomai apps run mlocked, so the stack memory is actually committed when
a real-time thread emerges, which would mean 8Mb consumed every time a
new thread is instantiated this way.

Depending on how much memory you have on the target, and how many
threads are instantiated, you may well end up with a RAM shortage,
causing all sorts of fireworks. This is only guesswork at this stage,
but that is plausible.

-- 
Philippe.