Re: Issue with std::mutex when porting POSIX program to Xenomai 3

Re: Issue with std::mutex when porting POSIX program to Xenomai 3

[Sam Chiu]

Thank you, Julien. I will give it a try.

Additionally, I'm curious if, besides the issue with the mutex, you've encountered any significant challenges during the process of porting your POSIX program. 

Of course, I understand if there are details you'd prefer not to disclose.


Best regards,
Sam

Re: Issue with std::mutex when porting POSIX program to Xenomai 3

[Julien Blanc]

Le jeudi 28 mars 2024 à 02:31 +0000, Sam Chiu a écrit :
> 
> Thank you, Julien. I will give it a try.
> 
> Additionally, I'm curious if, besides the issue with the mutex, you've encountered any significant challenges during the process
> of porting your POSIX program. 

It was not a port, targetting xenomai was a requirement from the start of the projet, so i guess it makes things a lot easier. We
just learned that using any C++ standard synchronization/thread facilities is a no-go on xenomai. Since we already had some
higher-level facilities (we have some shared code that also targets freertos using this platform abstraction layer), it was pretty
easy for us.

Cobalt wrappers can be tricky sometimes, so we've learned to use __RT o __STD macros when needed (our apps are often a mixed bag
of RT and non-RT threads).


Regards,

Julien

Re: Issue with std::mutex when porting POSIX program to Xenomai 3

[Sam Chiu]

Dear Julien,

Yes. Maybe we should first isolate the real-time aspects of the program. Thank you.

Best regards,
Sam

Re: Issue with std::mutex when porting POSIX program to Xenomai 3

[Julien Blanc]

Le mardi 26 mars 2024 à 09:13 +0000, Sam Chiu a écrit :
> Dear Xenomai Community,
> 
> 
> So maybe I should consider replacing std::mutex with pthread_mutex.  However, before proceeding, I would like to inquire if
> there are better
>  approaches or solutions for porting POSIX C++ programs that utilize std::mutex.

std::mutex is not posix, it's C++. Whether or not it will make posix calls, and which calls it will make, is completely up to your
libc++ implementation.

If you plan on using std::xx calls from xenomai threads, it will usually not work: you would need a xenomai-compatible libc++, and
i'm not aware of any. Some calls might happen to work, some won't, and there is just no guarantee that a lib update will not break
things. If you plan on using std::xx calls from regular linux threads, inside a xenomai application, you will need to disable
cobalt wrappers.

The way we solved this is by implementing our own xenomai::mutex, which resembles std's one, and uses wrapped pthread_mutex_xx
calls. This also allows us to compile to regular posix, for test purposes.

Regards,

Julien

Issue with std::mutex when porting POSIX program to Xenomai 3

[Sam Chiu]

Dear Xenomai Community,

I am in the process of porting a POSIX program(srsRAN) to Xenomai and encountered an issue related to std::mutex when compiling the project with the following command:

```
cmake ../ -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS="-I/usr/xenomai/include/cobalt -I/usr/xenomai/include -D_GNU_SOURCE -D_REENTRANT -fasynchronous-unwind-tables -D__COBALT__ -D__COBALT_WRAP__" -DCMAKE_EXE_LINKER_FLAGS="-Wl,--no-as-needed -Wl,@/usr/xenomai/lib/cobalt.wrappers -Wl,@/usr/xenomai/lib/modechk.wrappers  /usr/xenomai/lib/xenomai/bootstrap.o -Wl,--wrap=main -Wl,--dynamic-list=/usr/xenomai/lib/dynlist.ld -L/usr/xenomai/lib -lcobalt -lmodechk -pthread -lrt"
```

Upon running the test program, it crashed, displaying the following error message:

```
terminate called after throwing an instance of 'std::system_error'
  what():  Operation not permitted
```

To diagnose the issue, I recompiled the program with debug symbols and observed the problem might be related to the use of std::mutex. 

A backtrace captured after the crash indicates a failure within the mutex operations:

```
#0  __pthread_kill_implementation (threadid=281474842415168, signo=signo@entry=6, no_tid=no_tid@entry=0)
    at ./nptl/pthread_kill.c:44
#1  0x0000fffff7a90a24 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  0x0000fffff7a4a76c in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x0000fffff7a374bc in __GI_abort () at ./stdlib/abort.c:79
#4  0x0000fffff7d449c8 in __gnu_cxx::__verbose_terminate_handler() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#5  0x0000fffff7d423bc in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x0000fffff7d412b4 in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#7  0x0000fffff7d41ab8 in __gxx_personality_v0 () from /lib/aarch64-linux-gnu/libstdc++.so.6
#8  0x0000fffff7bce47c in ?? () from /lib/aarch64-linux-gnu/libgcc_s.so.1
#9  0x0000fffff7bce874 in _Unwind_RaiseException () from /lib/aarch64-linux-gnu/libgcc_s.so.1
#10 0x0000fffff7d426f8 in __cxa_throw () from /lib/aarch64-linux-gnu/libstdc++.so.6
#11 0x0000fffff7d3e364 in std::__throw_system_error(int) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#12 0x0000aaaaaab40320 in std::mutex::lock (this=0xffffffffeb58) at /usr/include/c++/12/bits/std_mutex.h:104
#13 0x0000aaaaaab5b2b8 in std::unique_lock<std::mutex>::lock (this=0xfffffffc5c10)
    at /usr/include/c++/12/bits/unique_lock.h:139
#14 0x0000aaaaaab4de2c in std::unique_lock<std::mutex>::unique_lock (this=0xfffffffc5c10, __m=...)
    at /usr/include/c++/12/bits/unique_lock.h:69
#15 0x0000aaaaab5dceb4 in srsran::detail::queue_impl<srsran::unique_function<void (), 64ul, false>, (srsran::concurrent_queue_policy)3, (srsran::concurrent_queue_wait_policy)0>::try_push<srsran::unique_function<void (), 64ul, false> >(srsran::unique_function<void (), 64ul, false>&&) (this=0xffffffffeb00, elem=...)
```

After reading the tips for porting POSIX programs to Xenomai on the official website, I learned that mutexes should be dynamically initialized. 

It seems the C++ std::mutex is being initialized statically. Checking the GNU library, the default initialization for locks seems to be static. 
```
#ifdef _GLIBCXX_HAS_GTHREADS
  /// @cond undocumented
  // Common base class for std::mutex and std::timed_mutex
  class __mutex_base
  {
  protected:
    typedef __gthread_mutex_t             __native_type;
#ifdef __GTHREAD_MUTEX_INIT
    __native_type  _M_mutex = __GTHREAD_MUTEX_INIT;
    constexpr __mutex_base() noexcept = default;
#else
    __native_type  _M_mutex;
    __mutex_base() noexcept
    {
      // XXX EAGAIN, ENOMEM, EPERM, EBUSY(may), EINVAL(may)
      __GTHREAD_MUTEX_INIT_FUNCTION(&_M_mutex);
    }
    ~__mutex_base() noexcept { __gthread_mutex_destroy(&_M_mutex); }
#endif
```
And the macro of `__GTHREAD_MUTEX_INIT` is defined as `#define __GTHREAD_MUTEX_INIT PTHREAD_MUTEX_INITIALIZER`.

So maybe I should consider replacing std::mutex with pthread_mutex.  However, before proceeding, I would like to inquire if there are better
 approaches or solutions for porting POSIX C++ programs that utilize std::mutex.

Some details about my environment:
* Xenomai version : Xenomai/cobalt v3.2.1
* Kernel version : Linux raspberrypi 5.15.59xenomai aarch64 GNU/Linux
* Device : raspiberry Pi 4b
* Cobalt building args : --enable-smp
* The project repo : https://github.com/srsran/srsRAN_Project


I greatly appreciate any suggestions or help. 


Best regards,
Sam