[Xenomai] "Ctrl + C" makes system crashed when exitting from "cross-link"

[Xenomai] "Ctrl + C" makes system crashed when exitting from "cross-link"

[Jack Lee]

Hello, everyone!
  My computer is Advantech MIO-5251, (the J1900 version, 
http://www.advantech.com/products/bc6786b7-b69b-4fdd-9b63-79180dbd5dcb/mio-5251/mod_ff8534cc-e858-40ba-9bc2-386f19bfee4a)
 linux kernel version is 4.1.18, xenomai version is 3.0.2, system is lubuntu 16.04.
I remove the original UART drivers and modprobe the xeno_16550A driver. cross-link demo
works fine, but when I type in "Ctrl + C" to exit from it, the terminal outputs these and the system
just dies.
...
main: /dev/rtdm/rtser1 (read) -> closed
main: /dev/rtdm/rtser0 (write) -> closed
main: delete write_task
write_task: error on rt_task_wait_period, Interrupted system call
write_task: exit
main: delete read_task
---

Sometimes, it outputs these:
...
main: exit
---
The system does not die, but the cross-link does not exit actually.
I run "cat /proc/xenomai/sched/thread" to get the pid and kill it,
the terminal just outputs "main: exit" repeatedly.

What's wrong, and  how can I solve it? Thanks!

Re: [Xenomai] "Ctrl + C" makes system crashed when exitting from "cross-link"

[Philippe Gerum]

On 09/11/2016 01:12 PM, Jack Lee wrote:
> Hello, everyone!
>   My computer is Advantech MIO-5251, (the J1900 version, 
> http://www.advantech.com/products/bc6786b7-b69b-4fdd-9b63-79180dbd5dcb/mio-5251/mod_ff8534cc-e858-40ba-9bc2-386f19bfee4a)
>  linux kernel version is 4.1.18, xenomai version is 3.0.2, system is lubuntu 16.04.
> I remove the original UART drivers and modprobe the xeno_16550A driver. cross-link demo
> works fine, but when I type in "Ctrl + C" to exit from it, the terminal outputs these and the system
> just dies.
> ...
> main: /dev/rtdm/rtser1 (read) -> closed
> main: /dev/rtdm/rtser0 (write) -> closed
> main: delete write_task
> write_task: error on rt_task_wait_period, Interrupted system call
> write_task: exit
> main: delete read_task
> ---
> 
> Sometimes, it outputs these:
> ...
> main: exit
> ---
> The system does not die, but the cross-link does not exit actually.
> I run "cat /proc/xenomai/sched/thread" to get the pid and kill it,
> the terminal just outputs "main: exit" repeatedly.
> 
> What's wrong, and  how can I solve it? Thanks!
> 

The reader task may be hanging around although it should have received a
deletion signal, causing the main thread to wait for all its children
indefinitely. This said, issuing rt_task_delete() and friends over a
signal handler is not legit, so the demo code has a problem anyway.

You could run the cross-link application over gdb, then check the stack
backtrace of the reader task at exit (or any task that appears to be
stuck), to find out where that task is stuck (^C, then switch to the
reader task context using gdb's "thread" command, then "bt").

-- 
Philippe.

[Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"

[Jack Lee]

> ------------------ Original ------------------
> From:  "Philippe Gerum";<rpm@xenomai.org>;
> Date:  Sep 11, 2016
> To:  "Jack Lee"<jacklee11@foxmail.com>; "xenomai"<xenomai@xenomai.org>;
> Subject:  Re: [Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"
> 
> On 09/11/2016 01:12 PM, Jack Lee wrote:
> > Hello, everyone!
> >   My computer is Advantech MIO-5251, (the J1900 version, 
> > http://www.advantech.com/products/bc6786b7-b69b-4fdd-9b63-79180dbd5dcb/mio-5251/mod_ff8534cc-e858-40ba-9bc2-386f19bfee4a)
> >  linux kernel version is 4.1.18, xenomai version is 3.0.2, system is lubuntu 16.04.
> > I remove the original UART drivers and modprobe the xeno_16550A driver. cross-link demo
> > works fine, but when I type in "Ctrl + C" to exit from it, the terminal outputs these and the system
> > just dies.
> > ...
> > main: /dev/rtdm/rtser1 (read) -> closed
> > main: /dev/rtdm/rtser0 (write) -> closed
> > main: delete write_task
> > write_task: error on rt_task_wait_period, Interrupted system call
> > write_task: exit
> > main: delete read_task
> > ---
> > 
> > Sometimes, it outputs these:
> > ...
> > main: exit
> > ---
> > The system does not die, but the cross-link does not exit actually.
> > I run "cat /proc/xenomai/sched/thread" to get the pid and kill it,
> > the terminal just outputs "main: exit" repeatedly.
> > 
> > What's wrong, and  how can I solve it? Thanks!
> > 
> 
> The reader task may be hanging around although it should have received a
> deletion signal, causing the main thread to wait for all its children
> indefinitely. This said, issuing rt_task_delete() and friends over a
> signal handler is not legit, so the demo code has a problem anyway.
> 
> You could run the cross-link application over gdb, then check the stack
> backtrace of the reader task at exit (or any task that appears to be
> stuck), to find out where that task is stuck (^C, then switch to the
> reader task context using gdb's "thread" command, then "bt").
> 
> -- 
> Philippe.

Hi, Philippe,

I run the "gdb cross-link" "Ctrl + C" and other instructions you showed, 
results are beblow:

^C
Thread 1 "cross-link" received signal SIGINT, Interrupt.
0x00007ffff7586d1d in pause () at ../sysdeps/unix/syscall-template.S:84
84	../sysdeps/unix/syscall-template.S: No such file or directory.

(gdb) thread find read
Thread 1 has target id 'Thread 0x7ffff7fde740 (LWP 3382)'
Thread 2 has target id 'Thread 0x7ffff6fa4700 (LWP 3383)'
Thread 3 has target id 'Thread 0x7ffff7ec9700 (LWP 3384)'
Thread 4 has target name 'read_task'
Thread 4 has target id 'Thread 0x7ffff7eb8700 (LWP 3385)'

(gdb) thread 4
[Switching to thread 4 (Thread 0x7ffff7eb8700 (LWP 3385))]
#0  0x00007ffff77a390f in do_ioctl () from /usr/xenomai/lib/libcobalt.so.2

(gdb) bt
#0  0x00007ffff77a390f in do_ioctl () from /usr/xenomai/lib/libcobalt.so.2
#1  0x00007ffff77a3d84 in __wrap_ioctl () from /usr/xenomai/lib/libcobalt.so.2
#2  0x0000000000401459 in read_task_proc (arg=0x0) at cross-link.c:200
#3  0x00007ffff7bd0b1a in task_entry () from /usr/xenomai/lib/libalchemy.so.0
#4  0x00007ffff79bd0ab in thread_trampoline ()
   from /usr/xenomai/lib/libcopperplate.so.0
#5  0x00007ffff77a5acc in cobalt_thread_trampoline ()
   from /usr/xenomai/lib/libcobalt.so.2
#6  0x00007ffff757d6fa in start_thread (arg=0x7ffff7eb8700)
    at pthread_create.c:333
#7  0x00007ffff72b3b5d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Is there something wrong?

Re: [Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"

[Philippe Gerum]

On 09/12/2016 05:30 AM, Jack Lee wrote:
>> ------------------ Original ------------------
>> From:  "Philippe Gerum";<rpm@xenomai.org>;
>> Date:  Sep 11, 2016
>> To:  "Jack Lee"<jacklee11@foxmail.com>; "xenomai"<xenomai@xenomai.org>;
>> Subject:  Re: [Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"
>>
>> On 09/11/2016 01:12 PM, Jack Lee wrote:
>>> Hello, everyone!
>>>   My computer is Advantech MIO-5251, (the J1900 version, 
>>> http://www.advantech.com/products/bc6786b7-b69b-4fdd-9b63-79180dbd5dcb/mio-5251/mod_ff8534cc-e858-40ba-9bc2-386f19bfee4a)
>>>  linux kernel version is 4.1.18, xenomai version is 3.0.2, system is lubuntu 16.04.
>>> I remove the original UART drivers and modprobe the xeno_16550A driver. cross-link demo
>>> works fine, but when I type in "Ctrl + C" to exit from it, the terminal outputs these and the system
>>> just dies.
>>> ...
>>> main: /dev/rtdm/rtser1 (read) -> closed
>>> main: /dev/rtdm/rtser0 (write) -> closed
>>> main: delete write_task
>>> write_task: error on rt_task_wait_period, Interrupted system call
>>> write_task: exit
>>> main: delete read_task
>>> ---
>>>
>>> Sometimes, it outputs these:
>>> ...
>>> main: exit
>>> ---
>>> The system does not die, but the cross-link does not exit actually.
>>> I run "cat /proc/xenomai/sched/thread" to get the pid and kill it,
>>> the terminal just outputs "main: exit" repeatedly.
>>>
>>> What's wrong, and  how can I solve it? Thanks!
>>>
>>
>> The reader task may be hanging around although it should have received a
>> deletion signal, causing the main thread to wait for all its children
>> indefinitely. This said, issuing rt_task_delete() and friends over a
>> signal handler is not legit, so the demo code has a problem anyway.
>>
>> You could run the cross-link application over gdb, then check the stack
>> backtrace of the reader task at exit (or any task that appears to be
>> stuck), to find out where that task is stuck (^C, then switch to the
>> reader task context using gdb's "thread" command, then "bt").
>>
>> -- 
>> Philippe.
> 
> Hi, Philippe,
> 
> I run the "gdb cross-link" "Ctrl + C" and other instructions you showed, 
> results are beblow:
> 
> ^C
> Thread 1 "cross-link" received signal SIGINT, Interrupt.
> 0x00007ffff7586d1d in pause () at ../sysdeps/unix/syscall-template.S:84
> 84	../sysdeps/unix/syscall-template.S: No such file or directory.
> 
> (gdb) thread find read
> Thread 1 has target id 'Thread 0x7ffff7fde740 (LWP 3382)'
> Thread 2 has target id 'Thread 0x7ffff6fa4700 (LWP 3383)'
> Thread 3 has target id 'Thread 0x7ffff7ec9700 (LWP 3384)'
> Thread 4 has target name 'read_task'
> Thread 4 has target id 'Thread 0x7ffff7eb8700 (LWP 3385)'
> 
> (gdb) thread 4
> [Switching to thread 4 (Thread 0x7ffff7eb8700 (LWP 3385))]
> #0  0x00007ffff77a390f in do_ioctl () from /usr/xenomai/lib/libcobalt.so.2
> 
> (gdb) bt
> #0  0x00007ffff77a390f in do_ioctl () from /usr/xenomai/lib/libcobalt.so.2
> #1  0x00007ffff77a3d84 in __wrap_ioctl () from /usr/xenomai/lib/libcobalt.so.2
> #2  0x0000000000401459 in read_task_proc (arg=0x0) at cross-link.c:200
> #3  0x00007ffff7bd0b1a in task_entry () from /usr/xenomai/lib/libalchemy.so.0
> #4  0x00007ffff79bd0ab in thread_trampoline ()
>    from /usr/xenomai/lib/libcopperplate.so.0
> #5  0x00007ffff77a5acc in cobalt_thread_trampoline ()
>    from /usr/xenomai/lib/libcobalt.so.2
> #6  0x00007ffff757d6fa in start_thread (arg=0x7ffff7eb8700)
>     at pthread_create.c:333
> #7  0x00007ffff72b3b5d in clone ()
>     at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> 
> Is there something wrong?
> 

Can't tell yet. What does "info threads" say?

Also, is the issue translates to a hard lockup without any message or
did you notice any warning or kernel oops on the console channel due to
a crash/BUG assertion?

If this is a silent, hard lockup, then you may want to enable
XENO_OPT_WATCHDOG in the Xenomai debug options of the kernel
configuration. This would catch any runaway Xenomai thread which may
freeze the box before it's too late (4 secs by default), giving us a
hint about the issue we are chasing.

-- 
Philippe.

[Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"

[Jack Lee]

> On 09/12/2016 05:30 AM, Jack Lee wrote:
> >> ------------------ Original ------------------
> >> From:  "Philippe Gerum";<rpm@xenomai.org>;
> >> Date:  Sep 11, 2016
> >> To:  "Jack Lee"<jacklee11@foxmail.com>; "xenomai"<xenomai@xenomai.org>;
> >> Subject:  Re: [Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"
> >>
> >> On 09/11/2016 01:12 PM, Jack Lee wrote:
> >>> Hello, everyone!
> >>>   My computer is Advantech MIO-5251, (the J1900 version, 
> >>> http://www.advantech.com/products/bc6786b7-b69b-4fdd-9b63-79180dbd5dcb/mio-5251/mod_ff8534cc-e858-40ba-9bc2-386f19bfee4a)
> >>>  linux kernel version is 4.1.18, xenomai version is 3.0.2, system is lubuntu 16.04.
> >>> I remove the original UART drivers and modprobe the xeno_16550A driver. cross-link demo
> >>> works fine, but when I type in "Ctrl + C" to exit from it, the terminal outputs these and the system
> >>> just dies.
> >>> ...
> >>> main: /dev/rtdm/rtser1 (read) -> closed
> >>> main: /dev/rtdm/rtser0 (write) -> closed
> >>> main: delete write_task
> >>> write_task: error on rt_task_wait_period, Interrupted system call
> >>> write_task: exit
> >>> main: delete read_task
> >>> ---
> >>>
> >>> Sometimes, it outputs these:
> >>> ...
> >>> main: exit
> >>> ---
> >>> The system does not die, but the cross-link does not exit actually.
> >>> I run "cat /proc/xenomai/sched/thread" to get the pid and kill it,
> >>> the terminal just outputs "main: exit" repeatedly.
> >>>
> >>> What's wrong, and  how can I solve it? Thanks!
> >>>
> >>
> >> The reader task may be hanging around although it should have received a
> >> deletion signal, causing the main thread to wait for all its children
> >> indefinitely. This said, issuing rt_task_delete() and friends over a
> >> signal handler is not legit, so the demo code has a problem anyway.
> >>
> >> You could run the cross-link application over gdb, then check the stack
> >> backtrace of the reader task at exit (or any task that appears to be
> >> stuck), to find out where that task is stuck (^C, then switch to the
> >> reader task context using gdb's "thread" command, then "bt").
> >>
> >> -- 
> >> Philippe.
> >  
> > Hi, Philippe,
> > 
> > I run the "gdb cross-link" "Ctrl + C" and other instructions you showed, 
> > results are beblow:
> > 
> > ^C
> > Thread 1 "cross-link" received signal SIGINT, Interrupt.
> > 0x00007ffff7586d1d in pause () at ../sysdeps/unix/syscall-template.S:84
> > 84	../sysdeps/unix/syscall-template.S: No such file or directory.
> > 
> > (gdb) thread find read
> > Thread 1 has target id 'Thread 0x7ffff7fde740 (LWP 3382)'
> > Thread 2 has target id 'Thread 0x7ffff6fa4700 (LWP 3383)'
> > Thread 3 has target id 'Thread 0x7ffff7ec9700 (LWP 3384)'
> > Thread 4 has target name 'read_task'
> > Thread 4 has target id 'Thread 0x7ffff7eb8700 (LWP 3385)'
> > 
> > (gdb) thread 4
> > [Switching to thread 4 (Thread 0x7ffff7eb8700 (LWP 3385))]
> > #0  0x00007ffff77a390f in do_ioctl () from /usr/xenomai/lib/libcobalt.so.2
> > 
> > (gdb) bt
> > #0  0x00007ffff77a390f in do_ioctl () from /usr/xenomai/lib/libcobalt.so.2
> > #1  0x00007ffff77a3d84 in __wrap_ioctl () from /usr/xenomai/lib/libcobalt.so.2
> > #2  0x0000000000401459 in read_task_proc (arg=0x0) at cross-link.c:200
> > #3  0x00007ffff7bd0b1a in task_entry () from /usr/xenomai/lib/libalchemy.so.0
> > #4  0x00007ffff79bd0ab in thread_trampoline ()
> >    from /usr/xenomai/lib/libcopperplate.so.0
> > #5  0x00007ffff77a5acc in cobalt_thread_trampoline ()
> >    from /usr/xenomai/lib/libcobalt.so.2
> > #6  0x00007ffff757d6fa in start_thread (arg=0x7ffff7eb8700)
> >     at pthread_create.c:333
> > #7  0x00007ffff72b3b5d in clone ()
> >     at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
> > 
> > Is there something wrong?
> > 
> 
> Can't tell yet. What does "info threads" say?
> 
> Also, is the issue translates to a hard lockup without any message or
> did you notice any warning or kernel oops on the console channel due to
> a crash/BUG assertion?
> 
> If this is a silent, hard lockup, then you may want to enable
> XENO_OPT_WATCHDOG in the Xenomai debug options of the kernel
> configuration. This would catch any runaway Xenomai thread which may
> freeze the box before it's too late (4 secs by default), giving us a
> hint about the issue we are chasing.
> 
> -- 
> Philippe.

Hi, Philippe, 

Sometimes, the system just dies absolutely, and I could only press the power key to power off and restart the computer.
Sometimes, "cross-link" can not exit even using the "kill" command, as described above. No crash assertion /warning reported.

I am not familiar with xenomai yet, the experiments may be done in the future.
Also, real-time performance of the advantech MIO-5251 is quite bad. I want to use this
system as a robot controller (eg. 6 axises.) Heavy duty of calculations is required. Is x-window running possible?
Do you have any board to recommend? Performance is preferred. Thanks!

Re: [Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"

[Philippe Gerum]

On 09/13/2016 11:13 AM, Jack Lee wrote:
> Sometimes, the system just dies absolutely, and I could only press the power key to power off and restart the computer.
> Sometimes, "cross-link" can not exit even using the "kill" command, as described above. No crash assertion /warning reported.
>

The watchdog may help in situation #1 if a runaway thread is at work.

> I am not familiar with xenomai yet, the experiments may be done in the future.
> Also, real-time performance of the advantech MIO-5251 is quite bad. I want to use this

Meaning with Xenomai in? What are the figures if so, any output of the
latency program?

Did you consider this information?
http://xenomai.org/troubleshooting-a-dual-kernel-configuration/#the_latency_test_shows_high_latencies

> system as a robot controller (eg. 6 axises.) Heavy duty of calculations is required. Is x-window running possible?

X-windows is possible if neither of your graphic card nor X driver
causes pathological latencies. I worked on several projects coupling
real-time acquisition and graphics on the same x86 SBC over the years
with no issues; conversely, a few graphic hw support did introduce
roadblocks in other cases, YYMV. You may definitely want to benchmark
the hw before buying.

> Do you have any board to recommend? Performance is preferred. Thanks!
> 

A bad BIOS is often involved in bad real-time performances (including
SMI woes), affecting dual kernel and native preemption configurations
the same way. Likewise, CONFIG_PERF_EVENTS on x86 may kill real-time
performances when NMI-based events are involved.

People of this list running x86 systems in production may hopefully give
you some hints about decent hardware for this usage.

-- 
Philippe.

[Xenomai] "Ctrl + C" makes system crashed when exitting from"cross-link"

[Jack Lee]

On 09/13/2016 09:00 PM, Philippe Gerum wrote:
> On 09/13/2016 11:13 AM, Jack Lee wrote:
>> Sometimes, the system just dies absolutely, and I could only press the power key to power off and restart the computer.
>> Sometimes, "cross-link" can not exit even using the "kill" command, as described above. No crash assertion /warning reported.
>>
> The watchdog may help in situation #1 if a runaway thread is at work.
>
>> I am not familiar with xenomai yet, the experiments may be done in the future.
>> Also, real-time performance of the advantech MIO-5251 is quite bad. I want to use this
> Meaning with Xenomai in? What are the figures if so, any output of the
> latency program?
>
> Did you consider this information?
>
>
>> system as a robot controller (eg. 6 axises.) Heavy duty of calculations is required. Is x-window running possible?
> X-windows is possible if neither of your graphic card nor X driver
> causes pathological latencies. I worked on several projects coupling
> real-time acquisition and graphics on the same x86 SBC over the years
> with no issues; conversely, a few graphic hw support did introduce
> roadblocks in other cases, YYMV. You may definitely want to benchmark
> the hw before buying.
>
>> Do you have any board to recommend? Performance is preferred. Thanks!
>>
> A bad BIOS is often involved in bad real-time performances (including
> SMI woes), affecting dual kernel and native preemption configurations
> the same way. Likewise, CONFIG_PERF_EVENTS on x86 may kill real-time
> performances when NMI-based events are involved.
>
> People of this list running x86 systems in production may hopefully give
> you some hints about decent hardware for this usage.

  * make sure that you carefully followed the“Kernel configuration”
    section
    <http://xenomai.org/troubleshooting-a-dual-kernel-configuration/#kconf>.

Checked.

  * if running on a x86 platform, make sure that you do not have an
    issue with SMIs, see thesection about SMIs
    <http://xenomai.org/troubleshooting-a-dual-kernel-configuration/#SMI>.

SMI walkaround enabled. nmi.watchdog was set to be 0.

  * if running on a x86 platform with a/legacy USB/switch available from
    the BIOS configuration, try disabling it.

Disabling it makes the keyboard not work.

  * if you observe high latencies while running X-window, try disabling
    hardware acceleration in the X-window server file. With recent
    versions of X-window, try using the/fbdev/driver. Install it (Debian
    package named/xserver-xorg-video-fbdev/for instance), then modifiy
    theDevicesection to use this driver in//etc/X11/xorg.conf/, as in:

fbdev is enabled.


If I have any progress on this issue, I will back.
Thanks for your endeavors!