[Xenomai] POSIX skin, task crashes Linux when run as RT task

[Xenomai] POSIX skin, task crashes Linux when run as RT task

[Steve B]

Hello,

I have a task running at 100Hz on a Beaglebone Black that seems to crash
sporadically when I set its parameters to SCHED_FIFO with any priority, and
it does not produce the crash when I set it to SCHED_OTHER.
However when I set it to SCHED_OTHER the jitter is much larger, so we
really want to get it running stable on SCHED_FIFO if possible.
Jitter is not currently measured precisely, but the difference is such that
it can be eyeballed on an oscilloscope watching the messages go back and
forth.
Looking at /proc/xenomai/stat this task tends to draw up to about 30% of
CPU at full load.

The symptom is that the Linux system becomes largely unresponsive, and the
task still runs and generates occasional printouts.
But it quickly goes from 100Hz (self-measured and reported once every few
seconds) to less than 1Hz and I usually can't kill it off without resetting
the board.

I know that the task is doing quite a lot of mode switches due to
interfacing with hardware that doesn't have an RTDM driver available.
I may be able to bump that part off to a non-RT task triggered with a
pthread_cond_wait or a semaphore or something, but I was sort of planning
to accept the mode switches as long as the timing is still "good enough."

Are there any techniques I can employ to troubleshoot this, or is it a
simple matter of the task being too CPU intensive and starving out Linux,
or that I really need to be getting rid of the mode switches?
Or is it possible that there's a bug in Xenomai that can be dug out from
this?

Cheers,

Steve

Re: [Xenomai] POSIX skin, task crashes Linux when run as RT task

[Philippe Gerum]

On 01/31/2015 12:52 AM, Steve B wrote:
> Hello,
> 
> I have a task running at 100Hz on a Beaglebone Black that seems to crash
> sporadically when I set its parameters to SCHED_FIFO with any priority, and

AFAICT, this should be the basic issue to investigate. Is there any
feedback on the kernel console? any crash report? Or is it just locking
up hard?

> it does not produce the crash when I set it to SCHED_OTHER.
> However when I set it to SCHED_OTHER the jitter is much larger, so we
> really want to get it running stable on SCHED_FIFO if possible.

As expected, SCHED_OTHER won't give you any rt guarantee, hence the
jitter. A SCHED_OTHER thread driven by Xenomai is allowed to wait for
Xenomai resources (e.g. pending on a sema4, waiting on a condvar,
blocking on a message queue), but won't be able to compete with
SCHED_FIFO threads for the CPU.

> Jitter is not currently measured precisely, but the difference is such that
> it can be eyeballed on an oscilloscope watching the messages go back and
> forth.
> Looking at /proc/xenomai/stat this task tends to draw up to about 30% of
> CPU at full load.
> 

You mean when running into the SCHED_OTHER class? If so, the mode
switches might explain such load only if the thread calls a Xenomai
service from its work loop, which forces the transition to primary mode,
which is required for competing for Xenomai resources.

Once the syscall has been handled, a SCHED_OTHER thread is moved back to
secondary/linux mode. So we have two mode switches per Xenomai syscall
which involves a transition for SCHED_OTHER threads. This differs from
SCHED_FIFO threads which switch lazily to the required mode, and stay
there until another transition to the converse mode is required.

> The symptom is that the Linux system becomes largely unresponsive, and the
> task still runs and generates occasional printouts.
> But it quickly goes from 100Hz (self-measured and reported once every few
> seconds) to less than 1Hz and I usually can't kill it off without resetting
> the board.
> 
> I know that the task is doing quite a lot of mode switches due to
> interfacing with hardware that doesn't have an RTDM driver available.
> I may be able to bump that part off to a non-RT task triggered with a
> pthread_cond_wait or a semaphore or something, but I was sort of planning
> to accept the mode switches as long as the timing is still "good enough."

In the SCHED_OTHER case, calling a regular linux driver won't cause any
mode switch, since the thread "naturally" executes in the linux domain
(i.e. secondary mode). On the contrary, a SCHED_FIFO task will have to
switch mode in such situation though.

> 
> Are there any techniques I can employ to troubleshoot this, or is it a
> simple matter of the task being too CPU intensive and starving out Linux,
> or that I really need to be getting rid of the mode switches?
> Or is it possible that there's a bug in Xenomai that can be dug out from
> this?
> 

More information is needed to better understand the issue, including
details about what the work loop is actually doing (see
http://xenomai.org/asking-for-help/).

Enabling CONFIG_XENO_OPT_WATCHDOG might help detecting runaway threads
which belong to Xenomai's SCHED_FIFO class, but this won't apply to your
test with SCHED_OTHER.

Maybe CONFIG_LOCKUP_DETECTOR might trigger in case a linux-driven thread
overconsumes the CPU (i.e. either a Xenomai SCHED_OTHER thread in linux
mode, or a regular pthread).

It is definitely possible that your workload overconsumes CPU depending
on the available CPU horsepower on your platform, but we can't be sure
until we know more about your setup. This said, mode switching is a
costly operation, especially with Xenomai 2.x (compared to 3.x). So the
design should use only a very few of them in a normal work cycle if
absolutely required, and preferably none, unless the later is paced at a
reasonably slow rate in the timeline.

There are ways to exchange messages between a plain regular pthread and
a Xenomai SCHED_FIFO thread without involving any mode switch (see the
XDDP protocol from the RTIPC driver). Other ways exist to share
resources and/or synchronize between Xenomai SCHED_OTHER and SCHED_FIFO
threads, at the expense of mode switches for the former. The best pick
depends on the nature of your workload.

-- 
Philippe.

Re: [Xenomai] POSIX skin, task crashes Linux when run as RT task

[Steve B]

On Sat, Jan 31, 2015 at 2:11 AM, Philippe Gerum <rpm@xenomai.org> wrote:

> On 01/31/2015 12:52 AM, Steve B wrote:
> > Hello,
> >
> > I have a task running at 100Hz on a Beaglebone Black that seems to crash
> > sporadically when I set its parameters to SCHED_FIFO with any priority,
> and
>
> AFAICT, this should be the basic issue to investigate. Is there any
> feedback on the kernel console? any crash report? Or is it just locking
> up hard?
>
>
I tried 'dmesg -w' in another telnet session, the only kernel message I saw
after starting up my application was that one of my serial ports had a
buffer overrun once (I don't believe this is related). No messages at the
time of the crash. I would say it's mostly locking up hard as it takes a
very long and indeterminate amount of time to CTRL-C out of the
application, and my other telnet sessions don't respond well (or at all) to
inputs on the command line. The application still gives some printouts, but
about a hundred or to a thousand times slower than nominal.


> > it does not produce the crash when I set it to SCHED_OTHER.
> > However when I set it to SCHED_OTHER the jitter is much larger, so we
> > really want to get it running stable on SCHED_FIFO if possible.
>
> As expected, SCHED_OTHER won't give you any rt guarantee, hence the
> jitter. A SCHED_OTHER thread driven by Xenomai is allowed to wait for
> Xenomai resources (e.g. pending on a sema4, waiting on a condvar,
> blocking on a message queue), but won't be able to compete with
> SCHED_FIFO threads for the CPU.
>
> > Jitter is not currently measured precisely, but the difference is such
> that
> > it can be eyeballed on an oscilloscope watching the messages go back and
> > forth.
> > Looking at /proc/xenomai/stat this task tends to draw up to about 30% of
> > CPU at full load.
> >
>
> You mean when running into the SCHED_OTHER class? If so, the mode
> switches might explain such load only if the thread calls a Xenomai
> service from its work loop, which forces the transition to primary mode,
> which is required for competing for Xenomai resources.
>
>
I am using some XDDP writes as well as writes to non-RTDM hardware, so I
suppose that I will get a handful of mode switches no matter which type of
thread it is.

More information is needed to better understand the issue, including
> details about what the work loop is actually doing (see
> http://xenomai.org/asking-for-help/).
>
> Enabling CONFIG_XENO_OPT_WATCHDOG might help detecting runaway threads
> which belong to Xenomai's SCHED_FIFO class, but this won't apply to your
> test with SCHED_OTHER.
>
> Maybe CONFIG_LOCKUP_DETECTOR might trigger in case a linux-driven thread
> overconsumes the CPU (i.e. either a Xenomai SCHED_OTHER thread in linux
> mode, or a regular pthread).
>
> It is definitely possible that your workload overconsumes CPU depending
> on the available CPU horsepower on your platform, but we can't be sure
> until we know more about your setup. This said, mode switching is a
> costly operation, especially with Xenomai 2.x (compared to 3.x). So the
> design should use only a very few of them in a normal work cycle if
> absolutely required, and preferably none, unless the later is paced at a
> reasonably slow rate in the timeline.
>
>
I am seeing mode switches for that thread increasing at a rate of about 100
for each time I look at /proc/xenomai/stat by hand (less than one second),
so it seems like it's on the order of a couple or a few per cycle actually.
I've been hoping to eliminate those somehow.


> There are ways to exchange messages between a plain regular pthread and
> a Xenomai SCHED_FIFO thread without involving any mode switch (see the
> XDDP protocol from the RTIPC driver). Other ways exist to share
> resources and/or synchronize between Xenomai SCHED_OTHER and SCHED_FIFO
> threads, at the expense of mode switches for the former. The best pick
> depends on the nature of your workload.
>
> I actually found XDDP to be very handy for another part of my application,
but I hadn't thought of using it for the part where the main thread puts
its messages out to hardware. I may look into it for this as well, but I
have been thinking that porting the driver to RTDM is the actual best
solution if it turns out the mode switches are part of the critical path.

I'm not sure yet how much I can disclose about what our main working thread
is doing aside from "a whole lot of floating point operations." It is
actually a bit of a black box to me as well currently. I understand that I
can get the best help if I'm able to provide more info, but since I'm not
sure, I was first hoping to just get some generic troubleshooting
strategies so I can know where to look.

My top suspicion now is that the main thread is occasionally hogging CPU
and some percentage of the time it's enough to throw off Linux, but I am
just trying to figure out if there's some way I can prove or disprove that
easily. It seems to be independent of whether or not my other (non-RT)
threads are even running at all.


At any rate, thanks very much for the suggestions!

Re: [Xenomai] POSIX skin, task crashes Linux when run as RT task

[Philippe Gerum]

On 02/02/2015 10:12 PM, Steve B wrote:
> On Sat, Jan 31, 2015 at 2:11 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> 
>> On 01/31/2015 12:52 AM, Steve B wrote:
>>> Hello,
>>>
>>> I have a task running at 100Hz on a Beaglebone Black that seems to crash
>>> sporadically when I set its parameters to SCHED_FIFO with any priority,
>> and
>>
>> AFAICT, this should be the basic issue to investigate. Is there any
>> feedback on the kernel console? any crash report? Or is it just locking
>> up hard?
>>
>>
> I tried 'dmesg -w' in another telnet session, the only kernel message I saw
> after starting up my application was that one of my serial ports had a
> buffer overrun once (I don't believe this is related). No messages at the
> time of the crash. I would say it's mostly locking up hard as it takes a
> very long and indeterminate amount of time to CTRL-C out of the
> application, and my other telnet sessions don't respond well (or at all) to
> inputs on the command line. The application still gives some printouts, but
> about a hundred or to a thousand times slower than nominal.
> 

A typical issue is an error reported by some blocking call - for
instance when reading data from some RT device - going unnoticed by the
application, which then iterates and so on. Normally, this causes a hard
lockup due to the RT scheduling priority, but if your work loop also
switches mode due to plain linux calls, then the system might just be
brought to a crawl instead.

That would explain why the Xenomai watchdog does not trigger either,
since you do frequent transitions to secondary/linux mode, the real-time
core does not detect the lockup condition which is N secs spent in
primary/rt mode without transition to linux (N=4 by default).

> 
>>> it does not produce the crash when I set it to SCHED_OTHER.
>>> However when I set it to SCHED_OTHER the jitter is much larger, so we
>>> really want to get it running stable on SCHED_FIFO if possible.
>>
>> As expected, SCHED_OTHER won't give you any rt guarantee, hence the
>> jitter. A SCHED_OTHER thread driven by Xenomai is allowed to wait for
>> Xenomai resources (e.g. pending on a sema4, waiting on a condvar,
>> blocking on a message queue), but won't be able to compete with
>> SCHED_FIFO threads for the CPU.
>>
>>> Jitter is not currently measured precisely, but the difference is such
>> that
>>> it can be eyeballed on an oscilloscope watching the messages go back and
>>> forth.
>>> Looking at /proc/xenomai/stat this task tends to draw up to about 30% of
>>> CPU at full load.
>>>
>>
>> You mean when running into the SCHED_OTHER class? If so, the mode
>> switches might explain such load only if the thread calls a Xenomai
>> service from its work loop, which forces the transition to primary mode,
>> which is required for competing for Xenomai resources.
>>
>>
> I am using some XDDP writes as well as writes to non-RTDM hardware, so I
> suppose that I will get a handful of mode switches no matter which type of
> thread it is.
> 

Ok.

> More information is needed to better understand the issue, including
>> details about what the work loop is actually doing (see
>> http://xenomai.org/asking-for-help/).
>>
>> Enabling CONFIG_XENO_OPT_WATCHDOG might help detecting runaway threads
>> which belong to Xenomai's SCHED_FIFO class, but this won't apply to your
>> test with SCHED_OTHER.
>>
>> Maybe CONFIG_LOCKUP_DETECTOR might trigger in case a linux-driven thread
>> overconsumes the CPU (i.e. either a Xenomai SCHED_OTHER thread in linux
>> mode, or a regular pthread).
>>
>> It is definitely possible that your workload overconsumes CPU depending
>> on the available CPU horsepower on your platform, but we can't be sure
>> until we know more about your setup. This said, mode switching is a
>> costly operation, especially with Xenomai 2.x (compared to 3.x). So the
>> design should use only a very few of them in a normal work cycle if
>> absolutely required, and preferably none, unless the later is paced at a
>> reasonably slow rate in the timeline.
>>
>>
> I am seeing mode switches for that thread increasing at a rate of about 100
> for each time I look at /proc/xenomai/stat by hand (less than one second),
> so it seems like it's on the order of a couple or a few per cycle actually.
> I've been hoping to eliminate those somehow.
> 

In case you did not spot them all yet:
http://xenomai.org/2014/06/finding-spurious-relaxes/

> 
>> There are ways to exchange messages between a plain regular pthread and
>> a Xenomai SCHED_FIFO thread without involving any mode switch (see the
>> XDDP protocol from the RTIPC driver). Other ways exist to share
>> resources and/or synchronize between Xenomai SCHED_OTHER and SCHED_FIFO
>> threads, at the expense of mode switches for the former. The best pick
>> depends on the nature of your workload.
>>
>> I actually found XDDP to be very handy for another part of my application,
> but I hadn't thought of using it for the part where the main thread puts
> its messages out to hardware. I may look into it for this as well, but I
> have been thinking that porting the driver to RTDM is the actual best
> solution if it turns out the mode switches are part of the critical path.
> 

A dedicated RTDM driver is definitely the way to go for driving a device
in pure rt mode.

> I'm not sure yet how much I can disclose about what our main working thread
> is doing aside from "a whole lot of floating point operations." It is
> actually a bit of a black box to me as well currently. I understand that I
> can get the best help if I'm able to provide more info, but since I'm not
> sure, I was first hoping to just get some generic troubleshooting
> strategies so I can know where to look.
> 

Before anything else, I would definitely make sure that each and every
call to a blocking service in any rt work loop is checked for error
condition.

> My top suspicion now is that the main thread is occasionally hogging CPU
> and some percentage of the time it's enough to throw off Linux, but I am
> just trying to figure out if there's some way I can prove or disprove that
> easily. It seems to be independent of whether or not my other (non-RT)
> threads are even running at all.
>

This is my understanding as well.

> 
> At any rate, thanks very much for the suggestions!
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai
> 


-- 
Philippe.

Re: [Xenomai] POSIX skin, task crashes Linux when run as RT task

[Steve B]

A typical issue is an error reported by some blocking call - for
> instance when reading data from some RT device - going unnoticed by the
> application, which then iterates and so on. Normally, this causes a hard
> lockup due to the RT scheduling priority, but if your work loop also
> switches mode due to plain linux calls, then the system might just be
> brought to a crawl instead.
>
> That would explain why the Xenomai watchdog does not trigger either,
> since you do frequent transitions to secondary/linux mode, the real-time
> core does not detect the lockup condition which is N secs spent in
> primary/rt mode without transition to linux (N=4 by default).
>
>

> Before anything else, I would definitely make sure that each and every
> call to a blocking service in any rt work loop is checked for error
> condition.
>
>
After re-distributing the part of the main work loop that I suspected was
doing the mode switches (moved to another thread which I tried both
SCHED_FIFO and SCHED_OTHER), all of the mode switches just went along to
the other thread, the main loop does about 10 during initialization and
then never increases.
Interestingly this freed things up at crash time just enough to get this
sort of message on my dmesg:
"[sched_delayed] sched: RT throttling activated"

If I just prevent that routine from running, thus getting rid of all the
recurring mode switches, the crash still happens.

The blocking call error thing is interesting. I do have some supplementary
threads (all SCHED_FIFO) that are reading off of the hardware with blocking
calls and feeding messages to the main work thread to be processed.
Interestingly these blocking reads (to non-RTDM devices) are not causing
any mode switches, so my assumption was that these threads are actually
keeping in secondary mode once they get started.
However it looks like the reads are rigged to report when the return value
is less than zero, which I have not seen happen so far.

> My top suspicion now is that the main thread is occasionally hogging CPU
> > and some percentage of the time it's enough to throw off Linux, but I am
> > just trying to figure out if there's some way I can prove or disprove
> that
> > easily. It seems to be independent of whether or not my other (non-RT)
> > threads are even running at all.
> >
>
> This is my understanding as well.
>

Since my blocking reads are set up to report errors and exit program, and I
still see the problem when I eliminate all of the mode switches, it seems
this points to the work loop not leaving enough time for Linux to do what
it needs to do.. how we can fix that is another issue. Thanks for all of
the info, this has been very educational!

Steve

Re: [Xenomai] POSIX skin, task crashes Linux when run as RT task

[Philippe Gerum]

On 02/03/2015 05:58 PM, Steve B wrote:
> A typical issue is an error reported by some blocking call - for
>> instance when reading data from some RT device - going unnoticed by the
>> application, which then iterates and so on. Normally, this causes a hard
>> lockup due to the RT scheduling priority, but if your work loop also
>> switches mode due to plain linux calls, then the system might just be
>> brought to a crawl instead.
>>
>> That would explain why the Xenomai watchdog does not trigger either,
>> since you do frequent transitions to secondary/linux mode, the real-time
>> core does not detect the lockup condition which is N secs spent in
>> primary/rt mode without transition to linux (N=4 by default).
>>
>>
> 
>> Before anything else, I would definitely make sure that each and every
>> call to a blocking service in any rt work loop is checked for error
>> condition.
>>
>>
> After re-distributing the part of the main work loop that I suspected was
> doing the mode switches (moved to another thread which I tried both
> SCHED_FIFO and SCHED_OTHER), all of the mode switches just went along to
> the other thread, the main loop does about 10 during initialization and
> then never increases.
> Interestingly this freed things up at crash time just enough to get this
> sort of message on my dmesg:
> "[sched_delayed] sched: RT throttling activated"

see Documentation/scheduler/sched-rt-group.txt, specifically the
discussion about
/proc/sys/kernel/sched_rt_period_us and
/proc/sys/kernel/sched_rt_runtime_us. This throttling only applies to
regular pthreads, or Xenomai threads running in secondary mode. When
controlled by the Xenomai scheduler (i.e. from primary mode), a
real-time thread cannot be detected by this mechanism.

> 
> If I just prevent that routine from running, thus getting rid of all the
> recurring mode switches, the crash still happens.
> 
> The blocking call error thing is interesting. I do have some supplementary
> threads (all SCHED_FIFO) that are reading off of the hardware with blocking
> calls and feeding messages to the main work thread to be processed.
> Interestingly these blocking reads (to non-RTDM devices) are not causing
> any mode switches, so my assumption was that these threads are actually
> keeping in secondary mode once they get started.
> However it looks like the reads are rigged to report when the return value
> is less than zero, which I have not seen happen so far.
> 
>> My top suspicion now is that the main thread is occasionally hogging CPU
>>> and some percentage of the time it's enough to throw off Linux, but I am
>>> just trying to figure out if there's some way I can prove or disprove
>> that
>>> easily. It seems to be independent of whether or not my other (non-RT)
>>> threads are even running at all.
>>>
>>
>> This is my understanding as well.
>>
> 
> Since my blocking reads are set up to report errors and exit program, and I
> still see the problem when I eliminate all of the mode switches, it seems
> this points to the work loop not leaving enough time for Linux to do what
> it needs to do.. how we can fix that is another issue.

Now that you moved away all reasons for Xenomai FIFO threads for
switching to secondary mode from their work loop,
CONFIG_XENO_OPT_WATCHDOG should help you detect runaway threads
overconsuming CPU in primary mode.

 Thanks for all of
> the info, this has been very educational!
> 
> Steve
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai
> 


-- 
Philippe.

Re: [Xenomai] POSIX skin, task crashes Linux when run as RT task

[Steve B]

> >> I actually found XDDP to be very handy for another part of my
> application,
> > but I hadn't thought of using it for the part where the main thread puts
> > its messages out to hardware. I may look into it for this as well, but I
> > have been thinking that porting the driver to RTDM is the actual best
> > solution if it turns out the mode switches are part of the critical path.
> >
>
> A dedicated RTDM driver is definitely the way to go for driving a device
> in pure rt mode.
>
>
>
Phillippe, thanks for the help on trying to track this issue down!
I built a simple Xenomai application to over consume CPU sporadically, and
found that if I set it to 100% CPU it would trigger the watchdog as it was
supposed to, so my main work thread taking up a lot of CPU could not alone
be the root cause for this. In the minimum configuration of my real
software where I was getting the issue, the only other things running were
threads to listen to a UART and to listen to the CAN bus, which I was doing
with the regular old Linux device drivers because RTDM ones did not seem to
be available.
I was able to reproduce the crash with my simple CPU-burner application by
adding the UART and CAN bus routines to it, and I was able to make it not
crash by taking away the sporadic deadline misses. So I came to the
conclusion that the combination of the deadline miss and the UART/CAN
receiver loops was somehow causing my issue. Unfortunately I'm not exactly
sure how to pinpoint it any deeper than that.

I bit the bullet and spent a couple days to port the Beaglebone Black CAN
bus driver to work with RTDM, and after switching to that my problem seems
to have been solved.
I would like to offer to contribute my device driver to Xenomai if anybody
is interested in having it. Will check with my employer/customer and make
sure there isn't any red tape I have to go through and then I will start a
new thread on that later.

Steve