[Xenomai] Losing Real Time Response During ARP Flood

[Xenomai] Losing Real Time Response During ARP Flood

[Steve Deiters]

We seem to be losing real time response during an ARP flood.  We recently had a misconfigured switch that caused an ARP flood and caused a lot of our devices to reboot.  After looking at it a bit we found the external watchdog circuit was resetting the boards.  We have a design where there is a low priority thread toggling a GPIO to service an external watchdog.  We’ve found that this thread is getting delayed significantly during the ARP flood.

The Ethernet driver and TCP stack is just the plain Linux ones.  We’ve had PCs running Windows and Linux that were on the network at the same time malfunction as well.  However with Xenomai I would think you would be somewhat guarded against any issues with Linux handling the networking.

We haven’t found any sure fire fix to get around this yet.  We can change some of the priorities around and it seems to alleviate the issue.  If we increase the priority of the watchdog thread it doesn’t happen.  We link against the POSIX skin so we have real time threads from all our pthread_create calls.  Anything relying on network communication I would expect to be running mostly in the Linux domain.  These mostly just receive data and pass it on to other real time threads through shared memory and memory queues.

We are trying to get any ideas on what may cause this behavior.  We would think a pure Linux Ethernet driver would not be able to hold off the real time threads.  We’ve speculated it may be purely due to CPU loading due to the high number of interrupts being processed, but we haven’t been able to prove this.

We’re going to try the I-Pipe tracer and our JTAG profiler next.  I was wondering if there are any ideas though on where to start looking.  Is there any way to rate limit an incoming interrupt source?

We are running a PowerPC platform with 2.6.33.5 Linux kernel and 2.5.4 Xenomai.


Any ideas are appreciated..


Steve Deiters
Senior Software Engineer
Basler Electric Company
12570 State Route 143
Highland, IL 62249-1074
United States
Phone: +1 618 654 2341 Ext. 406
Fax: +1 618 654 2351
Email: SteveDeiters@BASLER.com<mailto:SteveDeiters@BASLER.com>
www.basler.com



CONFIDENTIALITY NOTICE: This message may include confidential and/or legally protected information and is intended solely for the use of its intended recipient(s).  If you are not the intended recipient, any distribution or copying of this message (including any attachments) is strictly prohibited.  If you are not the intended recipient, please delete it (including any attachments) from your system without copying or forwarding it, and notify the sender of the error by reply e-mail.

Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it would be received, it is the responsibility of the recipient to ensure that the e-mail is virus free and no responsibility is accepted by Basler Electric Company for any damage or loss arising from its use.

Re: [Xenomai] Losing Real Time Response During ARP Flood

[Gilles Chanteperdrix]

On 02/15/2013 11:48 PM, Steve Deiters wrote:

> We seem to be losing real time response during an ARP flood.  We
> recently had a misconfigured switch that caused an ARP flood and
> caused a lot of our devices to reboot.  After looking at it a bit we
> found the external watchdog circuit was resetting the boards.  We
> have a design where there is a low priority thread toggling a GPIO to
> service an external watchdog.  We’ve found that this thread is
> getting delayed significantly during the ARP flood.
> 
> The Ethernet driver and TCP stack is just the plain Linux ones.
> We’ve had PCs running Windows and Linux that were on the network at
> the same time malfunction as well.  However with Xenomai I would
> think you would be somewhat guarded against any issues with Linux
> handling the networking.
> 
> We haven’t found any sure fire fix to get around this yet.  We can
> change some of the priorities around and it seems to alleviate the
> issue.  If we increase the priority of the watchdog thread it doesn’t
> happen.  We link against the POSIX skin so we have real time threads
> from all our pthread_create calls.  Anything relying on network
> communication I would expect to be running mostly in the Linux
> domain.  These mostly just receive data and pass it on to other real
> time threads through shared memory and memory queues.
> 
> We are trying to get any ideas on what may cause this behavior.  We
> would think a pure Linux Ethernet driver would not be able to hold
> off the real time threads.  We’ve speculated it may be purely due to
> CPU loading due to the high number of interrupts being processed, but
> we haven’t been able to prove this.
> 
> We’re going to try the I-Pipe tracer and our JTAG profiler next.  I
> was wondering if there are any ideas though on where to start
> looking.  Is there any way to rate limit an incoming interrupt
> source?
> 
> We are running a PowerPC platform with 2.6.33.5 Linux kernel and

> 2.5.4 Xenomai.

I am not sure I understand, if the watchdog is a plain linux task, then
naturally, it will be delayed by the ARP flood, with or without Xenomai.
So, I have to asssume that it is in fact a Xenomai task.

I can think of two ways of creating such a priority inversion:
- If you have root domain priority coupling enabled
(CONFIG_XENO_OPT_PRIOCPL), and a Xenomai thread running in secondary
mode has a higher priority than the watchdog thread;
- If a Xenomai thread running in secondary mode shares a mutex with the
watchdog, but the mutex does not have priority inheritance enabled.


-- 
                                                                Gilles.