[Xenomai] RPi: Possible solution for freezes

[Xenomai] RPi: Possible solution for freezes

[Geber Villa Fernández]

I was having the same problem of freezes in RPi (RPi3 in this case) for
quite some time as explained in the following topic (kernel 4.1.21 with
xenomai 3.0.3):
http://www.xenomai.org/pipermail/xenomai/2017-February/037090.html

I think that I achieved to solve the problem by disabling the FIQ
capabilities of the USB driver. There are problems by using the FIQ USB
driver with RT patches as shown in:
https://community.emlid.com/t/rpi-2-fiq-usb-driver-problems/508/8
https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=39951&start=75

The solution is as simple as adding the following parameters in the
"cmdline.txt" file in the boot partition:
dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0

If these parameters are loaded, introducing "dmesg | grep dwc_otg" in a
terminal (e.g. by SSH) will give you the following information:
[    0.511322] dwc_otg: FIQ disabled
[    0.511327] dwc_otg: NAK holdoff disabled
[    0.511331] dwc_otg: FIQ split-transaction FSM disabled

I was doing several tests which consist of copying several folders with
GBs of data from a PC to the RPi SD through WiFi (using rsync). With the
USB FIQ driver, RPi freezes after some time (less than an hour). Without
the USB FIQ driver, I have not had any freeze at the moment.

More info about this driver (USB FIQ) can be obtained in:
https://www.raspberrypi.org/forums/viewtopic.php?t=70437

Regards,
Geber

Re: [Xenomai] RPi: Possible solution for freezes

[Adam Vaughan]

Hi Gerber,

I haven't tried Xenomai on the RPi in a few years (I've been working
on BeagleBone Xenomai lately), but I have experimented with Xenomai
2.6 patches on the RPi 1 in the configuration you describe.

There's more detail in section 7.5 in my dissertation [1], but I
strongly suggest checking that the USB driver isn't affecting your
overall system latency numbers under load.  Under heavy USB Ethernet
usage with the FIQ turned off I measured system stalls of up to 15
milliseconds under Xenomai with an oscilloscope.

The only fix I had for this was to use PREEMPT_RT to lower the USB
driver thread priority.  At the time, I couldn't get PREEMPT_RT
patches to cleanly apply to the available Xenomai kernel patches, so I
ended up just using PREEMPT_RT exclusively.  This patch situation is
likely different these days.

-Adam

[1] https://deepblue.lib.umich.edu/handle/2027.42/111333

On Thu, Apr 27, 2017 at 2:17 AM, Geber Villa Fernández
<gebervf@gmail.com> wrote:
> I was having the same problem of freezes in RPi (RPi3 in this case) for
> quite some time as explained in the following topic (kernel 4.1.21 with
> xenomai 3.0.3):
> http://www.xenomai.org/pipermail/xenomai/2017-February/037090.html
>
> I think that I achieved to solve the problem by disabling the FIQ
> capabilities of the USB driver. There are problems by using the FIQ USB
> driver with RT patches as shown in:
> https://community.emlid.com/t/rpi-2-fiq-usb-driver-problems/508/8
> https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=39951&start=75
>
> The solution is as simple as adding the following parameters in the
> "cmdline.txt" file in the boot partition:
> dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0
>
> If these parameters are loaded, introducing "dmesg | grep dwc_otg" in a
> terminal (e.g. by SSH) will give you the following information:
> [    0.511322] dwc_otg: FIQ disabled
> [    0.511327] dwc_otg: NAK holdoff disabled
> [    0.511331] dwc_otg: FIQ split-transaction FSM disabled
>
> I was doing several tests which consist of copying several folders with
> GBs of data from a PC to the RPi SD through WiFi (using rsync). With the
> USB FIQ driver, RPi freezes after some time (less than an hour). Without
> the USB FIQ driver, I have not had any freeze at the moment.
>
> More info about this driver (USB FIQ) can be obtained in:
> https://www.raspberrypi.org/forums/viewtopic.php?t=70437
>
> Regards,
> Geber
>
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> https://xenomai.org/mailman/listinfo/xenomai

Re: [Xenomai] RPi: Possible solution for freezes

[Philippe Gerum]

On 04/27/2017 06:09 PM, Adam Vaughan wrote:
> Hi Gerber,
> 
> I haven't tried Xenomai on the RPi in a few years (I've been working
> on BeagleBone Xenomai lately), but I have experimented with Xenomai
> 2.6 patches on the RPi 1 in the configuration you describe.
> 
> There's more detail in section 7.5 in my dissertation [1], but I
> strongly suggest checking that the USB driver isn't affecting your
> overall system latency numbers under load.  Under heavy USB Ethernet
> usage with the FIQ turned off I measured system stalls of up to 15
> milliseconds under Xenomai with an oscilloscope.
> 
> The only fix I had for this was to use PREEMPT_RT to lower the USB
> driver thread priority.  At the time, I couldn't get PREEMPT_RT
> patches to cleanly apply to the available Xenomai kernel patches, so I
> ended up just using PREEMPT_RT exclusively.  This patch situation is
> likely different these days.

I don't think any regular driver thread priority could interfere with
the dual kernel operating in primary/real-time mode, that is actually a
selling point of the pipelined IRQ architecture which decouples the
regular kernel logic from the real-time core.

Using the I-pipe tracer combined with the -f option from the latency
test may give some hints about the issue.

-- 
Philippe.

Re: [Xenomai] RPi: Possible solution for freezes

[Philippe Gerum]

On 04/27/2017 06:27 PM, Philippe Gerum wrote:
> On 04/27/2017 06:09 PM, Adam Vaughan wrote:
>> Hi Gerber,
>>
>> I haven't tried Xenomai on the RPi in a few years (I've been working
>> on BeagleBone Xenomai lately), but I have experimented with Xenomai
>> 2.6 patches on the RPi 1 in the configuration you describe.
>>
>> There's more detail in section 7.5 in my dissertation [1], but I
>> strongly suggest checking that the USB driver isn't affecting your
>> overall system latency numbers under load.  Under heavy USB Ethernet
>> usage with the FIQ turned off I measured system stalls of up to 15
>> milliseconds under Xenomai with an oscilloscope.
>>
>> The only fix I had for this was to use PREEMPT_RT to lower the USB
>> driver thread priority.  At the time, I couldn't get PREEMPT_RT
>> patches to cleanly apply to the available Xenomai kernel patches, so I
>> ended up just using PREEMPT_RT exclusively.  This patch situation is
>> likely different these days.
> 
> I don't think any regular driver thread priority could interfere with
> the dual kernel operating in primary/real-time mode, that is actually a
> selling point of the pipelined IRQ architecture which decouples the
> regular kernel logic from the real-time core.
> 
> Using the I-pipe tracer combined with the -f option from the latency
> test may give some hints about the issue.
> 

Btw, there is a validated port of kernel 4.1 to the RPI Zero available
from git://git.xenomai.org/ipipe.git, vendors/beaglebone/ipipe-4.1.

-- 
Philippe.

Re: [Xenomai] RPi: Possible solution for freezes

[Adam Vaughan]

> I don't think any regular driver thread priority could interfere with
> the dual kernel operating in primary/real-time mode, that is actually a
> selling point of the pipelined IRQ architecture which decouples the
> regular kernel logic from the real-time core.

That's a good point.

I'm looking at my old code with a fresh set of eyes, and I think I
found a possible cause.  In debugging months prior I had apparently
switched from using ioctl_rt to ioctl_nrt, but never reverted back.
The latency numbers were apparently fine until I saw an influence of
heavy network usage.

On Thu, Apr 27, 2017 at 9:32 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> On 04/27/2017 06:27 PM, Philippe Gerum wrote:
>> On 04/27/2017 06:09 PM, Adam Vaughan wrote:
>>> Hi Gerber,
>>>
>>> I haven't tried Xenomai on the RPi in a few years (I've been working
>>> on BeagleBone Xenomai lately), but I have experimented with Xenomai
>>> 2.6 patches on the RPi 1 in the configuration you describe.
>>>
>>> There's more detail in section 7.5 in my dissertation [1], but I
>>> strongly suggest checking that the USB driver isn't affecting your
>>> overall system latency numbers under load.  Under heavy USB Ethernet
>>> usage with the FIQ turned off I measured system stalls of up to 15
>>> milliseconds under Xenomai with an oscilloscope.
>>>
>>> The only fix I had for this was to use PREEMPT_RT to lower the USB
>>> driver thread priority.  At the time, I couldn't get PREEMPT_RT
>>> patches to cleanly apply to the available Xenomai kernel patches, so I
>>> ended up just using PREEMPT_RT exclusively.  This patch situation is
>>> likely different these days.
>>
>> I don't think any regular driver thread priority could interfere with
>> the dual kernel operating in primary/real-time mode, that is actually a
>> selling point of the pipelined IRQ architecture which decouples the
>> regular kernel logic from the real-time core.
>>
>> Using the I-pipe tracer combined with the -f option from the latency
>> test may give some hints about the issue.
>>
>
> Btw, there is a validated port of kernel 4.1 to the RPI Zero available
> from git://git.xenomai.org/ipipe.git, vendors/beaglebone/ipipe-4.1.
>
> --
> Philippe.

Re: [Xenomai] RPi: Possible solution for freezes

[Philippe Gerum]

On 04/27/2017 07:24 PM, Adam Vaughan wrote:
>> I don't think any regular driver thread priority could interfere with
>> the dual kernel operating in primary/real-time mode, that is actually a
>> selling point of the pipelined IRQ architecture which decouples the
>> regular kernel logic from the real-time core.
> 
> That's a good point.
> 
> I'm looking at my old code with a fresh set of eyes, and I think I
> found a possible cause.  In debugging months prior I had apparently
> switched from using ioctl_rt to ioctl_nrt, but never reverted back.
> The latency numbers were apparently fine until I saw an influence of
> heavy network usage.
> 

Ack. Downgrading to regular non-rt mode with a call to ioctl_nrt would
definitely explain the figures, which would be those delivered by a
stock kernel by definition, not Xenomai's.

-- 
Philippe.