3.18 RT freezes on Raspberry Pi 2

3.18 RT freezes on Raspberry Pi 2

[mikhail.avkhimenia]

Hello,

Recently, Raspberry Pi Foundation presented and started selling new Raspberry Pi 2 SBC. It’s based on BCM2709 processor, which is an upgraded version of BCM2708. The main difference is that old BCM2708 has one core and its architecture is ARMv6, while the new BCM2709 has 4 cores and ARMv7 architecture. Most of the peripheral blocks are the same - DMA, I2C, UART, etc. 

RT patch has been working very well (after fixing the sd driver a couple of years ago) with Raspberry Pi 1 and kernel 3.12. 
With the release of the new board Raspberry Pi Foundation also switched to the new kernel version - 3.18, support for Raspberry Pi 2 has been introduced in that version.
I applied patch-3.18.7-rt2 on RPi’s kernel fork and built two kernels - one for Raspberry Pi 1 and another one for Raspberry Pi 2.
On older Raspberry Pi 1 3.18.7-rt2 seems to work without any problems, I ran cyclictest and stress and it performed well. 
But on new Raspberry Pi 2 I’m experiencing frequent freezes. 

Steps to reproduce: 
1. Clone https://github.com/raspberrypi/linux , checkout 3.18.7 - 0be82f722c097340632a59b879fbfee9c6148f53
2. Apply patch-3.18.7-rt2
3. (optionally) apply “[RT PATCH] mmc: sdhci: don't provide hard irq handler”
4. Make menuconfig bcm2709_defconfig
5 Configure: full rt preemption, periodic ticks, performance governor as default and optionally preemption debugging
6. Compile kernel & modules
7. Copy Image to /boot/kernel7.img and arch/arm/boot/dts/bcm2709-rpi-2-b.dtb to /boot/bcm2709-rpi-2-b.dtb, copy modules.
Boot and run: sudo cyclictest --smp -p51 -m

The bad thing is that it freezes without any message, even with preemption debug information and lockup detection enabled.

Here’s how it happens:
1. I connect to Raspberry Pi 2 over serial
2. I connect to Raspberry Pi 2 over Ethernet or WiFi
3. In the ssh terminal I run “sudo cyclictest --smp -p51 -m”, it works for 30 seconds and the terminal hangs. LEDs on WiFi or Ethernet connection stop blinking.
4. Serial terminal is still working, but after running a couple of commands it also freezes. If I try to run cyclictest on serial in that case it mostly freezes immediately, but a couple of times it ran and displayed that a couple of kernel are unavailable and after some time also froze.

If I run cyclictest on serial connection it also freezes regardless if WiFi or Ethernet are connected.

If I run “sudo cyclictest --smp -p49 -m” it does not freeze. So seems like it’s only affected when you run something with priority equal or higher than the priority of peripheral interrupt handlers.

Even though affinity mask for peripheral interrupts shows “f” it seems that they all run on cpu0 by checking /proc/interrupts. I tried adding “isolcpus=0” to /boot/cmdline.txt and with it the kernel does not freeze on “sudo cyclictest --smp -p51 -m” anymore. But it still freezes occasionally while working, it just takes much longer now to catch the freeze (30-60 minutes).

I’m trying to understand what causes such behaviour, as most of the drivers for Raspberry Pi 1 and 2 are the same. I’ve ruled out mmc by completely disabling it in the kernel and running from USB flash. 

Any ideas what can be done to understand what is causing this?
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3.18 RT freezes on Raspberry Pi 2

[Jeff Epler]

It occurs to me that you could adjust the priority of IRQs (by sudo
chrt).  With cyclictest running at priority 49, adjust each irq down
from 50 to 48 in turn and see if you get your hang.  Once you have a
specific irq, perhaps that will in turn point toward an actual cause.

Jeff

Re: 3.18 RT freezes on Raspberry Pi 2

[mikhail.avkhimenia]

Thanks for the good hint!

I started setting priority 49 for interrupt threads (while running cyclictest -p49) and most of them showed no effect, but as soon as I did that to one of the three dwc_otg interrupts it hanged. 
Tried with the other two dwc_otg interrupt threads and also got a hang. 

I've found that dwc_otg driver was updated for BCM2709 and probably that's the source of a problem:
https://github.com/Clouded/linux-rt-rpi2/commits/rpi-3.18.7-rt2/drivers/usb/host/dwc_otg

-- 
Best regards,
Mikhail

05.03.2015, 16:01, "Jeff Epler" <jepler@unpythonic.net>:
>   It occurs to me that you could adjust the priority of IRQs (by sudo
>   chrt).  With cyclictest running at priority 49, adjust each irq down
>   from 50 to 48 in turn and see if you get your hang.  Once you have a
>   specific irq, perhaps that will in turn point toward an actual cause.
>
>   Jeff

-- 
Best regards,
Mikhail Avkhimenia
Emlid Ltd
emlid.com


05.03.2015, 16:01, "Jeff Epler" <jepler@unpythonic.net>:
>  It occurs to me that you could adjust the priority of IRQs (by sudo
>  chrt).  With cyclictest running at priority 49, adjust each irq down
>  from 50 to 48 in turn and see if you get your hang.  Once you have a
>  specific irq, perhaps that will in turn point toward an actual cause.
>
>  Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3.18 RT freezes on Raspberry Pi 2

[Julien Ciesla]

 <mikhail.avkhimenia <at> emlid.com> writes:

> 
> Hello,
> 
> Recently, Raspberry Pi Foundation presented and started selling new 
Raspberry Pi 2 SBC. It’s based on
> BCM2709 processor, which is an upgraded version of BCM2708. The main 
difference is that old BCM2708 has
> one core and its architecture is ARMv6, while the new BCM2709 has 4 cores 
and ARMv7 architecture. Most of
> the peripheral blocks are the same - DMA, I2C, UART, etc. 
> 
> RT patch has been working very well (after fixing the sd driver a couple 
of years ago) with Raspberry Pi 1 and
> kernel 3.12. 
> With the release of the new board Raspberry Pi Foundation also switched 
to the new kernel version - 3.18,
> support for Raspberry Pi 2 has been introduced in that version.
> I applied patch-3.18.7-rt2 on RPi’s kernel fork and built two kernels - 
one for Raspberry Pi 1 and another
> one for Raspberry Pi 2.
> On older Raspberry Pi 1 3.18.7-rt2 seems to work without any problems, I 
ran cyclictest and stress and it
> performed well. 
> But on new Raspberry Pi 2 I’m experiencing frequent freezes. 
> 
> Steps to reproduce: 
> 1. Clone https://github.com/raspberrypi/linux , checkout 3.18.7 - 
0be82f722c097340632a59b879fbfee9c6148f53
> 2. Apply patch-3.18.7-rt2
> 3. (optionally) apply “[RT PATCH] mmc: sdhci: don't provide hard irq 
handler”
> 4. Make menuconfig bcm2709_defconfig
> 5 Configure: full rt preemption, periodic ticks, performance governor as 
default and optionally
> preemption debugging
> 6. Compile kernel & modules
> 7. Copy Image to /boot/kernel7.img and arch/arm/boot/dts/bcm2709-rpi-
2-b.dtb to
> /boot/bcm2709-rpi-2-b.dtb, copy modules.
> Boot and run: sudo cyclictest --smp -p51 -m
> 
> The bad thing is that it freezes without any message, even with 
preemption debug information and lockup
> detection enabled.
> 
> Here’s how it happens:
> 1. I connect to Raspberry Pi 2 over serial
> 2. I connect to Raspberry Pi 2 over Ethernet or WiFi
> 3. In the ssh terminal I run “sudo cyclictest --smp -p51 -m”, it works 
for 30 seconds and the terminal
> hangs. LEDs on WiFi or Ethernet connection stop blinking.
> 4. Serial terminal is still working, but after running a couple of 
commands it also freezes. If I try to run
> cyclictest on serial in that case it mostly freezes immediately, but a 
couple of times it ran and displayed
> that a couple of kernel are unavailable and after some time also froze.
> 
> If I run cyclictest on serial connection it also freezes regardless if 
WiFi or Ethernet are connected.
> 
> If I run “sudo cyclictest --smp -p49 -m” it does not freeze. So seems 
like it’s only affected when you
> run something with priority equal or higher than the priority of 
peripheral interrupt handlers.
> 
> Even though affinity mask for peripheral interrupts shows “f” it seems 
that they all run on cpu0 by
> checking /proc/interrupts. I tried adding “isolcpus=0” 
to /boot/cmdline.txt and with it the
> kernel does not freeze on “sudo cyclictest --smp -p51 -m” anymore. But it 
still freezes occasionally
> while working, it just takes much longer now to catch the freeze (30-60 
minutes).
> 
> I’m trying to understand what causes such behaviour, as most of the 
drivers for Raspberry Pi 1 and 2 are the
> same. I’ve ruled out mmc by completely disabling it in the kernel and 
running from USB flash. 
> 
> Any ideas what can be done to understand what is causing this?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" 
in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 


Hello Mikhaïl, I have the same issue, let me now if you find anything.

Re: 3.18 RT freezes on Raspberry Pi 2

[Julien Ciesla]

Same issue on linux raspberrypi 3.18.9 with patch rt4.

Re: 3.18 RT freezes on Raspberry Pi 2

[Sebastian Andrzej Siewior]

* Julien Ciesla | 2015-03-09 01:55:49 [+0000]:

>Same issue on linux raspberrypi 3.18.9 with patch rt4.

Then disable USB completely.

Sebastian

Re: 3.18 RT freezes on Raspberry Pi 2

[Mikhail Avkhimenia]

Hello,

I think I found the solution. 
The problem is that new USB driver for Raspberry Pi uses FIQ in a manner that leads to lockups on SMP systems, while booting of the older non-FIQ driver was broken. 
The following patch fixes an error and allows to boot the non-FIQ driver:

---
 drivers/usb/host/dwc_otg/dwc_otg_hcd.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/drivers/usb/host/dwc_otg/dwc_otg_hcd.c b/drivers/usb/host/dwc_otg/dwc_otg_hcd.c
index 68534d4..6a006c5 100644
--- a/drivers/usb/host/dwc_otg/dwc_otg_hcd.c
+++ b/drivers/usb/host/dwc_otg/dwc_otg_hcd.c
@@ -1414,8 +1414,12 @@ static void assign_and_init_hc(dwc_otg_hcd_t * hcd, dwc_otg_qh_t * qh)
 	dwc_otg_hc_init(hcd->core_if, hc);
 
 	local_irq_save(flags);
-	local_fiq_disable();
-	fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+
+	if (fiq_enable) {
+		local_fiq_disable();
+		fiq_fsm_spin_lock(&hcd->fiq_state->lock);
+	}
+
 	/* Enable the top level host channel interrupt. */
 	intr_enable = (1 << hc->hc_num);
 	DWC_MODIFY_REG32(&hcd->core_if->host_if->host_global_regs->haintmsk, 0, intr_enable);
@@ -1423,8 +1427,12 @@ static void assign_and_init_hc(dwc_otg_hcd_t * hcd, dwc_otg_qh_t * qh)
 	/* Make sure host channel interrupts are enabled. */
 	gintmsk.b.hcintr = 1;
 	DWC_MODIFY_REG32(&hcd->core_if->core_global_regs->gintmsk, 0, gintmsk.d32);
-	fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
-	local_fiq_enable();
+
+	if (fiq_enable) {
+		fiq_fsm_spin_unlock(&hcd->fiq_state->lock);
+		local_fiq_enable();
+	}
+	
 	local_irq_restore(flags);
 	hc->qh = qh;
 }
--


To boot the with the non-FIQ driver add this to the /boot/cmdline.txt:
dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0
 

Best regards,
Mikhail Avkhimenia


09.03.2015, 17:13, "Sebastian Andrzej Siewior" <bigeasy@linutronix.de>:
> * Julien Ciesla | 2015-03-09 01:55:49 [+0000]:
>> Same issue on linux raspberrypi 3.18.9 with patch rt4.
>
> Then disable USB completely.
>
> Sebastian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 3.18 RT freezes on Raspberry Pi 2

[Julien Ciesla]

Hello Mikhail,

With my own branch 3.18.9, I applyed your solution with "patch-3.18.9-
rt5" but my Pi2 was not able to boot anymore. So I synced to the latest 
3.18.y commit on raspberry linux 
(https://github.com/raspberrypi/linux/commit/2e529746b6de7633ffd02c27c1f
70a1df30fe243), which is on linux 3.18.10. I updated the "patch-3.18.9-
rt5" to fit on the latest kernel and now, All is working well. Let me 
know if you want the updated RT5 patch for kernel 3.18.10.

Regards,

Julien Ciesla

Re: 3.18 RT freezes on Raspberry Pi 2

[Oliver]

Hi

Same issue in RPi 2 running linux 4.1.6 with PREEMPT_RT_FULL.

Kodi-video playback gives skipped frames and freezes after a few minutes
which does not happen when rt is disabled (with FIQ enabled or disabled, it
does not matter).

Turning off FIQ as suggested by Mikhail gives the expected behaviour for
cyclictest and also prevents the freezes in kodi which is great, so thanks
for that hint!

However, I still get skipped frames in kodi.

Is it possible there is another/related issue?

Oliver

Re: 3.18 RT freezes on Raspberry Pi 2

[Jeremy Jongepier]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

Hello Oliver,

On 09/27/2015 05:44 PM, Oliver wrote:
> Hi
> 
> Same issue in RPi 2 running linux 4.1.6 with PREEMPT_RT_FULL.
> 
> Kodi-video playback gives skipped frames and freezes after a few minutes
> which does not happen when rt is disabled (with FIQ enabled or disabled, it
> does not matter).
> 
> Turning off FIQ as suggested by Mikhail gives the expected behaviour for
> cyclictest and also prevents the freezes in kodi which is great, so thanks
> for that hint!

Try removing all FIQ related stuff from your cmdline.txt and just use
the defaults. Works fine for me with 4.1.7-rt8.

Jeremy

> 
> However, I still get skipped frames in kodi.
> 
> Is it possible there is another/related issue?
> 
> Oliver



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: 3.18 RT freezes on Raspberry Pi 2

[Oliver]

Hi Jeremy

> Try removing all FIQ related stuff from your cmdline.txt and just use
> the defaults. Works fine for me with 4.1.7-rt8.

Well I only ended up disabled FIQ because with it enabled I get the freezes
described by Mikhail (and kodi video playback freezes as well). With FIQ
disabled I non longer have the freezes, but the skipped frames (that I don't
get when rt-preempt is disabled) remain.

I will try 4.1.7.rt-8 though.

Oliver

Re: 3.18 RT freezes on Raspberry Pi 2

[Oliver]

Hi there

Tested again. I still get the behaviour described by Mikhail in Linux 4.1.10
using the 4.1.7-rt-8 patch (I am runningn OpenADK, http://www.openadk.org )

I.e.
$ cyclictest --smp -p51 -m
makes the rpi 2 freeze after a few minutes at most.

Adding dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0
to cmdline fixes this (the patch mentioned by Mikhail is already in place).

@ Jeremy,
If this does no longer happen to you in 4.1.7-rt-8, it must have been fixed
somewhere else. Any idea where?

In addition, as already mentioned:
While the freezes are fixed, I still get trouble with kodi video playback
with rt enabled. I found out that this only happens when playing over lan,
not when playing directly from the sd!
This may be due to the non-FIQ driver giving poorer network performance. But
it still only happens with rt enabled.

Any chance the incomnpatibility with the rt patch will be fixed in the FIQ
driver?

Oliver

Re: 3.18 RT freezes on Raspberry Pi 2

[Jeremy Jongepier]

[-- Attachment #1: Type: text/plain, Size: 262 bytes --]

Hi Oliver,

On 10/11/2015 02:28 PM, Oliver wrote:
> @ Jeremy,
> If this does no longer happen to you in 4.1.7-rt-8, it must have been fixed
> somewhere else. Any idea where?

Maybe the RPi firmware? When was the last time you ran rpi-update?

Jeremy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: 3.18 RT freezes on Raspberry Pi 2

[oliver]

Hi Jeremy

I was on an older revision.
But in ba7a8fb cyclictest still gives a freeze...
Any other ideas?

Oliver

Re: 3.18 RT freezes on Raspberry Pi 2

[Jeremy Jongepier]

[-- Attachment #1: Type: text/plain, Size: 202 bytes --]

On 10/11/2015 02:28 PM, Oliver wrote:
> $ cyclictest --smp -p51 -m
> makes the rpi 2 freeze after a few minutes at most.

Running now for more than 5 minutes and no freeze (4.1.7-rt8).

Jeremy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: 3.18 RT freezes on Raspberry Pi 2

[Oliver]

Strange...

What are you installing exactly on your SD?

Re: 3.18 RT freezes on Raspberry Pi 2

[Jeremy Jongepier]

[-- Attachment #1: Type: text/plain, Size: 151 bytes --]

Hello Oliver,

On 10/19/2015 05:33 PM, Oliver wrote:
> Strange...
> 
> What are you installing exactly on your SD?

Debian Jessie.

Jeremy


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: 3.18 RT freezes on Raspberry Pi 2

[Oliver]

Ok, I am going to try this.
Did you use this image:
http://sjoerd.luon.net/posts/2015/02/debian-jessie-on-rpi2/

I have installed it. It is on 3.18.5
How did you get to 4.1.7? Did you use some unstable repository/which one?
And then you installed 4.1.7-rt8 the way Mikhail described?

Oliver

Re: 3.18 RT freezes on Raspberry Pi 2

[Jeremy Jongepier]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]

Hello Oliver,

On 10/24/2015 01:18 PM, Oliver wrote:
> Ok, I am going to try this.
> Did you use this image:
> http://sjoerd.luon.net/posts/2015/02/debian-jessie-on-rpi2/

No, I installed it myself. Since the Raspberry Pi 2 uses aan ARMv7 CPU
you can install stock Debian on it.

> 
> I have installed it. It is on 3.18.5
> How did you get to 4.1.7?

I used the official Raspberry Pi Github kernel repository, applied the
RT patchset and then compiled the kernel and modules.

Jeremy





[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]