Question about a freeze under Xenomai 3.2.1

Question about a freeze under Xenomai 3.2.1

[Julien Aube]

Hello,


I'm currently porting an old code from Xenomai 2 to Xenomai 3/Dovetail.

I'm using kernel 5.10.177-dovetail1 as this is the latest version I 
could work with xenomai 3.2.1.
The driver I'm porting makes heavy use of RTDM.

The platform is a Zynq 7030 (dual cortex-9).

I uses buildroot 2022.11 to build.

I hit a strange issue and ask for an advice before having to setup JTAG 
/ opencd , as you may have seen
the same issue before.

On the aforementioned platform , everything goes well as long as I do 
not enable SMP.
If I do, what happens is that Ethernet's phy detection hangup forever :

> Registering platform device 'Fixed MDIO bus.0'. Parent at platform
> device: 'Fixed MDIO bus.0': device_add
> bus: 'platform': add device Fixed MDIO bus.0
> device: 'fixed-0': device_add
> bus: 'mdio_bus': add driver Micrel KS8737
> [...]
> bus: 'mdio_bus': add driver Microchip KSZ9477
> bus: 'platform': add driver macb
> bus: 'platform': driver_probe_device: matched device e000c000.ethernet 
> with driver macb
> bus: 'platform': really_probe: probing driver macb with device 
> e000c000.ethernet
> macb e000c000.ethernet: no pinctrl handle
> device: 'e000c000.ethernet-ffffffff': device_add

If I disable SMP in the linux config kernel , here is what I have :

> Registering platform device 'Fixed MDIO bus.0'. Parent at platform
> device: 'Fixed MDIO bus.0': device_add
> bus: 'platform': add device Fixed MDIO bus.0
> device: 'fixed-0': device_add
> bus: 'mdio_bus': add driver Micrel KS8737
> [...]
> bus: 'mdio_bus': add driver Microchip KSZ9477
> bus: 'platform': add driver macb
> bus: 'platform': driver_probe_device: matched device e000c000.ethernet 
> with driver macb
> bus: 'platform': really_probe: probing driver macb with device 
> e000c000.ethernet
> macb e000c000.ethernet: no pinctrl handle
> device: 'e000c000.ethernet-ffffffff': device_add
> device: 'e000c000.ethernet-ffffffff:01': device_add
> bus: 'mdio_bus': add device e000c000.ethernet-ffffffff:01
> bus: 'mdio_bus': driver_probe_device: matched device 
> e000c000.ethernet-ffffffff:01 with driver Micrel KSZ9031 Gigabit PHY
> bus: 'mdio_bus': really_probe: probing driver Micrel KSZ9031 Gigabit 
> PHY with device e000c000.ethernet-ffffffff:01
> Micrel KSZ9031 Gigabit PHY e000c000.ethernet-ffffffff:01: no pinctrl 
> handle
> driver: 'Micrel KSZ9031 Gigabit PHY': driver_bound: bound to device 
> 'e000c000.ethernet-ffffffff:01'
> bus: 'mdio_bus': really_probe: bound device 
> e000c000.ethernet-ffffffff:01 to driver Micrel KSZ9031 Gigabit PHY
> device: 'eth0': device_add
> macb e000c000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000c000 
> irq 24 (xx:xx:xx:xx:xx:xx)

Kernel command line is rather simple :

 > console=ttyPS0,115200 earlyprintk root=/dev/nfs rw 
nfsroot=192.168.101.56:/home/rootfs/rootfs_2022,vers=4,tcp ip=dhcp


=> Is this something you already faced ?
If not , do you have any insight on debugging method besides JTAG ?

Thanks


Julien Aube

Re: Question about a freeze under Xenomai 3.2.1

[Jan Kiszka]

On 03.05.23 15:43, Julien Aube wrote:
> Hello,
> 
> 
> I'm currently porting an old code from Xenomai 2 to Xenomai 3/Dovetail.
> 
> I'm using kernel 5.10.177-dovetail1 as this is the latest version I
> could work with xenomai 3.2.1.

Well, 3.2.3 would also work with 5.15.y.

> The driver I'm porting makes heavy use of RTDM.
> 
> The platform is a Zynq 7030 (dual cortex-9).
> 
> I uses buildroot 2022.11 to build.
> 
> I hit a strange issue and ask for an advice before having to setup JTAG
> / opencd , as you may have seen
> the same issue before.
> 
> On the aforementioned platform , everything goes well as long as I do
> not enable SMP.
> If I do, what happens is that Ethernet's phy detection hangup forever :
> 
>> Registering platform device 'Fixed MDIO bus.0'. Parent at platform
>> device: 'Fixed MDIO bus.0': device_add
>> bus: 'platform': add device Fixed MDIO bus.0
>> device: 'fixed-0': device_add
>> bus: 'mdio_bus': add driver Micrel KS8737
>> [...]
>> bus: 'mdio_bus': add driver Microchip KSZ9477
>> bus: 'platform': add driver macb
>> bus: 'platform': driver_probe_device: matched device e000c000.ethernet
>> with driver macb
>> bus: 'platform': really_probe: probing driver macb with device
>> e000c000.ethernet
>> macb e000c000.ethernet: no pinctrl handle
>> device: 'e000c000.ethernet-ffffffff': device_add
> 
> If I disable SMP in the linux config kernel , here is what I have :
> 
>> Registering platform device 'Fixed MDIO bus.0'. Parent at platform
>> device: 'Fixed MDIO bus.0': device_add
>> bus: 'platform': add device Fixed MDIO bus.0
>> device: 'fixed-0': device_add
>> bus: 'mdio_bus': add driver Micrel KS8737
>> [...]
>> bus: 'mdio_bus': add driver Microchip KSZ9477
>> bus: 'platform': add driver macb
>> bus: 'platform': driver_probe_device: matched device e000c000.ethernet
>> with driver macb
>> bus: 'platform': really_probe: probing driver macb with device
>> e000c000.ethernet
>> macb e000c000.ethernet: no pinctrl handle
>> device: 'e000c000.ethernet-ffffffff': device_add
>> device: 'e000c000.ethernet-ffffffff:01': device_add
>> bus: 'mdio_bus': add device e000c000.ethernet-ffffffff:01
>> bus: 'mdio_bus': driver_probe_device: matched device
>> e000c000.ethernet-ffffffff:01 with driver Micrel KSZ9031 Gigabit PHY
>> bus: 'mdio_bus': really_probe: probing driver Micrel KSZ9031 Gigabit
>> PHY with device e000c000.ethernet-ffffffff:01
>> Micrel KSZ9031 Gigabit PHY e000c000.ethernet-ffffffff:01: no pinctrl
>> handle
>> driver: 'Micrel KSZ9031 Gigabit PHY': driver_bound: bound to device
>> 'e000c000.ethernet-ffffffff:01'
>> bus: 'mdio_bus': really_probe: bound device
>> e000c000.ethernet-ffffffff:01 to driver Micrel KSZ9031 Gigabit PHY
>> device: 'eth0': device_add
>> macb e000c000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000c000
>> irq 24 (xx:xx:xx:xx:xx:xx)
> 
> Kernel command line is rather simple :
> 
>> console=ttyPS0,115200 earlyprintk root=/dev/nfs rw
> nfsroot=192.168.101.56:/home/rootfs/rootfs_2022,vers=4,tcp ip=dhcp
> 
> 
> => Is this something you already faced ?
> If not , do you have any insight on debugging method besides JTAG ?

First of all, can you reproduce the issue when disabling Xenomai? And
when also disabling dovetail? And when using the corresponding LTS
kernel as is?

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: Question about a freeze under Xenomai 3.2.1

[Julien Aube]

Hello,

Le 03/05/2023 à 16:57, Jan Kiszka a écrit :
> Well, 3.2.3 would also work with 5.15.y.


Thanks. Following this comment I tried the kernel 
5.15.109-dovetail1-rebased on all the following tests.
I just had to update a few unrelated things like the DTS.

The behaviour with 5.15 is exactly the same.


> First of all, can you reproduce the issue when disabling Xenomai? And
> when also disabling dovetail? And when using the corresponding LTS
> kernel as is?


Here is what I have tested:

* LTS kernel (5.10.177 & 5.15.109) : works
* Dovetail kernel (5.10.177-dovetail1 & 5.15.109-dovetail1-rebase)  / 
xenomai 3.2.1
     * with CONFIG_DOVETAIL=n + CONFIG_SMP=y : works
     * with CONFIG_DOVETAIL=y + CONFIG_XENOMAI=n + CONFIG_SMP=y : works
     * with CONFIG_DOVETAIL=y + CONFIG_XENOMAI=y + CONFIG_SMP=n : works
     * with CONFIG_DOVETAIL=y + CONFIG_XENOMAI=y + CONFIG_SMP=y : fails

As an additional test, I tried with CONFIG_SMP=y but nr_cpus=1 on the 
cmdline.
In this configuration the failure still happens.


=> For me there is a problem specific to the SMP behaviour, since the 
tests are working fine in the non SMP case.
It might be that I do not understand the full extend of the SMP vs 
xenomai setting though.

As a sidenote I've added all I the debug flags I could think of.
Unfortunaly I can't use KDB since my serial console (usb-based on 
motherboard) does not translate a break correctly and I cannot fall into 
kdb.


Julien

Re: Question about a freeze under Xenomai 3.2.1

[Jan Kiszka]

On 04.05.23 15:33, Julien Aube wrote:
> Hello,
> 
> Le 03/05/2023 à 16:57, Jan Kiszka a écrit :
>> Well, 3.2.3 would also work with 5.15.y.
> 
> 
> Thanks. Following this comment I tried the kernel
> 5.15.109-dovetail1-rebased on all the following tests.
> I just had to update a few unrelated things like the DTS.
> 
> The behaviour with 5.15 is exactly the same.
> 
> 
>> First of all, can you reproduce the issue when disabling Xenomai? And
>> when also disabling dovetail? And when using the corresponding LTS
>> kernel as is?
> 
> 
> Here is what I have tested:
> 
> * LTS kernel (5.10.177 & 5.15.109) : works
> * Dovetail kernel (5.10.177-dovetail1 & 5.15.109-dovetail1-rebase)  /
> xenomai 3.2.1
>     * with CONFIG_DOVETAIL=n + CONFIG_SMP=y : works
>     * with CONFIG_DOVETAIL=y + CONFIG_XENOMAI=n + CONFIG_SMP=y : works
>     * with CONFIG_DOVETAIL=y + CONFIG_XENOMAI=y + CONFIG_SMP=n : works
>     * with CONFIG_DOVETAIL=y + CONFIG_XENOMAI=y + CONFIG_SMP=y : fails
> 
> As an additional test, I tried with CONFIG_SMP=y but nr_cpus=1 on the
> cmdline.
> In this configuration the failure still happens.
> 
> 
> => For me there is a problem specific to the SMP behaviour, since the
> tests are working fine in the non SMP case.
> It might be that I do not understand the full extend of the SMP vs
> xenomai setting though.
> 
> As a sidenote I've added all I the debug flags I could think of.
> Unfortunaly I can't use KDB since my serial console (usb-based on
> motherboard) does not translate a break correctly and I cannot fall into
> kdb.
> 

Is there any own driver involved, or can you reproduce the issue without
it as well?

I'm furthermore no longer sure if the Zynq 7030 is officially / fully
supported by dovetail. Maybe Philippe can comment on that.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: Question about a freeze under Xenomai 3.2.1

[Julien Aube]

> Is there any own driver involved, or can you reproduce the issue without
> it as well?

I could not reproduce the issue since for now I'm booting on an NFS 
filesystem, and the hang happens when the GEM driver does the detection 
of the phy (thus maybe the issue is rather in the phy layer rather than 
the ethernet driver).

The crash might happen elsewere but I have not seen it.

I'll try to create a setup were the rootfs is on the storage, disable 
ethernet driver and see what happens. Thanks for the idea.

> I'm furthermore no longer sure if the Zynq 7030 is officially / fully
> supported by dovetail. Maybe Philippe can comment on that.

It's a dual cortex-a9 , and it was working in xenomai 2 / kernel 3.9 , 
so if it's not working I hope the issue is a small one.

In the meantime I'll see what I can do with a jtag probe.

Thanks !

Julien

Re: Question about a freeze under Xenomai 3.2.1

[Greg Gallagher]

On Thu, May 4, 2023 at 10:27 AM Julien Aube <julien.aube@smile.fr> wrote:
>
> > Is there any own driver involved, or can you reproduce the issue without
> > it as well?
>
> I could not reproduce the issue since for now I'm booting on an NFS
> filesystem, and the hang happens when the GEM driver does the detection
> of the phy (thus maybe the issue is rather in the phy layer rather than
> the ethernet driver).
>
> The crash might happen elsewere but I have not seen it.
>
> I'll try to create a setup were the rootfs is on the storage, disable
> ethernet driver and see what happens. Thanks for the idea.
>
> > I'm furthermore no longer sure if the Zynq 7030 is officially / fully
> > supported by dovetail. Maybe Philippe can comment on that.
>
> It's a dual cortex-a9 , and it was working in xenomai 2 / kernel 3.9 ,
> so if it's not working I hope the issue is a small one.
>
> In the meantime I'll see what I can do with a jtag probe.
>
> Thanks !
>
> Julien
>
>
I think we saw this issue a while ago.  I can't remember if a proper
patch was submitted.  Let me take a look at my setup here.  I test
ipipe on both zynq7000 and Ultrascale boards.  I had a dovetail
environment to test zynq 7000 since there were some small issues
reported in the past. I should have some time to boot something
similar this weekend and see.

-Greg

Re: Question about a freeze under Xenomai 3.2.1

[Julien Aube]

Le 04/05/2023 à 17:00, Greg Gallagher a écrit :
> I think we saw this issue a while ago.  I can't remember if a proper
> patch was submitted.  Let me take a look at my setup here.  I test
> ipipe on both zynq7000 and Ultrascale boards.  I had a dovetail
> environment to test zynq 7000 since there were some small issues
> reported in the past. I should have some time to boot something
> similar this weekend and see.
>
> -Greg

Thanks for your help.
As an additional test I tried a RedPitaya that I own (very useful tool 
btw) , based on a 7010. Same issue occurs in the same setup.

Interesting to know that some issues occurs in the past. I'll try to see 
if I can find some insights in the mailing list archive.

Julien

RE: Question about a freeze under Xenomai 3.2.1

[Schuman Eelco (DC-AE/ESW5)]

Hi,

>> I think we saw this issue a while ago.  I can't remember if a proper 
>> patch was submitted.  Let me take a look at my setup here.  I test 
>> ipipe on both zynq7000 and Ultrascale boards.  I had a dovetail 
>> environment to test zynq 7000 since there were some small issues 
>> reported in the past. I should have some time to boot something 
>> similar this weekend and see.
>>
>> -Greg

>Thanks for your help.
>As an additional test I tried a RedPitaya that I own (very useful tool
>btw) , based on a 7010. Same issue occurs in the same setup.?
>
>Interesting to know that some issues occurs in the past. I'll try to see if I can find some insights in the mailing list archive.
>
>Julien

I had an issue like this some time ago, also with a Zynq 7000, Linux Dovetail 5.15, Xenomai 3.2 and a Buildroot build.  Processes would freeze after executing
some kind of delay e.g. sleep(). It looked like the default timer for the Zynq did not generate timer events for the proxy tick interrupts. I'm now trying a setup with 
the Zynq global timer as source of events. That seems to run Ok so far.

Eelco Schuman

Re: Question about a freeze under Xenomai 3.2.1

[Jan Kiszka]

On 05.05.23 09:36, Schuman Eelco (DC-AE/ESW5) wrote:
> Hi,
> 
>>> I think we saw this issue a while ago.  I can't remember if a proper 
>>> patch was submitted.  Let me take a look at my setup here.  I test 
>>> ipipe on both zynq7000 and Ultrascale boards.  I had a dovetail 
>>> environment to test zynq 7000 since there were some small issues 
>>> reported in the past. I should have some time to boot something 
>>> similar this weekend and see.
>>>
>>> -Greg
> 
>> Thanks for your help.
>> As an additional test I tried a RedPitaya that I own (very useful tool
>> btw) , based on a 7010. Same issue occurs in the same setup.?
>>
>> Interesting to know that some issues occurs in the past. I'll try to see if I can find some insights in the mailing list archive.
>>
>> Julien
> 
> I had an issue like this some time ago, also with a Zynq 7000, Linux Dovetail 5.15, Xenomai 3.2 and a Buildroot build.  Processes would freeze after executing
> some kind of delay e.g. sleep(). It looked like the default timer for the Zynq did not generate timer events for the proxy tick interrupts. I'm now trying a setup with 
> the Zynq global timer as source of events. That seems to run Ok so far.
> 

Now I recall: There is very likely some adaptation needed for that
default timer /wrt dovetail. Someone would have to dig into the details,
looking left and right on other board/SoC enablements.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

Re: Question about a freeze under Xenomai 3.2.1

[Julien Aube]

Le 05/05/2023 à 15:21, Jan Kiszka a écrit :
>> I had an issue like this some time ago, also with a Zynq 7000, Linux Dovetail 5.15, Xenomai 3.2 and a Buildroot build.  Processes would freeze after executing
>> some kind of delay e.g. sleep(). It looked like the default timer for the Zynq did not generate timer events for the proxy tick interrupts. I'm now trying a setup with
>> the Zynq global timer as source of events. That seems to run Ok so far.
>>
> Now I recall: There is very likely some adaptation needed for that
> default timer /wrt dovetail. Someone would have to dig into the details,
> looking left and right on other board/SoC enablements.
>

Thank you .

I'll have a look on this. I've never plunged this far in the internal of 
xenomai, I just used it so far.
If I find something it may end up to a possible patch for zynq support.


Julien

Re: Question about a freeze under Xenomai 3.2.1

[Julien Aube]

Hello,

Le 05/05/2023 à 15:21, Jan Kiszka a écrit :
> On 05.05.23 09:36, Schuman Eelco (DC-AE/ESW5) wrote:
>> Hi,
>>
>>>> I think we saw this issue a while ago.  I can't remember if a proper
>>>> patch was submitted.  Let me take a look at my setup here.  I test
>>>> ipipe on both zynq7000 and Ultrascale boards.  I had a dovetail
>>>> environment to test zynq 7000 since there were some small issues
>>>> reported in the past. I should have some time to boot something
>>>> similar this weekend and see.
>>>>
>>>> -Greg
>> I had an issue like this some time ago, also with a Zynq 7000, Linux Dovetail 5.15, Xenomai 3.2 and a Buildroot build.  Processes would freeze after executing
>> some kind of delay e.g. sleep(). It looked like the default timer for the Zynq did not generate timer events for the proxy tick interrupts. I'm now trying a setup with
>> the Zynq global timer as source of events. That seems to run Ok so far.
>>
> Now I recall: There is very likely some adaptation needed for that
> default timer /wrt dovetail. Someone would have to dig into the details,
> looking left and right on other board/SoC enablements.


With your suggestions I finally had a working setup with SMP:

The culprit shown itself when I printed the original timer that is 
proxied by the timer_proxy,

On the non-SMP kernel the timer is set to "arm_global_timer", while on 
SMP it was set to "local_timer" :


 > IRQ pipeline: high-priority Xenomai stage added.
 > CPU0: proxy tick device registered (333.33MHz)
 > CPU0: proxy original clock: arm_global_timer

vs

 >CPU0: proxy original clock: local_timer
 >CPU1: proxy original clock: local_timer

 From what I understood, there is one timer per CPU , while the 
arm_global_timer is global to the system.

Digging a little more, I found out that the local_timer was implemented 
by the "scutimer" on the zynq7000 platform.
As a first approach, I just disabled this timer in the DTS.

With that the proxy original clock is now arm_glocal_timer for both CPU 
and it work.

However.... I have a bad feeling about this, it look like I did not go 
down to the real issue which is that the scutimer does not
generate timer events as it should.

Eelco , is that how you resolved the situation on your side ?

Many thanks for your help an tips for this issue,

Julien Aube

RE: Question about a freeze under Xenomai 3.2.1

[Schuman Eelco (DC-AE/ESW5)]

Hi,

>With your suggestions I finally had a working setup with SMP:

>The culprit shown itself when I printed the original timer that is 
>proxied by the timer_proxy,

>On the non-SMP kernel the timer is set to "arm_global_timer", while on SMP it was set to "local_timer" :


> From what I understood, there is one timer per CPU , while the 
>arm_global_timer is global to the system.

>Digging a little more, I found out that the local_timer was implemented 
>by the "scutimer" on the zynq7000 platform.
>As a first approach, I just disabled this timer in the DTS.
>
>With that the proxy original clock is now arm_glocal_timer for both CPU 
>and it work.

>However.... I have a bad feeling about this, it look like I did not go 
>down to the real issue which is that the scutimer does not generate 
>timer events as it should.

I disabled only the generation of clockevents for the default timer ( arch/arm/kernel/smp_twd.c ). Both timers are running but the global timer is now used for the clockevents. I'm not sure if this is the best solution or  if it has an impact on performance. 

static void twd_timer_setup(void)
{
	struct clock_event_device *clk = raw_cpu_ptr(twd_evt);
	int cpu = smp_processor_id();
...
...
	//clockevents_config_and_register(clk, twd_timer_rate,                do not use this timer for clockevents
	//				0xf, 0xffffffff);
	enable_percpu_irq(clk->irq, 0);
}

This is with linux-v5.10.153-dovetail1.

Eelco Schuman