* rtnet rt_igb I210 not working with ARM
@ 2021-12-02 14:35 Yann Le Chevoir
2021-12-02 18:25 ` Jan Kiszka
0 siblings, 1 reply; 8+ messages in thread
From: Yann Le Chevoir @ 2021-12-02 14:35 UTC (permalink / raw)
To: xenomai
Hello,
I have an issue with rt_igb not working with an Intel I210 on ARM (see
system configuration below).
With Wireshark, looks like rtping is sending garbage on the network (see
below).
I notice there is no interrupt in /proc/xenomai/irq (see below).
Looks like other people got issues with I210 in the past but I am not sure
if they were similar and I am not able to find any conclusion.
Below, I show you how I am inserting the kernel module + how I am able to
have it working with igb (no realtime) and rt_fec (different driver for the
other Ethernet controller on the same board).
What can I do to solve this issue?
Is there anything else I can try to give you more data?
Thanks for your help,
Yann
########################
# System configuration #
########################
ARM cortexa9t2hf, quad core, 32bits.
Custom board inspired from an i.MX6Q SABRE Lite.
Two Ethernet controllers: Freescale FEC + Intel I210 on the PCIe bus.
Kernel: Freescale/linux-fslc v4.9.67
(https://github.com/Freescale/linux-fslc/tree/953c6e30c9701fda69ef08e2476c541dc4fb1453)
Xenomai: v3.0.9 and/or v3.0.12 (same results).
I-pipe: based on ipipe-core-4.9.51-arm-4.patch. It is slightly different
for gpc.c to applied on Freescale/linux-fslc v4.9.67.
/proc/cmdline does not seem relevant:
"console=ttymxc1,115200 earlyprintk=serial,ttymxc1,115200
root=PARTUUID=0002973b-03 rootwait ro ldo_active=on
video=mxcfb0:dev=hdmi,if=RGB24,bpp=32"
#####################################################################
# Wireshark trace when 10.0.0.1 (rtnet + rt_igb) tries to ping #
# 10.0.0.10 (Windows) #
# Note: there is no traffic when 10.0.0.1 does not do the rtping #
# all the trace is generated during the rtping (some seconds) #
#####################################################################
No. Time Source Destination Protocol Length Info
1 0.000000000 00.00.00 00.00.00 FC 98 Unknown frame
2 5.000397000 39.15.80 00.00.00 FC 98 Unknown frame
3 6.000411000 80:e3:f5:a8:88:e3 61:bf:b8:f1:61:bf 0xf5a8 98 Ethernet II
4 12.000545000 1e:14:21:27:1e:14 XagylCom_14:11:27 0x3127 98 Ethernet II
5 13.000501000 01:d8:fb:4f:01:dc fb:4f:01:ec:fb:4f 0xfb4f 98 Ethernet II
6 15.000537000 ImcNetwo_05:5c:ed 7a:e5:38:39:81:00 0x4358 98 Ethernet II
7 16.000589000 05:4f:6e:65:49:46 Nadatel_64:09:00 LLC 98 U F, func=UA; DSAP 0x6c Group, SSAP 0x64 Response
8 17.000552000 ad:00:07:74:69:6d 69:64:05:00:00:03 0x656f 98 Ethernet II
#################################################
# /proc/xenomai/irq after performing the rtping #
#################################################
# cat /proc/xenomai/irq
IRQ CPU0 CPU1 CPU2 CPU3
16: 865166 741239 1388290 2146141 [timer/0]
301: 0 0 0 0 rteth0
1032: 0 0 0 0 [sync]
1033: 0 1 1 1 [timer-ipi]
1034: 0 1 2 1 [reschedule]
1035: 0 0 0 0 [virtual]
1039: 1 0 31 0 [virtual]
######################################################################
# Interruptions are well bound to rt_igb. I insured that igb was not #
# loaded at boot time. #
######################################################################
# ls /sys/bus/pci/drivers/rt_igb/
0000:01:00.0 bind module new_id remove_id uevent unbind
##################################
# Inserting the module: #
# rtnet + rt_igb, *not* working #
##################################
# modprobe rt_igb
*** RTnet for Xenomai v3.0.9 ***
RTnet: initialising real-time networking
rt_igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.18-k
rt_igb: Copyright (c) 2007-2014 Intel Corporation.
rt_igb 0000:01:00.0: enabling device (0140 -> 0142)
RTnet: registered rteth0
rt_igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
rt_igb 0000:01:00.0: rteth0: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
rt_igb 0000:01:00.0: rteth0: PBA No: FFFFFF-0FF
rt_igb 0000:01:00.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
# modprobe rtudp
# modprobe rtpacket
# lsmod
Module Size Used by
rtpacket 9111 0
rtudp 14160 0
rtipv4 31184 1 rtudp
rt_igb 84337 0
rtnet 52246 4 rt_igb,rtipv4,rtpacket,rtudp
(fec 40019 0) <-- Other NIC (see below)
# rtifconfig rteth0 up 10.0.0.1
rt_igb: rteth0: igb: rteth0 NIC Link is Up 100 Mbps Full Duplex,
Flow Control: RX/TX
# rtroute add 10.0.0.10 FF:FF:FF:FF:FF:FF dev rteth0
# rtping 10.0.0.10
Real-time PING 10.0.0.10 56(84) bytes of data.
... (*not* working) ^C
--- 10.0.0.10 rtping statistics ---
5 packets transmitted, 0 received, 100% packet loss
worst case rtt = 0.0 us
#############################################
# Inserting the module: #
# igb (no rt), same Xenomai kernel, working #
#############################################
# modprobe igb
igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
igb: Copyright (c) 2007-2014 Intel Corporation.
igb 0000:01:00.0: enabling device (0140 -> 0142)
pps pps1: new PPS source ptp1
igb 0000:01:00.0: added PHC on eth1
igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:01:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
igb 0000:01:00.0: eth1: PBA No: FFFFFF-0FF
igb 0000:01:00.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
# lsmod
Module Size Used by
igb 145781 0
(fec 40019 0) <-- Other NIC (see below)
# ifconfig eth1 up 10.0.0.1
IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
8021q: adding VLAN 0 to HW filter on device eth1
igb 0000:01:00.0 eth1: igb: eth1 NIC Link is Up 100 Mbps Full Duplex,
Flow Control: RX/TX
IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
# ping 10.0.0.10
... (working)
########################################################
# Inserting the module: #
# rtnet + rt_fec (second Ethernet controller), working #
########################################################
# modprobe rt_fec
...
RTnet: registered rteth0
# modprobe rtudp
# modprobe rtpacket
# lsmod
Module Size Used by
rtpacket 9111 0
rtudp 14160 0
rtipv4 31184 1 rtudp
rt_fec 15832 1
rtnet 52182 4 rtipv4,rtpacket,rt_fec,rtudp
# rtifconfig rteth0 up 10.0.0.1
Generic PHY 2188000.ethernet-1:10: attached PHY driver [Generic PHY]
(mii_bus:phy_addr=2188000.ethernet-1:10, irq=-1)
# rtroute add 10.0.0.10 FF:FF:FF:FF:FF:FF dev rteth0
# rtping 10.0.0.10
... (working)
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: rtnet rt_igb I210 not working with ARM
2021-12-02 14:35 rtnet rt_igb I210 not working with ARM Yann Le Chevoir
@ 2021-12-02 18:25 ` Jan Kiszka
2021-12-02 19:42 ` Yann Le Chevoir
2022-01-05 23:39 ` Yann Le Chevoir
0 siblings, 2 replies; 8+ messages in thread
From: Jan Kiszka @ 2021-12-02 18:25 UTC (permalink / raw)
To: Yann Le Chevoir, xenomai
On 02.12.21 15:35, Yann Le Chevoir via Xenomai wrote:
> Hello,
>
> I have an issue with rt_igb not working with an Intel I210 on ARM (see
> system configuration below).
> With Wireshark, looks like rtping is sending garbage on the network (see
> below).
> I notice there is no interrupt in /proc/xenomai/irq (see below).
> Looks like other people got issues with I210 in the past but I am not sure
> if they were similar and I am not able to find any conclusion.
>
> Below, I show you how I am inserting the kernel module + how I am able to
> have it working with igb (no realtime) and rt_fec (different driver for the
> other Ethernet controller on the same board).
>
> What can I do to solve this issue?
> Is there anything else I can try to give you more data?
>
Could you check if that card works in a standard PC (x86-64) with RTnet?
Could you also check if latest 3.2.1 makes any difference? Not all fixes
on RTnet have been backported to very-old-stable.
Jan
--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: rtnet rt_igb I210 not working with ARM
2021-12-02 18:25 ` Jan Kiszka
@ 2021-12-02 19:42 ` Yann Le Chevoir
2022-01-05 23:39 ` Yann Le Chevoir
1 sibling, 0 replies; 8+ messages in thread
From: Yann Le Chevoir @ 2021-12-02 19:42 UTC (permalink / raw)
To: xenomai
> > Hello,
> >
> > I have an issue with rt_igb not working with an Intel I210 on ARM (see
> > system configuration below).
> > With Wireshark, looks like rtping is sending garbage on the network
> > (see below).
> > I notice there is no interrupt in /proc/xenomai/irq (see below).
> > Looks like other people got issues with I210 in the past but I am not
> > sure if they were similar and I am not able to find any conclusion.
> >
> > Below, I show you how I am inserting the kernel module + how I am able
> > to have it working with igb (no realtime) and rt_fec (different driver
> > for the other Ethernet controller on the same board).
> >
> > What can I do to solve this issue?
> > Is there anything else I can try to give you more data?
> >
>
> Could you check if that card works in a standard PC (x86-64) with RTnet?
> Could you also check if latest 3.2.1 makes any difference? Not all fixes on
> RTnet have been backported to very-old-stable.
>
> Jan
>
Thanks Jan,
Testing it in a x86-64 may be difficult because the Ethernet chip is soldered on our board, that's not an external card with a PCIe connector. I will check if we can do something.
I will give a new try with version 3.2.1. I already tried (and did not mention to not confuse anyone) before to send my question but it was not compiling. I will investigate the compilation issue is that's what you suggest. And come back to you.
Thanks,
Yann
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: rtnet rt_igb I210 not working with ARM
2021-12-02 18:25 ` Jan Kiszka
2021-12-02 19:42 ` Yann Le Chevoir
@ 2022-01-05 23:39 ` Yann Le Chevoir
2022-01-06 8:18 ` Jan Kiszka
1 sibling, 1 reply; 8+ messages in thread
From: Yann Le Chevoir @ 2022-01-05 23:39 UTC (permalink / raw)
To: xenomai
> > Hello,
> >
> > I have an issue with rt_igb not working with an Intel I210 on ARM (see
> > system configuration below).
> > With Wireshark, looks like rtping is sending garbage on the network
> > (see below).
> > I notice there is no interrupt in /proc/xenomai/irq (see below).
> > Looks like other people got issues with I210 in the past but I am not
> > sure if they were similar and I am not able to find any conclusion.
> >
> > Below, I show you how I am inserting the kernel module + how I am able
> > to have it working with igb (no realtime) and rt_fec (different driver
> > for the other Ethernet controller on the same board).
> >
> > What can I do to solve this issue?
> > Is there anything else I can try to give you more data?
> >
>
> Could you check if that card works in a standard PC (x86-64) with RTnet?
> Could you also check if latest 3.2.1 makes any difference? Not all fixes on
> RTnet have been backported to very-old-stable.
>
> Jan
Hello Jan, all,
Happy new year!
Sorry for the delay, I needed to update my Linux kernel
(Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be able
to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?
I notice that the type of interruption changed since the previous version:
we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".
And there is this new error: "Error 2 getting interrupt" which prevents to
have rteth0 configured (see below). That's different from last time.
I notice that the driver igb itself does not work with the new Xenomai
kernel (see below). That's also different from last time. I confirmed that
it works with the non-xenomai kernel though (same kernel 5.4.147 without
the I-pipe patch).
Note: to have the driver igb working with the Xenomai kernel, I can use
the legacy interrupts ("pci=nomsi" on the kernel command line).
Does this "error 2" sound familiar? Is there anything else I can try?
I did not try with a standard x86-64 PC yet. Do you think it may still
help? I have the feeling that this irq issue is very specific to my arm
SOC.
Thanks for your help!
Yann
#################################################
# Inserting the rt_igb module with Xenomai 3.2: #
# We have the error "Error 2 getting interrupt" #
#################################################
# modprobe rt_igb
*** RTnet for Xenomai v3.2.1 ***
RTnet: initialising real-time networking
rt_igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.18-k
rt_igb: Copyright (c) 2007-2014 Intel Corporation.
rt_igb 0000:01:00.0: enabling device (0140 -> 0142)
RTnet: registered rteth0
rt_igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
rt_igb 0000:01:00.0: rteth0: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
rt_igb 0000:01:00.0: rteth0: PBA No: FFFFFF-0FF
rt_igb 0000:01:00.0: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
# modprobe rtudp
# modprobe rtpacket
# rtifconfig rteth0 up 10.0.0.1
rt_igb 0000:01:00.0: Error 2 getting interrupt
########################################################
# Inserting the igb (no rt) module with Xenomai 3.2: #
# We have the kernel trace + the console hangs #
# (I understand that irq were not disabled - not sure) #
# Note: with "pci=nomsi", igb works. rt_igb does not #
########################################################
# modprobe igb
igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
igb: Copyright (c) 2007-2014 Intel Corporation.
igb 0000:01:00.0: enabling device (0140 -> 0142)
pps pps0: new PPS source ptp0
igb 0000:01:00.0: added PHC on eth0
igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
# ifconfig eth0 up 10.0.0.1
------------[ cut here ]------------
WARNING: CPU: 0 PID: 32 at kernel/irq/handle.c:153 __handle_irq_event_percpu+0x12c/0x130
irq 303 handler igb_msix_ring+0x0/0x38 [igb] enabled interrupts
...
I-pipe domain: Linux
Workqueue: events igb_watchdog_task [igb]
...
[<80bca0e4>] (warn_slowpath_fmt) from [<80176580>] (__handle_irq_event_percpu+0x12c/0x130)
[<80176580>] (__handle_irq_event_percpu) from [<801765b0>] (handle_irq_event_percpu+0x2c/0x80)
[<801765b0>] (handle_irq_event_percpu) from [<8017663c>] (handle_irq_event+0x38/0x5c)
[<8017663c>] (handle_irq_event) from [<8017b6a4>] (handle_edge_irq+0xac/0x200)
[<8017b6a4>] (handle_edge_irq) from [<801756c8>] (generic_handle_irq+0x24/0x34)
[<801756c8>] (generic_handle_irq) from [<80504b14>] (dw_handle_msi_irq+0x88/0xdc)
[<80504b14>] (dw_handle_msi_irq) from [<80504b8c>] (dw_chained_msi_isr+0x24/0x70)
[<80504b8c>] (dw_chained_msi_isr) from [<801c4b54>] (__ipipe_dispatch_irq+0x84/0x1f0)
[<801c4b54>] (__ipipe_dispatch_irq) from [<80114c84>] (__ipipe_grab_irq+0x64/0x7c)
[<80114c84>] (__ipipe_grab_irq) from [<804c8bb4>] (gic_handle_irq+0x4c/0x90)
[<804c8bb4>] (gic_handle_irq) from [<80101b8c>] (__irq_svc+0x6c/0x90)
[<80101b8c>] (__irq_svc) from [<7f006910>] (igb_rd32+0x20/0xb0 [igb])
[<7f006910>] (igb_rd32 [igb]) from [<7f02140c>] (igb_ptp_rx_hang+0x1c/0xa8 [igb])
[<7f02140c>] (igb_ptp_rx_hang [igb]) from [<7f00df6c>] (igb_watchdog_task+0x1a8/0x72c [igb])
[<7f00df6c>] (igb_watchdog_task [igb]) from [<80147670>] (process_one_work+0x204/0x458)
...
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: rtnet rt_igb I210 not working with ARM
2022-01-05 23:39 ` Yann Le Chevoir
@ 2022-01-06 8:18 ` Jan Kiszka
2022-01-28 23:31 ` Yann Le Chevoir
0 siblings, 1 reply; 8+ messages in thread
From: Jan Kiszka @ 2022-01-06 8:18 UTC (permalink / raw)
To: Yann Le Chevoir, xenomai
On 06.01.22 00:39, Yann Le Chevoir via Xenomai wrote:
>>> Hello,
>>>
>>> I have an issue with rt_igb not working with an Intel I210 on ARM (see
>>> system configuration below).
>>> With Wireshark, looks like rtping is sending garbage on the network
>>> (see below).
>>> I notice there is no interrupt in /proc/xenomai/irq (see below).
>>> Looks like other people got issues with I210 in the past but I am not
>>> sure if they were similar and I am not able to find any conclusion.
>>>
>>> Below, I show you how I am inserting the kernel module + how I am able
>>> to have it working with igb (no realtime) and rt_fec (different driver
>>> for the other Ethernet controller on the same board).
>>>
>>> What can I do to solve this issue?
>>> Is there anything else I can try to give you more data?
>>>
>>
>> Could you check if that card works in a standard PC (x86-64) with RTnet?
>> Could you also check if latest 3.2.1 makes any difference? Not all fixes on
>> RTnet have been backported to very-old-stable.
>>
>> Jan
>
> Hello Jan, all,
>
> Happy new year!
>
> Sorry for the delay, I needed to update my Linux kernel
> (Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be able
> to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?
Yes, everything up to and including 5.4 is I-pipe, 5.10 and later is
dovetail.
> I notice that the type of interruption changed since the previous version:
> we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".
Maybe the host controller of your SoC gained MSI-X support betwen 4.9
and 5.4? Then both the standard driver and rt_igb would try to use that
instead (MSI is legacy).
> And there is this new error: "Error 2 getting interrupt" which prevents to
> have rteth0 configured (see below). That's different from last time.
> I notice that the driver igb itself does not work with the new Xenomai
> kernel (see below). That's also different from last time. I confirmed that
> it works with the non-xenomai kernel though (same kernel 5.4.147 without
> the I-pipe patch).
5.4.147 - that is no official ipipe-arm release. Did you integrate the
patch yourself base on which I-pipe core revision? And how about
ipipe-core-5.4.151-arm-4?
> Note: to have the driver igb working with the Xenomai kernel, I can use
> the legacy interrupts ("pci=nomsi" on the kernel command line).
> Does this "error 2" sound familiar? Is there anything else I can try?
ENOENT - you could try search the code for sources of this error code.
But it may also come from the PCI or MSI core.
>
> I did not try with a standard x86-64 PC yet. Do you think it may still
> help? I have the feeling that this irq issue is very specific to my arm
> SOC.
Not unlikely. I'm afraid there is no way around digging for the source
of the error code. That can help to understand better what goes wrong
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 8+ messages in thread* RE: rtnet rt_igb I210 not working with ARM
2022-01-06 8:18 ` Jan Kiszka
@ 2022-01-28 23:31 ` Yann Le Chevoir
2022-02-07 7:18 ` Jan Kiszka
0 siblings, 1 reply; 8+ messages in thread
From: Yann Le Chevoir @ 2022-01-28 23:31 UTC (permalink / raw)
To: Jan Kiszka, xenomai
>>>> I have an issue with rt_igb not working with an Intel I210 on ARM
>>>> (see system configuration below).
>>>> With Wireshark, looks like rtping is sending garbage on the network
>>>> (see below).
>>>> I notice there is no interrupt in /proc/xenomai/irq (see below).
>>>> Looks like other people got issues with I210 in the past but I am
>>>> not sure if they were similar and I am not able to find any conclusion.
>>>>
>>>> Below, I show you how I am inserting the kernel module + how I am
>>>> able to have it working with igb (no realtime) and rt_fec (different
>>>> driver for the other Ethernet controller on the same board).
>>>>
>>>> What can I do to solve this issue?
>>>> Is there anything else I can try to give you more data?
>>>>
>>> Could you check if that card works in a standard PC (x86-64) with RTnet?
>>> Could you also check if latest 3.2.1 makes any difference? Not all
>>> fixes on RTnet have been backported to very-old-stable.
>>>
>>> Jan
>>
>> Sorry for the delay, I needed to update my Linux kernel
>> (Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be
>> able to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?
>
> Yes, everything up to and including 5.4 is I-pipe, 5.10 and later is dovetail.
>
>> I notice that the type of interruption changed since the previous version:
>> we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".
>
> Maybe the host controller of your SoC gained MSI-X support betwen 4.9 and
> 5.4? Then both the standard driver and rt_igb would try to use that instead
> (MSI is legacy).
>
>> And there is this new error: "Error 2 getting interrupt" which
>> prevents to have rteth0 configured (see below). That's different from last
> time.
>> I notice that the driver igb itself does not work with the new Xenomai
>> kernel (see below). That's also different from last time. I confirmed
>> that it works with the non-xenomai kernel though (same kernel 5.4.147
>> without the I-pipe patch).
>
> 5.4.147 - that is no official ipipe-arm release. Did you integrate the patch
> yourself base on which I-pipe core revision? And how about ipipe-core-
> 5.4.151-arm-4?
Hello Jan, all,
What I "always" did for kernels linux-fslc-imx
4.1.38 (Yocto Morty),
4.9.67 (Yocto Thud)
and 5.4.147 (Yocto Dunfell)
is that I tried to apply the closest arm I-pipe patch as possible:
- Given an example, I used ipipe-core-5.4.151-arm-4.patch to patch
Yocto Dunfell.
- Then, I manually resolved the conflicts.
- That went well (happy with the performances and the rtcan driver).
- Idea was to stay as close as possible of the official Yocto branch
we are using. Does that make sense?
By reading your answer, I wondered if I did right. So I also tried to
compile the kernel from the ipipe-arm repo:
- Branch ipipe/5.4.y.
- That compiles and boots.
- But, for the rt_igb, same issue as previous procedure:
"Error 2 getting interrupt".
>> Note: to have the driver igb working with the Xenomai kernel, I can
>> use the legacy interrupts ("pci=nomsi" on the kernel command line).
>> Does this "error 2" sound familiar? Is there anything else I can try?
>
> ENOENT - you could try search the code for sources of this error code.
> But it may also come from the PCI or MSI core.
>
>>
>> I did not try with a standard x86-64 PC yet. Do you think it may still
>> help? I have the feeling that this irq issue is very specific to my
>> arm SOC.
>
> Not unlikely. I'm afraid there is no way around digging for the source of the
> error code. That can help to understand better what goes wrong
I tried so many different versions.
I finally came back to my initial version:
linux-fslc-imx 4.9.67
+ ipipe-core-4.9.51-arm-4.patch (adapted)
+ Xenomai 3.0.9 or 3.0.12.
- With this one, at least, the non-Xenomai igb driver works with the
I-pipe patch.
- I figured out that by disabling MSI (pci=nomsi), the irq counter in
/proc/xenomai/irq started to increment for rt_igb.
- Then, I confirmed that the buffers (ICMP or ARP frames) were correct
(printk in the igb_main.c source code).
- I also realized that the RX was working.
The RTnet stack was preparing an ARP response to my Windows ping.
- Only the TX was not good (garbage on Wireshark).
I finally have something working with the patch bellow.
So:
- pci=nomsi to have the irq counter incrementing.
- dma_sync_single_for_device in igb_tx_map to have rt_igb sending
good data.
My questions:
What are your feeling about that?
Is dma_sync real-time? Is it a good fix/workaround?
Why do I need to add such a dma_sync with my arm SoC?
- I suspect that this driver would work on a regular x86...
- Can you tell if it works on other arm SoC?
Any suggestion to do work the driver with MSI instead of legacy interrupts?
Any suggestion to do work rt_igb with Kernel 5.4 + Xenomai 3.2?
The patch (igb_tx_map with Xenomai 3.0.12):
--- kernel.ori/drivers/net/drivers/igb/igb_main.c
+++ kernel/drivers/net/drivers/igb/igb_main.c
@@ -4095,6 +4095,7 @@
size = skb->len;
dma = rtskb_data_dma_addr(skb, 0);
+ dma_sync_single_for_device(tx_ring->dev, dma, size, DMA_BIDIRECTIONAL);
Thank you,
Yann
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: rtnet rt_igb I210 not working with ARM
2022-01-28 23:31 ` Yann Le Chevoir
@ 2022-02-07 7:18 ` Jan Kiszka
2022-03-29 18:42 ` Yann Le Chevoir
0 siblings, 1 reply; 8+ messages in thread
From: Jan Kiszka @ 2022-02-07 7:18 UTC (permalink / raw)
To: Yann Le Chevoir, xenomai
On 29.01.22 00:31, Yann Le Chevoir wrote:
>>>>> I have an issue with rt_igb not working with an Intel I210 on ARM
>>>>> (see system configuration below).
>>>>> With Wireshark, looks like rtping is sending garbage on the network
>>>>> (see below).
>>>>> I notice there is no interrupt in /proc/xenomai/irq (see below).
>>>>> Looks like other people got issues with I210 in the past but I am
>>>>> not sure if they were similar and I am not able to find any conclusion.
>>>>>
>>>>> Below, I show you how I am inserting the kernel module + how I am
>>>>> able to have it working with igb (no realtime) and rt_fec (different
>>>>> driver for the other Ethernet controller on the same board).
>>>>>
>>>>> What can I do to solve this issue?
>>>>> Is there anything else I can try to give you more data?
>>>>>
>>>> Could you check if that card works in a standard PC (x86-64) with RTnet?
>>>> Could you also check if latest 3.2.1 makes any difference? Not all
>>>> fixes on RTnet have been backported to very-old-stable.
>>>>
>>>> Jan
>>>
>>> Sorry for the delay, I needed to update my Linux kernel
>>> (Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be
>>> able to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?
>>
>> Yes, everything up to and including 5.4 is I-pipe, 5.10 and later is dovetail.
>>
>>> I notice that the type of interruption changed since the previous version:
>>> we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".
>>
>> Maybe the host controller of your SoC gained MSI-X support betwen 4.9 and
>> 5.4? Then both the standard driver and rt_igb would try to use that instead
>> (MSI is legacy).
>>
>>> And there is this new error: "Error 2 getting interrupt" which
>>> prevents to have rteth0 configured (see below). That's different from last
>> time.
>>> I notice that the driver igb itself does not work with the new Xenomai
>>> kernel (see below). That's also different from last time. I confirmed
>>> that it works with the non-xenomai kernel though (same kernel 5.4.147
>>> without the I-pipe patch).
>>
>> 5.4.147 - that is no official ipipe-arm release. Did you integrate the patch
>> yourself base on which I-pipe core revision? And how about ipipe-core-
>> 5.4.151-arm-4?
>
> Hello Jan, all,
>
> What I "always" did for kernels linux-fslc-imx
> 4.1.38 (Yocto Morty),
> 4.9.67 (Yocto Thud)
> and 5.4.147 (Yocto Dunfell)
> is that I tried to apply the closest arm I-pipe patch as possible:
> - Given an example, I used ipipe-core-5.4.151-arm-4.patch to patch
> Yocto Dunfell.
> - Then, I manually resolved the conflicts.
> - That went well (happy with the performances and the rtcan driver).
> - Idea was to stay as close as possible of the official Yocto branch
> we are using. Does that make sense?
I would not blindly use downstream kernels (Yocto, Freescale/NXP etc.)
unless you know why you need them.
> By reading your answer, I wondered if I did right. So I also tried to
> compile the kernel from the ipipe-arm repo:
> - Branch ipipe/5.4.y.
> - That compiles and boots.
> - But, for the rt_igb, same issue as previous procedure:
> "Error 2 getting interrupt".
>
>>> Note: to have the driver igb working with the Xenomai kernel, I can
>>> use the legacy interrupts ("pci=nomsi" on the kernel command line).
>>> Does this "error 2" sound familiar? Is there anything else I can try?
>>
>> ENOENT - you could try search the code for sources of this error code.
>> But it may also come from the PCI or MSI core.
>>
>>>
>>> I did not try with a standard x86-64 PC yet. Do you think it may still
>>> help? I have the feeling that this irq issue is very specific to my
>>> arm SOC.
>>
>> Not unlikely. I'm afraid there is no way around digging for the source of the
>> error code. That can help to understand better what goes wrong
>
> I tried so many different versions.
> I finally came back to my initial version:
> linux-fslc-imx 4.9.67
> + ipipe-core-4.9.51-arm-4.patch (adapted)
> + Xenomai 3.0.9 or 3.0.12.
> - With this one, at least, the non-Xenomai igb driver works with the
> I-pipe patch.
> - I figured out that by disabling MSI (pci=nomsi), the irq counter in
> /proc/xenomai/irq started to increment for rt_igb.
> - Then, I confirmed that the buffers (ICMP or ARP frames) were correct
> (printk in the igb_main.c source code).
> - I also realized that the RX was working.
> The RTnet stack was preparing an ARP response to my Windows ping.
> - Only the TX was not good (garbage on Wireshark).
>
> I finally have something working with the patch bellow.
> So:
> - pci=nomsi to have the irq counter incrementing.
> - dma_sync_single_for_device in igb_tx_map to have rt_igb sending
> good data.
>
> My questions:
> What are your feeling about that?
> Is dma_sync real-time? Is it a good fix/workaround?
In the best case, it's just a barrier (but there is already a wmb()). In
the worst case, Linux code is triggered that actually does more, e.g.
because DMA is not cache-coherent. You will have to analyze what is
actually done in that function on your platform.
> Why do I need to add such a dma_sync with my arm SoC?
> - I suspect that this driver would work on a regular x86...
> - Can you tell if it works on other arm SoC?
I don't think this was tried on ARM so far, but then the topic will
actually be rather SoC-specific.
> Any suggestion to do work the driver with MSI instead of legacy interrupts?
> Any suggestion to do work rt_igb with Kernel 5.4 + Xenomai 3.2?
>
> The patch (igb_tx_map with Xenomai 3.0.12):
> --- kernel.ori/drivers/net/drivers/igb/igb_main.c
> +++ kernel/drivers/net/drivers/igb/igb_main.c
> @@ -4095,6 +4095,7 @@
> size = skb->len;
>
> dma = rtskb_data_dma_addr(skb, 0);
> + dma_sync_single_for_device(tx_ring->dev, dma, size, DMA_BIDIRECTIONAL);
>
Comparing the code against the related Linux driver regarding DMA and
possibly also regarding configuration for MSI-X will be needed. There
will be no way around understanding how DMA works as well as how MSI-X
is delivered. Then it should become clearer what is missing in the
rt_igb driver or possibly also in ARM-enablement of Xenomai for that SoC.
Jan
--
Siemens AG, Technology
Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 8+ messages in thread* RE: rtnet rt_igb I210 not working with ARM
2022-02-07 7:18 ` Jan Kiszka
@ 2022-03-29 18:42 ` Yann Le Chevoir
0 siblings, 0 replies; 8+ messages in thread
From: Yann Le Chevoir @ 2022-03-29 18:42 UTC (permalink / raw)
To: xenomai
> > I finally have something working with the patch bellow.
> > So:
> > - pci=nomsi to have the irq counter incrementing.
> > - dma_sync_single_for_device in igb_tx_map to have rt_igb sending
> > good data.
> >
> > My questions:
> > What are your feeling about that?
> > Is dma_sync real-time? Is it a good fix/workaround?
>
> In the best case, it's just a barrier (but there is already a wmb()). In the worst
> case, Linux code is triggered that actually does more, e.g.
> because DMA is not cache-coherent. You will have to analyze what is actually
> done in that function on your platform.
>
> > Why do I need to add such a dma_sync with my arm SoC?
> > - I suspect that this driver would work on a regular x86...
> > - Can you tell if it works on other arm SoC?
>
> I don't think this was tried on ARM so far, but then the topic will actually be
> rather SoC-specific.
Hi,
Maybe worth mentioning for those who face the same problem:
I did not go deeper with the DMA yet. I was happy with the tx performances.
But I faced a different issue with the rx, with 1 packets over 500 which was lost at high frequency.
I got to do a similar patch (dma_sync) for the rx. See below (tx + rx). That works for me and performances are validated.
--- a/kernel/drivers/net/drivers/igb/igb_main.c
+++ b/kernel/drivers/net/drivers/igb/igb_main.c
@@ -4143,6 +4143,7 @@ static void igb_tx_map(struct igb_ring *tx_ring,
* We also need this memory barrier to make certain all of the
* status bits have been updated before next_to_watch is written.
*/
+ dma_sync_single_for_device(tx_ring->dev, dma, size, DMA_TO_DEVICE);
wmb();
if (skb->xmit_stamp)
@@ -4764,13 +4765,19 @@ static struct rtskb *igb_fetch_rx_buffer(struct igb_ring *rx_ring,
{
struct igb_rx_buffer *rx_buffer;
struct rtskb *skb;
+ dma_addr_t dma;
+ unsigned int size;
rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean];
skb = rx_buffer->skb;
prefetchw(skb->data);
+ dma = rtskb_data_dma_addr(skb, 0);
+ size = le16_to_cpu(rx_desc->wb.upper.length);
+ dma_sync_single_for_cpu(rx_ring->dev, dma, size, DMA_FROM_DEVICE);
+
/* pull the header of the skb in */
- rtskb_put(skb, le16_to_cpu(rx_desc->wb.upper.length));
+ rtskb_put(skb, size);
rx_buffer->skb = NULL;
rx_buffer->dma = 0;
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-03-29 18:42 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-12-02 14:35 rtnet rt_igb I210 not working with ARM Yann Le Chevoir
2021-12-02 18:25 ` Jan Kiszka
2021-12-02 19:42 ` Yann Le Chevoir
2022-01-05 23:39 ` Yann Le Chevoir
2022-01-06 8:18 ` Jan Kiszka
2022-01-28 23:31 ` Yann Le Chevoir
2022-02-07 7:18 ` Jan Kiszka
2022-03-29 18:42 ` Yann Le Chevoir
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.