rtnet rt_igb I210 not working with ARM

rtnet rt_igb I210 not working with ARM

[Yann Le Chevoir]

Hello,

I have an issue with rt_igb not working with an Intel I210 on ARM (see
system configuration below).
With Wireshark, looks like rtping is sending garbage on the network (see
below).
I notice there is no interrupt in /proc/xenomai/irq (see below).
Looks like other people got issues with I210 in the past but I am not sure
if they were similar and I am not able to find any conclusion.

Below, I show you how I am inserting the kernel module + how I am able to
have it working with igb (no realtime) and rt_fec (different driver for the
other Ethernet controller on the same board).

What can I do to solve this issue?
Is there anything else I can try to give you more data?

Thanks for your help,

Yann

########################
# System configuration #
########################
ARM cortexa9t2hf, quad core, 32bits.
Custom board inspired from an i.MX6Q SABRE Lite.
Two Ethernet controllers: Freescale FEC + Intel I210 on the PCIe bus.
Kernel: Freescale/linux-fslc v4.9.67
(https://github.com/Freescale/linux-fslc/tree/953c6e30c9701fda69ef08e2476c541dc4fb1453)
Xenomai: v3.0.9 and/or v3.0.12 (same results).
I-pipe: based on ipipe-core-4.9.51-arm-4.patch. It is slightly different
        for gpc.c to applied on Freescale/linux-fslc v4.9.67.
/proc/cmdline does not seem relevant:
    "console=ttymxc1,115200 earlyprintk=serial,ttymxc1,115200
    root=PARTUUID=0002973b-03 rootwait ro ldo_active=on
    video=mxcfb0:dev=hdmi,if=RGB24,bpp=32"

#####################################################################
# Wireshark trace when 10.0.0.1 (rtnet + rt_igb) tries to ping      #
# 10.0.0.10 (Windows)                                               #
# Note: there is no traffic when 10.0.0.1 does not do the rtping    #
#       all the trace is generated during the rtping (some seconds) #
#####################################################################

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000000    00.00.00              00.00.00              FC       98     Unknown frame
      2 5.000397000    39.15.80              00.00.00              FC       98     Unknown frame
      3 6.000411000    80:e3:f5:a8:88:e3     61:bf:b8:f1:61:bf     0xf5a8   98     Ethernet II
      4 12.000545000   1e:14:21:27:1e:14     XagylCom_14:11:27     0x3127   98     Ethernet II
      5 13.000501000   01:d8:fb:4f:01:dc     fb:4f:01:ec:fb:4f     0xfb4f   98     Ethernet II
      6 15.000537000   ImcNetwo_05:5c:ed     7a:e5:38:39:81:00     0x4358   98     Ethernet II
      7 16.000589000   05:4f:6e:65:49:46     Nadatel_64:09:00      LLC      98     U F, func=UA; DSAP 0x6c Group, SSAP 0x64 Response
      8 17.000552000   ad:00:07:74:69:6d     69:64:05:00:00:03     0x656f   98     Ethernet II

#################################################
# /proc/xenomai/irq after performing the rtping #
#################################################

# cat /proc/xenomai/irq
  IRQ       CPU0        CPU1        CPU2        CPU3
   16:    865166      741239     1388290     2146141         [timer/0]
  301:         0           0           0           0         rteth0
 1032:         0           0           0           0         [sync]
 1033:         0           1           1           1         [timer-ipi]
 1034:         0           1           2           1         [reschedule]
 1035:         0           0           0           0         [virtual]
 1039:         1           0          31           0         [virtual]

######################################################################
# Interruptions are well bound to rt_igb. I insured that igb was not #
# loaded at boot time.                                               #
######################################################################

# ls /sys/bus/pci/drivers/rt_igb/
0000:01:00.0  bind  module  new_id  remove_id  uevent  unbind

##################################
# Inserting the module:          #
# rtnet + rt_igb, *not* working  #
##################################

# modprobe rt_igb
    *** RTnet for Xenomai v3.0.9 ***
    RTnet: initialising real-time networking
    rt_igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.18-k
    rt_igb: Copyright (c) 2007-2014 Intel Corporation.
    rt_igb 0000:01:00.0: enabling device (0140 -> 0142)
    RTnet: registered rteth0
    rt_igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
    rt_igb 0000:01:00.0: rteth0: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
    rt_igb 0000:01:00.0: rteth0: PBA No: FFFFFF-0FF
    rt_igb 0000:01:00.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
# modprobe rtudp
# modprobe rtpacket
# lsmod
    Module                  Size  Used by
    rtpacket                9111  0
    rtudp                  14160  0
    rtipv4                 31184  1 rtudp
    rt_igb                 84337  0
    rtnet                  52246  4 rt_igb,rtipv4,rtpacket,rtudp
    (fec                    40019  0) <-- Other NIC (see below)
# rtifconfig rteth0 up 10.0.0.1
    rt_igb: rteth0: igb: rteth0 NIC Link is Up 100 Mbps Full Duplex,
	                                             Flow Control: RX/TX
# rtroute add 10.0.0.10 FF:FF:FF:FF:FF:FF dev rteth0
# rtping 10.0.0.10
    Real-time PING 10.0.0.10 56(84) bytes of data.
    ... (*not* working) ^C
    --- 10.0.0.10 rtping statistics ---
    5 packets transmitted, 0 received, 100% packet loss
    worst case rtt = 0.0 us


#############################################
# Inserting the module:                     #
# igb (no rt), same Xenomai kernel, working #
#############################################

# modprobe igb
    igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
    igb: Copyright (c) 2007-2014 Intel Corporation.
    igb 0000:01:00.0: enabling device (0140 -> 0142)
    pps pps1: new PPS source ptp1
    igb 0000:01:00.0: added PHC on eth1
    igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
    igb 0000:01:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
    igb 0000:01:00.0: eth1: PBA No: FFFFFF-0FF
    igb 0000:01:00.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)
# lsmod
    Module                  Size  Used by
    igb                   145781  0
    (fec                    40019  0) <-- Other NIC (see below)
# ifconfig eth1 up 10.0.0.1
    IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
    8021q: adding VLAN 0 to HW filter on device eth1
    igb 0000:01:00.0 eth1: igb: eth1 NIC Link is Up 100 Mbps Full Duplex,
	                                                  Flow Control: RX/TX
    IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
# ping 10.0.0.10
    ... (working)


########################################################
# Inserting the module:                                #
# rtnet + rt_fec (second Ethernet controller), working #
########################################################

# modprobe rt_fec
    ...
    RTnet: registered rteth0
# modprobe rtudp
# modprobe rtpacket
# lsmod
    Module                  Size  Used by
    rtpacket                9111  0
    rtudp                  14160  0
    rtipv4                 31184  1 rtudp
    rt_fec                 15832  1
    rtnet                  52182  4 rtipv4,rtpacket,rt_fec,rtudp
# rtifconfig rteth0 up 10.0.0.1
    Generic PHY 2188000.ethernet-1:10: attached PHY driver [Generic PHY]
	                    (mii_bus:phy_addr=2188000.ethernet-1:10, irq=-1)
# rtroute add 10.0.0.10 FF:FF:FF:FF:FF:FF dev rteth0
# rtping 10.0.0.10
    ... (working)

Re: rtnet rt_igb I210 not working with ARM

[Jan Kiszka]

On 02.12.21 15:35, Yann Le Chevoir via Xenomai wrote:
> Hello,
> 
> I have an issue with rt_igb not working with an Intel I210 on ARM (see
> system configuration below).
> With Wireshark, looks like rtping is sending garbage on the network (see
> below).
> I notice there is no interrupt in /proc/xenomai/irq (see below).
> Looks like other people got issues with I210 in the past but I am not sure
> if they were similar and I am not able to find any conclusion.
> 
> Below, I show you how I am inserting the kernel module + how I am able to
> have it working with igb (no realtime) and rt_fec (different driver for the
> other Ethernet controller on the same board).
> 
> What can I do to solve this issue?
> Is there anything else I can try to give you more data?
> 

Could you check if that card works in a standard PC (x86-64) with RTnet?
Could you also check if latest 3.2.1 makes any difference? Not all fixes
on RTnet have been backported to very-old-stable.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

RE: rtnet rt_igb I210 not working with ARM

[Yann Le Chevoir]

> > Hello,
> >
> > I have an issue with rt_igb not working with an Intel I210 on ARM (see
> > system configuration below).
> > With Wireshark, looks like rtping is sending garbage on the network
> > (see below).
> > I notice there is no interrupt in /proc/xenomai/irq (see below).
> > Looks like other people got issues with I210 in the past but I am not
> > sure if they were similar and I am not able to find any conclusion.
> >
> > Below, I show you how I am inserting the kernel module + how I am able
> > to have it working with igb (no realtime) and rt_fec (different driver
> > for the other Ethernet controller on the same board).
> >
> > What can I do to solve this issue?
> > Is there anything else I can try to give you more data?
> >
> 
> Could you check if that card works in a standard PC (x86-64) with RTnet?
> Could you also check if latest 3.2.1 makes any difference? Not all fixes on
> RTnet have been backported to very-old-stable.
> 
> Jan
> 


Thanks Jan,

Testing it in a x86-64 may be difficult because the Ethernet chip is soldered on our board, that's not an external card with a PCIe connector.  I will check if we can do something.
I will give a new try with version 3.2.1. I already tried (and did not mention to not confuse anyone) before to send my question but it was not compiling. I will investigate the compilation issue is that's what you suggest. And come back to you.

Thanks,

Yann

RE: rtnet rt_igb I210 not working with ARM

[Yann Le Chevoir]

> > Hello,
> >
> > I have an issue with rt_igb not working with an Intel I210 on ARM (see
> > system configuration below).
> > With Wireshark, looks like rtping is sending garbage on the network
> > (see below).
> > I notice there is no interrupt in /proc/xenomai/irq (see below).
> > Looks like other people got issues with I210 in the past but I am not
> > sure if they were similar and I am not able to find any conclusion.
> >
> > Below, I show you how I am inserting the kernel module + how I am able
> > to have it working with igb (no realtime) and rt_fec (different driver
> > for the other Ethernet controller on the same board).
> >
> > What can I do to solve this issue?
> > Is there anything else I can try to give you more data?
> >
> 
> Could you check if that card works in a standard PC (x86-64) with RTnet?
> Could you also check if latest 3.2.1 makes any difference? Not all fixes on
> RTnet have been backported to very-old-stable.
> 
> Jan

Hello Jan, all,

Happy new year!

Sorry for the delay, I needed to update my Linux kernel
(Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be able
to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?
I notice that the type of interruption changed since the previous version:
we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".
And there is this new error: "Error 2 getting interrupt" which prevents to
have rteth0 configured (see below). That's different from last time.
I notice that the driver igb itself does not work with the new Xenomai
kernel (see below). That's also different from last time. I confirmed that
it works with the non-xenomai kernel though (same kernel 5.4.147 without
the I-pipe patch).
Note: to have the driver igb working with the Xenomai kernel, I can use
the legacy interrupts ("pci=nomsi" on the kernel command line).
Does this "error 2" sound familiar? Is there anything else I can try?

I did not try with a standard x86-64 PC yet. Do you think it may still
help? I have the feeling that this irq issue is very specific to my arm
SOC.

Thanks for your help!

Yann


#################################################
# Inserting the rt_igb module with Xenomai 3.2: #
# We have the error "Error 2 getting interrupt" #
#################################################

# modprobe rt_igb
    *** RTnet for Xenomai v3.2.1 ***
    RTnet: initialising real-time networking
    rt_igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.18-k
    rt_igb: Copyright (c) 2007-2014 Intel Corporation.
    rt_igb 0000:01:00.0: enabling device (0140 -> 0142)
    RTnet: registered rteth0
    rt_igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
    rt_igb 0000:01:00.0: rteth0: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
    rt_igb 0000:01:00.0: rteth0: PBA No: FFFFFF-0FF
    rt_igb 0000:01:00.0: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s)
# modprobe rtudp
# modprobe rtpacket
# rtifconfig rteth0 up 10.0.0.1
    rt_igb 0000:01:00.0: Error 2 getting interrupt


########################################################
# Inserting the igb (no rt) module with Xenomai 3.2:   #
# We have the kernel trace + the console hangs         #
# (I understand that irq were not disabled - not sure) #
# Note: with "pci=nomsi", igb works. rt_igb does not   #
########################################################

# modprobe igb
    igb: Intel(R) Gigabit Ethernet Network Driver - version 5.6.0-k
    igb: Copyright (c) 2007-2014 Intel Corporation.
    igb 0000:01:00.0: enabling device (0140 -> 0142)
    pps pps0: new PPS source ptp0
    igb 0000:01:00.0: added PHC on eth0
    igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
    igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:50:64:00:d5:10
    igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
    igb 0000:01:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
# ifconfig eth0 up 10.0.0.1
    ------------[ cut here ]------------
    WARNING: CPU: 0 PID: 32 at kernel/irq/handle.c:153 __handle_irq_event_percpu+0x12c/0x130
    irq 303 handler igb_msix_ring+0x0/0x38 [igb] enabled interrupts
    ...
    I-pipe domain: Linux
    Workqueue: events igb_watchdog_task [igb]
	...
    [<80bca0e4>] (warn_slowpath_fmt) from [<80176580>] (__handle_irq_event_percpu+0x12c/0x130)
    [<80176580>] (__handle_irq_event_percpu) from [<801765b0>] (handle_irq_event_percpu+0x2c/0x80)
    [<801765b0>] (handle_irq_event_percpu) from [<8017663c>] (handle_irq_event+0x38/0x5c)
    [<8017663c>] (handle_irq_event) from [<8017b6a4>] (handle_edge_irq+0xac/0x200)
    [<8017b6a4>] (handle_edge_irq) from [<801756c8>] (generic_handle_irq+0x24/0x34)
    [<801756c8>] (generic_handle_irq) from [<80504b14>] (dw_handle_msi_irq+0x88/0xdc)
    [<80504b14>] (dw_handle_msi_irq) from [<80504b8c>] (dw_chained_msi_isr+0x24/0x70)
    [<80504b8c>] (dw_chained_msi_isr) from [<801c4b54>] (__ipipe_dispatch_irq+0x84/0x1f0)
    [<801c4b54>] (__ipipe_dispatch_irq) from [<80114c84>] (__ipipe_grab_irq+0x64/0x7c)
    [<80114c84>] (__ipipe_grab_irq) from [<804c8bb4>] (gic_handle_irq+0x4c/0x90)
    [<804c8bb4>] (gic_handle_irq) from [<80101b8c>] (__irq_svc+0x6c/0x90)
    [<80101b8c>] (__irq_svc) from [<7f006910>] (igb_rd32+0x20/0xb0 [igb])
    [<7f006910>] (igb_rd32 [igb]) from [<7f02140c>] (igb_ptp_rx_hang+0x1c/0xa8 [igb])
    [<7f02140c>] (igb_ptp_rx_hang [igb]) from [<7f00df6c>] (igb_watchdog_task+0x1a8/0x72c [igb])
    [<7f00df6c>] (igb_watchdog_task [igb]) from [<80147670>] (process_one_work+0x204/0x458)
    ...

Re: rtnet rt_igb I210 not working with ARM

[Jan Kiszka]

On 06.01.22 00:39, Yann Le Chevoir via Xenomai wrote:
>>> Hello,
>>>
>>> I have an issue with rt_igb not working with an Intel I210 on ARM (see
>>> system configuration below).
>>> With Wireshark, looks like rtping is sending garbage on the network
>>> (see below).
>>> I notice there is no interrupt in /proc/xenomai/irq (see below).
>>> Looks like other people got issues with I210 in the past but I am not
>>> sure if they were similar and I am not able to find any conclusion.
>>>
>>> Below, I show you how I am inserting the kernel module + how I am able
>>> to have it working with igb (no realtime) and rt_fec (different driver
>>> for the other Ethernet controller on the same board).
>>>
>>> What can I do to solve this issue?
>>> Is there anything else I can try to give you more data?
>>>
>>
>> Could you check if that card works in a standard PC (x86-64) with RTnet?
>> Could you also check if latest 3.2.1 makes any difference? Not all fixes on
>> RTnet have been backported to very-old-stable.
>>
>> Jan
> 
> Hello Jan, all,
> 
> Happy new year!
> 
> Sorry for the delay, I needed to update my Linux kernel
> (Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be able
> to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?

Yes, everything up to and including 5.4 is I-pipe, 5.10 and later is
dovetail.

> I notice that the type of interruption changed since the previous version:
> we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".

Maybe the host controller of your SoC gained MSI-X support betwen 4.9
and 5.4? Then both the standard driver and rt_igb would try to use that
instead (MSI is legacy).

> And there is this new error: "Error 2 getting interrupt" which prevents to
> have rteth0 configured (see below). That's different from last time.
> I notice that the driver igb itself does not work with the new Xenomai
> kernel (see below). That's also different from last time. I confirmed that
> it works with the non-xenomai kernel though (same kernel 5.4.147 without
> the I-pipe patch).

5.4.147 - that is no official ipipe-arm release. Did you integrate the
patch yourself base on which I-pipe core revision? And how about
ipipe-core-5.4.151-arm-4?

> Note: to have the driver igb working with the Xenomai kernel, I can use
> the legacy interrupts ("pci=nomsi" on the kernel command line).
> Does this "error 2" sound familiar? Is there anything else I can try?

ENOENT - you could try search the code for sources of this error code.
But it may also come from the PCI or MSI core.

> 
> I did not try with a standard x86-64 PC yet. Do you think it may still
> help? I have the feeling that this irq issue is very specific to my arm
> SOC.

Not unlikely. I'm afraid there is no way around digging for the source
of the error code. That can help to understand better what goes wrong

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: rtnet rt_igb I210 not working with ARM

[Yann Le Chevoir]

>>>> I have an issue with rt_igb not working with an Intel I210 on ARM
>>>> (see system configuration below).
>>>> With Wireshark, looks like rtping is sending garbage on the network
>>>> (see below).
>>>> I notice there is no interrupt in /proc/xenomai/irq (see below).
>>>> Looks like other people got issues with I210 in the past but I am
>>>> not sure if they were similar and I am not able to find any conclusion.
>>>>
>>>> Below, I show you how I am inserting the kernel module + how I am
>>>> able to have it working with igb (no realtime) and rt_fec (different
>>>> driver for the other Ethernet controller on the same board).
>>>>
>>>> What can I do to solve this issue?
>>>> Is there anything else I can try to give you more data?
>>>>
>>> Could you check if that card works in a standard PC (x86-64) with RTnet?
>>> Could you also check if latest 3.2.1 makes any difference? Not all
>>> fixes on RTnet have been backported to very-old-stable.
>>>
>>> Jan
>>
>> Sorry for the delay, I needed to update my Linux kernel
>> (Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be
>> able to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?
>
> Yes, everything up to and including 5.4 is I-pipe, 5.10 and later is dovetail.
> 
>> I notice that the type of interruption changed since the previous version:
>> we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".
> 
> Maybe the host controller of your SoC gained MSI-X support betwen 4.9 and
> 5.4? Then both the standard driver and rt_igb would try to use that instead
> (MSI is legacy).
> 
>> And there is this new error: "Error 2 getting interrupt" which
>> prevents to have rteth0 configured (see below). That's different from last
> time.
>> I notice that the driver igb itself does not work with the new Xenomai
>> kernel (see below). That's also different from last time. I confirmed
>> that it works with the non-xenomai kernel though (same kernel 5.4.147
>> without the I-pipe patch).
> 
> 5.4.147 - that is no official ipipe-arm release. Did you integrate the patch
> yourself base on which I-pipe core revision? And how about ipipe-core-
> 5.4.151-arm-4?

Hello Jan, all,

What I "always" did for kernels linux-fslc-imx
    4.1.38 (Yocto Morty),
    4.9.67 (Yocto Thud)
    and 5.4.147 (Yocto Dunfell)
is that I tried to apply the closest arm I-pipe patch as possible:
    - Given an example, I used ipipe-core-5.4.151-arm-4.patch to patch
      Yocto Dunfell.
    - Then, I manually resolved the conflicts.
    - That went well (happy with the performances and the rtcan driver).
    - Idea was to stay as close as possible of the official Yocto branch
      we are using. Does that make sense?
By reading your answer, I wondered if I did right. So I also tried to
compile the kernel from the ipipe-arm repo:
    - Branch ipipe/5.4.y.
    - That compiles and boots.
    - But, for the rt_igb, same issue as previous procedure:
      "Error 2 getting interrupt".

>> Note: to have the driver igb working with the Xenomai kernel, I can
>> use the legacy interrupts ("pci=nomsi" on the kernel command line).
>> Does this "error 2" sound familiar? Is there anything else I can try?
> 
> ENOENT - you could try search the code for sources of this error code.
> But it may also come from the PCI or MSI core.
> 
>>
>> I did not try with a standard x86-64 PC yet. Do you think it may still
>> help? I have the feeling that this irq issue is very specific to my
>> arm SOC.
> 
> Not unlikely. I'm afraid there is no way around digging for the source of the
> error code. That can help to understand better what goes wrong

I tried so many different versions.
I finally came back to my initial version:
    linux-fslc-imx 4.9.67
      + ipipe-core-4.9.51-arm-4.patch (adapted)
      + Xenomai 3.0.9 or 3.0.12.
    - With this one, at least, the non-Xenomai igb driver works with the
      I-pipe patch.
    - I figured out that by disabling MSI (pci=nomsi), the irq counter in
      /proc/xenomai/irq started to increment for rt_igb.
    - Then, I confirmed that the buffers (ICMP or ARP frames) were correct
      (printk in the igb_main.c source code).
    - I also realized that the RX was working.
      The RTnet stack was preparing an ARP response to my Windows ping.
    - Only the TX was not good (garbage on Wireshark).

I finally have something working with the patch bellow.
So:
    - pci=nomsi to have the irq counter incrementing.
    - dma_sync_single_for_device in igb_tx_map to have rt_igb sending
      good data.

My questions:
What are your feeling about that?
Is dma_sync real-time? Is it a good fix/workaround?
Why do I need to add such a dma_sync with my arm SoC?
    - I suspect that this driver would work on a regular x86...
    - Can you tell if it works on other arm SoC?
Any suggestion to do work the driver with MSI instead of legacy interrupts?
Any suggestion to do work rt_igb with Kernel 5.4 + Xenomai 3.2?

The patch (igb_tx_map with Xenomai 3.0.12):
--- kernel.ori/drivers/net/drivers/igb/igb_main.c
+++ kernel/drivers/net/drivers/igb/igb_main.c
@@ -4095,6 +4095,7 @@
 	size = skb->len;

   dma = rtskb_data_dma_addr(skb, 0);
+  dma_sync_single_for_device(tx_ring->dev, dma, size, DMA_BIDIRECTIONAL);

Thank you,

Yann

Re: rtnet rt_igb I210 not working with ARM

[Jan Kiszka]

On 29.01.22 00:31, Yann Le Chevoir wrote:
>>>>> I have an issue with rt_igb not working with an Intel I210 on ARM
>>>>> (see system configuration below).
>>>>> With Wireshark, looks like rtping is sending garbage on the network
>>>>> (see below).
>>>>> I notice there is no interrupt in /proc/xenomai/irq (see below).
>>>>> Looks like other people got issues with I210 in the past but I am
>>>>> not sure if they were similar and I am not able to find any conclusion.
>>>>>
>>>>> Below, I show you how I am inserting the kernel module + how I am
>>>>> able to have it working with igb (no realtime) and rt_fec (different
>>>>> driver for the other Ethernet controller on the same board).
>>>>>
>>>>> What can I do to solve this issue?
>>>>> Is there anything else I can try to give you more data?
>>>>>
>>>> Could you check if that card works in a standard PC (x86-64) with RTnet?
>>>> Could you also check if latest 3.2.1 makes any difference? Not all
>>>> fixes on RTnet have been backported to very-old-stable.
>>>>
>>>> Jan
>>>
>>> Sorry for the delay, I needed to update my Linux kernel
>>> (Freescale/linux-fslc) from version 4.9.67 to version 5.4.147 to be
>>> able to compile Xenomai 3.2.1. I am still using I-pipe, not Dovetail, right?
>>
>> Yes, everything up to and including 5.4 is I-pipe, 5.10 and later is dovetail.
>>
>>> I notice that the type of interruption changed since the previous version:
>>> we now have "Using MSI-X interrupts" instead of "Using MSI interrupts".
>>
>> Maybe the host controller of your SoC gained MSI-X support betwen 4.9 and
>> 5.4? Then both the standard driver and rt_igb would try to use that instead
>> (MSI is legacy).
>>
>>> And there is this new error: "Error 2 getting interrupt" which
>>> prevents to have rteth0 configured (see below). That's different from last
>> time.
>>> I notice that the driver igb itself does not work with the new Xenomai
>>> kernel (see below). That's also different from last time. I confirmed
>>> that it works with the non-xenomai kernel though (same kernel 5.4.147
>>> without the I-pipe patch).
>>
>> 5.4.147 - that is no official ipipe-arm release. Did you integrate the patch
>> yourself base on which I-pipe core revision? And how about ipipe-core-
>> 5.4.151-arm-4?
> 
> Hello Jan, all,
> 
> What I "always" did for kernels linux-fslc-imx
>     4.1.38 (Yocto Morty),
>     4.9.67 (Yocto Thud)
>     and 5.4.147 (Yocto Dunfell)
> is that I tried to apply the closest arm I-pipe patch as possible:
>     - Given an example, I used ipipe-core-5.4.151-arm-4.patch to patch
>       Yocto Dunfell.
>     - Then, I manually resolved the conflicts.
>     - That went well (happy with the performances and the rtcan driver).
>     - Idea was to stay as close as possible of the official Yocto branch
>       we are using. Does that make sense?

I would not blindly use downstream kernels (Yocto, Freescale/NXP etc.)
unless you know why you need them.

> By reading your answer, I wondered if I did right. So I also tried to
> compile the kernel from the ipipe-arm repo:
>     - Branch ipipe/5.4.y.
>     - That compiles and boots.
>     - But, for the rt_igb, same issue as previous procedure:
>       "Error 2 getting interrupt".
> 
>>> Note: to have the driver igb working with the Xenomai kernel, I can
>>> use the legacy interrupts ("pci=nomsi" on the kernel command line).
>>> Does this "error 2" sound familiar? Is there anything else I can try?
>>
>> ENOENT - you could try search the code for sources of this error code.
>> But it may also come from the PCI or MSI core.
>>
>>>
>>> I did not try with a standard x86-64 PC yet. Do you think it may still
>>> help? I have the feeling that this irq issue is very specific to my
>>> arm SOC.
>>
>> Not unlikely. I'm afraid there is no way around digging for the source of the
>> error code. That can help to understand better what goes wrong
> 
> I tried so many different versions.
> I finally came back to my initial version:
>     linux-fslc-imx 4.9.67
>       + ipipe-core-4.9.51-arm-4.patch (adapted)
>       + Xenomai 3.0.9 or 3.0.12.
>     - With this one, at least, the non-Xenomai igb driver works with the
>       I-pipe patch.
>     - I figured out that by disabling MSI (pci=nomsi), the irq counter in
>       /proc/xenomai/irq started to increment for rt_igb.
>     - Then, I confirmed that the buffers (ICMP or ARP frames) were correct
>       (printk in the igb_main.c source code).
>     - I also realized that the RX was working.
>       The RTnet stack was preparing an ARP response to my Windows ping.
>     - Only the TX was not good (garbage on Wireshark).
> 
> I finally have something working with the patch bellow.
> So:
>     - pci=nomsi to have the irq counter incrementing.
>     - dma_sync_single_for_device in igb_tx_map to have rt_igb sending
>       good data.
> 
> My questions:
> What are your feeling about that?
> Is dma_sync real-time? Is it a good fix/workaround?

In the best case, it's just a barrier (but there is already a wmb()). In
the worst case, Linux code is triggered that actually does more, e.g.
because DMA is not cache-coherent. You will have to analyze what is
actually done in that function on your platform.

> Why do I need to add such a dma_sync with my arm SoC?
>     - I suspect that this driver would work on a regular x86...
>     - Can you tell if it works on other arm SoC?

I don't think this was tried on ARM so far, but then the topic will
actually be rather SoC-specific.

> Any suggestion to do work the driver with MSI instead of legacy interrupts?
> Any suggestion to do work rt_igb with Kernel 5.4 + Xenomai 3.2?
> 
> The patch (igb_tx_map with Xenomai 3.0.12):
> --- kernel.ori/drivers/net/drivers/igb/igb_main.c
> +++ kernel/drivers/net/drivers/igb/igb_main.c
> @@ -4095,6 +4095,7 @@
>  	size = skb->len;
> 
>    dma = rtskb_data_dma_addr(skb, 0);
> +  dma_sync_single_for_device(tx_ring->dev, dma, size, DMA_BIDIRECTIONAL);
> 

Comparing the code against the related Linux driver regarding DMA and
possibly also regarding configuration for MSI-X will be needed. There
will be no way around understanding how DMA works as well as how MSI-X
is delivered. Then it should become clearer what is missing in the
rt_igb driver or possibly also in ARM-enablement of Xenomai for that SoC.

Jan

-- 
Siemens AG, Technology
Competence Center Embedded Linux

RE: rtnet rt_igb I210 not working with ARM

[Yann Le Chevoir]

> > I finally have something working with the patch bellow.
> > So:
> >     - pci=nomsi to have the irq counter incrementing.
> >     - dma_sync_single_for_device in igb_tx_map to have rt_igb sending
> >       good data.
> >
> > My questions:
> > What are your feeling about that?
> > Is dma_sync real-time? Is it a good fix/workaround?
> 
> In the best case, it's just a barrier (but there is already a wmb()). In the worst
> case, Linux code is triggered that actually does more, e.g.
> because DMA is not cache-coherent. You will have to analyze what is actually
> done in that function on your platform.
> 
> > Why do I need to add such a dma_sync with my arm SoC?
> >     - I suspect that this driver would work on a regular x86...
> >     - Can you tell if it works on other arm SoC?
> 
> I don't think this was tried on ARM so far, but then the topic will actually be
> rather SoC-specific.

Hi,
Maybe worth mentioning for those who face the same problem:
I did not go deeper with the DMA yet. I was happy with the tx performances.
But I faced a different issue with the rx, with 1 packets over 500 which was lost at high frequency.
I got to do a similar patch (dma_sync) for the rx. See below (tx + rx). That works for me and performances are validated.

--- a/kernel/drivers/net/drivers/igb/igb_main.c
+++ b/kernel/drivers/net/drivers/igb/igb_main.c
@@ -4143,6 +4143,7 @@ static void igb_tx_map(struct igb_ring *tx_ring,
 	 * We also need this memory barrier to make certain all of the
 	 * status bits have been updated before next_to_watch is written.
 	 */
+	dma_sync_single_for_device(tx_ring->dev, dma, size, DMA_TO_DEVICE);
 	wmb();
 
 	if (skb->xmit_stamp)
@@ -4764,13 +4765,19 @@ static struct rtskb *igb_fetch_rx_buffer(struct igb_ring *rx_ring,
 {
 	struct igb_rx_buffer *rx_buffer;
 	struct rtskb *skb;
+	dma_addr_t dma;
+	unsigned int size;
 
 	rx_buffer = &rx_ring->rx_buffer_info[rx_ring->next_to_clean];
 	skb = rx_buffer->skb;
 	prefetchw(skb->data);
 
+	dma = rtskb_data_dma_addr(skb, 0);
+	size = le16_to_cpu(rx_desc->wb.upper.length);
+	dma_sync_single_for_cpu(rx_ring->dev, dma, size, DMA_FROM_DEVICE);
+
 	/* pull the header of the skb in */
-	rtskb_put(skb, le16_to_cpu(rx_desc->wb.upper.length));
+	rtskb_put(skb, size);
 	rx_buffer->skb = NULL;
 	rx_buffer->dma = 0;