Re: [Intel-wired-lan] Supermicro AOC-STGN-I1S (Intel 82599EN based 10G adapter) - poor network perfomance after moving to Debian 11.5

[Paul Menzel <pmenzel@molgen.mpg.de>]

Dear Bartek,


Am 23.01.23 um 19:58 schrieb Bartek Kois:

> W dniu 23.01.2023 o 19:53, Paul Menzel pisze:

>> Am 23.01.23 um 19:38 schrieb Bartek Kois:
>>>
>>> W dniu 22.01.2023 o 21:28, Paul Menzel pisze:
>>>> Dear Bartek,
>>>>
>>>>
>>>> Am 19.01.23 um 18:17 schrieb Bartek Kois:
>>>>> W dniu 19.01.2023 o 18:09, Paul Menzel pisze:
>>>>
>>>>>> Am 19.01.23 um 17:58 schrieb Bartek Kois:
>>>>>>> W dniu 19.01.2023 o 13:24, Bartek Kois pisze:
>>>>>>>>
>>>>>>>> W dniu 19.01.2023 o 11:17, Paul Menzel pisze:
>>>>>>>>>
>>>>>>>>> #regzbot ^introduced: 4.9.88..5.10.149
>>>>>>
>>>>>>>>> Am 14.01.23 um 11:23 schrieb Bartek Kois:
>>>>>>>>>
>>>>>>>>>> After moving from Debian 9.7 to 11.5 as soon as I perform "ip 
>>>>>>>>>> link set enp1s0 up" for my 10G adapter (AOC-STGN-I1S - Intel 
>>>>>>>>>> 82599EN based 10G adapter) I am experiencing high cpu load 
>>>>>>>>>> (even if no traffic is passing through the adapter) and 
>>>>>>>>>> network performance is low (when network is connected).
>>>>>>>>>
>>>>>>>>> How do you test the network performance? Please give exact 
>>>>>>>>> numbers for comparison.
>>>>>>>>>
>>>>>>>> I am using this server as a router for my subscribers with 
>>>>>>>> iptables (for NAT and firewall) and hfsc (for QoS). First I 
>>>>>>>> encountered this problem while migrating form Debian 9.7 to 
>>>>>>>> 11.5. Routers based  on Supermicro X11SSL-F (Intel® C232 
>>>>>>>> chipset) works with no problems after that migration, but 
>>>>>>>> routers based on Supermicro X9SCL (Intel C202 PCH) and 
>>>>>>>> Supermicro X10SLL+-F (Intel C222 Express PCH) starts behaving 
>>>>>>>> strangely with high cpu load (0.5-0.8 while before it was around 
>>>>>>>> 0.0-0.1) and subscribers not being able to utilize their plans. 
>>>>>>>> I tried to strip down the problem and ends up with clean system 
>>>>>>>> with no iptables or hfsc rules behaving the same (higher load) 
>>>>>>>> right after setting the 10G link upeven if no traffic is passing 
>>>>>>>> by.
>>>>>>>>
>>>>>>>>>> The cpu load is oscillating between 0.1 and 0.3 on vanilla system
>>>>>>>>>> with no network attached. The problem can be observed on the 
>>>>>>>>>> following platforms: Supermicro X9SCL (Intel C202 PCH) and
>>>>>>>>>> Supermicro X10SLL+-F (Intel C222 Express PCH), but for the 
>>>>>>>>>> Supermicro
>>>>>>>>>> X11SSL-F (Intel® C232 chipset) everything is working well.
>>>>>>>>>>
>>>>>>>>>> Tested environments:
>>>>>>>>>> Debian 9.7 - Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 
>>>>>>>>>> (2018-05-07) x86_64 GNU/Linux [all platforms working well with 
>>>>>>>>>> no problems: Supermicro X9SCL (Intel C202 PCH), Supermicro 
>>>>>>>>>> X10SLL+-F (Intel C222 Express PCH), Supermicro X11SSL-F 
>>>>>>>>>> (Intel® C232 chipset)]
>>>>>>>>>
>>>>>>>>>> Debian 11.5 - Linux 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 
>>>>>>>>>> (2022-10-21) x86_64 GNU/Linux [older platforms: Supermicro 
>>>>>>>>>> X9SCL (Intel C202 PCH), Supermicro X10SLL+-F (Intel C222 
>>>>>>>>>> Express PCH) behave problematic as described above | newer 
>>>>>>>>>> platform: Supermicro X11SSL-F (Intel® C232 chipset) working 
>>>>>>>>>> well with no problems]
>>>>>>>>>
>>>>>>>>> Maybe create a bug at the Linux kernel bug tracker [1], where 
>>>>>>>>> you can attach all the logs (`dmesg`, `lspci -nnk -s …`, …).
>>>>>>>>>
>>>>>>>> I`ve already reported that to the Debian team 
>>>>>>>> ttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024763, but so 
>>>>>>>> far nobody took care of this issue so far.
>>>>>>>>
>>>>>>>>>> So far to solve the problem I was trying to upgrade system to 
>>>>>>>>>> the newest stable version, upgrade kernel to version 6.x, 
>>>>>>>>>> upgrade ixgbe driver to the newest version but with no luck.
>>>>>>>>>
>>>>>>>>> Thank you for checking that. Too bad it’s still present. To 
>>>>>>>>> rule out some user space problem, could you test Debian 9.7 
>>>>>>>>> with a stable Linux release, currently 6.1.7?
>>>>>>>>>
>>>>>>>>> What does `sudo perf top --sort comm,dso` show, where the time 
>>>>>>>>> is spent?
>>>>>>>>
>>>>>>>> During my first test in real enviroment with subscribers I 
>>>>>>>> gether the following data through the perf:
>>>>>>>>
>>>>>>>>   27.83%  [kernel]                   [k] strncpy
>>>>>>>>   14.80%  [kernel]                   [k] nft_do_chain
>>>>>>>>    7.61%  [kernel]                   [k] memcmp
>>>>>>>>    5.63%  [kernel]                   [k] nft_meta_get_eval
>>>>>>>>    3.14%  [kernel]                   [k] nft_cmp_eval
>>>>>>>>    2.79%  [kernel]                   [k] asm_exc_nmi
>>>>>>>>    1.07%  [kernel]                   [k] module_get_kallsym
>>>>>>>>    0.92%  [kernel]                   [k] 
>>>>>>>> kallsyms_expand_symbol.constprop.0
>>>>>>>>    0.85%  [kernel]                   [k] ixgbe_poll
>>>>>>>>    0.75%  [kernel]                   [k] format_decode
>>>>>>>>    0.61%  [kernel]                   [k] number
>>>>>>>>    0.56%  [kernel]                   [k] menu_select
>>>>>>>>    0.54%  [kernel]                   [k] clflush_cache_range
>>>>>>>>    0.52%  [kernel]                   [k] cpuidle_enter_state
>>>>>>>>    0.51%  [kernel]                   [k] vsnprintf
>>>>>>>>    0.50%  [kernel]                   [k] u32_classify
>>>>>>>>    0.49%  [kernel]                   [k] fib_table_lookup
>>>>>>>>    0.40%  [kernel]                   [k] dma_pte_clear_level
>>>>>>>>    0.39%  [kernel]                   [k] domain_mapping
>>>>>>>>    0.36%  [kernel]                   [k] ixgbe_xmit_fram
>>>>>>>>
>>>>>>>>
>>>>>>>>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM 
>>>>>>>> TIME+ COMMAND
>>>>>>>>      18 root      20   0       0      0      0 S  28.2 0.0 
>>>>>>>> 7:06.27 ksoftirqd/1
>>>>>>>>      12 root      20   0       0      0      0 R  12.0 0.0 
>>>>>>>> 4:10.88 ksoftirqd/0
>>>>>>
>>>>>> […]
>>>>>>
>>>>>> Do you see different behavior in `/proc/interrupts`?
>>>>>>
>>>>> This is how it looks like for Debian 11.5 - Linux 5.10.0-19-amd64 
>>>>> #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux on 
>>>>> Supermicro X10SLL+-F (Intel C222 Express PCH):
>>>>>
>>>>>        1 root      20   0  163948  10288   7696 S   0.0 0.1 0:39.58 
>>>>> systemd
>>>>
>>>> […]
>>>>
>>>> The content of `/proc/interrupts` has a different format on my system.
>>>>
>>>> ```
>>>> $ head -3 /proc/interrupts
>>>>            CPU0       CPU1       CPU2       CPU3
>>>>   1:      55560          0        113          0  IR-IO-APIC 1-edge 
>>>> i8042
>>>>   8:          0          0          0          0  IR-IO-APIC 8-edge 
>>>> rtc0
>>>> ```
>>>> […]
>>>>
>>>>> and for Debian 9.7 - Linux 4.9.0-6-amd64 #1 SMP Debian 
>>>>> 4.9.88-1+deb9u1 on Supermicro X10SLL+-F (Intel C222 Express PCH)
>>>>>
>>>>> 31659 root      20   0       0      0      0 S   0.3  0.0 0:00.92 
>>>>> kworker/7:0
>>>>>      1 root      20   0   57032   6736   5256 S   0.0  0.1 2:28.14 
>>>>> systemd
>>>>
>>>> […]
>>>>>>>>>> Supermicro support suggested as follows:
>>>>>>>>>> it might be kernel related debian 11.5 has kernel 5.10 which 
>>>>>>>>>> is a recent kernel it might not properly support the chipsets 
>>>>>>>>>> for X9 therefore i suggest to use RHEL or CentOS as they use 
>>>>>>>>>> much older kernel versions. I expect that with ubuntu 20.04 
>>>>>>>>>> you see the same problem it uses kernel 5.4
>>>>>>>>> >>> Testing another GNU/Linux distribution for another data 
>>>>>>>>> point, might be a good idea.
>>>>>>>>>
>>>>>>>>> As nobody has responded yet, bisecting the issue is probably 
>>>>>>>>> the fastest way to get to the bottom of this. Luckily the 
>>>>>>>>> problem seems reproducible and you seem to be able to build a 
>>>>>>>>> Linux kernel yourself, so that should work. (For testing 
>>>>>>>>> purposes you could also test with Ubuntu, as they provide Linux 
>>>>>>>>> kernel builds for (almost) all releases in their Linux kernel 
>>>>>>>>> mainline PPA [2].)
>>>>>>>>>
>>>>>>>> Of course  I can try Ubuntu and report how it is working.
>>>>>>>>
>>>>>>> Ubuntu (5.15.0-43-generic) seems to be working in the same way 
>>>>>>> generating higher load after executing "ip link set enp1s0 up".
>>>>>>
>>>>>> That is good to know. (Is this Ubuntu 22.04?) What about Ubuntu 
>>>>>> 20.04 with Linux 5.4, and Ubuntu 18.04 with 4.15?
>>>>>>
>>>>>> Anyway, I think, you won’t come around bisecting. Another hint, 
>>>>>> make sure that you can build a 4.9 Linux kernel yourself, that 
>>>>>> does not exhibit that issue.
>>>>>>
>>>>> That`s right, it is 22.04. I don`t have to build it. Standard 
>>>>> kernel Linux 4.9.0-6-amd64 from Debian 9.7 worked without problems 
>>>>> for past 4 years.
>>>>
>>>> If nobody of the developers/maintainers is going to step up, you are 
>>>> on your own. Again, as you can reproduce this easily, the fastest 
>>>> way is to bisect the issue, which you can do on your own.
>>>
>>> How can I investigate that further?
>>
>> I repeat myself, please bisect the issue. It’s the fastest way.
>>
>>> I thought about trying to change some of the parameters related to 
>>> ixgbe driver and observe if anything is changing, but when I am 
>>> trying to do:
>>>
>>> sudo modprobe ixgbe IntMode=0
>>>
>>> I get the following error in the dmesg:
>>>
>>> [ 2137.324772] ixgbe: unknown parameter 'IntMode' ignored <<<<<<<<<
>>
>> […]
>>
>> `modinfo ixgbe` shows the supported parameters.

>> PS: If you need help bisecting, please ask. Otherwise, I am out of 
>> this thread.
> 
> Ok, how exactly I can bisect this issue?

What have you tried so far? As written in the past, I’d first try more 
distributions, for example, older Ubuntu versions. Then, if you have 
some range, I’d use the Ubuntu PPA, and then between the release 
candidate versions, only then start doing `git bisect` as documented in 
the documentation [3].


Kind regards,

Paul


>>>>>>>>> [1]: https://bugzilla.kernel.org/
>>>>>>>>> [2]: https://kernel.ubuntu.com/~kernel-ppa/mainline/
[3]: https://www.kernel.org/doc/html/latest/admin-guide/bug-bisect.html