Re: [Intel-wired-lan] Supermicro AOC-STGN-I1S (Intel 82599EN based 10G adapter) - poor network perfomance after moving to Debian 11.5

[Bartek Kois <bartek.kois@gmail.com>]

W dniu 19.01.2023 o 13:24, Bartek Kois pisze:
>
> W dniu 19.01.2023 o 11:17, Paul Menzel pisze:
>>
>> #regzbot ^introduced: 4.9.88..5.10.149
>>
>> Dear Bartek,
>>
>>
>> Am 14.01.23 um 11:23 schrieb Bartek Kois:
>>
>>> After moving from Debian 9.7 to 11.5 as soon as I perform "ip link 
>>> set enp1s0 up" for my 10G adapter (AOC-STGN-I1S - Intel 82599EN 
>>> based 10G adapter) I am experiencing high cpu load (even if no 
>>> traffic is passing through the adapter) and network performance is 
>>> low (when network is connected).
>>
>> How do you test the network performance? Please give exact numbers 
>> for comparison.
>>
> I am using this server as a router for my subscribers with iptables 
> (for NAT and firewall) and hfsc (for QoS). First I encountered this 
> problem while migrating form Debian 9.7 to 11.5. Routers based  on 
> Supermicro X11SSL-F (Intel® C232 chipset) works with no problems after 
> that migration, but routers based on Supermicro X9SCL (Intel C202 PCH) 
> and Supermicro X10SLL+-F (Intel C222 Express PCH) starts behaving 
> strangely with high cpu load (0.5-0.8 while before it was around 
> 0.0-0.1) and subscribers not being able to utilize their plans. I 
> tried to strip down the problem and ends up with clean system with no 
> iptables or hfsc rules behaving the same (higher load) right after 
> setting the 10G link upeven if no traffic is passing by.
>
>>> The cpu load is oscillating between 0.1 and 0.3 on vanilla system
>>> with no network attached. The problem can be observed on the 
>>> following platforms: Supermicro X9SCL (Intel C202 PCH) and
>>> Supermicro X10SLL+-F (Intel C222 Express PCH), but for the Supermicro
>>> X11SSL-F (Intel® C232 chipset) everything is working well.
>>>
>>> Tested environments:
>>> Debian 9.7 - Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 
>>> (2018-05-07) x86_64 GNU/Linux [all platforms working well with no 
>>> problems: Supermicro X9SCL (Intel C202 PCH), Supermicro X10SLL+-F 
>>> (Intel C222 Express PCH), Supermicro X11SSL-F (Intel® C232 chipset)]
>>
>>> Debian 11.5 - Linux 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 
>>> (2022-10-21) x86_64 GNU/Linux  [older platforms: Supermicro X9SCL 
>>> (Intel C202 PCH), Supermicro X10SLL+-F (Intel C222 Express PCH) 
>>> behave problematic as described above | newer platform: Supermicro 
>>> X11SSL-F (Intel® C232 chipset) working well with no problems]
>>
>> Maybe create a bug at the Linux kernel bug tracker [1], where you can 
>> attach all the logs (`dmesg`, `lspci -nnk -s …`, …).
>>
> I`ve already reported that to the Debian team 
> ttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1024763, but so far 
> nobody took care of this issue so far.
>
>>> So far to solve the problem I was trying to upgrade system to the 
>>> newest stable version, upgrade kernel to version 6.x, upgrade ixgbe 
>>> driver to the newest version but with no luck.
>>
>> Thank you for checking that. Too bad it’s still present. To rule out 
>> some user space problem, could you test Debian 9.7 with a stable 
>> Linux release, currently 6.1.7?
>>
>> What does `sudo perf top --sort comm,dso` show, where the time is spent?
>
> During my first test in real enviroment with subscribers I gether the 
> following data through the perf:
>
>   27.83%  [kernel]                   [k] strncpy
>   14.80%  [kernel]                   [k] nft_do_chain
>    7.61%  [kernel]                   [k] memcmp
>    5.63%  [kernel]                   [k] nft_meta_get_eval
>    3.14%  [kernel]                   [k] nft_cmp_eval
>    2.79%  [kernel]                   [k] asm_exc_nmi
>    1.07%  [kernel]                   [k] module_get_kallsym
>    0.92%  [kernel]                   [k] 
> kallsyms_expand_symbol.constprop.0
>    0.85%  [kernel]                   [k] ixgbe_poll
>    0.75%  [kernel]                   [k] format_decode
>    0.61%  [kernel]                   [k] number
>    0.56%  [kernel]                   [k] menu_select
>    0.54%  [kernel]                   [k] clflush_cache_range
>    0.52%  [kernel]                   [k] cpuidle_enter_state
>    0.51%  [kernel]                   [k] vsnprintf
>    0.50%  [kernel]                   [k] u32_classify
>    0.49%  [kernel]                   [k] fib_table_lookup
>    0.40%  [kernel]                   [k] dma_pte_clear_level
>    0.39%  [kernel]                   [k] domain_mapping
>    0.36%  [kernel]                   [k] ixgbe_xmit_fram
>
>
>     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM TIME+ 
> COMMAND
>      18 root      20   0       0      0      0 S  28.2   0.0 7:06.27 
> ksoftirqd/1
>      12 root      20   0       0      0      0 R  12.0   0.0 4:10.88 
> ksoftirqd/0
>      23 root      20   0       0      0      0 S   6.0   0.0 4:36.08 
> ksoftirqd/2
>      28 root      20   0       0      0      0 S   5.3   0.0 6:46.47 
> ksoftirqd/3
>  846449 root      20   0       0      0      0 I   1.0   0.0 0:01.61 
> kworker/0:0-events_power_efficient
>      13 root      20   0       0      0      0 I   0.3   0.0 0:13.50 
> rcu_sched
>    8264 root      20   0  101536   6944   4824 S   0.3   0.2 0:07.77 
> dhcpd
>       1 root      20   0  164048  10184   7672 S   0.0   0.3 0:04.52 
> systemd
>       2 root      20   0       0      0      0 S   0.0   0.0 0:00.00 
> kthreadd
>       3 root       0 -20       0      0      0 I   0.0   0.0 0:00.00 
> rcu_gp
>       4 root       0 -20       0      0      0 I   0.0   0.0 0:00.00 
> rcu_par_gp
>       6 root       0 -20       0      0      0 I   0.0   0.0 0:00.00 
> kworker/0:0H-events_highpri
>       9 root       0 -20       0      0      0 I   0.0   0.0 0:00.00 
> mm_percpu_wq
>      10 root      20   0       0      0      0 S   0.0   0.0 0:00.00 
> rcu_tasks_rude_
>      11 root      20   0       0      0      0 S   0.0   0.0 0:00.00 
> rcu_tasks_trace
>      14 root      rt   0       0      0      0 S   0.0   0.0 0:00.26 
> migration/0
>
>>
>>> Supermicro support suggested as follows:
>>> it might be kernel related debian 11.5 has kernel 5.10 which is a 
>>> recent kernel it might not properly support the chipsets for X9 
>>> therefore i suggest to use RHEL or CentOS as they use much older 
>>> kernel versions. I expect that with ubuntu 20.04 you see the same 
>>> problem it uses kernel 5.4
>> Testing another GNU/Linux distribution for another data point, might 
>> be a good idea.
>>
>> As nobody has responded yet, bisecting the issue is probably the 
>> fastest way to get to the bottom of this. Luckily the problem seems 
>> reproducible and you seem to be able to build a Linux kernel 
>> yourself, so that should work. (For testing purposes you could also 
>> test with Ubuntu, as they provide Linux kernel builds for (almost) 
>> all releases in their Linux kernel mainline PPA [2].)
>>
> Of course  I can try Ubuntu and report how it is working.
>
Ubuntu (5.15.0-43-generic) seems to be working in the same way 
generating higher load after executing "ip link set enp1s0 up".

Best regards

Bartek Kois

> Best regards
>
> Bartek Kois
>
>>
>> Kind regards,
>>
>> Paul
>>
>>
>> [1]: https://bugzilla.kernel.org/
>> [2]: https://kernel.ubuntu.com/~kernel-ppa/mainline/