Re: [Intel-wired-lan] Supermicro AOC-STGN-I1S (Intel 82599EN based 10G adapter) - poor network perfomance after moving to Debian 11.5

From: Paul Menzel <pmenzel@molgen.mpg.de>
To: Bartek Kois <bartek.kois@gmail.com>
Cc: intel-wired-lan@osuosl.org, regressions@lists.linux.dev
Subject: Re: [Intel-wired-lan] Supermicro AOC-STGN-I1S (Intel 82599EN based 10G adapter) - poor network perfomance after moving to Debian 11.5
Date: Thu, 19 Jan 2023 11:17:42 +0100	[thread overview]
Message-ID: <652bf236-d97e-832c-e0f3-24927a46d7ad@molgen.mpg.de> (raw)
In-Reply-To: <d1530cba-1a72-cae8-6a04-ed8ec0f82e6e@gmail.com>

#regzbot ^introduced: 4.9.88..5.10.149

Dear Bartek,

Am 14.01.23 um 11:23 schrieb Bartek Kois:

> After moving from Debian 9.7 to 11.5 as soon as I perform "ip link set 
> enp1s0 up" for my 10G adapter (AOC-STGN-I1S - Intel 82599EN based 10G 
> adapter) I am experiencing high cpu load (even if no traffic is passing 
> through the adapter) and network performance is low (when network is 
> connected).

How do you test the network performance? Please give exact numbers for 
comparison.

> The cpu load is oscillating between 0.1 and 0.3 on vanilla system
> with no network attached. The problem can be observed on the 
> following platforms: Supermicro X9SCL (Intel C202 PCH) and
> Supermicro X10SLL+-F (Intel C222 Express PCH), but for the Supermicro
> X11SSL-F (Intel® C232 chipset) everything is working well.
> 
> Tested environments:
> Debian 9.7 - Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 
> (2018-05-07) x86_64 GNU/Linux [all platforms working well with no 
> problems: Supermicro X9SCL (Intel C202 PCH), Supermicro X10SLL+-F (Intel 
> C222 Express PCH), Supermicro X11SSL-F (Intel® C232 chipset)]

> Debian 11.5 - Linux 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 
> (2022-10-21) x86_64 GNU/Linux  [older platforms: Supermicro X9SCL (Intel 
> C202 PCH), Supermicro X10SLL+-F (Intel C222 Express PCH) behave 
> problematic as described above | newer platform: Supermicro X11SSL-F 
> (Intel® C232 chipset) working well with no problems]

Maybe create a bug at the Linux kernel bug tracker [1], where you can 
attach all the logs (`dmesg`, `lspci -nnk -s …`, …).

> So far to solve the problem I was trying to upgrade system to the newest 
> stable version, upgrade kernel to version 6.x, upgrade ixgbe driver to 
> the newest version but with no luck.

Thank you for checking that. Too bad it’s still present. To rule out 
some user space problem, could you test Debian 9.7 with a stable Linux 
release, currently 6.1.7?

What does `sudo perf top --sort comm,dso` show, where the time is spent?

> Supermicro support suggested as follows:
> it might be kernel related debian 11.5 has kernel 5.10 which is a 
> recent kernel it might not properly support the chipsets for X9 
> therefore i suggest to use RHEL or CentOS as they use much older kernel 
> versions. I expect that with ubuntu 20.04 you see the same problem it 
> uses kernel 5.4
Testing another GNU/Linux distribution for another data point, might be 
a good idea.

As nobody has responded yet, bisecting the issue is probably the fastest 
way to get to the bottom of this. Luckily the problem seems reproducible 
and you seem to be able to build a Linux kernel yourself, so that should 
work. (For testing purposes you could also test with Ubuntu, as they 
provide Linux kernel builds for (almost) all releases in their Linux 
kernel mainline PPA [2].)

Kind regards,

Paul

[1]: https://bugzilla.kernel.org/
[2]: https://kernel.ubuntu.com/~kernel-ppa/mainline/
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan