From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx3.molgen.mpg.de (mx3.molgen.mpg.de [141.14.17.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E48453C26 for ; Thu, 19 Jan 2023 10:22:19 +0000 (UTC) Received: from [192.168.0.2] (ip5f5ae989.dynamic.kabel-deutschland.de [95.90.233.137]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) (Authenticated sender: pmenzel) by mx.molgen.mpg.de (Postfix) with ESMTPSA id A386F60027FD0; Thu, 19 Jan 2023 11:22:17 +0100 (CET) Message-ID: Date: Thu, 19 Jan 2023 11:22:17 +0100 Precedence: bulk X-Mailing-List: regressions@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Subject: Re: [Intel-wired-lan] Supermicro AOC-STGN-I1S (Intel 82599EN based 10G adapter) - poor network performance after moving to Debian 11.5 Content-Language: en-US From: Paul Menzel To: Bartek Kois Cc: intel-wired-lan@osuosl.org, regressions@lists.linux.dev References: <652bf236-d97e-832c-e0f3-24927a46d7ad@molgen.mpg.de> In-Reply-To: <652bf236-d97e-832c-e0f3-24927a46d7ad@molgen.mpg.de> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Dear Bartek, Am 19.01.23 um 11:17 schrieb Paul Menzel: > #regzbot ^introduced: 4.9.88..5.10.149 > Am 14.01.23 um 11:23 schrieb Bartek Kois: > >> After moving from Debian 9.7 to 11.5 as soon as I perform "ip link set >> enp1s0 up" for my 10G adapter (AOC-STGN-I1S - Intel 82599EN based 10G >> adapter) I am experiencing high cpu load (even if no traffic is >> passing through the adapter) and network performance is low (when >> network is connected). > > How do you test the network performance? Please give exact numbers for > comparison. > >> The cpu load is oscillating between 0.1 and 0.3 on vanilla system >> with no network attached. The problem can be observed on the following >> platforms: Supermicro X9SCL (Intel C202 PCH) and >> Supermicro X10SLL+-F (Intel C222 Express PCH), but for the Supermicro >> X11SSL-F (Intel® C232 chipset) everything is working well. >> >> Tested environments: >> Debian 9.7 - Linux 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 >> (2018-05-07) x86_64 GNU/Linux [all platforms working well with no >> problems: Supermicro X9SCL (Intel C202 PCH), Supermicro X10SLL+-F >> (Intel C222 Express PCH), Supermicro X11SSL-F (Intel® C232 chipset)] > >> Debian 11.5 - Linux 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 >> (2022-10-21) x86_64 GNU/Linux  [older platforms: Supermicro X9SCL >> (Intel C202 PCH), Supermicro X10SLL+-F (Intel C222 Express PCH) behave >> problematic as described above | newer platform: Supermicro X11SSL-F >> (Intel® C232 chipset) working well with no problems] > > Maybe create a bug at the Linux kernel bug tracker [1], where you can > attach all the logs (`dmesg`, `lspci -nnk -s …`, …). > >> So far to solve the problem I was trying to upgrade system to the >> newest stable version, upgrade kernel to version 6.x, upgrade ixgbe >> driver to the newest version but with no luck. > > Thank you for checking that. Too bad it’s still present. To rule out > some user space problem, could you test Debian 9.7 with a stable Linux > release, currently 6.1.7? > > What does `sudo perf top --sort comm,dso` show, where the time is spent? > >> Supermicro support suggested as follows: >> it might be kernel related debian 11.5 has kernel 5.10 which is a >> recent kernel it might not properly support the chipsets for X9 >> therefore i suggest to use RHEL or CentOS as they use much older >> kernel versions. I expect that with ubuntu 20.04 you see the same >> problem it uses kernel 5.4 > > Testing another GNU/Linux distribution for another data point, might be > a good idea. > > As nobody has responded yet, bisecting the issue is probably the fastest > way to get to the bottom of this. Luckily the problem seems reproducible > and you seem to be able to build a Linux kernel yourself, so that should > work. (For testing purposes you could also test with Ubuntu, as they > provide Linux kernel builds for (almost) all releases in their Linux > kernel mainline PPA [2].) You could also try to do that in a virtual machine by passing through the network device to the VM. If that reproduces the issue, that’s quite a fast setup for bisecting a regression, as start times are really fast. (For example, you can pass the Linux kernel directly to a QEMU VM with the `-kernel` switch.) Kind regards, Paul > [1]: https://bugzilla.kernel.org/ > [2]: https://kernel.ubuntu.com/~kernel-ppa/mainline/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5ADD1C00A5A for ; Thu, 19 Jan 2023 10:22:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id EAAD982131; Thu, 19 Jan 2023 10:22:24 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org EAAD982131 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org; s=default; t=1674123745; bh=s8g9B+zHSe9b6n5ETZxBOmwcMsCT2Rxw/NC6JZY4BAM=; h=Date:From:To:References:In-Reply-To:Subject:List-Id: List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe: Cc:From; b=C/A6YPCl8qUk4XyC+1M5RI48IIYDe8mUhKVOPPRjOHiflKqByzDJtMDAOd3wBjNc+ uDg0Ey50JvxBw/iQ/mxX2FXbCPJLy9+BE5AeAv2LSqmQflvAsv/Hc/tfbCIOX/MQdw npZyaT2cjYX+s8GMtrqD8Pe7ZVdFvxkwXK5Qw+xZC9d4OTDM5WsWP8bLSou9Q9R8dc yODQcmecnw6ErmRZ4ZNuK+cDcfu1Ctghy0PdSlhUMVJ/J3luKI91Ps934dWCn+DZgh SRxYt8BMiu7V/i+TcRp1fShXJTstz6KYrUZh0mrTDlm3AzipLFluXgozxnpWZW/qs4 jG1DsPrEFfPgg== X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YdiTgJIGoCXY; Thu, 19 Jan 2023 10:22:24 +0000 (UTC) Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34]) by smtp1.osuosl.org (Postfix) with ESMTP id EE36182297; Thu, 19 Jan 2023 10:22:23 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org EE36182297 Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by ash.osuosl.org (Postfix) with ESMTP id C861E1BF400 for ; Thu, 19 Jan 2023 10:22:21 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id AF21882297 for ; Thu, 19 Jan 2023 10:22:21 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org AF21882297 X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z6f4E8pU_Fvq for ; Thu, 19 Jan 2023 10:22:19 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0 DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 913C582131 Received: from mx3.molgen.mpg.de (mx3.molgen.mpg.de [141.14.17.11]) by smtp1.osuosl.org (Postfix) with ESMTPS id 913C582131 for ; Thu, 19 Jan 2023 10:22:19 +0000 (UTC) Received: from [192.168.0.2] (ip5f5ae989.dynamic.kabel-deutschland.de [95.90.233.137]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) (Authenticated sender: pmenzel) by mx.molgen.mpg.de (Postfix) with ESMTPSA id A386F60027FD0; Thu, 19 Jan 2023 11:22:17 +0100 (CET) Message-ID: Date: Thu, 19 Jan 2023 11:22:17 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Content-Language: en-US From: Paul Menzel To: Bartek Kois References: <652bf236-d97e-832c-e0f3-24927a46d7ad@molgen.mpg.de> In-Reply-To: <652bf236-d97e-832c-e0f3-24927a46d7ad@molgen.mpg.de> Subject: Re: [Intel-wired-lan] Supermicro AOC-STGN-I1S (Intel 82599EN based 10G adapter) - poor network performance after moving to Debian 11.5 X-BeenThere: intel-wired-lan@osuosl.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel Wired Ethernet Linux Kernel Driver Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: intel-wired-lan@osuosl.org, regressions@lists.linux.dev Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Errors-To: intel-wired-lan-bounces@osuosl.org Sender: "Intel-wired-lan" RGVhciBCYXJ0ZWssCgoKQW0gMTkuMDEuMjMgdW0gMTE6MTcgc2NocmllYiBQYXVsIE1lbnplbDoK PiAjcmVnemJvdCBeaW50cm9kdWNlZDogNC45Ljg4Li41LjEwLjE0OQoKPiBBbSAxNC4wMS4yMyB1 bSAxMToyMyBzY2hyaWViIEJhcnRlayBLb2lzOgo+IAo+PiBBZnRlciBtb3ZpbmcgZnJvbSBEZWJp YW4gOS43IHRvIDExLjUgYXMgc29vbiBhcyBJIHBlcmZvcm0gImlwIGxpbmsgc2V0IAo+PiBlbnAx czAgdXAiIGZvciBteSAxMEcgYWRhcHRlciAoQU9DLVNUR04tSTFTIC0gSW50ZWwgODI1OTlFTiBi YXNlZCAxMEcgCj4+IGFkYXB0ZXIpIEkgYW0gZXhwZXJpZW5jaW5nIGhpZ2ggY3B1IGxvYWQgKGV2 ZW4gaWYgbm8gdHJhZmZpYyBpcyAKPj4gcGFzc2luZyB0aHJvdWdoIHRoZSBhZGFwdGVyKSBhbmQg bmV0d29yayBwZXJmb3JtYW5jZSBpcyBsb3cgKHdoZW4gCj4+IG5ldHdvcmsgaXMgY29ubmVjdGVk KS4KPiAKPiBIb3cgZG8geW91IHRlc3QgdGhlIG5ldHdvcmsgcGVyZm9ybWFuY2U/IFBsZWFzZSBn aXZlIGV4YWN0IG51bWJlcnMgZm9yIAo+IGNvbXBhcmlzb24uCj4gCj4+IFRoZSBjcHUgbG9hZCBp cyBvc2NpbGxhdGluZyBiZXR3ZWVuIDAuMSBhbmQgMC4zIG9uIHZhbmlsbGEgc3lzdGVtCj4+IHdp dGggbm8gbmV0d29yayBhdHRhY2hlZC4gVGhlIHByb2JsZW0gY2FuIGJlIG9ic2VydmVkIG9uIHRo ZSBmb2xsb3dpbmcgCj4+IHBsYXRmb3JtczogU3VwZXJtaWNybyBYOVNDTCAoSW50ZWwgQzIwMiBQ Q0gpIGFuZAo+PiBTdXBlcm1pY3JvIFgxMFNMTCstRiAoSW50ZWwgQzIyMiBFeHByZXNzIFBDSCks IGJ1dCBmb3IgdGhlIFN1cGVybWljcm8KPj4gWDExU1NMLUYgKEludGVswq4gQzIzMiBjaGlwc2V0 KSBldmVyeXRoaW5nIGlzIHdvcmtpbmcgd2VsbC4KPj4KPj4gVGVzdGVkIGVudmlyb25tZW50czoK Pj4gRGViaWFuIDkuNyAtIExpbnV4IDQuOS4wLTYtYW1kNjQgIzEgU01QIERlYmlhbiA0LjkuODgt MStkZWI5dTEgCj4+ICgyMDE4LTA1LTA3KSB4ODZfNjQgR05VL0xpbnV4IFthbGwgcGxhdGZvcm1z IHdvcmtpbmcgd2VsbCB3aXRoIG5vIAo+PiBwcm9ibGVtczogU3VwZXJtaWNybyBYOVNDTCAoSW50 ZWwgQzIwMiBQQ0gpLCBTdXBlcm1pY3JvIFgxMFNMTCstRiAKPj4gKEludGVsIEMyMjIgRXhwcmVz cyBQQ0gpLCBTdXBlcm1pY3JvIFgxMVNTTC1GIChJbnRlbMKuIEMyMzIgY2hpcHNldCldCj4gCj4+ IERlYmlhbiAxMS41IC0gTGludXggNS4xMC4wLTE5LWFtZDY0ICMxIFNNUCBEZWJpYW4gNS4xMC4x NDktMiAKPj4gKDIwMjItMTAtMjEpIHg4Nl82NCBHTlUvTGludXjCoCBbb2xkZXIgcGxhdGZvcm1z OiBTdXBlcm1pY3JvIFg5U0NMIAo+PiAoSW50ZWwgQzIwMiBQQ0gpLCBTdXBlcm1pY3JvIFgxMFNM TCstRiAoSW50ZWwgQzIyMiBFeHByZXNzIFBDSCkgYmVoYXZlIAo+PiBwcm9ibGVtYXRpYyBhcyBk ZXNjcmliZWQgYWJvdmUgfCBuZXdlciBwbGF0Zm9ybTogU3VwZXJtaWNybyBYMTFTU0wtRiAKPj4g KEludGVswq4gQzIzMiBjaGlwc2V0KSB3b3JraW5nIHdlbGwgd2l0aCBubyBwcm9ibGVtc10KPiAK PiBNYXliZSBjcmVhdGUgYSBidWcgYXQgdGhlIExpbnV4IGtlcm5lbCBidWcgdHJhY2tlciBbMV0s IHdoZXJlIHlvdSBjYW4gCj4gYXR0YWNoIGFsbCB0aGUgbG9ncyAoYGRtZXNnYCwgYGxzcGNpIC1u bmsgLXMg4oCmYCwg4oCmKS4KPiAKPj4gU28gZmFyIHRvIHNvbHZlIHRoZSBwcm9ibGVtIEkgd2Fz IHRyeWluZyB0byB1cGdyYWRlIHN5c3RlbSB0byB0aGUgCj4+IG5ld2VzdCBzdGFibGUgdmVyc2lv biwgdXBncmFkZSBrZXJuZWwgdG8gdmVyc2lvbiA2LngsIHVwZ3JhZGUgaXhnYmUgCj4+IGRyaXZl ciB0byB0aGUgbmV3ZXN0IHZlcnNpb24gYnV0IHdpdGggbm8gbHVjay4KPiAKPiBUaGFuayB5b3Ug Zm9yIGNoZWNraW5nIHRoYXQuIFRvbyBiYWQgaXTigJlzIHN0aWxsIHByZXNlbnQuIFRvIHJ1bGUg b3V0IAo+IHNvbWUgdXNlciBzcGFjZSBwcm9ibGVtLCBjb3VsZCB5b3UgdGVzdCBEZWJpYW4gOS43 IHdpdGggYSBzdGFibGUgTGludXggCj4gcmVsZWFzZSwgY3VycmVudGx5IDYuMS43Pwo+IAo+IFdo YXQgZG9lcyBgc3VkbyBwZXJmIHRvcCAtLXNvcnQgY29tbSxkc29gIHNob3csIHdoZXJlIHRoZSB0 aW1lIGlzIHNwZW50Pwo+IAo+PiBTdXBlcm1pY3JvIHN1cHBvcnQgc3VnZ2VzdGVkIGFzIGZvbGxv d3M6Cj4+IGl0IG1pZ2h0IGJlIGtlcm5lbCByZWxhdGVkIGRlYmlhbiAxMS41IGhhcyBrZXJuZWwg NS4xMCB3aGljaCBpcyBhIAo+PiByZWNlbnQga2VybmVsIGl0IG1pZ2h0IG5vdCBwcm9wZXJseSBz dXBwb3J0IHRoZSBjaGlwc2V0cyBmb3IgWDkgCj4+IHRoZXJlZm9yZSBpIHN1Z2dlc3QgdG8gdXNl IFJIRUwgb3IgQ2VudE9TIGFzIHRoZXkgdXNlIG11Y2ggb2xkZXIgCj4+IGtlcm5lbCB2ZXJzaW9u cy4gSSBleHBlY3QgdGhhdCB3aXRoIHVidW50dSAyMC4wNCB5b3Ugc2VlIHRoZSBzYW1lIAo+PiBw cm9ibGVtIGl0IHVzZXMga2VybmVsIDUuNAo+IAo+IFRlc3RpbmcgYW5vdGhlciBHTlUvTGludXgg ZGlzdHJpYnV0aW9uIGZvciBhbm90aGVyIGRhdGEgcG9pbnQsIG1pZ2h0IGJlIAo+IGEgZ29vZCBp ZGVhLgo+IAo+IEFzIG5vYm9keSBoYXMgcmVzcG9uZGVkIHlldCwgYmlzZWN0aW5nIHRoZSBpc3N1 ZSBpcyBwcm9iYWJseSB0aGUgZmFzdGVzdCAKPiB3YXkgdG8gZ2V0IHRvIHRoZSBib3R0b20gb2Yg dGhpcy4gTHVja2lseSB0aGUgcHJvYmxlbSBzZWVtcyByZXByb2R1Y2libGUgCj4gYW5kIHlvdSBz ZWVtIHRvIGJlIGFibGUgdG8gYnVpbGQgYSBMaW51eCBrZXJuZWwgeW91cnNlbGYsIHNvIHRoYXQg c2hvdWxkIAo+IHdvcmsuIChGb3IgdGVzdGluZyBwdXJwb3NlcyB5b3UgY291bGQgYWxzbyB0ZXN0 IHdpdGggVWJ1bnR1LCBhcyB0aGV5IAo+IHByb3ZpZGUgTGludXgga2VybmVsIGJ1aWxkcyBmb3Ig KGFsbW9zdCkgYWxsIHJlbGVhc2VzIGluIHRoZWlyIExpbnV4IAo+IGtlcm5lbCBtYWlubGluZSBQ UEEgWzJdLikKCllvdSBjb3VsZCBhbHNvIHRyeSB0byBkbyB0aGF0IGluIGEgdmlydHVhbCBtYWNo aW5lIGJ5IHBhc3NpbmcgdGhyb3VnaCAKdGhlIG5ldHdvcmsgZGV2aWNlIHRvIHRoZSBWTS4gSWYg dGhhdCByZXByb2R1Y2VzIHRoZSBpc3N1ZSwgdGhhdOKAmXMgcXVpdGUgCmEgZmFzdCBzZXR1cCBm b3IgYmlzZWN0aW5nIGEgcmVncmVzc2lvbiwgYXMgc3RhcnQgdGltZXMgYXJlIHJlYWxseSBmYXN0 LiAKKEZvciBleGFtcGxlLCB5b3UgY2FuIHBhc3MgdGhlIExpbnV4IGtlcm5lbCBkaXJlY3RseSB0 byBhIFFFTVUgVk0gd2l0aCAKdGhlIGAta2VybmVsYCBzd2l0Y2guKQoKCktpbmQgcmVnYXJkcywK ClBhdWwKCgo+IFsxXTogaHR0cHM6Ly9idWd6aWxsYS5rZXJuZWwub3JnLwo+IFsyXTogaHR0cHM6 Ly9rZXJuZWwudWJ1bnR1LmNvbS9+a2VybmVsLXBwYS9tYWlubGluZS8KX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KSW50ZWwtd2lyZWQtbGFuIG1haWxpbmcg bGlzdApJbnRlbC13aXJlZC1sYW5Ab3N1b3NsLm9yZwpodHRwczovL2xpc3RzLm9zdW9zbC5vcmcv bWFpbG1hbi9saXN0aW5mby9pbnRlbC13aXJlZC1sYW4K