From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=0.7 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FUZZY_XPILL, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E9E8C282D8 for ; Fri, 1 Feb 2019 06:33:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DCD7920857 for ; Fri, 1 Feb 2019 06:33:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WqNjP/+i" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727574AbfBAGdK (ORCPT ); Fri, 1 Feb 2019 01:33:10 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:43972 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725876AbfBAGdK (ORCPT ); Fri, 1 Feb 2019 01:33:10 -0500 Received: by mail-wr1-f68.google.com with SMTP id r10so5764848wrs.10 for ; Thu, 31 Jan 2019 22:33:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=byVDMAF82wgKDYk4hCbqtT52By2tT+sbQl93kRJ1l6M=; b=WqNjP/+ia2V++5IPGTn2fSE5WMiGpDg7F9xasSB6OZdOiPtwep0J2CnOWs6AInYYrT xlpA23jUaDNqmL28gwvMit2eYszLzyh9bYN6BbKrtO6AiKNj3baCAc3LYfTinNyQa0NB iPYhQ80k6oeh+V2qPiniB0EgNQ3Vzdy0KS6dywn1jdp39B8EFO0ZpW+2r5mI6V/Criju VzjLfxBqKfKS0uSwi4uyRmpBHxJ1PyF4rkHW2ES4VHRvRQsacf/aYdMF3O9r+r3MKNi/ p19raejemQll9+jFxhwamCtemIgDL4n4Ya8zosePGvlUrt0JiLvPCHXmGP1BcCa4Hu65 cCYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=byVDMAF82wgKDYk4hCbqtT52By2tT+sbQl93kRJ1l6M=; b=EoaxApyy+UvoJf2kHJBFGK+QCf53rBh9W8j3ktj8gdFq0euNM4Xmh8xUwfZdXle/M1 MsR7+FyKvfxMVfRsH5KOVHrMs3CQmxWdntkFmGIb3t+N5TVoMp0rvMYGnpNmADa2boSS erBGM1R9pZ4c05pgfStkp/nZr+RO1T0AlDz5OJdlLdSZiRRUYRe/seOvZiR4f7XwRVY1 PYU/AfwXQbgjurUy836mUcFF51uDKBmNfgNjtvt2KoYrB1lJidApSaa2a7A59WKgwPaE 0PnkI/c0wGg+6i0iWd1YpOz/IqUIqqoLnXLWa0JYhnCx3tBYMKC11MUl4zH3FUFNavuC CDOg== X-Gm-Message-State: AHQUAubS3PSZyXLtDQJvohCE8dEbPa1eCY5gKkL0AH+u7hzDpMuaxUBa dzczb3M9hZ9x+vqDc68eXpw= X-Google-Smtp-Source: AHgI3IbxwK3bJ5/VaGjnKl4mAsrPc0mJun3IZy2nYxjZMoiUSwol+YKELCdDqki6tOTVscOQ+l57XA== X-Received: by 2002:a5d:444a:: with SMTP id x10mr23651652wrr.162.1549002786295; Thu, 31 Jan 2019 22:33:06 -0800 (PST) Received: from ?IPv6:2003:ea:8bf1:e200:81f8:3fab:207f:b0f2? (p200300EA8BF1E20081F83FAB207FB0F2.dip0.t-ipconnect.de. [2003:ea:8bf1:e200:81f8:3fab:207f:b0f2]) by smtp.googlemail.com with ESMTPSA id y138sm1958747wmc.16.2019.01.31.22.33.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 31 Jan 2019 22:33:04 -0800 (PST) Subject: Re: r8169 Driver - Poor Network Performance Since Kernel 4.19 To: David Chang Cc: Peter Ceiley , Realtek linux nic maintainers , netdev@vger.kernel.org, Martti Laaksonen References: <172787aa-9ef5-091d-f70f-baf89fe0b1ee@gmail.com> <20190131023240.GF25745@linux-kyyb.suse> <4d832c16-8830-b746-a818-6026c2e6725c@gmail.com> <20190201042907.GB7193@linux-kyyb.suse> From: Heiner Kallweit Message-ID: Date: Fri, 1 Feb 2019 07:32:57 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.5.0 MIME-Version: 1.0 In-Reply-To: <20190201042907.GB7193@linux-kyyb.suse> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Thanks, however the register diff is a little hard to read. Usually ethtool -d outputs something like this: RealTek RTL8168g/8111g registers: -------------------------------------------------------- 0x00: MAC Address 00:01:2e:83:90:11 0x08: Multicast Address Filter 0x100000c0 0x00000084 0x10: Dump Tally Counter Command 0x78c43000 0x00000001 0x20: Tx Normal Priority Ring Addr 0x77cc4000 0x00000001 0x28: Tx High Priority Ring Addr 0x00000000 0x00000000 0x30: Flash memory read/write 0x00000000 0x34: Early Rx Byte Count 0 0x36: Early Rx Status 0x00 0x37: Command 0x0c Rx on, Tx on 0x3C: Interrupt Mask 0x003f LinkChg RxNoBuf TxErr TxOK RxErr RxOK 0x3E: Interrupt Status 0x0000 0x40: Tx Configuration 0x4f000f80 0x44: Rx Configuration 0x0002cf0e 0x48: Timer count 0x00000000 0x4C: Missed packet counter 0x000000 0x50: EEPROM Command 0x10 0x51: Config 0 0x00 0x52: Config 1 0xcf 0x53: Config 2 0x9c 0x54: Config 3 0x60 0x55: Config 4 0x50 0x56: Config 5 0x01 0x58: Timer interrupt 0x00000000 0x5C: Multiple Interrupt Select 0x0000 0x60: PHY access 0x00000000 0x64: TBI control and status 0x00000000 0x68: TBI Autonegotiation advertisement (ANAR) 0x0000 0x6A: TBI Link partner ability (LPAR) 0x0000 0x6C: PHY status 0xf3 .. On 01.02.2019 05:29, David Chang wrote: > On Jan 31, 2019 at 07:35:30 +0100, Heiner Kallweit wrote: >> Hi David, two more things: >> >> 1. Could you please test a recent linux-next kernel? > > Not tested yet. Will do if possible. > >> 2. Please get a register dump (ethtool -d ) from 4.18 and 4.19 >> and compare them. > > For your informaiton. > > [with pcie_aspm=off] > --- v4.18.15 2019-02-01 12:11:56.019051828 +0800 > +++ v4.9.11 2019-02-01 12:12:26.827439645 +0800 > @@ -3,18 +3,19 @@ > Offset Values > ------ ------ > 0x0000: ec 8e b5 5a 2c f5 00 00 48 00 40 00 80 00 80 00 > -0x0010: 00 10 38 0e 04 00 00 00 78 00 06 00 00 00 00 00 > -0x0020: 00 f0 9b f6 03 00 00 00 00 00 00 00 00 00 00 00 > +0x0010: 00 f0 ba 0d 04 00 00 00 78 00 06 00 00 00 00 00 > +0x0020: 00 d0 35 f7 03 00 00 00 00 00 00 00 00 00 00 00 > 0x0030: 00 00 00 00 00 00 00 0c 00 00 00 00 3f 80 00 00 > -0x0040: 80 0f 10 57 0e cf 02 00 00 cf ba 34 00 00 00 00 > -0x0050: 10 00 cf 18 60 11 00 01 11 11 11 00 00 00 00 00 > +0x0040: 80 0f 10 57 0e cf 02 00 00 d8 c7 50 00 00 00 00 > +0x0050: 10 00 cf 98 60 11 01 01 11 11 11 00 00 00 00 00 > 0x0060: 00 00 00 00 3c 10 00 81 2c f0 00 80 93 00 80 f0 > -0x0070: 00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 00 00 > +0x0070: 00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 76 d0 > 0x0080: 8b 06 01 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > -0x00b0: 7f 04 00 00 00 00 00 00 e1 c1 05 d2 00 00 00 00 > +0x00b0: 7f 04 00 00 00 00 00 00 ad 79 01 d2 00 00 00 00 > 0x00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x00d0: 21 00 00 32 0e 00 00 00 00 00 00 40 06 11 fd 00 > -0x00e0: e1 20 51 51 00 30 94 f6 03 00 00 00 27 00 00 00 > +0x00e0: e1 20 51 51 00 e0 35 f7 03 00 00 00 27 00 00 00 > 0x00f0: 3f 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 > > [pcie_aspm.policy=performance] > --- v4.18.15-p 2019-02-01 12:18:46.919221060 +0800 > +++ v4.9.11-p 2019-02-01 12:19:09.207474824 +0800 > @@ -3,18 +3,19 @@ > Offset Values > ------ ------ > 0x0000: ec 8e b5 5a 2c f5 00 00 48 00 40 00 80 00 80 00 > -0x0010: 00 f0 bc 0d 04 00 00 00 78 00 06 00 00 00 00 00 > -0x0020: 00 60 2e f7 03 00 00 00 00 00 00 00 00 00 00 00 > +0x0010: 00 c0 22 09 04 00 00 00 78 00 06 00 00 00 00 00 > +0x0020: 00 f0 e5 f4 03 00 00 00 00 00 00 00 00 00 00 00 > 0x0030: 00 00 00 00 00 00 00 0c 00 00 00 00 3f 80 00 00 > -0x0040: 80 0f 10 57 0e cf 02 00 00 53 50 1a 00 00 00 00 > -0x0050: 10 00 cf 18 60 11 00 01 11 11 11 00 00 00 00 00 > +0x0040: 80 0f 10 57 0e cf 02 00 00 d2 35 7b 00 00 00 00 > +0x0050: 10 00 cf 98 60 11 01 01 11 11 11 00 00 00 00 00 > 0x0060: 00 00 00 00 3c 10 00 81 2c f0 00 80 93 00 80 f0 > -0x0070: 00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 00 00 > +0x0070: 00 6f 00 c4 b0 31 00 00 07 00 00 00 00 00 a4 a0 > 0x0080: 8b 06 01 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x0090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x00a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > -0x00b0: 7f 04 00 00 00 00 00 00 e1 c1 05 d2 00 00 00 00 > +0x00b0: 7f 04 00 00 00 00 00 00 ad 79 01 d2 00 00 00 00 > 0x00c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > 0x00d0: 21 00 00 32 0e 00 00 00 00 00 00 40 06 11 fd 00 > -0x00e0: e1 20 51 51 00 70 2e f7 03 00 00 00 27 00 00 00 > +0x00e0: e1 20 51 51 00 00 e6 f4 03 00 00 00 27 00 00 00 > 0x00f0: 3f 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 > > Thanks, > David > >> Heiner >> >> >> On 31.01.2019 07:21, Heiner Kallweit wrote: >>> David, thanks for the link to the bug ticket. >>> I think only a proper bisect can help to find the offending commit. >>> >>> Heiner >>> >>> >>> On 31.01.2019 03:32, David Chang wrote: >>>> Hi, >>>> >>>> We had a similr case here. >>>> - Realtek r8169 receive performance regression in kernel 4.19 >>>> https://bugzilla.suse.com/show_bug.cgi?id=1119649 >>>> >>>> kernel: r8169 0000:01:00.0 eth0: RTL8168h/8111h, XID 54100880 >>>> The major symptom is there are many rx_missed count. >>>> >>>> >>>> On Jan 30, 2019 at 20:15:45 +0100, Heiner Kallweit wrote: >>>>> Hi Peter, >>>>> >>>>> recently I had somebody where pcie_aspm=off for whatever reason didn't >>>>> do the trick, can you also check with pcie_aspm.policy=performance. >>>> >>>> We will give it a try later. >>>> >>>>> And please check with "ethtool -S " whether the chip statistics >>>>> show a significant number of errors. >>>>> >>>>> If this doesn't help you may have to bisect to find the offending commit. >>>> >>>> We had tried fallback driver to a few previous commits as following, >>>> but with no luck. >>>> >>>> 9675931e6b65 r8169: re-enable MSI-X on RTL8168g (v4.19) >>>> 098b01ad9837 r8169: don't include asm headers directly (v4.19-rc1) >>>> a2965f12fde6 r8169: remove rtl8169_set_speed_xmii (v4.19-rc1) >>>> 6fcf9b1d4d6c r8169: fix runtime suspend (v4.19-rc1) >>>> e397286b8e89 r8169: remove TBI 1000BaseX support (v4.19-rc1) >>>> >>>> Thanks, >>>> David Chang >>>> >>>>> >>>>> Heiner >>>>> >>>>> >>>>> On 30.01.2019 10:59, Peter Ceiley wrote: >>>>>> Hi Heiner, >>>>>> >>>>>> I tried disabling the ASPM using the pcie_aspm=off kernel parameter >>>>>> and this made no difference. >>>>>> >>>>>> I tried compiling the 4.18.16 r8169.c with the 4.19.18 source and >>>>>> subsequently loaded the module in the running 4.19.18 kernel. I can >>>>>> confirm that this immediately resolved the issue and access to the NFS >>>>>> shares operated as expected. >>>>>> >>>>>> I presume this means it is an issue with the r8169 driver included in >>>>>> 4.19 onwards? >>>>>> >>>>>> To answer your last questions: >>>>>> >>>>>> Base Board Information >>>>>> Manufacturer: Alienware >>>>>> Product Name: 0PGRP5 >>>>>> Version: A02 >>>>>> >>>>>> ... and yes, the RTL8168 is the onboard network chip. >>>>>> >>>>>> Regards, >>>>>> >>>>>> Peter. >>>>>> >>>>>> On Tue, 29 Jan 2019 at 17:44, Heiner Kallweit wrote: >>>>>>> >>>>>>> Hi Peter, >>>>>>> >>>>>>> I think the vendor driver doesn't enable ASPM per default. >>>>>>> So it's worth a try to disable ASPM in the BIOS or via sysfs. >>>>>>> Few older systems seem to have issues with ASPM, what kind of >>>>>>> system / mainboard are you using? The RTL8168 is the onboard >>>>>>> network chip? >>>>>>> >>>>>>> Rgds, Heiner >>>>>>> >>>>>>> >>>>>>> On 29.01.2019 07:20, Peter Ceiley wrote: >>>>>>>> Hi Heiner, >>>>>>>> >>>>>>>> Thanks, I'll do some more testing. It might not be the driver - I >>>>>>>> assumed it was due to the fact that using the r8168 driver 'resolves' >>>>>>>> the issue. I'll see if I can test the r8169.c on top of 4.19 - this is >>>>>>>> a good idea. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> Peter. >>>>>>>> >>>>>>>> On Tue, 29 Jan 2019 at 17:16, Heiner Kallweit wrote: >>>>>>>>> >>>>>>>>> Hi Peter, >>>>>>>>> >>>>>>>>> at a first glance it doesn't look like a typical driver issue. >>>>>>>>> What you could do: >>>>>>>>> >>>>>>>>> - Test the r8169.c from 4.18 on top of 4.19. >>>>>>>>> >>>>>>>>> - Check whether disabling ASPM (/sys/module/pcie_aspm) has an effect. >>>>>>>>> >>>>>>>>> - Bisect between 4.18 and 4.19 to find the offending commit. >>>>>>>>> >>>>>>>>> Any specific reason why you think root cause is in the driver and not >>>>>>>>> elsewhere in the network subsystem? >>>>>>>>> >>>>>>>>> Heiner >>>>>>>>> >>>>>>>>> >>>>>>>>> On 28.01.2019 23:10, Peter Ceiley wrote: >>>>>>>>>> Hi Heiner, >>>>>>>>>> >>>>>>>>>> Thanks for getting back to me. >>>>>>>>>> >>>>>>>>>> No, I don't use jumbo packets. >>>>>>>>>> >>>>>>>>>> Bandwidth is *generally* good, and iperf results to my NAS provide >>>>>>>>>> over 900 Mbits/s in both circumstances. The issue seems to appear when >>>>>>>>>> establishing a connection and is most notable, for example, on my >>>>>>>>>> mounted NFS shares where it takes seconds (up to 10's of seconds on >>>>>>>>>> larger directories) to list the contents of each directory. Once a >>>>>>>>>> transfer begins on a file, I appear to get good bandwidth. >>>>>>>>>> >>>>>>>>>> I'm unsure of the best scientific data to provide you in order to >>>>>>>>>> troubleshoot this issue. Running the following >>>>>>>>>> >>>>>>>>>> netstat -s |grep retransmitted >>>>>>>>>> >>>>>>>>>> shows a steady increase in retransmitted segments each time I list the >>>>>>>>>> contents of a remote directory, for example, running 'ls' on a >>>>>>>>>> directory containing 345 media files did the following using kernel >>>>>>>>>> 4.19.18: >>>>>>>>>> >>>>>>>>>> increased retransmitted segments by 21 and the 'time' command showed >>>>>>>>>> the following: >>>>>>>>>> real 0m19.867s >>>>>>>>>> user 0m0.012s >>>>>>>>>> sys 0m0.036s >>>>>>>>>> >>>>>>>>>> The same command shows no retransmitted segments running kernel >>>>>>>>>> 4.18.16 and 'time' showed: >>>>>>>>>> real 0m0.300s >>>>>>>>>> user 0m0.004s >>>>>>>>>> sys 0m0.007s >>>>>>>>>> >>>>>>>>>> ifconfig does not show any RX/TX errors nor dropped packets in either case. >>>>>>>>>> >>>>>>>>>> dmesg XID: >>>>>>>>>> [ 2.979984] r8169 0000:03:00.0 eth0: RTL8168g/8111g, >>>>>>>>>> f8:b1:56:fe:67:e0, XID 4c000800, IRQ 32 >>>>>>>>>> >>>>>>>>>> # lspci -vv >>>>>>>>>> 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. >>>>>>>>>> RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c) >>>>>>>>>> Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller >>>>>>>>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- >>>>>>>>>> ParErr- Stepping- SERR- FastB2B- DisINTx+ >>>>>>>>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- >>>>>>>>>> SERR- >>>>>>>>> Latency: 0, Cache Line Size: 64 bytes >>>>>>>>>> Interrupt: pin A routed to IRQ 19 >>>>>>>>>> Region 0: I/O ports at d000 [size=256] >>>>>>>>>> Region 2: Memory at f7b00000 (64-bit, non-prefetchable) [size=4K] >>>>>>>>>> Region 4: Memory at f2100000 (64-bit, prefetchable) [size=16K] >>>>>>>>>> Capabilities: [40] Power Management version 3 >>>>>>>>>> Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA >>>>>>>>>> PME(D0+,D1+,D2+,D3hot+,D3cold+) >>>>>>>>>> Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- >>>>>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ >>>>>>>>>> Address: 0000000000000000 Data: 0000 >>>>>>>>>> Capabilities: [70] Express (v2) Endpoint, MSI 01 >>>>>>>>>> DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s >>>>>>>>>> <512ns, L1 <64us >>>>>>>>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- >>>>>>>>>> SlotPowerLimit 10.000W >>>>>>>>>> DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- >>>>>>>>>> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- >>>>>>>>>> MaxPayload 128 bytes, MaxReadReq 4096 bytes >>>>>>>>>> DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- >>>>>>>>>> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit >>>>>>>>>> Latency L0s unlimited, L1 <64us >>>>>>>>>> ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ >>>>>>>>>> LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+ >>>>>>>>>> ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- >>>>>>>>>> LnkSta: Speed 2.5GT/s (ok), Width x1 (ok) >>>>>>>>>> TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- >>>>>>>>>> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, >>>>>>>>>> OBFF Via message/WAKE# >>>>>>>>>> AtomicOpsCap: 32bit- 64bit- 128bitCAS- >>>>>>>>>> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, >>>>>>>>>> OBFF Disabled >>>>>>>>>> AtomicOpsCtl: ReqEn- >>>>>>>>>> LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- >>>>>>>>>> Transmit Margin: Normal Operating Range, >>>>>>>>>> EnterModifiedCompliance- ComplianceSOS- >>>>>>>>>> Compliance De-emphasis: -6dB >>>>>>>>>> LnkSta2: Current De-emphasis Level: -6dB, >>>>>>>>>> EqualizationComplete-, EqualizationPhase1- >>>>>>>>>> EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- >>>>>>>>>> Capabilities: [b0] MSI-X: Enable+ Count=4 Masked- >>>>>>>>>> Vector table: BAR=4 offset=00000000 >>>>>>>>>> PBA: BAR=4 offset=00000800 >>>>>>>>>> Capabilities: [d0] Vital Product Data >>>>>>>>>> pcilib: sysfs_read_vpd: read failed: Input/output error >>>>>>>>>> Not readable >>>>>>>>>> Capabilities: [100 v1] Advanced Error Reporting >>>>>>>>>> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >>>>>>>>>> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >>>>>>>>>> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- >>>>>>>>>> RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- >>>>>>>>>> UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- >>>>>>>>>> RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- >>>>>>>>>> CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ AdvNonFatalErr- >>>>>>>>>> CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ >>>>>>>>>> AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- >>>>>>>>>> ECRCChkCap+ ECRCChkEn- >>>>>>>>>> MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- >>>>>>>>>> HeaderLog: 00000000 00000000 00000000 00000000 >>>>>>>>>> Capabilities: [140 v1] Virtual Channel >>>>>>>>>> Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 >>>>>>>>>> Arb: Fixed- WRR32- WRR64- WRR128- >>>>>>>>>> Ctrl: ArbSelect=Fixed >>>>>>>>>> Status: InProgress- >>>>>>>>>> VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- >>>>>>>>>> Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- >>>>>>>>>> Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 >>>>>>>>>> Status: NegoPending- InProgress- >>>>>>>>>> Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00 >>>>>>>>>> Capabilities: [170 v1] Latency Tolerance Reporting >>>>>>>>>> Max snoop latency: 71680ns >>>>>>>>>> Max no snoop latency: 71680ns >>>>>>>>>> Kernel driver in use: r8169 >>>>>>>>>> Kernel modules: r8169 >>>>>>>>>> >>>>>>>>>> Please let me know if you have any other ideas in terms of testing. >>>>>>>>>> >>>>>>>>>> Thanks! >>>>>>>>>> >>>>>>>>>> Peter. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, 29 Jan 2019 at 05:28, Heiner Kallweit wrote: >>>>>>>>>>> >>>>>>>>>>> On 28.01.2019 12:13, Peter Ceiley wrote: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I have been experiencing very poor network performance since Kernel >>>>>>>>>>>> 4.19 and I'm confident it's related to the r8169 driver. >>>>>>>>>>>> >>>>>>>>>>>> I have no issue with kernel versions 4.18 and prior. I am experiencing >>>>>>>>>>>> this issue in kernels 4.19 and 4.20 (currently running/testing with >>>>>>>>>>>> 4.20.4 & 4.19.18). >>>>>>>>>>>> >>>>>>>>>>>> If someone could guide me in the right direction, I'm happy to help >>>>>>>>>>>> troubleshoot this issue. Note that I have been keeping an eye on one >>>>>>>>>>>> issue related to loading of the PHY driver, however, my symptoms >>>>>>>>>>>> differ in that I still have a network connection. I have attempted to >>>>>>>>>>>> reload the driver on a running system, but this does not improve the >>>>>>>>>>>> situation. >>>>>>>>>>>> >>>>>>>>>>>> Using the proprietary r8168 driver returns my device to proper working order. >>>>>>>>>>>> >>>>>>>>>>>> lshw shows: >>>>>>>>>>>> description: Ethernet interface >>>>>>>>>>>> product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller >>>>>>>>>>>> vendor: Realtek Semiconductor Co., Ltd. >>>>>>>>>>>> physical id: 0 >>>>>>>>>>>> bus info: pci@0000:03:00.0 >>>>>>>>>>>> logical name: enp3s0 >>>>>>>>>>>> version: 0c >>>>>>>>>>>> serial: >>>>>>>>>>>> size: 1Gbit/s >>>>>>>>>>>> capacity: 1Gbit/s >>>>>>>>>>>> width: 64 bits >>>>>>>>>>>> clock: 33MHz >>>>>>>>>>>> capabilities: pm msi pciexpress msix vpd bus_master cap_list >>>>>>>>>>>> ethernet physical tp aui bnc mii fibre 10bt 10bt-fd 100bt 100bt-fd >>>>>>>>>>>> 1000bt-fd autonegotiation >>>>>>>>>>>> configuration: autonegotiation=on broadcast=yes driver=r8169 >>>>>>>>>>>> duplex=full firmware=rtl8168g-2_0.0.1 02/06/13 ip=192.168.1.25 >>>>>>>>>>>> latency=0 link=yes multicast=yes port=MII speed=1Gbit/s >>>>>>>>>>>> resources: irq:19 ioport:d000(size=256) >>>>>>>>>>>> memory:f7b00000-f7b00fff memory:f2100000-f2103fff >>>>>>>>>>>> >>>>>>>>>>>> Kind Regards, >>>>>>>>>>>> >>>>>>>>>>>> Peter. >>>>>>>>>>>> >>>>>>>>>>> Hi Peter, >>>>>>>>>>> >>>>>>>>>>> the description "poor network performance" is quite vague, therefore: >>>>>>>>>>> >>>>>>>>>>> - Can you provide any measurements? >>>>>>>>>>> - iperf results before and after >>>>>>>>>>> - statistics about dropped packets (rx and/or tx) >>>>>>>>>>> - Do you use jumbo packets? >>>>>>>>>>> >>>>>>>>>>> Also help would be a "lspci -vv" output for the network card and >>>>>>>>>>> the dmesg output line with the chip XID. >>>>>>>>>>> >>>>>>>>>>> Heiner >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >