From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIMWL_WL_MED,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83BA1C169C4 for ; Tue, 29 Jan 2019 06:20:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 40B6E2148E for ; Tue, 29 Jan 2019 06:20:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ceiley-net.20150623.gappssmtp.com header.i=@ceiley-net.20150623.gappssmtp.com header.b="OUQ3rroa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727545AbfA2GU6 (ORCPT ); Tue, 29 Jan 2019 01:20:58 -0500 Received: from mail-ed1-f68.google.com ([209.85.208.68]:41815 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725832AbfA2GU4 (ORCPT ); Tue, 29 Jan 2019 01:20:56 -0500 Received: by mail-ed1-f68.google.com with SMTP id a20so15019583edc.8 for ; Mon, 28 Jan 2019 22:20:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ceiley-net.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=uX4zmiougHHQAUHkfQpAwxsAYZVCUqvkMuwFBIRSLSc=; b=OUQ3rroaZ58KgGxRM0uUfee1fptk10seqAmhFQzLyzspxcW1z7+F2KP0t5Rn5JYSQF zDl620+XQaHdHrB/e7RK+FSltfB06TUznVKw40QIFxsrcl4ddzpOHuMw8n5Q4mbOtHou MEPUct9ydNEWQjuIllS5r9dvymZB/N7+fTdXYtir2Y/tCgMwCFGP0JfLml4QBNACf0OV sLMEk3+MMx2Tnw8JWt2bDBfycJ5Bqv0Bz2DVHM858Fw2FiFMsCw2xaM37fYLSh9Sqx0s xK0zkzJu7F52htoYgnwvgCS7cFaqVG2QOBN4yxQgWGl8sElLOm6BNktIzNBsGmP4e4TE +Ekg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=uX4zmiougHHQAUHkfQpAwxsAYZVCUqvkMuwFBIRSLSc=; b=W1rAz7uHKLMsWjjJgefbUfnUllRwWdp/5k2ChszMChQuBXhH+4AKFW+m5KoEHNfTSr Hg8Vor46oL+VF+gnGOHcyqPS//kVByYYDl2Z7gKJUSB6uwIQaglNqCiPiC+lCqqk9LAd 3V/DCNudMyCeZ1NeadnlkuD9h2JFXmXZ8ioakPFe71KtY7EQhVpTbHKdWsmXYYzczFnA G34UvRUQPkMxnKlljXEeEiwuxKdu2XNjv6excQdJ9ujN9tXqgjF+1PQHhF5nqPUQxmdk FH0rHxkJeidRPg4yw54zGwLOVMe/8m1FU3zf0WwS5kMVzelUHvMb1N9bL8xyTOdbuXU4 JwaA== X-Gm-Message-State: AJcUukf6lyUu8je0zM+jECG5V8Bog/j3Y3cv2KL/TOu/rBxtvQ9xDQhe swenvEKXYutMywva5gXoeTmch1nP6FjfjQgRuRWIXw== X-Google-Smtp-Source: ALg8bN6LUjNa+8QILT2Rdm/PpbOCDAYpd4x10cTan7V++uZQJLagg5xmN4JcboPh6HCvXkr2Ys9PMqOR3aseTB/iwi4= X-Received: by 2002:aa7:c703:: with SMTP id i3mr24408232edq.170.1548742853757; Mon, 28 Jan 2019 22:20:53 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Peter Ceiley Date: Tue, 29 Jan 2019 17:20:40 +1100 Message-ID: Subject: Re: r8169 Driver - Poor Network Performance Since Kernel 4.19 To: Heiner Kallweit Cc: Realtek linux nic maintainers , netdev@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Hi Heiner, Thanks, I'll do some more testing. It might not be the driver - I assumed it was due to the fact that using the r8168 driver 'resolves' the issue. I'll see if I can test the r8169.c on top of 4.19 - this is a good idea. Cheers, Peter. On Tue, 29 Jan 2019 at 17:16, Heiner Kallweit wrote: > > Hi Peter, > > at a first glance it doesn't look like a typical driver issue. > What you could do: > > - Test the r8169.c from 4.18 on top of 4.19. > > - Check whether disabling ASPM (/sys/module/pcie_aspm) has an effect. > > - Bisect between 4.18 and 4.19 to find the offending commit. > > Any specific reason why you think root cause is in the driver and not > elsewhere in the network subsystem? > > Heiner > > > On 28.01.2019 23:10, Peter Ceiley wrote: > > Hi Heiner, > > > > Thanks for getting back to me. > > > > No, I don't use jumbo packets. > > > > Bandwidth is *generally* good, and iperf results to my NAS provide > > over 900 Mbits/s in both circumstances. The issue seems to appear when > > establishing a connection and is most notable, for example, on my > > mounted NFS shares where it takes seconds (up to 10's of seconds on > > larger directories) to list the contents of each directory. Once a > > transfer begins on a file, I appear to get good bandwidth. > > > > I'm unsure of the best scientific data to provide you in order to > > troubleshoot this issue. Running the following > > > > netstat -s |grep retransmitted > > > > shows a steady increase in retransmitted segments each time I list the > > contents of a remote directory, for example, running 'ls' on a > > directory containing 345 media files did the following using kernel > > 4.19.18: > > > > increased retransmitted segments by 21 and the 'time' command showed > > the following: > > real 0m19.867s > > user 0m0.012s > > sys 0m0.036s > > > > The same command shows no retransmitted segments running kernel > > 4.18.16 and 'time' showed: > > real 0m0.300s > > user 0m0.004s > > sys 0m0.007s > > > > ifconfig does not show any RX/TX errors nor dropped packets in either case. > > > > dmesg XID: > > [ 2.979984] r8169 0000:03:00.0 eth0: RTL8168g/8111g, > > f8:b1:56:fe:67:e0, XID 4c000800, IRQ 32 > > > > # lspci -vv > > 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > > RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c) > > Subsystem: Dell RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- > > ParErr- Stepping- SERR- FastB2B- DisINTx+ > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > > SERR- > Latency: 0, Cache Line Size: 64 bytes > > Interrupt: pin A routed to IRQ 19 > > Region 0: I/O ports at d000 [size=256] > > Region 2: Memory at f7b00000 (64-bit, non-prefetchable) [size=4K] > > Region 4: Memory at f2100000 (64-bit, prefetchable) [size=16K] > > Capabilities: [40] Power Management version 3 > > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA > > PME(D0+,D1+,D2+,D3hot+,D3cold+) > > Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ > > Address: 0000000000000000 Data: 0000 > > Capabilities: [70] Express (v2) Endpoint, MSI 01 > > DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s > > <512ns, L1 <64us > > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > > SlotPowerLimit 10.000W > > DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- > > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- > > MaxPayload 128 bytes, MaxReadReq 4096 bytes > > DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- > > LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit > > Latency L0s unlimited, L1 <64us > > ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ > > LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+ > > ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt- > > LnkSta: Speed 2.5GT/s (ok), Width x1 (ok) > > TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- > > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, > > OBFF Via message/WAKE# > > AtomicOpsCap: 32bit- 64bit- 128bitCAS- > > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, > > OBFF Disabled > > AtomicOpsCtl: ReqEn- > > LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- > > Transmit Margin: Normal Operating Range, > > EnterModifiedCompliance- ComplianceSOS- > > Compliance De-emphasis: -6dB > > LnkSta2: Current De-emphasis Level: -6dB, > > EqualizationComplete-, EqualizationPhase1- > > EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- > > Capabilities: [b0] MSI-X: Enable+ Count=4 Masked- > > Vector table: BAR=4 offset=00000000 > > PBA: BAR=4 offset=00000800 > > Capabilities: [d0] Vital Product Data > > pcilib: sysfs_read_vpd: read failed: Input/output error > > Not readable > > Capabilities: [100 v1] Advanced Error Reporting > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- > > RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > > CESta: RxErr+ BadTLP+ BadDLLP+ Rollover- Timeout+ AdvNonFatalErr- > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ > > AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- > > ECRCChkCap+ ECRCChkEn- > > MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap- > > HeaderLog: 00000000 00000000 00000000 00000000 > > Capabilities: [140 v1] Virtual Channel > > Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 > > Arb: Fixed- WRR32- WRR64- WRR128- > > Ctrl: ArbSelect=Fixed > > Status: InProgress- > > VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- > > Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256- > > Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=01 > > Status: NegoPending- InProgress- > > Capabilities: [160 v1] Device Serial Number 01-00-00-00-68-4c-e0-00 > > Capabilities: [170 v1] Latency Tolerance Reporting > > Max snoop latency: 71680ns > > Max no snoop latency: 71680ns > > Kernel driver in use: r8169 > > Kernel modules: r8169 > > > > Please let me know if you have any other ideas in terms of testing. > > > > Thanks! > > > > Peter. > > > > > > > > > > > > > > > > > > > > On Tue, 29 Jan 2019 at 05:28, Heiner Kallweit wrote: > >> > >> On 28.01.2019 12:13, Peter Ceiley wrote: > >>> Hi, > >>> > >>> I have been experiencing very poor network performance since Kernel > >>> 4.19 and I'm confident it's related to the r8169 driver. > >>> > >>> I have no issue with kernel versions 4.18 and prior. I am experiencing > >>> this issue in kernels 4.19 and 4.20 (currently running/testing with > >>> 4.20.4 & 4.19.18). > >>> > >>> If someone could guide me in the right direction, I'm happy to help > >>> troubleshoot this issue. Note that I have been keeping an eye on one > >>> issue related to loading of the PHY driver, however, my symptoms > >>> differ in that I still have a network connection. I have attempted to > >>> reload the driver on a running system, but this does not improve the > >>> situation. > >>> > >>> Using the proprietary r8168 driver returns my device to proper working order. > >>> > >>> lshw shows: > >>> description: Ethernet interface > >>> product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller > >>> vendor: Realtek Semiconductor Co., Ltd. > >>> physical id: 0 > >>> bus info: pci@0000:03:00.0 > >>> logical name: enp3s0 > >>> version: 0c > >>> serial: > >>> size: 1Gbit/s > >>> capacity: 1Gbit/s > >>> width: 64 bits > >>> clock: 33MHz > >>> capabilities: pm msi pciexpress msix vpd bus_master cap_list > >>> ethernet physical tp aui bnc mii fibre 10bt 10bt-fd 100bt 100bt-fd > >>> 1000bt-fd autonegotiation > >>> configuration: autonegotiation=on broadcast=yes driver=r8169 > >>> duplex=full firmware=rtl8168g-2_0.0.1 02/06/13 ip=192.168.1.25 > >>> latency=0 link=yes multicast=yes port=MII speed=1Gbit/s > >>> resources: irq:19 ioport:d000(size=256) > >>> memory:f7b00000-f7b00fff memory:f2100000-f2103fff > >>> > >>> Kind Regards, > >>> > >>> Peter. > >>> > >> Hi Peter, > >> > >> the description "poor network performance" is quite vague, therefore: > >> > >> - Can you provide any measurements? > >> - iperf results before and after > >> - statistics about dropped packets (rx and/or tx) > >> - Do you use jumbo packets? > >> > >> Also help would be a "lspci -vv" output for the network card and > >> the dmesg output line with the chip XID. > >> > >> Heiner > > >