* r8169/r8168 slow internet upload
@ 2014-05-11 23:19 Marc Duponcheel
2014-05-12 5:44 ` Francois Romieu
2014-06-23 22:46 ` Marc Duponcheel
0 siblings, 2 replies; 5+ messages in thread
From: Marc Duponcheel @ 2014-05-11 23:19 UTC (permalink / raw)
To: netdev; +Cc: Marc Duponcheel
Hi all
I have an issue with linux box acting as router (in fact forwarder)
with 2 Realtek RTL8111/8168/8411 cards.
In short: large file uploads are way too slow and, during such
uploads, I see TcpRetransSegs and TcpOutRsts on sending host.
System is bleeding edge gentoo.
I replaced linux r8169 by vendor r8168 driver. That improved some
uploads (see below).
- Details
--------
Setup is as follows (VDSL2) with down/up 50Mbps/6Mbps
intranet - server - router - modem - internet
modem =
b-box3 technicolor box in bridge mode
router =
The linux box in question doing pppoe on eth towards internet (where
it receives an address for p-t-p to next hop) and having public /30
network on eth towards intranet.
server =
Also a bleeding edge gentoo gentoo box having one public address from
the /30 network (router has the other one) and the doing NAT to
devices on intranet.
** note **
Before having the [router - modem] combo, I used a CISCO886VA-K9 box
and I had expected 50/6 throughput. But provider, who migrates to
VDSL2 vectoring, does not allow cisco on its network so I use b-box3
as bridge (the box they allow) and put a linux router behind.
- Symptoms
---------
I test ftp between my server and clients few or several hops away.
* r8169:
Incoming speed is OK (50Mbps) but outgoing is only 200Kbps. 30 times
too slow.
Now I set eth speeds to 100Mbps (on linux-router, I never touch
linux-server who works fine anyway for years with cisco-router):
Outgoing speed to few hops away is 2Mbps and to many hops away still 200Kbps.
* replace r8169 with r8168:
Outgoing to few hops away is 3Mbps and to many hops away is 1.5Mbps.
As said I always notice TcpRetransSegs and TcpOutRsts on server. I
use
# watch -n 5 'nstat | grep ^Tcp'
to inspect.
I see few (really few) router drops on eth Rx towards intranet. In
fact I am not sure if those increase during upload.
If I increase txqueuelen from 1000 to 10000 on router and flush tcp
metrics on router and server, outgoing speed starts 6Mbps but soon
drops back to steady 3Mbps and 1.5Mbps. I reverted back to 1000.
FYI: ookla speedtests on all hosts show 50/6 (only with with r8168 on
router), and, perhaps, for most work, everything is fine, but large
file uploads, in particular from our ftp server to internet many
hops away, is still 4 times too slow.
If I do
watch 'ethtool --statistics iface | grep -v ' 0''
on both router ethernets and server ethernet towards internet, I do not see
any error count increase.
Note that realtek driver does not support setting eth flow control so
I could not try to turn it off (not that I expect it to help).
Also PPP MTU/MRU are 1492 and TCPMSS clamp to PMTU is on server and router
(but that did not matter).
- Conclusions
------------
Setup is both driver and eth-speed sensitive. I also suspect TCP to
kick in to compensate for whatever issues.
- Questions
----------
Am I missing something obvious? Or are the RTL8111/8168/8411
cards/drivers not good for this setup? Any other hints? I am happy to
do whatever tests and provide whatever outputs that may help drive
discussion.
*** thanks ****
--
Marc Duponcheel
Velodroomstraat 74 - 2600 Berchem - Belgium
+32 (0)478 68.10.91 - marc@offline.be
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: r8169/r8168 slow internet upload
2014-05-11 23:19 r8169/r8168 slow internet upload Marc Duponcheel
@ 2014-05-12 5:44 ` Francois Romieu
[not found] ` <20140513005023.GA24091@offline.be>
2014-06-23 22:46 ` Marc Duponcheel
1 sibling, 1 reply; 5+ messages in thread
From: Francois Romieu @ 2014-05-12 5:44 UTC (permalink / raw)
To: Marc Duponcheel; +Cc: netdev
Marc Duponcheel <marc@offline.be> :
[...]
> I have an issue with linux box acting as router (in fact forwarder)
> with 2 Realtek RTL8111/8168/8411 cards.
>
> In short: large file uploads are way too slow and, during such
> uploads, I see TcpRetransSegs and TcpOutRsts on sending host.
>
> System is bleeding edge gentoo.
Please specify kernel version.
[...]
> * r8169:
>
> Incoming speed is OK (50Mbps) but outgoing is only 200Kbps. 30 times
> too slow.
Please send plain ethtool output and network capture file for the outgoing
traffic.
> Now I set eth speeds to 100Mbps (on linux-router, I never touch
> linux-server who works fine anyway for years with cisco-router):
>
> Outgoing speed to few hops away is 2Mbps and to many hops away still 200Kbps.
A capture file for each of those would be welcome.
[...]
> * replace r8169 with r8168:
>
> Outgoing to few hops away is 3Mbps and to many hops away is 1.5Mbps.
Same capture as above will be welcome.
[...]
> Am I missing something obvious? Or are the RTL8111/8168/8411
> cards/drivers not good for this setup? Any other hints? I am happy to
> do whatever tests and provide whatever outputs that may help drive
> discussion.
Nothing obvious on the driver side. There could be an r816x hardware bug
thus some (small) packets loss.
Please send a complete dmesg including the 'XID' line from the r8169
driver.
Thanks.
--
Ueimor
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: r8169/r8168 slow internet upload
[not found] ` <20140513005023.GA24091@offline.be>
@ 2014-05-14 22:03 ` Francois Romieu
[not found] ` <20140516020441.GB25757@offline.be>
0 siblings, 1 reply; 5+ messages in thread
From: Francois Romieu @ 2014-05-14 22:03 UTC (permalink / raw)
To: Marc Duponcheel; +Cc: netdev
Marc Duponcheel <marc@offline.be> :
[...]
> not something to do on a router)). That should be fine since r8168
> gave better results.
Is r8169 unusable even at 100 ?
[...]
> * case 100 (when router enp{2,3}s0 are 100Mbps)
> interestingly enough, speed took off at expected 6Mbps but dropped to
> stable 1.5Mbps
I haven't found a clear packet loss pattern yet.
Can you take the *same* (100) capture at enp3s0 ?
If the network speed test is fine as long as the client is near, I'd
seriously question your (c*gent ?) transit.
mtr output from far client to server would be welcome.
--
Ueimor
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: r8169/r8168 slow internet upload
[not found] ` <20140516020441.GB25757@offline.be>
@ 2014-05-16 2:55 ` Marc Duponcheel
0 siblings, 0 replies; 5+ messages in thread
From: Marc Duponcheel @ 2014-05-16 2:55 UTC (permalink / raw)
To: Francois Romieu; +Cc: netdev
Here is TCP nstat after some time (days) of normal. mixed user traffic
are TcpRetransSegs and TcpOutRsts reasonable ?
Thanks
opteron ~ # nstat | grep ^Tcp
TcpActiveOpens 218771 0.0
TcpPassiveOpens 245187 0.0
TcpAttemptFails 961 0.0
TcpEstabResets 1803 0.0
TcpInSegs 5539041 0.0
TcpOutSegs 5883556 0.0
TcpRetransSegs 38828 0.0
TcpOutRsts 369036 0.0
TcpExtSyncookiesFailed 112 0.0
TcpExtEmbryonicRsts 802 0.0
TcpExtTW 218409 0.0
TcpExtPAWSEstab 5 0.0
TcpExtDelayedACKs 32173 0.0
TcpExtDelayedACKLocked 50 0.0
TcpExtDelayedACKLost 1052 0.0
TcpExtListenDrops 7 0.0
TcpExtTCPPrequeued 216710 0.0
TcpExtTCPDirectCopyFromBacklog 2218150 0.0
TcpExtTCPDirectCopyFromPrequeue 31007060 0.0
TcpExtTCPHPHits 1189383 0.0
TcpExtTCPHPHitsToUser 23700 0.0
TcpExtTCPPureAcks 1265234 0.0
TcpExtTCPHPAcks 1373386 0.0
TcpExtTCPSackRecovery 10161 0.0
TcpExtTCPSACKReneging 1 0.0
TcpExtTCPFullUndo 2 0.0
TcpExtTCPDSACKUndo 4 0.0
TcpExtTCPLossUndo 103 0.0
TcpExtTCPLostRetransmit 23 0.0
TcpExtTCPSackFailures 7 0.0
TcpExtTCPLossFailures 46 0.0
TcpExtTCPFastRetrans 29705 0.0
TcpExtTCPForwardRetrans 144 0.0
TcpExtTCPSlowStartRetrans 186 0.0
TcpExtTCPTimeouts 914 0.0
TcpExtTCPLossProbes 2094 0.0
TcpExtTCPLossProbeRecovery 250 0.0
TcpExtTCPSackRecoveryFail 63 0.0
TcpExtTCPSchedulerFailed 1 0.0
TcpExtTCPDSACKOldSent 1041 0.0
TcpExtTCPDSACKOfoSent 55 0.0
TcpExtTCPDSACKRecv 967 0.0
TcpExtTCPAbortOnData 1520 0.0
TcpExtTCPAbortOnClose 185 0.0
TcpExtTCPAbortOnTimeout 256 0.0
TcpExtTCPDSACKIgnoredNoUndo 85 0.0
TcpExtTCPSpuriousRTOs 11 0.0
TcpExtTCPSackShifted 28438 0.0
TcpExtTCPSackMerged 26733 0.0
TcpExtTCPSackShiftFallback 32560 0.0
TcpExtTCPDeferAcceptDrop 335 0.0
TcpExtIPReversePathFilter 24 0.0
TcpExtTCPRetransFail 114 0.0
TcpExtTCPRcvCoalesce 338862 0.0
TcpExtTCPOFOQueue 5712 0.0
TcpExtTCPOFOMerge 55 0.0
TcpExtTCPChallengeACK 20 0.0
TcpExtTCPSpuriousRtxHostQueues 463 0.0
TcpExtTCPAutoCorking 48213 0.0
--
Marc Duponcheel
Velodroomstraat 74 - 2600 Berchem - Belgium
+32 (0)478 68.10.91 - marc@offline.be
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: r8169/r8168 slow internet upload
2014-05-11 23:19 r8169/r8168 slow internet upload Marc Duponcheel
2014-05-12 5:44 ` Francois Romieu
@ 2014-06-23 22:46 ` Marc Duponcheel
1 sibling, 0 replies; 5+ messages in thread
From: Marc Duponcheel @ 2014-06-23 22:46 UTC (permalink / raw)
To: netdev; +Cc: Marc Duponcheel
FYI
it seems upgrade to 3.14.5 -> 3.15.1 fixed issue ...
not sure which change implements fix but, in any case:
THANKS
On 2014 May 12, Marc Duponcheel wrote:
> Hi all
>
> I have an issue with linux box acting as router (in fact forwarder)
> with 2 Realtek RTL8111/8168/8411 cards.
>
> In short: large file uploads are way too slow and, during such
> uploads, I see TcpRetransSegs and TcpOutRsts on sending host.
>
> System is bleeding edge gentoo.
>
> I replaced linux r8169 by vendor r8168 driver. That improved some
> uploads (see below).
>
>
> - Details
> --------
>
> Setup is as follows (VDSL2) with down/up 50Mbps/6Mbps
>
>
> intranet - server - router - modem - internet
>
>
> modem =
> b-box3 technicolor box in bridge mode
>
> router =
> The linux box in question doing pppoe on eth towards internet (where
> it receives an address for p-t-p to next hop) and having public /30
> network on eth towards intranet.
>
> server =
> Also a bleeding edge gentoo gentoo box having one public address from
> the /30 network (router has the other one) and the doing NAT to
> devices on intranet.
>
> ** note **
>
> Before having the [router - modem] combo, I used a CISCO886VA-K9 box
> and I had expected 50/6 throughput. But provider, who migrates to
> VDSL2 vectoring, does not allow cisco on its network so I use b-box3
> as bridge (the box they allow) and put a linux router behind.
>
>
> - Symptoms
> ---------
>
> I test ftp between my server and clients few or several hops away.
>
> * r8169:
>
> Incoming speed is OK (50Mbps) but outgoing is only 200Kbps. 30 times
> too slow.
>
> Now I set eth speeds to 100Mbps (on linux-router, I never touch
> linux-server who works fine anyway for years with cisco-router):
>
> Outgoing speed to few hops away is 2Mbps and to many hops away still 200Kbps.
>
> * replace r8169 with r8168:
>
> Outgoing to few hops away is 3Mbps and to many hops away is 1.5Mbps.
>
> As said I always notice TcpRetransSegs and TcpOutRsts on server. I
> use
> # watch -n 5 'nstat | grep ^Tcp'
> to inspect.
>
>
> I see few (really few) router drops on eth Rx towards intranet. In
> fact I am not sure if those increase during upload.
>
>
> If I increase txqueuelen from 1000 to 10000 on router and flush tcp
> metrics on router and server, outgoing speed starts 6Mbps but soon
> drops back to steady 3Mbps and 1.5Mbps. I reverted back to 1000.
>
> FYI: ookla speedtests on all hosts show 50/6 (only with with r8168 on
> router), and, perhaps, for most work, everything is fine, but large
> file uploads, in particular from our ftp server to internet many
> hops away, is still 4 times too slow.
>
> If I do
> watch 'ethtool --statistics iface | grep -v ' 0''
> on both router ethernets and server ethernet towards internet, I do not see
> any error count increase.
>
> Note that realtek driver does not support setting eth flow control so
> I could not try to turn it off (not that I expect it to help).
>
> Also PPP MTU/MRU are 1492 and TCPMSS clamp to PMTU is on server and router
> (but that did not matter).
>
> - Conclusions
> ------------
>
> Setup is both driver and eth-speed sensitive. I also suspect TCP to
> kick in to compensate for whatever issues.
>
>
> - Questions
> ----------
>
> Am I missing something obvious? Or are the RTL8111/8168/8411
> cards/drivers not good for this setup? Any other hints? I am happy to
> do whatever tests and provide whatever outputs that may help drive
> discussion.
>
>
> *** thanks ****
--
Marc Duponcheel
Velodroomstraat 74 - 2600 Berchem - Belgium
+32 (0)478 68.10.91 - marc@offline.be
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2014-06-23 23:05 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-05-11 23:19 r8169/r8168 slow internet upload Marc Duponcheel
2014-05-12 5:44 ` Francois Romieu
[not found] ` <20140513005023.GA24091@offline.be>
2014-05-14 22:03 ` Francois Romieu
[not found] ` <20140516020441.GB25757@offline.be>
2014-05-16 2:55 ` Marc Duponcheel
2014-06-23 22:46 ` Marc Duponcheel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).