TCP send behaviour leads to cable modem woes

TCP send behaviour leads to cable modem woes

[Svein Ove Aas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

My internet connection is via a cable modem, and thereon from Telenor. (A 
Norwegian ISP.)

In general, when I download something I can get up to 1400-1500 Kb/s, which is 
pretty good for a 1024/256 account. (They don't appear to oversubscribe their 
lines (yahoo!), but mine is also uncapped when there is spare capacity. Think 
traffic-control.)

So far, so good.

My account includes 4 IP addresses, and when I actually have four computers 
directly connected I can easily get 7-8Kb/s upload from each of them.
Oddly, when one of them is acting as a firewall/bridge for the others or I'm 
just uploading from one, I get 7-8Kb/s for *all* of them. (Or the one.)

This is, dare I say, *not* expected behaviour.
I've been in contact with telenor about it, and have garnered the following 
information.

(A)	Although the line appear to be straight Ethernet attached to a 
packet-filtering switch (just ARP-filtering, actually), it's *actually* an 
ATM-based line. This should come as no surprise.

(B)	Whatever they have allocating the ATM cells for transfer is doing it in 
bursts of about 16KB. Or possibly 32KB. Well, the tech I talked to was pretty 
sure it was a power of two, at least.

(C)	This means that while I get 8 bursts (or more) of 16KB per second on 
download (empirically confirmed, but the cable modem will tend to space it 
out when the line is at capacity), giving me a latency of 128-256 ms and so 
on and so forth (which I have), I get only *two* bursts per second to upload 
things. I think. You may want to apply a multiplier somewhere.

And, finally, (D):

This thoroughly screws up TCP/IP for uploading purposes. It *completely* 
breaks Realtek cards, screws up uploading speeds in Linux and Windows XP (I 
assume they think there is a lot of intermittent packet loss because of the 
delay), and has no apparent effect on Windows 9x/2000.

The cable modem in question is manufactured by Coresma and is marked NeMo. 
It's also supposed to have a pretty large send buffer, so if I could just 
force Linux to send at some user-defined speed without being so paranoid 
about overloading the line, I could get a lot more use out of it.

For the curious, if I do just that with UDP, I can indeed send at up to 30KB/s 
without losing packets. They *do* come in bursts, though.


Please, save me before I lose my mind!

- - Svein Ove Aas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE+/IsG9OlFkai3rMARAmZ4AKCeGIXGhREfh0kcA4Dr8FJs9fNuFgCg1sTb
1bk3+ipUs9tS35oZidxcY4I=
=Zz5P
-----END PGP SIGNATURE-----

Re: TCP send behaviour leads to cable modem woes

[Mika Liljeberg]

Try enabling tcp_frto on your Linux box to see if that helps the
uploads.

	MikaL

On Fri, 2003-06-27 at 21:20, Svein Ove Aas wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> My internet connection is via a cable modem, and thereon from Telenor. (A 
> Norwegian ISP.)
> 
> In general, when I download something I can get up to 1400-1500 Kb/s, which is 
> pretty good for a 1024/256 account. (They don't appear to oversubscribe their 
> lines (yahoo!), but mine is also uncapped when there is spare capacity. Think 
> traffic-control.)
> 
> So far, so good.
> 
> My account includes 4 IP addresses, and when I actually have four computers 
> directly connected I can easily get 7-8Kb/s upload from each of them.
> Oddly, when one of them is acting as a firewall/bridge for the others or I'm 
> just uploading from one, I get 7-8Kb/s for *all* of them. (Or the one.)
> 
> This is, dare I say, *not* expected behaviour.
> I've been in contact with telenor about it, and have garnered the following 
> information.
> 
> (A)	Although the line appear to be straight Ethernet attached to a 
> packet-filtering switch (just ARP-filtering, actually), it's *actually* an 
> ATM-based line. This should come as no surprise.
> 
> (B)	Whatever they have allocating the ATM cells for transfer is doing it in 
> bursts of about 16KB. Or possibly 32KB. Well, the tech I talked to was pretty 
> sure it was a power of two, at least.
> 
> (C)	This means that while I get 8 bursts (or more) of 16KB per second on 
> download (empirically confirmed, but the cable modem will tend to space it 
> out when the line is at capacity), giving me a latency of 128-256 ms and so 
> on and so forth (which I have), I get only *two* bursts per second to upload 
> things. I think. You may want to apply a multiplier somewhere.
> 
> And, finally, (D):
> 
> This thoroughly screws up TCP/IP for uploading purposes. It *completely* 
> breaks Realtek cards, screws up uploading speeds in Linux and Windows XP (I 
> assume they think there is a lot of intermittent packet loss because of the 
> delay), and has no apparent effect on Windows 9x/2000.
> 
> The cable modem in question is manufactured by Coresma and is marked NeMo. 
> It's also supposed to have a pretty large send buffer, so if I could just 
> force Linux to send at some user-defined speed without being so paranoid 
> about overloading the line, I could get a lot more use out of it.
> 
> For the curious, if I do just that with UDP, I can indeed send at up to 30KB/s 
> without losing packets. They *do* come in bursts, though.
> 
> 
> Please, save me before I lose my mind!
> 
> - - Svein Ove Aas
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.2 (GNU/Linux)
> 
> iD8DBQE+/IsG9OlFkai3rMARAmZ4AKCeGIXGhREfh0kcA4Dr8FJs9fNuFgCg1sTb
> 1bk3+ipUs9tS35oZidxcY4I=
> =Zz5P
> -----END PGP SIGNATURE-----
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: TCP send behaviour leads to cable modem woes

[Svein Ove Aas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

fredag 27. juni 2003, 21:02, skrev Mika Liljeberg:
> Try enabling tcp_frto on your Linux box to see if that helps the
> uploads.

I'm runing 2.4.20-gentoo at the moment, but I'll try that later.

Incidentally, while googling I heard someone saying that only works if it's 
enabled on both ends? Of course, that might be if upload/download are 'both' 
affected, in which case it wouldn't apply to me.

Well, I'll try it, anyway. Sounds interesting.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE+/J7Q9OlFkai3rMARAmYpAJ4mSwEONKs2h/8SC02pq/TTTs+1BgCfXaAu
ZNcnbYAm8Rl3BE87jB1Z8gY=
=Vlbk
-----END PGP SIGNATURE-----

Re: TCP send behaviour leads to cable modem woes

[Mika Liljeberg]

On Fri, 2003-06-27 at 22:45, Svein Ove Aas wrote:
> Incidentally, while googling I heard someone saying that only works if it's 
> enabled on both ends? Of course, that might be if upload/download are 'both' 
> affected, in which case it wouldn't apply to me.

It's a sender side only algorithm, so enabling it at your end should be
enough to help the uploads. For downloads it needs to be on at the other
end, of course.

	MikaL

Re: TCP send behaviour leads to cable modem woes

[Bryan Whitehead]

Take a look at the wondershaper script.

It's helped my DSL get good rates from home both up and down...

http://lartc.org/wondershaper/

Svein Ove Aas wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> My internet connection is via a cable modem, and thereon from Telenor. (A 
> Norwegian ISP.)
> 
> In general, when I download something I can get up to 1400-1500 Kb/s, which is 
> pretty good for a 1024/256 account. (They don't appear to oversubscribe their 
> lines (yahoo!), but mine is also uncapped when there is spare capacity. Think 
> traffic-control.)
> 
> So far, so good.
> 
> My account includes 4 IP addresses, and when I actually have four computers 
> directly connected I can easily get 7-8Kb/s upload from each of them.
> Oddly, when one of them is acting as a firewall/bridge for the others or I'm 
> just uploading from one, I get 7-8Kb/s for *all* of them. (Or the one.)
> 
> This is, dare I say, *not* expected behaviour.
> I've been in contact with telenor about it, and have garnered the following 
> information.
> 
> (A)	Although the line appear to be straight Ethernet attached to a 
> packet-filtering switch (just ARP-filtering, actually), it's *actually* an 
> ATM-based line. This should come as no surprise.
> 
> (B)	Whatever they have allocating the ATM cells for transfer is doing it in 
> bursts of about 16KB. Or possibly 32KB. Well, the tech I talked to was pretty 
> sure it was a power of two, at least.
> 
> (C)	This means that while I get 8 bursts (or more) of 16KB per second on 
> download (empirically confirmed, but the cable modem will tend to space it 
> out when the line is at capacity), giving me a latency of 128-256 ms and so 
> on and so forth (which I have), I get only *two* bursts per second to upload 
> things. I think. You may want to apply a multiplier somewhere.
> 
> And, finally, (D):
> 
> This thoroughly screws up TCP/IP for uploading purposes. It *completely* 
> breaks Realtek cards, screws up uploading speeds in Linux and Windows XP (I 
> assume they think there is a lot of intermittent packet loss because of the 
> delay), and has no apparent effect on Windows 9x/2000.
> 
> The cable modem in question is manufactured by Coresma and is marked NeMo. 
> It's also supposed to have a pretty large send buffer, so if I could just 
> force Linux to send at some user-defined speed without being so paranoid 
> about overloading the line, I could get a lot more use out of it.
> 
> For the curious, if I do just that with UDP, I can indeed send at up to 30KB/s 
> without losing packets. They *do* come in bursts, though.
> 
> 
> Please, save me before I lose my mind!
> 
> - - Svein Ove Aas
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.2 (GNU/Linux)
> 
> iD8DBQE+/IsG9OlFkai3rMARAmZ4AKCeGIXGhREfh0kcA4Dr8FJs9fNuFgCg1sTb
> 1bk3+ipUs9tS35oZidxcY4I=
> =Zz5P
> -----END PGP SIGNATURE-----
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


-- 
Bryan Whitehead
SysAdmin - JPL - Interferometry and Large Optical Systems
Phone: 818 354 2903
driver@jpl.nasa.gov

Re: TCP send behaviour leads to cable modem woes

[Svein Ove Aas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

fredag 27. juni 2003, 22:09, skrev Bryan Whitehead:
> Take a look at the wondershaper script.
>
> It's helped my DSL get good rates from home both up and down...
>
> http://lartc.org/wondershaper/

I wrote something like that myself once.
It's a good shaper, but it works by *capping* up/download speeds and 
rearranging the priorities locally, which wouldn't help me a bit.

My problem seems to be the severe burstiness of my connection - read the head 
post for a full explanation.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE+/Kfj9OlFkai3rMARAsINAKCCNHifkbI098wUxyzXX8Tp+V9KUgCfbxDR
JUKHOvOjTpXMWZLQ6nw6E4E=
=1x7f
-----END PGP SIGNATURE-----

Re: TCP send behaviour leads to cable modem woes

[Andre Tomt]

On fre, 2003-06-27 at 22:24, Svein Ove Aas wrote:
> > http://lartc.org/wondershaper/
> 
> I wrote something like that myself once.
> It's a good shaper, but it works by *capping* up/download speeds and 
> rearranging the priorities locally, which wouldn't help me a bit.

By capping the speed below the link speed most modems will usually avoid
bursting. IMHO it's mostly a net gain in usability even though you don't
get the same raw download speeds as without capping.

-- 
Cheers,
André Tomt
andre@tomt.net

Re: TCP send behaviour leads to cable modem woes

[Svein Ove Aas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

fredag 27. juni 2003, 22:43, skrev Andre Tomt:
> On fre, 2003-06-27 at 22:24, Svein Ove Aas wrote:
> > > http://lartc.org/wondershaper/
> >
> > I wrote something like that myself once.
> > It's a good shaper, but it works by *capping* up/download speeds and
> > rearranging the priorities locally, which wouldn't help me a bit.
>
> By capping the speed below the link speed most modems will usually avoid
> bursting. IMHO it's mostly a net gain in usability even though you don't
> get the same raw download speeds as without capping.

I'll try it if the F-RTO option doesn't work. Might as well.

It might actually work; the problem at the moment is that as far as the kernel 
is concerned my usual uplink works at 10Mbit/s++ for a fraction of a second 
and then downright drops most of the rest of the data it's sent until the 
next burst. If I cap it sufficiently that it can't overflow either the line 
(on average) or my cable modem's buffers, that should work.

It makes sense.

Now, how come I didn't think of that myself?

- - Svein Ove Aas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE+/MiN9OlFkai3rMARArzXAKDTITRw3swGcINfEBAlteJlCS2CiACgpVIw
FpqXUkhx8iJct7nEuLaZ3Xc=
=GisS
-----END PGP SIGNATURE-----

Re: TCP send behaviour leads to cable modem woes

[Svein Ove Aas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

fredag 27. juni 2003, 21:57, skrev Mika Liljeberg:
> On Fri, 2003-06-27 at 22:45, Svein Ove Aas wrote:
> > Incidentally, while googling I heard someone saying that only works if
> > it's enabled on both ends? Of course, that might be if upload/download
> > are 'both' affected, in which case it wouldn't apply to me.
>
> It's a sender side only algorithm, so enabling it at your end should be
> enough to help the uploads. For downloads it needs to be on at the other
> end, of course.

Well, it doesn't appear to have any effect.
(What is it *supposed* to do? Something about spurious retransmission 
timeouts, was it?)

I'm going to research this a bit more on my own, if none of you have any 
further ideas. If I find a solution it should probably be in a HOWTO 
somewhere, so I'll get back to you if/when that happens.

- - Svein Ove Aas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE+/aCD9OlFkai3rMARAtzFAKCyfXKjWF9yA6wuQZiJvo11RsIs9gCcCBW/
Si1UTkOPaDEtXz5Pq+U64NM=
=kjUX
-----END PGP SIGNATURE-----

Re: TCP send behaviour leads to cable modem woes

[Mika Liljeberg]

On Sat, 2003-06-28 at 17:04, Svein Ove Aas wrote:
> Well, it doesn't appear to have any effect.
> (What is it *supposed* to do? Something about spurious retransmission 
> timeouts, was it?)

Yeah, frto should help if you're seeing unnecessary retransmission
timeouts caused by delay spikes. It won't do much good if you're also
losing packets, e.g., due to overflowing the modem buffers. From what I
gathered from your explanation, the cable link might also be bunching up
the incoming ACK packets into bursts, each of which causes the sending
TCP to inject a corresponding burst of new segments into the network. If
that's what is happening, rate capping is probably more effective. Even
if you set the rate cap a little high it should mitigate the effects of
the bursts.

	MikaL

Re: TCP send behaviour leads to cable modem woes

[Svein Ove Aas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

lørdag 28. juni 2003, 17:22, skrev Mika Liljeberg:
> On Sat, 2003-06-28 at 17:04, Svein Ove Aas wrote:
> > Well, it doesn't appear to have any effect.
> > (What is it *supposed* to do? Something about spurious retransmission
> > timeouts, was it?)
>
> Yeah, frto should help if you're seeing unnecessary retransmission
> timeouts caused by delay spikes. It won't do much good if you're also
> losing packets, e.g., due to overflowing the modem buffers. From what I
> gathered from your explanation, the cable link might also be bunching up
> the incoming ACK packets into bursts, each of which causes the sending
> TCP to inject a corresponding burst of new segments into the network. If
> that's what is happening, rate capping is probably more effective. Even
> if you set the rate cap a little high it should mitigate the effects of
> the bursts.
>
> 	MikaL

That Has Been Tried.

Didn't Work (tm).


At this point I'm starting to suspect there's a problem with my ISP. They keep 
saying there isn't, but...

Incidentally, capping the upload speed should be enough, shouldn't it? If it 
isn't then I should probably try that again.

- - Svein Ove Aas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE+/csL9OlFkai3rMARAobHAKCVxOQLk8u7ivgGO7M7mjQPbjKiRACeJU8s
9AGfL1lSeZ4tXxSnjHYNEwc=
=Do92
-----END PGP SIGNATURE-----