Re: bandwidth for bkbits.net (good news)

Re: bandwidth for bkbits.net (good news)

[Pascal Schmidt]

On Sat, 30 Aug 2003 18:20:11 +0200, you wrote in linux.kernel:

> In order to control incoming traffic, it is easiest to look at tcp.
> udp can work similarly, but it doesn't have to.  To throttle the
> stream of tcp packets, you could simply throw away the acks and the
> sending side will reduce it's speed.  So you have to measure one
> stream and control a related but different one.  Maybe this is
> possible, not sure.

All you have to do is drop the incoming packets if they exceed
a certain bandwidth. That will stop the corresponding ack automatically
since your TCP stack won't even see the packet.

I'm doing this on my ISDN line to limit the rest of the house to one
third of the bandwidth even if they're all busy downloading tons of
warez. I'm paying, I want the bandwidth when I need it. They can get
it all when there's no traffic for my machine.

No problem with the HTB queueing discipline.

[...]
> Third, there is the problem transition from continuous streams to
> discrete packets, when bandwidth is low.  It doesn't take a huge
> amount of large packets to create enough latency for your sad VOIP.

Yes, latency is a problem if you want to saturate your bandwidth. It is
easy to guarantee some bandwith for latency critical stuff - just
don't give out that piece of bandwith to something else. Of course,
then most of the time that piece is wasted... and it doesn't help with
problems somewhere along the net.

> And fourth, the possibility of resonance frequency.  If measurement
> and/or shaping intervals are too long and match nicely to the travel
> time to one of your peers, you are back at point three.

Dunno about that one.

-- 
Ciao,
Pascal

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 12:58:33AM +0200, Pascal Schmidt wrote:
> On Sat, 30 Aug 2003 18:20:11 +0200, you wrote in linux.kernel:
> 
> > In order to control incoming traffic, it is easiest to look at tcp.
> > udp can work similarly, but it doesn't have to.  To throttle the
> > stream of tcp packets, you could simply throw away the acks and the
> > sending side will reduce it's speed.  So you have to measure one
> > stream and control a related but different one.  Maybe this is
> > possible, not sure.
> 
> All you have to do is drop the incoming packets if they exceed
> a certain bandwidth. 

If you think we haven't done that, think again.  We're at the wrong end
of the pipe to do that, I'm pretty sure that what you are describing 
simply won't work.  We've already tried rate limiting all traffic except
for the phone traffic to 1/2 the T1 line and it doesn't work.

Remember, we're talking vioce here.  Retransmits don't work and aren't
even attempted.  I've got people in 5 or 6 states spread out from one 
end of the USA to the other.  So any disruption means things don't work.

All of the traffic shaping tends to be focussed on "good" versus "bad"
bandwidth utilization.  That's nice but it means that you are tolerant
of a "settling time" during which everyone is fighting for bandwidth.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Pascal Schmidt]

On Sat, 30 Aug 2003, Larry McVoy wrote:

>> All you have to do is drop the incoming packets if they exceed
>> a certain bandwidth. 
> If you think we haven't done that, think again.

> We're at the wrong end of the pipe to do that, I'm pretty sure that what 
> you are describing simply won't work. 

In a way, you're on the right end of the pipe because the system
that does your traffic shaping is part of the general network, viewed
from the machines behind the shaper.

Dropping the packets means that the sending side, at least if we're
talking TCP, will throttle its sending rate. But, depending on the
distance in hops to the sender, it may take up to a few seconds for
this to kick in. So I guess that's why it doesn't work for your
VoIP case - the senders don't notice fast enough that they should
slow down.

When my ISDN line is saturated with downloads from the rest of the
people here and data starts to come in for MY machine, it takes about
5 seconds until I get my bandwidth. Too slow for VoIP.

Traffic shaping on the side of your ISP will probably help since
they have a bigger incoming pipe. If they hadn't, they'd have the
same problem, though.

-- 
Ciao,
Pascal

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 03:05:37AM +0200, Pascal Schmidt wrote:
> On Sat, 30 Aug 2003, Larry McVoy wrote:
> 
> >> All you have to do is drop the incoming packets if they exceed
> >> a certain bandwidth. 
> > If you think we haven't done that, think again.
> 
> > We're at the wrong end of the pipe to do that, I'm pretty sure that what 
> > you are describing simply won't work. 
> 
> In a way, you're on the right end of the pipe because the system
> that does your traffic shaping is part of the general network, viewed
> from the machines behind the shaper.
> 
> Dropping the packets means that the sending side, at least if we're
> talking TCP, will throttle its sending rate. But, depending on the
> distance in hops to the sender, it may take up to a few seconds for
> this to kick in. So I guess that's why it doesn't work for your
> VoIP case - the senders don't notice fast enough that they should
> slow down.

that's because you don't limit the bkbits.net to a fixed rate. If you
want to give priorities, it won't work well because it takes time to be
effective, but if you rate limit hard both ways it has to work, unless
you're under syn-flood ;) The downside is that you will waste bandwith
(i.e. you will hurt the bkbits.net service even when you don't use
voip), but it will work.

This is what I use normally to limit my brother downloads, and it works
fine for me (though I don't often place calls through adsl myself, it's
basically useless since people only uses cellphones here, and last time
I chekced voip wasn't free for cellphones with my isp).

this is the script:

INTERNAL_NET=eth0
EXTERNAL_NET=ppp0

		# External net, prioritize with TOS and low prio traffic in MARK chan 6
		tc qdisc add dev $EXTERNAL_NET root handle 1: prio bands 5
		tc qdisc add dev $EXTERNAL_NET parent 1:1 handle 10: sfq
		tc qdisc add dev $EXTERNAL_NET parent 1:2 handle 20: sfq
		tc qdisc add dev $EXTERNAL_NET parent 1:3 handle 30: sfq
		tc qdisc add dev $EXTERNAL_NET parent 1:4 tbf rate 16kbit buffer 1600 limit 3000
		tc qdisc add dev $EXTERNAL_NET parent 1:5 tbf rate 16kbit buffer 1600 limit 3000

		tc filter add dev $EXTERNAL_NET protocol ip parent 1:0 prio 4 handle 6 fw flowid 1:4
		tc filter add dev $EXTERNAL_NET protocol ip parent 1:0 prio 5 handle 7 fw flowid 1:5

		tc qdisc add dev $INTERNAL_NET root handle 1: prio bands 5
		tc qdisc add dev $INTERNAL_NET parent 1:1 handle 10: sfq
		tc qdisc add dev $INTERNAL_NET parent 1:2 handle 20: sfq
		tc qdisc add dev $INTERNAL_NET parent 1:3 handle 30: sfq
		tc qdisc add dev $INTERNAL_NET parent 1:4 tbf rate 64kbit buffer 1600 limit 3000
		tc qdisc add dev $INTERNAL_NET parent 1:5 tbf rate 64kbit buffer 1600 limit 3000

		tc filter add dev $INTERNAL_NET protocol ip parent 1:0 prio 4 handle 8 fw flowid 1:4
		tc filter add dev $INTERNAL_NET protocol ip parent 1:0 prio 5 handle 9 fw flowid 1:5

		#tc qdisc add dev $EXTERNAL_NET root sfq perturb 30

btw, the sfq is a great idea to have anyways, to provide fariness to
the traffing going out of your network.

then the below scripts will turn it off/on (so he can get full bandwith
while I'm not at my desk or when he's not doing anything very network
intensive ;)

Basically you should run the below with 'start' before placing the call,
and running 'stop' after the call is complete. It may take a few seconds
to rate limit all the connections but it should work fine.

EXTERNAL_NET=ppp0
INTERNAL_NET=eth0

case "$1" in
	start)
		iptables -t mangle -A PREROUTING -t mangle -j MARK -s 192.168.1.9 --set-mark 6 -i $INTERNAL_NET
		iptables -t mangle -A POSTROUTING -t mangle -j MARK -d 192.168.1.9 --set-mark 8
		;;
	stop)
		iptables -t mangle -D PREROUTING -t mangle -j MARK -s 192.168.1.9 --set-mark 6 -i $INTERNAL_NET
		iptables -t mangle -D POSTROUTING -t mangle -j MARK -d 192.168.1.9 --set-mark 8
		;;
	status)
		iptables -t mangle -L -v
		;;
	*)
		echo "Usage: $0 {start|stop}"
		exit 1
		;;
esac

(he's IP 192.168.1.9 of course, you should replace it with bkbits.net
and tweak the 64kbit output/16k input with the input/output bandwith you
want to give to bkbits)

BTW, I still wonder how to rsync the bkcvs coherently? (and I wonder how
can Peter sync it synchronous too if you don't give him a pair of
sequence numbers or a fcntl lock, personally I prefer the sequence
numbers so it works for pure read only too like the rsync on kernel.org)

Andrea

Re: bandwidth for bkbits.net (good news)

[Jeff Sipek]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Saturday 30 August 2003 21:39, Andrea Arcangeli wrote:
> On Sun, Aug 31, 2003 at 03:05:37AM +0200, Pascal Schmidt wrote:
> > On Sat, 30 Aug 2003, Larry McVoy wrote:
> > >> All you have to do is drop the incoming packets if they exceed
> > >> a certain bandwidth.
> > >
> > > If you think we haven't done that, think again.
> > >
> > > We're at the wrong end of the pipe to do that, I'm pretty sure that
> > > what you are describing simply won't work.
> >
> > In a way, you're on the right end of the pipe because the system
> > that does your traffic shaping is part of the general network, viewed
> > from the machines behind the shaper.
> >
> > Dropping the packets means that the sending side, at least if we're
> > talking TCP, will throttle its sending rate. But, depending on the
> > distance in hops to the sender, it may take up to a few seconds for
> > this to kick in. So I guess that's why it doesn't work for your
> > VoIP case - the senders don't notice fast enough that they should
> > slow down.
>
> that's because you don't limit the bkbits.net to a fixed rate. If you
> want to give priorities, it won't work well because it takes time to be
> effective, but if you rate limit hard both ways it has to work, unless
> you're under syn-flood ;) The downside is that you will waste bandwith
> (i.e. you will hurt the bkbits.net service even when you don't use
> voip), but it will work.

How about giving something to voip as a hard limit and then using some shaper 
to give it more if needed.

Jeff.

- -- 
*NOTE: This message is ROT-13 encrypted twice for extra protection*
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/UVsBwFP0+seVj/4RAkmpAJ9YwjdPLZZsdD7fCRDRHyS+JrJnkgCdG/sc
sr5Mqx5Qii//AQPepCqHDcw=
=RoPR
-----END PGP SIGNATURE-----

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sat, Aug 30, 2003 at 10:18:37PM -0400, Jeff Sipek wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Saturday 30 August 2003 21:39, Andrea Arcangeli wrote:
> > On Sun, Aug 31, 2003 at 03:05:37AM +0200, Pascal Schmidt wrote:
> > > On Sat, 30 Aug 2003, Larry McVoy wrote:
> > > >> All you have to do is drop the incoming packets if they exceed
> > > >> a certain bandwidth.
> > > >
> > > > If you think we haven't done that, think again.
> > > >
> > > > We're at the wrong end of the pipe to do that, I'm pretty sure that
> > > > what you are describing simply won't work.
> > >
> > > In a way, you're on the right end of the pipe because the system
> > > that does your traffic shaping is part of the general network, viewed
> > > from the machines behind the shaper.
> > >
> > > Dropping the packets means that the sending side, at least if we're
> > > talking TCP, will throttle its sending rate. But, depending on the
> > > distance in hops to the sender, it may take up to a few seconds for
> > > this to kick in. So I guess that's why it doesn't work for your
> > > VoIP case - the senders don't notice fast enough that they should
> > > slow down.
> >
> > that's because you don't limit the bkbits.net to a fixed rate. If you
> > want to give priorities, it won't work well because it takes time to be
> > effective, but if you rate limit hard both ways it has to work, unless
> > you're under syn-flood ;) The downside is that you will waste bandwith
> > (i.e. you will hurt the bkbits.net service even when you don't use
> > voip), but it will work.
> 
> How about giving something to voip as a hard limit and then using some shaper 
> to give it more if needed.

it may take a few secs to rate limit the rest, the old tcp connections
will keep sending packets that will get rejected when voip kicks in,
that's the problem described in the email I answered to with the
solution (i.e.  limit bkbits.net hard). I'm pretty sure that rate
limting bkbits.net hard (like we do with my brother's computer when he
generates heavy traffic) will let voip to work flawlessy, but it has the
downside of limiting the bandwith available to bkbits users even when
nobody place phone calls. Upgrading the current line and using the
software rate limiting with the solution I proposed, or adding a new
line dedicated to bkbits.net, wouldn't be much different in practice (if
it was me I'd choose it only in function of what's more cost effective,
after double checking that my algorithm infact works fine for higher
bandwidth too, I only tested it on 640kbit).

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 03:39:28AM +0200, Andrea Arcangeli wrote:
> > Dropping the packets means that the sending side, at least if we're
> > talking TCP, will throttle its sending rate. But, depending on the
> > distance in hops to the sender, it may take up to a few seconds for
> > this to kick in. So I guess that's why it doesn't work for your
> > VoIP case - the senders don't notice fast enough that they should
> > slow down.
> 
> that's because you don't limit the bkbits.net to a fixed rate. 

We not only limited bkbits.net to a fixed rate, we limited all TCP based
traffic to a fixed rate and it still didn't work.

I'm pretty convinced we can't solve the problem at our end.  Maybe we can
but I'm voting for throwing another T1 at the problem, we'll try working
with the ISP suggested solution of trunking them and rate limiting at
their end and if that doesn't work then we'll split them and use one for
bkbits.net and other bitmover related TCP traffic and use the other one
for phones.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Alan Cox]

On Sul, 2003-08-31 at 03:56, Larry McVoy wrote:
> I'm pretty convinced we can't solve the problem at our end.  Maybe we can

For bursts of traffic you can't.

> but I'm voting for throwing another T1 at the problem, we'll try working

or switch to ATM ;)

> with the ISP suggested solution of trunking them and rate limiting at
> their end and if that doesn't work then we'll split them and use one for
> bkbits.net and other bitmover related TCP traffic and use the other one
> for phones.

Fractioning the line is also doable but less flexible with some kit. 

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote:
> On Sul, 2003-08-31 at 03:56, Larry McVoy wrote:
> > I'm pretty convinced we can't solve the problem at our end.  Maybe we can
> 
> For bursts of traffic you can't.

what's the difference of rejecting packets in software, or because the
link can't handle them? Assume the guaranteed bandwidth is much lower
than what the link can provide (very common here during the day), the
isp will have to do the software thing anyways to balance the bandwidth
across the different ip.

it works flawlessy for me and it's the same problem (I also use
streaming services), and they're unusable until I turn the shaping on,
I'm sure that if you use the script and you change 1kbyte/sec to
everything but voip it'll work fine for you too, since basically
everything else won't pass anymore, it will take ages to open an html
page and all the bkbits.net users will hang, and the link will be idle
99% of the time, so voip will take it over as much as it can. I don't
think it's a matter of "if it works or not", I think it's a matter of
how much you're ok to lose in terms of global bandwith for all the other
services but voip.

the only annoying problem I run into, is that tc is missing a flush
operation (like iptables -F) but I never need to tweak it anymore so I
don't mind too much.

Andrea

Re: bandwidth for bkbits.net (good news)

[Alan Cox]

On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote:
> On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote:
> > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote:
> > > I'm pretty convinced we can't solve the problem at our end.  Maybe we can
> > 
> > For bursts of traffic you can't.
> 
> what's the difference of rejecting packets in software, or because the
> link can't handle them? Assume the guaranteed bandwidth is much lower


It doesn't work when you dont control incoming. As a simple extreme
example if I pingflood you from a fast site then no amount of shaping
your end of the link will help, it has to be shaped at the ISP end.

Re: bandwidth for bkbits.net (good news)

[Jörn Engel]

On Sun, 31 August 2003 16:31:32 +0100, Alan Cox wrote:
> On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote:
> > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote:
> > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote:
> > > > I'm pretty convinced we can't solve the problem at our end.  Maybe we can
> > > 
> > > For bursts of traffic you can't.
> > 
> > what's the difference of rejecting packets in software, or because the
> > link can't handle them? Assume the guaranteed bandwidth is much lower
> 
> It doesn't work when you dont control incoming. As a simple extreme
> example if I pingflood you from a fast site then no amount of shaping
> your end of the link will help, it has to be shaped at the ISP end.

If someone wants to DOS you, he can.  Full stop.

iirc, Larry was worried about well behaved traffic still doing bad
things to his connection.

Jörn

-- 
Fancy algorithms are slow when n is small, and n is usually small.
Fancy algorithms have big constants. Until you know that n is
frequently going to be big, don't get fancy.
-- Rob Pike

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 05:43:01PM +0200, Jörn Engel wrote:
> On Sun, 31 August 2003 16:31:32 +0100, Alan Cox wrote:
> > On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote:
> > > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote:
> > > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote:
> > > > > I'm pretty convinced we can't solve the problem at our end.  Maybe we can
> > > > 
> > > > For bursts of traffic you can't.
> > > 
> > > what's the difference of rejecting packets in software, or because the
> > > link can't handle them? Assume the guaranteed bandwidth is much lower
> > 
> > It doesn't work when you dont control incoming. As a simple extreme
> > example if I pingflood you from a fast site then no amount of shaping
> > your end of the link will help, it has to be shaped at the ISP end.
> 
> If someone wants to DOS you, he can.  Full stop.

yes, that's unfixable at Larry's end, and normaly it's unfixable for the
ISP too.

> iirc, Larry was worried about well behaved traffic still doing bad
> things to his connection.

I also understood the problem were the legimate clones/checkouts
happening in established state with a single tcp connection generating a
burst of traffic.

If this is not the case and Larry is under attack when the voip doesn't
work, of course nothing can help him, most probably not even his ISP,
but I assumed this wasn't the case.

Andrea

Re: bandwidth for bkbits.net (good news)

[Florian Weimer]

Andrea Arcangeli <andrea@suse.de> writes:

> yes, that's unfixable at Larry's end, and normaly it's unfixable for the
> ISP too.

The ISP can do several things to prioritize production traffic or drop
malicious traffic.  However, this isn't trivial and requires careful
planning, and it's unlikely that anyone who is able to would want to
do this for a T1 customer (typically, it requires "unusual"
configuration of vital production routers with the fat pipes).

>> iirc, Larry was worried about well behaved traffic still doing bad
>> things to his connection.

In this case, it's a bit easier for the ISP because the tweaking can
be done on edge routers (which typically die during DoS attacks as
well, so countermeasures cannot be applied there).  Especially with
low-bandwidth links, it shouldn't be too hard to find a smallish,
geeky ISP who is willing to try some fiddling on the edge router to
improve performance (at least that's true in Germany, don't know about
the U.S.).

However, why can't bkbits.net distributed around the world, at least
for read access?  This would eliminate the choking point once and for
all.

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 07:06:55PM +0200, Florian Weimer wrote:
> Andrea Arcangeli <andrea@suse.de> writes:
> However, why can't bkbits.net distributed around the world, at least
> for read access?  

It's not just the read access that is the problem but there is absolutely
nothing stopping you or anyone else from setting up bkbackup.net as a 
cache.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Henning P. Schmiedehausen]

Florian Weimer <fw@deneb.enyo.de> writes:

>do this for a T1 customer (typically, it requires "unusual"
>configuration of vital production routers with the fat pipes).

You need a shaper connected to the ISP backbone which shapes the
outgoing traffic for you and a border router which talks to the T1
(C17xx or C26xx). Normally, if your ISP has some sort of clue, you
will also need a bastion router which can handle backbone <-> 100 MBit
traffic and does dynamic routing updates (EGP or OSPF) to the ISP
backbone (A C26xx or C37xx).

This isn't your run-of-the-mill setup but I know of plenty ISPs here
that will happily sell you this and also the consulting needed to
plan, setup and operate the link. However, you won't get it for
$199/month. :-)

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)

Re: bandwidth for bkbits.net (good news)

[Florian Weimer]

"Henning P. Schmiedehausen" <hps@intermeta.de> writes:

> Florian Weimer <fw@deneb.enyo.de> writes:
>
>>do this for a T1 customer (typically, it requires "unusual"
>>configuration of vital production routers with the fat pipes).
>
> You need a shaper connected to the ISP backbone which shapes the
> outgoing traffic for you and a border router which talks to the T1
> (C17xx or C26xx). Normally, if your ISP has some sort of clue, you
> will also need a bastion router which can handle backbone <-> 100 MBit
> traffic and does dynamic routing updates (EGP or OSPF) to the ISP
> backbone (A C26xx or C37xx).

C37xx can handle a maximum load of 225 kpps (data sheet number,
i.e. this value cannot be exceeded even under most favorable
conditions), the others handle even less.  Such routers are of no help
during a DoS attack.

Yes, I snipped the DoS context, and your approach would work in a
benign environment. 8-)

Re: bandwidth for bkbits.net (good news)

[Henning Schmiedehausen]

On Fri, 2003-09-05 at 10:10, Florian Weimer wrote:

> > You need a shaper connected to the ISP backbone which shapes the
> > outgoing traffic for you and a border router which talks to the T1
> > (C17xx or C26xx). Normally, if your ISP has some sort of clue, you
> > will also need a bastion router which can handle backbone <-> 100 MBit
> > traffic and does dynamic routing updates (EGP or OSPF) to the ISP
> > backbone (A C26xx or C37xx).
> 
> C37xx can handle a maximum load of 225 kpps (data sheet number,
> i.e. this value cannot be exceeded even under most favorable
> conditions), the others handle even less.  Such routers are of no help
> during a DoS attack.
> 
> Yes, I snipped the DoS context, and your approach would work in a
> benign environment. 8-)

225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec

100 MBit / 8 bit = 12,5 MBytes / sec

So, IMHO even with a small packet saturated 100 MBit link you won't
reach 225kpps. AFAIK this was Ciscos intention to publish this number.
It basically says "you will have filled your link before you fill our
router". 

I'm pretty sure that your 37xx won't do any routing updates anymore at
this point. And if you do _anything_ that forces the packets down the
slow path from the routing engine, you're toast anyway.

But I'm pretty sure that a C37xx would handle full 100 MBit traffic to a
busy website without any problems. In fact, I know that it does. ;-) (We
did switch to a C12000 shortly after, mainly because we went Gigabit).

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)

Re: bandwidth for bkbits.net (good news)

[Florian Weimer]

Henning Schmiedehausen <hps@intermeta.de> writes:

>> Yes, I snipped the DoS context, and your approach would work in a
>> benign environment. 8-)
>
> 225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec
>
> 100 MBit / 8 bit = 12,5 MBytes / sec
>
> So, IMHO even with a small packet saturated 100 MBit link you won't
> reach 225kpps. AFAIK this was Ciscos intention to publish this number.
> It basically says "you will have filled your link before you fill our
> router". 

Cisco sells Gigabit Ethernet linecards for this router, and the
situation is quite different in this case.  And with most attacks,
it's a futile exercise to try to combat them on a Fast Ethernet link,
anyway. 8-(

> But I'm pretty sure that a C37xx would handle full 100 MBit traffic to a
> busy website without any problems. In fact, I know that it does. ;-) (We
> did switch to a C12000 shortly after, mainly because we went Gigabit).

Of course it does.  After all, web traffic is somewhat different from
pure DoS traffic. 8-)

Re: bandwidth for bkbits.net (good news)

[P]

Henning Schmiedehausen wrote:
> On Fri, 2003-09-05 at 10:10, Florian Weimer wrote:
> 
>>>You need a shaper connected to the ISP backbone which shapes the
>>>outgoing traffic for you and a border router which talks to the T1
>>>(C17xx or C26xx). Normally, if your ISP has some sort of clue, you
>>>will also need a bastion router which can handle backbone <-> 100 MBit
>>>traffic and does dynamic routing updates (EGP or OSPF) to the ISP
>>>backbone (A C26xx or C37xx).
>>
>>C37xx can handle a maximum load of 225 kpps (data sheet number,
>>i.e. this value cannot be exceeded even under most favorable
>>conditions), the others handle even less.  Such routers are of no help
>>during a DoS attack.
>>
>>Yes, I snipped the DoS context, and your approach would work in a
>>benign environment. 8-)
> 
> 225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec
> 
> 100 MBit / 8 bit = 12,5 MBytes / sec
> 
> So, IMHO even with a small packet saturated 100 MBit link you won't
> reach 225kpps. AFAIK this was Ciscos intention to publish this number.
> It basically says "you will have filled your link before you fill our
> router". 

100Mb/s LAN @ 64 byte packets is 148500 pps half duplex
from my testing => 257Kpps full duplex.

Here's some interesting results I've got from the latest
intel e100 3.0.0_dev13 driver, showing how the receive
rate degrades at higher packet rates.

------------------------
send Kpps     recv Kpps
------------------------
126           126
126.5         118.4
128           115.7
130           102.7
135            99.1
140            90.6
148            88.2
------------------------

It's not a CPU issue as it can receive at
the same rate on another interface.
renicing ksoftirqd has no effect.
system is PIII 1.2GHz, i815, 2.4.20
NAPI is turned on in the driver.

Pádraig.

Re: bandwidth for bkbits.net (good news)

[Ricky Beam]

On 5 Sep 2003, Henning Schmiedehausen wrote:
>225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec
>
>100 MBit / 8 bit = 12,5 MBytes / sec
>
>So, IMHO even with a small packet saturated 100 MBit link you won't
>reach 225kpps. AFAIK this was Ciscos intention to publish this number.
>It basically says "you will have filled your link before you fill our
>router".

64B is the minimum ETHERNET frame size.  That isn't true for PPP, HDLC,
Frame relay, ATM, etc.

>I'm pretty sure that your 37xx won't do any routing updates anymore at
>this point. And if you do _anything_ that forces the packets down the
>slow path from the routing engine, you're toast anyway.

Sure it can.  Yeah, it'll be slow_er_, but not stopped.  CEF/PXF doesn't
require much CPU to switch a packet.

At process switching, the router is rated to a few K pps.  Process switching
SUCKS.

I've seen a 7206/200 _attempt_ to move 150kpps @ 64B each. (misbehaving
software and a misconfigured firewall...) The router was chuggin' right
along -- discarding UDP traffic like a mad man.  From the console, it was
working fine. *grin* It's rather hard for telnet to compete (and even
harder for ssh.)

All those machines are behind a 7401 now.  And it doesn't even blink at
such things.  (That thing's worth every penny we paid for it.)

--Ricky

Re: bandwidth for bkbits.net (good news)

[Henning Schmiedehausen]

On Fri, 2003-09-05 at 18:43, Ricky Beam wrote:

On 5 Sep 2003, Henning Schmiedehausen wrote:
> >225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec
> >
> >100 MBit / 8 bit = 12,5 MBytes / sec
> >
> >So, IMHO even with a small packet saturated 100 MBit link you won't
> >reach 225kpps. AFAIK this was Ciscos intention to publish this number.
> >It basically says "you will have filled your link before you fill our
> >router".
> 
> 64B is the minimum ETHERNET frame size.  That isn't true for PPP, HDLC,
> Frame relay, ATM, etc.

We were talking 100 MBit Ethernet, weren't we? ;-)

	Regards
		Henning

Re: bandwidth for bkbits.net (good news)

[Ricky Beam]

On 7 Sep 2003, Henning Schmiedehausen wrote:
>> 64B is the minimum ETHERNET frame size.  That isn't true for PPP, HDLC,
>> Frame relay, ATM, etc.
>
>We were talking 100 MBit Ethernet, weren't we? ;-)

Actually, we're talking about VoIP between remote offices.  We already
know this is not accomplished via fast ethernet WAN links.  The WAN
link is, in fact, a T1.  T1's do not have minimum frame sizes like
ethernet.

--Ricky

[COMPLETELY OFF-TOPIC] Re: bandwidth for bkbits.net (good news)

[Henning Schmiedehausen]

That is where the thread started. ;-) As we reached the max kpps on a
C37xx point of discussion, we were already on 100 MBit Ethernet. ;-) 

Let's end this thread.

	Regards
		Henning


On Sun, 2003-09-07 at 20:41, Ricky Beam wrote:
> On 7 Sep 2003, Henning Schmiedehausen wrote:
> >> 64B is the minimum ETHERNET frame size.  That isn't true for PPP, HDLC,
> >> Frame relay, ATM, etc.
> >
> >We were talking 100 MBit Ethernet, weren't we? ;-)
> 
> Actually, we're talking about VoIP between remote offices.  We already
> know this is not accomplished via fast ethernet WAN links.  The WAN
> link is, in fact, a T1.  T1's do not have minimum frame sizes like
> ethernet.
> 
> --Ricky
-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)

Re: bandwidth for bkbits.net (good news)

[Ricky Beam]

On Sun, 31 Aug 2003, Florian Weimer wrote:
>The ISP can do several things to prioritize production traffic or drop
>malicious traffic.  However, this isn't trivial and requires careful
>planning, and it's unlikely that anyone who is able to would want to
>do this for a T1 customer (typically, it requires "unusual"
>configuration of vital production routers with the fat pipes).

In the cisco world, all it takes is:
	interface <foo>
	  fair-queue

That's it.  That's all it takes on the ISPs part to provide a minimal
amount of QoS.  Cisco's documentation clearly lists how IOS handles
the queuing.  It's not full policy based traffic shaping with RSVP
and all that crap, but it's enough for a many situations.

Larry's situation is highly complicated by "the internet".  I've never
known anyone to be happy with VoIP across the internet.  It's hard
enough to make it work properly within a single ISP's network _with_
their assistance.  Across the wastelands, forget about it. (Why do you
think GRIC built their own global network for moving VoIP traffic?)

--Ricky

Re: bandwidth for bkbits.net (good news)

[Florian Weimer]

Ricky Beam <jfbeam@bluetronic.net> writes:

> On Sun, 31 Aug 2003, Florian Weimer wrote:
>>The ISP can do several things to prioritize production traffic or drop
>>malicious traffic.  However, this isn't trivial and requires careful
>>planning, and it's unlikely that anyone who is able to would want to
>>do this for a T1 customer (typically, it requires "unusual"
>>configuration of vital production routers with the fat pipes).
>
> In the cisco world, all it takes is:
> 	interface <foo>
> 	  fair-queue

WFQ is already the default for low-bandwidth links (and it obviously
can't work on high-end routers). 8-/

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 04:31:32PM +0100, Alan Cox wrote:
> On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote:
> > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote:
> > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote:
> > > > I'm pretty convinced we can't solve the problem at our end.  Maybe we can
> > > 
> > > For bursts of traffic you can't.
> > 
> > what's the difference of rejecting packets in software, or because the
> > link can't handle them? Assume the guaranteed bandwidth is much lower
> 
> 
> It doesn't work when you dont control incoming. As a simple extreme
> example if I pingflood you from a fast site then no amount of shaping
> your end of the link will help, it has to be shaped at the ISP end.

sure, that's why I said it won't work with synflood. I just doubt the
ping/syn floods distributed denial of services are an high percentage of
the traffic passing through on bkbits.net. I though it was legitimate
traffic, and I assume bitkeeer is somehow efficient in handling the
transfer, by using a single tcp connection for the whole transfer of the
data, just like pserver/cvs-ssh do. For example if bitkeeper would open
a new tcp connection for each file (similar to cvsps -p -g w/o the
--cvs-direct option that I asked for), it would be much harder to shape
that traffic. But I understood the traffic that hurts is all in
established state for several seconds, so it should be technically
possible to stop it to around 1kbyte/sec globally to give an huge margin
to voip.

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 05:44:50PM +0200, Andrea Arcangeli wrote:
> > It doesn't work when you dont control incoming. As a simple extreme
> > example if I pingflood you from a fast site then no amount of shaping
> > your end of the link will help, it has to be shaped at the ISP end.
> 
> sure, that's why I said it won't work with synflood. 

Someone syncs w/ bkbits every 19 seconds 24x7.  We also run our web server
here.   All traffic to from bitmover.com/bitkeeper.com/bkbits.net goes
through that T1 line.

You guys who are saying it can work are thinking (a) one connection of 
long duration (think about all the web hits on bkbits.net, those are all
short and new TCP connections) and (b) that a little settling time is OK.

There is a reason that the phone networks don't work like IP networks.
The bandwidth is allocated whether you use it or not and your phone works.
Doing optimistic allocation and then backing off means that the phones
don't work.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 09:22:43AM -0700, Larry McVoy wrote:
> On Sun, Aug 31, 2003 at 05:44:50PM +0200, Andrea Arcangeli wrote:
> > > It doesn't work when you dont control incoming. As a simple extreme
> > > example if I pingflood you from a fast site then no amount of shaping
> > > your end of the link will help, it has to be shaped at the ISP end.
> > 
> > sure, that's why I said it won't work with synflood. 
> 
> Someone syncs w/ bkbits every 19 seconds 24x7.  We also run our web server

1 syn every 19 seconds is nothing.

> You guys who are saying it can work are thinking (a) one connection of 
> long duration (think about all the web hits on bkbits.net, those are all

it doesn't need to be long duration, just longer than a syn or a ping.
If it goes in established for a few packets is should be enough to
throttle it just fine.

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 06:33:50PM +0200, Andrea Arcangeli wrote:
> On Sun, Aug 31, 2003 at 09:22:43AM -0700, Larry McVoy wrote:
> > On Sun, Aug 31, 2003 at 05:44:50PM +0200, Andrea Arcangeli wrote:
> > > > It doesn't work when you dont control incoming. As a simple extreme
> > > > example if I pingflood you from a fast site then no amount of shaping
> > > > your end of the link will help, it has to be shaped at the ISP end.
> > > 
> > > sure, that's why I said it won't work with synflood. 
> > 
> > Someone syncs w/ bkbits every 19 seconds 24x7.  We also run our web server
> 
> 1 syn every 19 seconds is nothing.

A sync != one connection.  And that doesn't include the auto backup that we
do to another server, doesn't include all the HTTP traffic, doesn't include
our web site traffic.

> > You guys who are saying it can work are thinking (a) one connection of 
> > long duration (think about all the web hits on bkbits.net, those are all
> 
> it doesn't need to be long duration, just longer than a syn or a ping.
> If it goes in established for a few packets is should be enough to
> throttle it just fine.

You are welcome to *demonstrate* something that works but telling me that it
works when we've tried what you said to try isn't very compelling.  I know
this doesn't work from both theory and practice.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 09:48:02AM -0700, Larry McVoy wrote:
> works when we've tried what you said to try isn't very compelling.  I know
> this doesn't work from both theory and practice.

Post your configure scripts so we can point you what you did wrong.

I already posted a perfectly working configuration that I depend totally
on, and it never failed once to throttle the traffic. If you don't care
to try it, at least post whatever you tried so far. Any other claim or
email that doesn't include scripts will go to /dev/null.

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 07:06:33PM +0200, Andrea Arcangeli wrote:
> On Sun, Aug 31, 2003 at 09:48:02AM -0700, Larry McVoy wrote:
> > works when we've tried what you said to try isn't very compelling.  I know
> > this doesn't work from both theory and practice.
> 
> Post your configure scripts so we can point you what you did wrong.

They are Cisco configuration, it won't do you much good.  All the traffic
goes in/out through a Cisco 2610, we have a full T1 and we clamped all 
TCP traffic at .75Mbit.  Still didn't help even though we verified that
it was indeed clamping down on the traffic.

I'm not sure why you are arguing this, if you have a fat pipe feeding into
a small pipe and you are trying to throttle at far end of the small pipe
isn't it obvious that you can't make that work?  It's not the packets we
send, it's the packets you send.  And all the flow control stuff is per
connection, not per pipe.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 02:18:55PM -0700, Larry McVoy wrote:
> On Sun, Aug 31, 2003 at 07:06:33PM +0200, Andrea Arcangeli wrote:
> > On Sun, Aug 31, 2003 at 09:48:02AM -0700, Larry McVoy wrote:
> > > works when we've tried what you said to try isn't very compelling.  I know
> > > this doesn't work from both theory and practice.
> > 
> > Post your configure scripts so we can point you what you did wrong.
> 
> They are Cisco configuration, it won't do you much good.  All the traffic

I don't trust anything but the sourcecode I can read, so please try
again with linux.

> I'm not sure why you are arguing this, if you have a fat pipe feeding into

you never tried with linux, how can you claim you know it doesn't work
in practice? The fact is that you never tried it, while we used it all
the time.

> a small pipe and you are trying to throttle at far end of the small pipe
> isn't it obvious that you can't make that work?  It's not the packets we
> send, it's the packets you send.  And all the flow control stuff is per

it's tcp, it's trivial to rate limit the packets we send, as far as you
go into established somehow.

Andrea

Re: bandwidth for bkbits.net (good news)

[Alan Cox]

On Sul, 2003-08-31 at 23:49, Andrea Arcangeli wrote:
> > I'm not sure why you are arguing this, if you have a fat pipe feeding into
> 
> you never tried with linux, how can you claim you know it doesn't work
> in practice? The fact is that you never tried it, while we used it all
> the time.

How about because any undergraduate can do the mathematical proof its
not possible. Unless he controls the ISP end of the link random bursts
of traffic, pingfloods, anything not respecting requests to slow down
will lose voice traffic.

Since he doesn't appear to control the ISP end (either directly or via
RSVP) he's lost. 

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote:
> On Sul, 2003-08-31 at 23:49, Andrea Arcangeli wrote:
> > > I'm not sure why you are arguing this, if you have a fat pipe feeding into
> > 
> > you never tried with linux, how can you claim you know it doesn't work
> > in practice? The fact is that you never tried it, while we used it all
> > the time.
> 
> How about because any undergraduate can do the mathematical proof its
> not possible. Unless he controls the ISP end of the link random bursts
> of traffic, pingfloods, anything not respecting requests to slow down
> will lose voice traffic.

Whoops, if I had read this I wouldn't have sent the other reply.  What 
Alan said is what I've been trying to say.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote:
> On Sul, 2003-08-31 at 23:49, Andrea Arcangeli wrote:
> > > I'm not sure why you are arguing this, if you have a fat pipe feeding into
> > 
> > you never tried with linux, how can you claim you know it doesn't work
> > in practice? The fact is that you never tried it, while we used it all
> > the time.
> 
> How about because any undergraduate can do the mathematical proof its
> not possible. Unless he controls the ISP end of the link random bursts
> of traffic, pingfloods, anything not respecting requests to slow down
> will lose voice traffic.

they are legitimate tcp connections, not udp or icmp. I'm not saying you
can control pingfloods or udp floods or syn floods.

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 01:02:19AM +0200, Andrea Arcangeli wrote:
> On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote:
> > How about because any undergraduate can do the mathematical proof its
> > not possible. Unless he controls the ISP end of the link random bursts
> > of traffic, pingfloods, anything not respecting requests to slow down
> > will lose voice traffic.
> 
> they are legitimate tcp connections, not udp or icmp. I'm not saying you
> can control pingfloods or udp floods or syn floods.

I'll try this one more time and then I'm giving up because as far as I 
can tell you haven't tried this with a busy server, I think you said you
did shaping on your home machine or something.  Hardly the same thing.

How about a series of tiny HTTP requests?  All 1.0, no connection reuse.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 04:07:28PM -0700, Larry McVoy wrote:
> On Mon, Sep 01, 2003 at 01:02:19AM +0200, Andrea Arcangeli wrote:
> > On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote:
> > > How about because any undergraduate can do the mathematical proof its
> > > not possible. Unless he controls the ISP end of the link random bursts
> > > of traffic, pingfloods, anything not respecting requests to slow down
> > > will lose voice traffic.
> > 
> > they are legitimate tcp connections, not udp or icmp. I'm not saying you
> > can control pingfloods or udp floods or syn floods.
> 
> I'll try this one more time and then I'm giving up because as far as I 
> can tell you haven't tried this with a busy server, I think you said you
> did shaping on your home machine or something.  Hardly the same thing.

the only difference I believe is that the connections are originated by
my end, but that only changes the syn packet that normally accounts for
a non significant amount of the bandwidth. And while this is located in
a house, this isn't what I would call an home user usage.

> How about a series of tiny HTTP requests?  All 1.0, no connection reuse.

on the long run, the way I recommend you to act, is to have a script
that turns the traffic shaping on on demand before/after using voip, so
you won't hurt the bkbits.net userbase when you're not at the phone. So
you will fix the unprofessional thing during conf calls completely and
the community should be ok with reasonably short slowdowns while you're
at the phone (if only voip would run under linux too you could automate it).

Just try for a test with my script changing it to give 1kbyte/sec both
ways (not 64k/16k) to everything but voip. I think I already posted all
the needed information you need to make it work. You will need to rework
the iptables rules carefully though, to include all the traffic but voip.

Hope this helps you too (I couldn't live without it),

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 01:22:24AM +0200, Andrea Arcangeli wrote:
> on the long run, the way I recommend you to act, is to have a script
> that turns the traffic shaping on on demand before/after using voip

That's a cool idea but unfortunately we are using the 3com NBX stuff
which is based on Wind River and we don't have any way to know that a
voice conversation is going on.  I suppose we could build a way based
on snooping the network though.  Have to think about that.

By the way, I'd love to replace the 3com stuff with something based
on Linux.  I strongly suspect that the phones could do better than they
are doing and we're getting the short end of the stick by going with some
closed answer.  But all the open answers don't come close to offering
what 3com does.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Jamie Lokier]

Larry McVoy wrote:
> But all the open answers don't come close to offering what 3com does.

Have you looked at what Asterisk and Bayonne do?
If so, is an important feature missing which you use?

Cheers,
-- Jamie

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 01:19:40AM +0100, Jamie Lokier wrote:
> Larry McVoy wrote:
> > But all the open answers don't come close to offering what 3com does.
> 
> Have you looked at what Asterisk and Bayonne do?
> If so, is an important feature missing which you use?

Hunt groups.  I want incoming sales and support calls to go to group
of people.  All the "virtual PBX" solutions do a lame hunt where they
ring the first guy 3 times, then the next guy 3 times, etc.  The 3com
rings everyone at the same time.  Believe it or not, if you guys call
for support I want someone to pick up the phone *right now*.  If you
or our commercial customers are having nasty BK problems I don't want
you to wait while the stupid phone system goes hunting around looking
for someone who isn't out to lunch or whatever.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Alan Cox]

On Llu, 2003-09-01 at 00:22, Andrea Arcangeli wrote:
> the only difference I believe is that the connections are originated by
> my end, but that only changes the syn packet that normally accounts for
> a non significant amount of the bandwidth. And while this is located in
> a house, this isn't what I would call an home user usage.

Each ACK that has caused previous delays generally opens up a 64K window
so you get bursts of data incoming. A sequence of acks can cause the
incoming pipe to fill enough to screw up voip very easily.

Window bending stuff does help and certainly the packeteer boxes use it
to good effect. I've never tried that on Linux but even then Im dubious
that just cutting the routes down to an 8K window would help

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Mon, Sep 01, 2003 at 12:43:56PM +0100, Alan Cox wrote:
> On Llu, 2003-09-01 at 00:22, Andrea Arcangeli wrote:
> > the only difference I believe is that the connections are originated by
> > my end, but that only changes the syn packet that normally accounts for
> > a non significant amount of the bandwidth. And while this is located in
> > a house, this isn't what I would call an home user usage.
> 
> Each ACK that has caused previous delays generally opens up a 64K window
> so you get bursts of data incoming. A sequence of acks can cause the

the congestion avoidance shouldn't allow what you say. It sends a few
packets immediatly (cwnd starts > 1 recently), and that's why
non-keepalive connections are bad, but after that the congestion window
will remain low if we drop the packets.

> incoming pipe to fill enough to screw up voip very easily.
> 
> Window bending stuff does help and certainly the packeteer boxes use it
> to good effect. I've never tried that on Linux but even then Im dubious
> that just cutting the routes down to an 8K window would help

The only object of my posts was to try to correct the misinformation
that was spreading from Larry's posts that implied you had to buy a
speparate connection to shape the bitkeeper connections doing clones of
the tree. That's what I and everybody else understood. Now apparently he
was wrong and the hurting factor aren't the bitkeeper clients cloning
the tree but the webserver. So I'm glad to have reached my object.

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 06:13:16PM +0200, Andrea Arcangeli wrote:
> The only object of my posts was to try to correct the misinformation
> that was spreading from Larry's posts that implied you had to buy a
> speparate connection to shape the bitkeeper connections doing clones of
> the tree. That's what I and everybody else understood. Now apparently he
> was wrong and the hurting factor aren't the bitkeeper clients cloning
> the tree but the webserver. So I'm glad to have reached my object.

I don't know what axe you are trying to grind but that's nonsense.  Yes,
I said when you start doing a clone the phones don't work but I never 
said "the only problem we have is that an out going clone causes problems".
That's clearly not true, there are incoming clones, pushes, outgoing pulls,
all sorts of HTTP traffic, etc.  

The problem is that you heard clone and you assumed that clone was the
ONLY source of traffic.  That's your problem, not mine, so don't go 
accusing me of spreading misinformation because you weren't thinking
or listening, please.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Alan Cox]

On Llu, 2003-09-01 at 17:13, Andrea Arcangeli wrote:
> > Each ACK that has caused previous delays generally opens up a 64K window
> > so you get bursts of data incoming. A sequence of acks can cause the
> 
> the congestion avoidance shouldn't allow what you say. It sends a few
> packets immediatly (cwnd starts > 1 recently), and that's why
> non-keepalive connections are bad, but after that the congestion window
> will remain low if we drop the packets.

You may trigger fast retransmit patterns. Thats why you have to bend the
window.

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Mon, Sep 01, 2003 at 05:28:31PM +0100, Alan Cox wrote:
> On Llu, 2003-09-01 at 17:13, Andrea Arcangeli wrote:
> > > Each ACK that has caused previous delays generally opens up a 64K window
> > > so you get bursts of data incoming. A sequence of acks can cause the
> > 
> > the congestion avoidance shouldn't allow what you say. It sends a few
> > packets immediatly (cwnd starts > 1 recently), and that's why
> > non-keepalive connections are bad, but after that the congestion window
> > will remain low if we drop the packets.
> 
> You may trigger fast retransmit patterns. Thats why you have to bend the
> window.

fast restransmit are good to trigger, they don't arrive in a flood like
if we had a huge cwnd, if we can send packets out of the network,
they're right to be sent. During congestions the outgoing acks will be
rejected too (it's limiting both ways).

there is no difference between artificial congestion generated by a
shaper and a true congestion. If tcp is correct it has to slowdown
immediatly when it notices congestion.

I also don't see anything specific to a busy http server non support
keepalive in this ack-fast-retransmit matter.

Andrea

Re: bandwidth for bkbits.net (good news)

[Roman Zippel]

Hi,

On Sun, 31 Aug 2003, Alan Cox wrote:

> How about because any undergraduate can do the mathematical proof its
> not possible. Unless he controls the ISP end of the link random bursts
> of traffic, pingfloods, anything not respecting requests to slow down
> will lose voice traffic.

At first Larry wasn't talking about incoming bursts: "We do VOIP phones 
and when you guys clone a repo our phones don't work".

bye, Roman

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 01:39:56AM +0200, Roman Zippel wrote:
> Hi,
> 
> On Sun, 31 Aug 2003, Alan Cox wrote:
> 
> > How about because any undergraduate can do the mathematical proof its
> > not possible. Unless he controls the ISP end of the link random bursts
> > of traffic, pingfloods, anything not respecting requests to slow down
> > will lose voice traffic.
> 
> At first Larry wasn't talking about incoming bursts: "We do VOIP phones 
> and when you guys clone a repo our phones don't work".

Hey, let me make something clear in case it isn't.  This isn't your problem,
you have every right to clone away as fast as you want.  So I am definitely
*not* asking you to ease up on bkbits.net.  In fact, I'd be unhappy if you
did, we get a fair amount of stress testing of new releases on bkbits.net.
So have at it.

But getting back to the problem space, it's incoming traffic, outgoing 
traffic, all the web hits (Google crawls the web interface to bkbits
for example, even though we've tried to turn that off through robots.txt
people have linkages into specific csets and then google will follow those
down and up), etc.

I'd love for shaping to be something that would help here but I agree 
with Alan and others who say that this isn't something we can control
at our end.  

The good news is that I'm 99% sure we have a deal for 2 T1 lines at
$500/each and we're currently paying $800/month for one T1 so I view
this as a slight increase in dollars for double the bandwidth.  The ISP
is willing to do the shaping at their end, where it will help, if that
works that means that everyone wins, you guys get more bandwidth.
If it doesn't work then you guys get a dedicated T1 line for bkbits
which is more than what you have now.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 05:09:08PM -0700, Larry McVoy wrote:
> On Mon, Sep 01, 2003 at 01:39:56AM +0200, Roman Zippel wrote:
> > At first Larry wasn't talking about incoming bursts: "We do VOIP phones 
> > and when you guys clone a repo our phones don't work".
> 
> Hey, let me make something clear in case it isn't.  This isn't your problem,
> you have every right to clone away as fast as you want.  

I forgot to add that once we have this sorted out we'll do two other things
that you'll like:

a) give you BK URL's that don't change (the current URL's are unstable, they
   are based on revisions and revision numbers are unstable)
b) make every changeset (or range of changesets) be something you can grab
   as a regular diff -Nur style patch.  So all the BK users can post to the
   kernel list and include a URL that you all can wget and there is the 
   patch.  No need to use BK at all unless you want to, the data is right
   there as a patch.

All I'm trying to do is to underscore the point that none of you should "be
nice" and not beat up on bkbits.net.  It's a service, we get at least some
benefit from you using the service so bang on it all you want.  We'll solve
the bandwidth problems.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 12:49:38AM +0200, Andrea Arcangeli wrote:
> > They are Cisco configuration, it won't do you much good.  All the traffic
> 
> I don't trust anything but the sourcecode I can read, so please try
> again with linux.

We did that as well.  We ran wondershaper on bkbits.net AND capped the 
Cisco.

> > I'm not sure why you are arguing this, if you have a fat pipe feeding into
> 
> you never tried with linux, how can you claim you know it doesn't work
> in practice? The fact is that you never tried it, while we used it all
> the time.

Not for VOIP with a busy server you don't.

> > a small pipe and you are trying to throttle at far end of the small pipe
> > isn't it obvious that you can't make that work?  It's not the packets we
> > send, it's the packets you send.  And all the flow control stuff is per
> 
> it's tcp, it's trivial to rate limit the packets we send, as far as you
> go into established somehow.

Let's see.  We've tried Linux.  It didn't work.  We tried again w/ the 
Cisco.  Didn't work.  Tried with both.  Didn't work.  Stopped hacking
and starting thinking and realized it's impossible to make it work at
just one end, especially at the wrong end.  Several people who I trust
(like Alan for example) pointed it that the theory matches the practice,
it can't work.

Yet you keep insisting it will work.  Why?  What is the theory that says
you can keep the other end of the T1 line from being congested when you
don't have control over that router?  And that router has several 100Mbit
links all of which can point at my 1.5Mbit link.  Explain to me how anything
you do can make that work.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 03:56:39PM -0700, Larry McVoy wrote:
> Yet you keep insisting it will work.  Why?  What is the theory that says
> you can keep the other end of the T1 line from being congested when you
> don't have control over that router?  And that router has several 100Mbit

it's absolutely trivial, your end only needs to drop 99% of your
outgoing acks and their incoming packets for every connection but voip
while you are at the phone, you won't kill the connections but everybody
but your voip will work. the exponential backoff and sstrash on the
other and will rate limit everything immediatly.

you will definitely control the other end.

Anyways, I was only trying to help, I know how it works, it works
perfectly for me, and the reason everything makes perfect sense is that
this directory exists in Alan's tree too:

	2.4.22ac1/net/sched/

All assuming you're not under attack, if that's the case you should
contact your ISP since that's unfixable on your end.

Andrea

Re: bandwidth for bkbits.net (good news)

[Jamie Lokier]

Andrea Arcangeli wrote:
> On Sun, Aug 31, 2003 at 03:56:39PM -0700, Larry McVoy wrote:
> > Yet you keep insisting it will work.  Why?  What is the theory that says
> > you can keep the other end of the T1 line from being congested when you
> > don't have control over that router?  And that router has several 100Mbit
> 
> it's absolutely trivial, your end only needs to drop 99% of your
> outgoing acks and their incoming packets for every connection but voip
> while you are at the phone, you won't kill the connections but everybody
> but your voip will work. the exponential backoff and sstrash on the
> other and will rate limit everything immediatly.

Let's work it out.  We assume 99% means drop virtually everything:

  Every 19 seconds on average, 24x7, a new HTTP connection.

  Rate is not uniform throughout the day.  Let's take a wild guess and
  say it is 10 times higher at peak times.

  That's one connection every 1.9 seconds.

  Let's assume you drop 99% of outgoing ACKs.

  Then all connecting remote clients will retry their SYNs until they
  get a connection or a timeout.  Default tcp_syn_retries (assuming Linux
  clients) is 5.

  That's one SYN every 0.38 seconds.   -> bad but not awful.

Plus existing connections.  Let's pretend each connection take 100 seconds.

  That's 100/1.9 or 52 concurrent connections, roughly.

  Each of those will retry with an ACK if it has any pending ACK or data.

  When you start using the phone, that's 100 ACKs total,
  approximately, with exponential backoff.    --> bad but not awful.

These calculations are horrendously inaccurate, but should be ok
within a couple of orders of magnitude.

So, near-total annihilation of bkbits.net when Larry or any of his
team are on the phone should work.  You can either integrate the phone
system with netfilter so it is automatic when a customer calls.
Trivial ;)  Or disable bkbits.net during Larry's working day. ;)

-- Jamie

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 01:18:19AM +0100, Jamie Lokier wrote:
>   Every 19 seconds on average, 24x7, a new HTTP connection.

Wrong.  Every 19 seconds is a new BK connection.  The HTTP connections are
at least an order of magnitude more frequent.  And increasing, more and 
more people are becoming aware that the source is there and you can 
point to it.

>   That's one connection every 1.9 seconds.

Ha.  Don't I wish.  Peak connection rates are a lot higher than that.  

> Trivial ;)  Or disable bkbits.net during Larry's working day. ;)

It's not my working day or the other people who work locally, all our
phones are on a 100Mbit switched net to the 3com before it goes out on
POTS.  It's the remote people.  And half our company is currently remote.
I could write a book on how well remote does and doesn't work.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Mon, Sep 01, 2003 at 01:18:19AM +0100, Jamie Lokier wrote:
> So, near-total annihilation of bkbits.net when Larry or any of his
> team are on the phone should work.  You can either integrate the phone

this is *exactly* my point.

Of course near-total annihilation of bkbits.net may not be acceptable to
the community, but still it should work as a proof of concept.

Depending on the connect/sec of the http server (not bkbits, for the
largest part of the conversation I couldn't know about the http server,
Larry only mentioned the bkbits.net clone until recently), the
"reservation" margin will have to change: the less connect/sec the
smaller margin Larry will need to reserve, the more connect/sec the
bigger marging will be necessary.

Andrea

Re: bandwidth for bkbits.net (good news)

[Jamie Lokier]

Andrea Arcangeli wrote:
> Depending on the connect/sec of the http server (not bkbits, for the
> largest part of the conversation I couldn't know about the http server,
> Larry only mentioned the bkbits.net clone until recently), the
> "reservation" margin will have to change: the less connect/sec the
> smaller margin Larry will need to reserve, the more connect/sec the
> bigger marging will be necessary.

Hi Andrea,

Above a certain connection rate, no amount of margin is enough.  The
connections are a SYN flood.

You need to take into account the peak rate, which varies from second
to second, due to simple statistics plus the tendency of the net to
make traffic burstier than it started.

Above a certain number of packets/sec, competing VoIP traffic degrades
- a little added latency is equivalent to dropping with VoIP.  That
translates to dropouts in the audio.  Acceptable for hackers talking
over a flakey line, but not corporate telephony.

At Larry's server, inbound connection rate resembles a low-level SYN
flood, which is enough to poke holes in VoIP latency.

Of course the real problem is lack of end to end QoS between Larry's
nodes.  Two T1s is just a workaround :)

-- Jamie

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Mon, Sep 01, 2003 at 01:50:41AM +0100, Jamie Lokier wrote:
> Andrea Arcangeli wrote:
> > Depending on the connect/sec of the http server (not bkbits, for the
> > largest part of the conversation I couldn't know about the http server,
> > Larry only mentioned the bkbits.net clone until recently), the
> > "reservation" margin will have to change: the less connect/sec the
> > smaller margin Larry will need to reserve, the more connect/sec the
> > bigger marging will be necessary.
> 
> Hi Andrea,
> 
> Above a certain connection rate, no amount of margin is enough.  The
> connections are a SYN flood.
> 
> You need to take into account the peak rate, which varies from second
> to second, due to simple statistics plus the tendency of the net to
> make traffic burstier than it started.
> 
> Above a certain number of packets/sec, competing VoIP traffic degrades
> - a little added latency is equivalent to dropping with VoIP.  That
> translates to dropouts in the audio.  Acceptable for hackers talking
> over a flakey line, but not corporate telephony.
> 
> At Larry's server, inbound connection rate resembles a low-level SYN
> flood, which is enough to poke holes in VoIP latency.

Yes. if you check my very first email in this thread, you will see I
said "it has to work, unless you're under syn-flood ;)".

this is obvious, I'm not arguing about that. From Larry's description of
the problem it couldn't be a syn flood, but it was a bkbits.net clone
thing or checkout or whatever non syn intensive.

now apparently bkbits.net has nothing to do with it, and it's all about
the http server. Then of course the syn overhead may become noticeable
but things have changed totally from the original description of the
problem (the bkbits.net clone thing).

however keep in mind you will somehow throttle the number of syns too,
unless every single syn arrives to the webserver from a different user
(unlikely).

Andrea

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Mon, Sep 01, 2003 at 03:10:55AM +0200, Andrea Arcangeli wrote:
> now apparently bkbits.net has nothing to do with it, and it's all about
> the http server. 

BK _is_ an http server.  The same daemon you talk to for clones is the
HTTP server.  That's one and the same machine.

> however keep in mind you will somehow throttle the number of syns too,
> unless every single syn arrives to the webserver from a different user
> (unlikely).

That's exactly the situation that any busy server has.  Which is
why I kept saying "tell me how you made this work for a busy server".
"Busy server" by definition in this context at least means a server that
is getting lots of connection requests from lots of different users.

Why that isn't obvious to you I don't understand.  This is bkbits.net.
Yeah, it's not slashdot or anything but there are something like 400
branches of the Linux kernel on it and that's ignoring all other projects.
It's the web server which provides insight into the Linux kernel source
base, of course it is busy.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Henning P. Schmiedehausen]

Andrea Arcangeli <andrea@suse.de> writes:

>> They are Cisco configuration, it won't do you much good.  All the traffic

>I don't trust anything but the sourcecode I can read, so please try
>again with linux.

Andrea, you have no clue at all. Please stop this. You might be
terrific at VM and linux kernel internals but you suck at network
topology and serious internet traffic routing.

You want to try rate limiting a DDoS attack from the wrong side of a
small pipe. You can't do this. Not with Linux, with BSD, with a C64 or
a Cisco. 

People have been trying this on the internet since 1995 and know what
they're talking about. This is not a case of "I have no clue, so I
didn't know that it was impossible and I did it". There are people out
there that got Ph.D.'s for proving that it's impossible to do what you
suggest and other people that got rich building devices that can do
this (from the right end of the pipe).

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)

Re: bandwidth for bkbits.net (good news)

[Henning P. Schmiedehausen]

Andrea Arcangeli <andrea@suse.de> writes:

>> It doesn't work when you dont control incoming. As a simple extreme
>> example if I pingflood you from a fast site then no amount of shaping
>> your end of the link will help, it has to be shaped at the ISP end.

>sure, that's why I said it won't work with synflood. I just doubt the
>ping/syn floods distributed denial of services are an high percentage of

The traffic on sites like www.microsoft.com, www.kernel.org or I'd say
even www.bkbits.net isn't distinguishable from a DDoS Synflood on a
regular day. A bazillion of different IP addresses from all over the
net trying to establish a TCP connection at the same time. And the
other sides are dog-slow modem/isdn/dsl connections which means high
latency, low bandwith, many retransmits and still they're legal
connections that you want to serve.

Forget it. You can't compare your homegrown ISDN/DSL line with the
traffic of a serious internet service. If you try, you get burned.

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Sun, Aug 31, 2003 at 04:31:32PM +0100, Alan Cox wrote:
> > what's the difference of rejecting packets in software, or because the
> > link can't handle them? Assume the guaranteed bandwidth is much lower
> 
> It doesn't work when you dont control incoming. As a simple extreme
> example if I pingflood you from a fast site then no amount of shaping
> your end of the link will help, it has to be shaped at the ISP end.

HTTP traffic is enough to simulate this, the connections are all small,
short lived, and there are a lot of them.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 09:23:37AM -0700, Larry McVoy wrote:
> On Sun, Aug 31, 2003 at 04:31:32PM +0100, Alan Cox wrote:
> > > what's the difference of rejecting packets in software, or because the
> > > link can't handle them? Assume the guaranteed bandwidth is much lower
> > 
> > It doesn't work when you dont control incoming. As a simple extreme
> > example if I pingflood you from a fast site then no amount of shaping
> > your end of the link will help, it has to be shaped at the ISP end.
> 
> HTTP traffic is enough to simulate this, the connections are all small,
> short lived, and there are a lot of them.

it's much harder to throttle http, but it should work too. you may need
bigger margin due the higher percentage of unthrottable packets like
syns.

Andrea

Re: bandwidth for bkbits.net (good news)

[Jörn Engel]

On Sun, 31 August 2003 16:45:05 +0200, Andrea Arcangeli wrote:
> On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote:
> > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote:
> > > I'm pretty convinced we can't solve the problem at our end.  Maybe we can
> > 
> > For bursts of traffic you can't.
> 
> what's the difference of rejecting packets in software, or because the
> link can't handle them?

Nothing, packets are always rejected in software. :)

> it works flawlessy for me and it's the same problem (I also use
> streaming services), and they're unusable until I turn the shaping on,
> I'm sure that if you use the script and you change 1kbyte/sec to
> everything but voip it'll work fine for you too, since basically
> everything else won't pass anymore, it will take ages to open an html
> page and all the bkbits.net users will hang, and the link will be idle
> 99% of the time, so voip will take it over as much as it can. I don't
> think it's a matter of "if it works or not", I think it's a matter of
> how much you're ok to lose in terms of global bandwith for all the other
> services but voip.

Yes, that should work for well behaved traffic.  You have to leave
more breathing room compared to your provider doing the traffic
shaping for you, but it is also more flexible than one line for each
VOIP and the rest.

Still, I don't like the idea of throwing away those packets that have
made it through the bottleneck. :)

Jörn

-- 
The strong give up and move away, while the weak give up and stay.
-- unknown

Re: bandwidth for bkbits.net (good news)

[Henning P. Schmiedehausen]

Larry McVoy <lm@bitmover.com> writes:

>I'm pretty convinced we can't solve the problem at our end.  Maybe we can

You're right. You can't no matter what all the "network experts"
around here say. It's either placing a shaper at the ISP or getting a
second line.

I still don't understand why you don't simply co-lo the bkbits.net
box. There should be plenty of housing centers in the Bay Area within
a 30 minute drive of your office. ;-)

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)

Re: bandwidth for bkbits.net (good news)

[Pascal Schmidt]

On Sun, 31 Aug 2003, Andrea Arcangeli wrote:

> This is what I use normally to limit my brother downloads, and it works
> fine for me (though I don't often place calls through adsl myself, it's
> basically useless since people only uses cellphones here, and last time
> I chekced voip wasn't free for cellphones with my isp).
> 
> this is the script:

Mine is similar, though I use tc filters instead of firewalling
rules (my machine is on 192.168.2.0/24, rest of the house is
on 192.168.3.0/24) I'm using the imq device on a 2.2 kernel to
have all traffic go through that device for shaping:

#!/bin/bash
# traffic shaping startup script
#
# (c) 2003 Pascal Schmidt <der.eremit@email.de>

. /etc/rc.d/rc.functions

PATH=/bin:/sbin:/usr/bin:/usr/sbin

case "$1" in
start)
        echo -n "Setting up traffic shaper: "

        # remove other qdisc if any
	tc qdisc del dev imq root &> /dev/null

	# use HTB as root
	tc qdisc add dev imq root handle 1:0 htb default 3

	# complete LAN
	tc class add dev imq parent 1:0 classid 1:1  htb \
	   rate 10mbit burst 15k

	# ISDN connection, split up into two "halves"
	tc class add dev imq parent 1:1 classid 1:2  htb \
	   rate 64kbit burst 15k
	tc class add dev imq parent 1:2 classid 1:21 htb \
	   rate 40kbit ceil 64kbit burst 15k
	tc class add dev imq parent 1:2 classid 1:22 htb \
	   rate 24kbit ceil 59kbit burst 15k

	# rest of LAN bandwidth for internal traffic
	tc class add dev imq parent 1:1 classid 1:3  htb \
	   rate  9mbit ceil 10mbit burst 15k

	# use SFQ on leaf classes
	tc qdisc add dev imq parent 1:21 handle 21:0 sfq perturb 10
	tc qdisc add dev imq parent 1:22 handle 22:0 sfq perturb 10
	tc qdisc add dev imq parent 1:3  handle  3:0 sfq perturb 10

	# LAN to LAN traffic goes to fast class
	tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \
	   match ip src 192.168.0.0/16 \
	   match ip dst 192.168.0.0/16 flowid 1:3

	# rest of traffic to LAN #1 (must be from Internet)
	# goes to first ISDN class (half ISDN channel)
	tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \
	   match ip dst 192.168.2.0/24 flowid 1:21

	# rest of traffic to LAN #2 (must be from Internet)
	# goes to second ISDN class (half ISDN channel)
	tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \
	   match ip dst 192.168.3.0/24 flowid 1:22

        check_status

	# bring up intermediate queue interface
	echo -n "Bringing up intermediate queue: "
	ifconfig imq up
	check_status
	;;
stop)
	# bring down intermediate queue interface
	echo -n "Taking down intermediate queue: "
	ifconfig imq down
	check_status
	
	# remove root qdisc
	echo -n "Disabling traffic shaper: "
	tc qdisc del dev imq root
	check_status
	;;
status)
	tc -d -s class show dev imq
	;;
restart)
	$0 stop
	$0 start
	;;
*)
	echo "Usage: $0 {start|stop|status|restart}"
	exit 1
esac

-- 
Ciao,
Pascal

Re: bandwidth for bkbits.net (good news)

[Andrea Arcangeli]

On Sun, Aug 31, 2003 at 03:47:17PM +0200, Pascal Schmidt wrote:
> On Sun, 31 Aug 2003, Andrea Arcangeli wrote:
> 
> > This is what I use normally to limit my brother downloads, and it works
> > fine for me (though I don't often place calls through adsl myself, it's
> > basically useless since people only uses cellphones here, and last time
> > I chekced voip wasn't free for cellphones with my isp).
> > 
> > this is the script:
> 
> Mine is similar, though I use tc filters instead of firewalling
> rules (my machine is on 192.168.2.0/24, rest of the house is
> on 192.168.3.0/24) I'm using the imq device on a 2.2 kernel to
> have all traffic go through that device for shaping:

the main reason I choosen to mark the packets through netfilter is that
I don't find `tc` very intuitive (the most annoying thing is that it
misses a flush command), so it'll be quicker to add future tweaks this
way (like adding more ips or port/ip combinations into the "shaped"
channel)

thanks,
Andrea

Re: bandwidth for bkbits.net (good news)

[Henning P. Schmiedehausen]

Pascal Schmidt <der.eremit@email.de> writes:

>In a way, you're on the right end of the pipe because the system
>that does your traffic shaping is part of the general network, viewed
>from the machines behind the shaper.

Stop argueing. You can shape only outgoing packets, which means that the
shaper must have a big uplink pipe and then shape that into a thin downlink
pipe. In Larrys' case, the shaper must be on the ISP network before the T1.

Once the packets hit the T1 (or your ISDN link), you've already
lost. No matter how much shaping, dropping, non-acking you're doing on
your side of the link.

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
hps@intermeta.de        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)

Re: bandwidth for bkbits.net (good news)

[Jörn Engel]

On Sun, 31 August 2003 00:58:33 +0200, Pascal Schmidt wrote:
> 
> All you have to do is drop the incoming packets if they exceed
> a certain bandwidth. That will stop the corresponding ack automatically
> since your TCP stack won't even see the packet.

You've cut away the part where I explained that you don't want to
throw away the incoming packet.  Yes, the stupid approach works, but
it is still stupid.

> I'm doing this on my ISDN line to limit the rest of the house to one
> third of the bandwidth even if they're all busy downloading tons of
> warez. I'm paying, I want the bandwidth when I need it. They can get
> it all when there's no traffic for my machine.
> 
> No problem with the HTB queueing discipline.

But latency just went boom.  On your side, your packets will come out
quickly because the queue is short.  But on your ISP's side, there is
a single queue for both your and the warez' traffic.  Your packets
will get thrown away just as much, as theirs.

> Yes, latency is a problem if you want to saturate your bandwidth. It is
> easy to guarantee some bandwith for latency critical stuff - just
> don't give out that piece of bandwith to something else. Of course,
> then most of the time that piece is wasted...

This works if your latency requirements are moderate or the pipe is
big.  Assume 5ms and ISDN and you have a window of 400 Bytes roughly.
Each time you happen to receive 400 Bytes of packets at the same time,
you hear a pause.  A single large packet is more than that.  Oops!

For Larry's T1 line, the numbers naturally go up, but it still doesn't
take a huge amount of packets.

> and it doesn't help with problems somewhere along the net.

Compared to ISDN or T1, the net usually has big pipes and people tend
to care about low latency, so that problem doesn't even exist compared
with the receiving end.

Jörn

-- 
Simplicity is prerequisite for reliability.
-- Edsger W. Dijkstra

Re: bandwidth for bkbits.net (good news)

[Pascal Schmidt]

On Tue, 02 Sep 2003 09:00:20 +0200, you wrote in linux.kernel:

> Stop argueing. You can shape only outgoing packets, which means that the
> shaper must have a big uplink pipe and then shape that into a thin downlink
> pipe. In Larrys' case, the shaper must be on the ISP network before the T1.

For Larry's case, you're right. Nothing you can do to shape lots of
incoming TCP connections. However, if you're facing a few long-lived
TCP connections, you can shape them by dropping incoming packets (that
is, shaping the outgoing packets from the router/shaper to the LAN).
The sending side will adjust its rate after at most a few seconds.

That works perfectly for the case of reserving bandwidth for critical
services in the face of ongoing large downloads. That's something
a lot of people want and need, so it's not all that worseless per se.

-- 
Ciao,
Pascal
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
On Tue, 02 Sep 2003 09:00:20 +0200, you wrote in linux.kernel:

> Stop argueing. You can shape only outgoing packets, which means that the
> shaper must have a big uplink pipe and then shape that into a thin downlink
> pipe. In Larrys' case, the shaper must be on the ISP network before the T1.

For Larry's case, you're right. Nothing you can do to shape lots of
incoming TCP connections. However, if you're facing a few long-lived
TCP connections, you can shape them by dropping incoming packets (that
is, shaping the outgoing packets from the router/shaper to the LAN).
The sending side will adjust its rate after at most a few seconds.

That works perfectly for the case of reserving bandwidth for critical
services in the face of ongoing large downloads. That's something
a lot of people want and need, so it's not all that worseless per se.

-- 
Ciao,
Pascal

bandwidth for bkbits.net (good news)

[Larry McVoy]

FYI - we're working with several vendors to try and get more bandwidth
for bkbits.net.  We think we have a line on a deal where we can get a
full T1 just for bkbits.net for about $500/month.  If we get that then
we'll turn on the patch server feature so you can hit any URL and get a
regular diff -Nur style patch for that changeset (or range of changesets).

We've avoiding turning on that feature in the past because we share the
T1 line that bkbits.net lives on with all the rest of bitmover and we are
partialy a distributed company.  We do VOIP phones and when you guys clone
a repo our phones don't work - that makes us look bad during a sales call.
I'm not complaining, we get nice stress testing from bkbits so you should
hammer on it all you want but I'd like it if we could really encourage
that more.  Turning on a patch server should do the trick.

ETA on this is a month.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Larry McVoy]

On Fri, Aug 29, 2003 at 06:29:49PM -0700, Larry McVoy wrote:
> We've avoiding turning on that feature in the past because we share the
> T1 line that bkbits.net lives on with all the rest of bitmover and we are
> partialy a distributed company.  We do VOIP phones and when you guys clone
> a repo our phones don't work - that makes us look bad during a sales call.

Many people have sent me mail saying that we should be using traffic
shaping to fix this problem.  We are using it and we can't seem to make
it work.  Our theory is that we have a network like

 ----- [ ISP ] ====== internet ======== [ ISP ] ----

wherein "-" means our skinny T1 or DSL and "=" means some fat internet 
connection on the backbone.  

We can shape all we want on our ends but if the internet is blasting us
then our skinny pipe gets full and our shaping doesn't work.  We really
need to have the ISP do the shaping so they can squelch the traffic 
before it gets to our pipe.

If there is someone out there who (a) is running VOIP over the public net
to a pile of different end points (T1 on both ends tends to work, T1 to DSL
or cable modem tends to get harder) and (b) has figured out traffic shaping
that works I'd love to know about it.  

But just saying QoS/wondershaper doesn't help much (though the thought is
appreciated), we've tried that already.

Thanks.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: bandwidth for bkbits.net (good news)

[Jörn Engel]

On Sat, 30 August 2003 08:03:11 -0700, Larry McVoy wrote:
> 
> Many people have sent me mail saying that we should be using traffic
> shaping to fix this problem.  We are using it and we can't seem to make
> it work.  Our theory is that we have a network like
> 
>  ----- [ ISP ] ====== internet ======== [ ISP ] ----
> 
> wherein "-" means our skinny T1 or DSL and "=" means some fat internet 
> connection on the backbone.  
> 
> We can shape all we want on our ends but if the internet is blasting us
> then our skinny pipe gets full and our shaping doesn't work.  We really
> need to have the ISP do the shaping so they can squelch the traffic 
> before it gets to our pipe.
> 
> If there is someone out there who (a) is running VOIP over the public net
> to a pile of different end points (T1 on both ends tends to work, T1 to DSL
> or cable modem tends to get harder) and (b) has figured out traffic shaping
> that works I'd love to know about it.  

By principle, you can only control your side of the wire.  Unless your
ISP does some decent shaping for you or allows you to place a machine
at the other end to do it, you are at the internets mercy.  If people
with enough bandwidth want to DOS you, they can.

For well behaved traffic you have some limited control over the
incoming traffic through your responses, so QoS should work in theory,
but setting it up is still very close to black magic.

> But just saying QoS/wondershaper doesn't help much (though the thought is
> appreciated), we've tried that already.

Wondershaper was magically configured once and works for everyone that
has similar needs like the original magician.  Plus, this magician had
a simple task, as the sending side is the bottleneck and that is the
side he has control over.  So unless you have a regular DSL line...

What you really need might not even exist in the kernel yet.  The last
couple of times I've tried setting it up and read the code, things
were just not adequate.

---<deeper technical stuff follows>---

In order to control incoming traffic, it is easiest to look at tcp.
udp can work similarly, but it doesn't have to.  To throttle the
stream of tcp packets, you could simply through away the acks and the
sending side will reduce it's speed.  So you have to measure one
stream and control a related but different one.  Maybe this is
possible, not sure.

Second, you usually don't want to through away the acks, as packets
would be retransmitted then.  This reduces the effective bandwidth and
limited bandwidth was the problem in the first place.  So we have to
delay them enough to slow things down, but not beyond the timeout.

Third, there is the problem transition from continuous streams to
discrete packets, when bandwidth is low.  It doesn't take a huge
amount of large packets to create enough latency for your sad VOIP.

And fourth, the possibility of resonance frequency.  If measurement
and/or shaping intervals are too long and match nicely to the travel
time to one of your peers, you are back at point three.

Overall, it is possible to do some decent traffic shaping, but it is
far from being simple.  And it might even be completely impossible, at
least in your case, with the code that is in the kernel today, be it
Linux or some BSD.

Jörn

PS: If anyone can prove how stupid I am and how simple QoS is, please
do!  Seriously!

-- 
Public Domain  - Free as in Beer
General Public - Free as in Speech
BSD License    - Free as in Enterprise
Shared Source  - Free as in "Work will make you..."

Re: bandwidth for bkbits.net (good news)

[Toon van der Pas]

On Sat, Aug 30, 2003 at 08:03:11AM -0700, Larry McVoy wrote:
> On Fri, Aug 29, 2003 at 06:29:49PM -0700, Larry McVoy wrote:
> 
> > We've avoiding turning on that feature in the past because we
> > share the T1 line that bkbits.net lives on with all the rest of
> > bitmover and we are partialy a distributed company.  We do VOIP
> > phones and when you guys clone a repo our phones don't work -
> > that makes us look bad during a sales call.
> 
> Many people have sent me mail saying that we should be using
> traffic shaping to fix this problem.  We are using it and we can't
> seem to make it work.  Our theory is that we have a network like
> 
>  ----- [ ISP ] ====== internet ======== [ ISP ] ----
> 
> wherein "-" means our skinny T1 or DSL and "=" means some fat
> internet connection on the backbone.
> 
> We can shape all we want on our ends but if the internet is
> blasting us then our skinny pipe gets full and our shaping doesn't
> work.  We really need to have the ISP do the shaping so they can
> squelch the traffic before it gets to our pipe.
> 
> If there is someone out there who (a) is running VOIP over the
> public net to a pile of different end points (T1 on both ends tends
> to work, T1 to DSL or cable modem tends to get harder) and (b) has
> figured out traffic shaping that works I'd love to know about it.

We are using a Packetshaper box from Packeteer (www.packeteer.com).
It solves the problem you are describing (shaping the incoming traffic)
by ajusting the windowsize in the outgoing acknowledgements, if I'm not
mistaken.  It has done miracles for us.

Could Linux do this?

(Sorry for the plug. I don't have stocks in the company, nor am I
involved with them in any other way. We are just a happy customer.)

> But just saying QoS/wondershaper doesn't help much (though the
> thought is appreciated), we've tried that already.
> 
> Thanks.
> -- 
> ---
> Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Regards,
Toon.