* Re: bandwidth for bkbits.net (good news) [not found] ` <qgCn.2y4.11@gated-at.bofh.it> @ 2003-08-30 22:58 ` Pascal Schmidt 2003-08-30 23:07 ` Larry McVoy 2003-08-31 6:30 ` Jörn Engel 0 siblings, 2 replies; 71+ messages in thread From: Pascal Schmidt @ 2003-08-30 22:58 UTC (permalink / raw) To: Jörn Engel; +Cc: linux-kernel On Sat, 30 Aug 2003 18:20:11 +0200, you wrote in linux.kernel: > In order to control incoming traffic, it is easiest to look at tcp. > udp can work similarly, but it doesn't have to. To throttle the > stream of tcp packets, you could simply throw away the acks and the > sending side will reduce it's speed. So you have to measure one > stream and control a related but different one. Maybe this is > possible, not sure. All you have to do is drop the incoming packets if they exceed a certain bandwidth. That will stop the corresponding ack automatically since your TCP stack won't even see the packet. I'm doing this on my ISDN line to limit the rest of the house to one third of the bandwidth even if they're all busy downloading tons of warez. I'm paying, I want the bandwidth when I need it. They can get it all when there's no traffic for my machine. No problem with the HTB queueing discipline. [...] > Third, there is the problem transition from continuous streams to > discrete packets, when bandwidth is low. It doesn't take a huge > amount of large packets to create enough latency for your sad VOIP. Yes, latency is a problem if you want to saturate your bandwidth. It is easy to guarantee some bandwith for latency critical stuff - just don't give out that piece of bandwith to something else. Of course, then most of the time that piece is wasted... and it doesn't help with problems somewhere along the net. > And fourth, the possibility of resonance frequency. If measurement > and/or shaping intervals are too long and match nicely to the travel > time to one of your peers, you are back at point three. Dunno about that one. -- Ciao, Pascal ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-30 22:58 ` bandwidth for bkbits.net (good news) Pascal Schmidt @ 2003-08-30 23:07 ` Larry McVoy 2003-08-31 1:05 ` Pascal Schmidt 2003-08-31 6:30 ` Jörn Engel 1 sibling, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-30 23:07 UTC (permalink / raw) To: Pascal Schmidt; +Cc: J?rn Engel, linux-kernel On Sun, Aug 31, 2003 at 12:58:33AM +0200, Pascal Schmidt wrote: > On Sat, 30 Aug 2003 18:20:11 +0200, you wrote in linux.kernel: > > > In order to control incoming traffic, it is easiest to look at tcp. > > udp can work similarly, but it doesn't have to. To throttle the > > stream of tcp packets, you could simply throw away the acks and the > > sending side will reduce it's speed. So you have to measure one > > stream and control a related but different one. Maybe this is > > possible, not sure. > > All you have to do is drop the incoming packets if they exceed > a certain bandwidth. If you think we haven't done that, think again. We're at the wrong end of the pipe to do that, I'm pretty sure that what you are describing simply won't work. We've already tried rate limiting all traffic except for the phone traffic to 1/2 the T1 line and it doesn't work. Remember, we're talking vioce here. Retransmits don't work and aren't even attempted. I've got people in 5 or 6 states spread out from one end of the USA to the other. So any disruption means things don't work. All of the traffic shaping tends to be focussed on "good" versus "bad" bandwidth utilization. That's nice but it means that you are tolerant of a "settling time" during which everyone is fighting for bandwidth. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-30 23:07 ` Larry McVoy @ 2003-08-31 1:05 ` Pascal Schmidt 2003-08-31 1:39 ` Andrea Arcangeli 2003-09-02 6:53 ` Henning P. Schmiedehausen 0 siblings, 2 replies; 71+ messages in thread From: Pascal Schmidt @ 2003-08-31 1:05 UTC (permalink / raw) To: Larry McVoy; +Cc: linux-kernel On Sat, 30 Aug 2003, Larry McVoy wrote: >> All you have to do is drop the incoming packets if they exceed >> a certain bandwidth. > If you think we haven't done that, think again. > We're at the wrong end of the pipe to do that, I'm pretty sure that what > you are describing simply won't work. In a way, you're on the right end of the pipe because the system that does your traffic shaping is part of the general network, viewed from the machines behind the shaper. Dropping the packets means that the sending side, at least if we're talking TCP, will throttle its sending rate. But, depending on the distance in hops to the sender, it may take up to a few seconds for this to kick in. So I guess that's why it doesn't work for your VoIP case - the senders don't notice fast enough that they should slow down. When my ISDN line is saturated with downloads from the rest of the people here and data starts to come in for MY machine, it takes about 5 seconds until I get my bandwidth. Too slow for VoIP. Traffic shaping on the side of your ISP will probably help since they have a bigger incoming pipe. If they hadn't, they'd have the same problem, though. -- Ciao, Pascal ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 1:05 ` Pascal Schmidt @ 2003-08-31 1:39 ` Andrea Arcangeli 2003-08-31 2:18 ` Jeff Sipek ` (2 more replies) 2003-09-02 6:53 ` Henning P. Schmiedehausen 1 sibling, 3 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 1:39 UTC (permalink / raw) To: Pascal Schmidt; +Cc: Larry McVoy, linux-kernel On Sun, Aug 31, 2003 at 03:05:37AM +0200, Pascal Schmidt wrote: > On Sat, 30 Aug 2003, Larry McVoy wrote: > > >> All you have to do is drop the incoming packets if they exceed > >> a certain bandwidth. > > If you think we haven't done that, think again. > > > We're at the wrong end of the pipe to do that, I'm pretty sure that what > > you are describing simply won't work. > > In a way, you're on the right end of the pipe because the system > that does your traffic shaping is part of the general network, viewed > from the machines behind the shaper. > > Dropping the packets means that the sending side, at least if we're > talking TCP, will throttle its sending rate. But, depending on the > distance in hops to the sender, it may take up to a few seconds for > this to kick in. So I guess that's why it doesn't work for your > VoIP case - the senders don't notice fast enough that they should > slow down. that's because you don't limit the bkbits.net to a fixed rate. If you want to give priorities, it won't work well because it takes time to be effective, but if you rate limit hard both ways it has to work, unless you're under syn-flood ;) The downside is that you will waste bandwith (i.e. you will hurt the bkbits.net service even when you don't use voip), but it will work. This is what I use normally to limit my brother downloads, and it works fine for me (though I don't often place calls through adsl myself, it's basically useless since people only uses cellphones here, and last time I chekced voip wasn't free for cellphones with my isp). this is the script: INTERNAL_NET=eth0 EXTERNAL_NET=ppp0 # External net, prioritize with TOS and low prio traffic in MARK chan 6 tc qdisc add dev $EXTERNAL_NET root handle 1: prio bands 5 tc qdisc add dev $EXTERNAL_NET parent 1:1 handle 10: sfq tc qdisc add dev $EXTERNAL_NET parent 1:2 handle 20: sfq tc qdisc add dev $EXTERNAL_NET parent 1:3 handle 30: sfq tc qdisc add dev $EXTERNAL_NET parent 1:4 tbf rate 16kbit buffer 1600 limit 3000 tc qdisc add dev $EXTERNAL_NET parent 1:5 tbf rate 16kbit buffer 1600 limit 3000 tc filter add dev $EXTERNAL_NET protocol ip parent 1:0 prio 4 handle 6 fw flowid 1:4 tc filter add dev $EXTERNAL_NET protocol ip parent 1:0 prio 5 handle 7 fw flowid 1:5 tc qdisc add dev $INTERNAL_NET root handle 1: prio bands 5 tc qdisc add dev $INTERNAL_NET parent 1:1 handle 10: sfq tc qdisc add dev $INTERNAL_NET parent 1:2 handle 20: sfq tc qdisc add dev $INTERNAL_NET parent 1:3 handle 30: sfq tc qdisc add dev $INTERNAL_NET parent 1:4 tbf rate 64kbit buffer 1600 limit 3000 tc qdisc add dev $INTERNAL_NET parent 1:5 tbf rate 64kbit buffer 1600 limit 3000 tc filter add dev $INTERNAL_NET protocol ip parent 1:0 prio 4 handle 8 fw flowid 1:4 tc filter add dev $INTERNAL_NET protocol ip parent 1:0 prio 5 handle 9 fw flowid 1:5 #tc qdisc add dev $EXTERNAL_NET root sfq perturb 30 btw, the sfq is a great idea to have anyways, to provide fariness to the traffing going out of your network. then the below scripts will turn it off/on (so he can get full bandwith while I'm not at my desk or when he's not doing anything very network intensive ;) Basically you should run the below with 'start' before placing the call, and running 'stop' after the call is complete. It may take a few seconds to rate limit all the connections but it should work fine. EXTERNAL_NET=ppp0 INTERNAL_NET=eth0 case "$1" in start) iptables -t mangle -A PREROUTING -t mangle -j MARK -s 192.168.1.9 --set-mark 6 -i $INTERNAL_NET iptables -t mangle -A POSTROUTING -t mangle -j MARK -d 192.168.1.9 --set-mark 8 ;; stop) iptables -t mangle -D PREROUTING -t mangle -j MARK -s 192.168.1.9 --set-mark 6 -i $INTERNAL_NET iptables -t mangle -D POSTROUTING -t mangle -j MARK -d 192.168.1.9 --set-mark 8 ;; status) iptables -t mangle -L -v ;; *) echo "Usage: $0 {start|stop}" exit 1 ;; esac (he's IP 192.168.1.9 of course, you should replace it with bkbits.net and tweak the 64kbit output/16k input with the input/output bandwith you want to give to bkbits) BTW, I still wonder how to rsync the bkcvs coherently? (and I wonder how can Peter sync it synchronous too if you don't give him a pair of sequence numbers or a fcntl lock, personally I prefer the sequence numbers so it works for pure read only too like the rsync on kernel.org) Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 1:39 ` Andrea Arcangeli @ 2003-08-31 2:18 ` Jeff Sipek 2003-08-31 2:42 ` Andrea Arcangeli 2003-08-31 2:56 ` Larry McVoy 2003-08-31 13:47 ` Pascal Schmidt 2 siblings, 1 reply; 71+ messages in thread From: Jeff Sipek @ 2003-08-31 2:18 UTC (permalink / raw) To: Andrea Arcangeli, Pascal Schmidt; +Cc: Larry McVoy, linux-kernel -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday 30 August 2003 21:39, Andrea Arcangeli wrote: > On Sun, Aug 31, 2003 at 03:05:37AM +0200, Pascal Schmidt wrote: > > On Sat, 30 Aug 2003, Larry McVoy wrote: > > >> All you have to do is drop the incoming packets if they exceed > > >> a certain bandwidth. > > > > > > If you think we haven't done that, think again. > > > > > > We're at the wrong end of the pipe to do that, I'm pretty sure that > > > what you are describing simply won't work. > > > > In a way, you're on the right end of the pipe because the system > > that does your traffic shaping is part of the general network, viewed > > from the machines behind the shaper. > > > > Dropping the packets means that the sending side, at least if we're > > talking TCP, will throttle its sending rate. But, depending on the > > distance in hops to the sender, it may take up to a few seconds for > > this to kick in. So I guess that's why it doesn't work for your > > VoIP case - the senders don't notice fast enough that they should > > slow down. > > that's because you don't limit the bkbits.net to a fixed rate. If you > want to give priorities, it won't work well because it takes time to be > effective, but if you rate limit hard both ways it has to work, unless > you're under syn-flood ;) The downside is that you will waste bandwith > (i.e. you will hurt the bkbits.net service even when you don't use > voip), but it will work. How about giving something to voip as a hard limit and then using some shaper to give it more if needed. Jeff. - -- *NOTE: This message is ROT-13 encrypted twice for extra protection* -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) iD8DBQE/UVsBwFP0+seVj/4RAkmpAJ9YwjdPLZZsdD7fCRDRHyS+JrJnkgCdG/sc sr5Mqx5Qii//AQPepCqHDcw= =RoPR -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 2:18 ` Jeff Sipek @ 2003-08-31 2:42 ` Andrea Arcangeli 0 siblings, 0 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 2:42 UTC (permalink / raw) To: Jeff Sipek; +Cc: Pascal Schmidt, Larry McVoy, linux-kernel On Sat, Aug 30, 2003 at 10:18:37PM -0400, Jeff Sipek wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Saturday 30 August 2003 21:39, Andrea Arcangeli wrote: > > On Sun, Aug 31, 2003 at 03:05:37AM +0200, Pascal Schmidt wrote: > > > On Sat, 30 Aug 2003, Larry McVoy wrote: > > > >> All you have to do is drop the incoming packets if they exceed > > > >> a certain bandwidth. > > > > > > > > If you think we haven't done that, think again. > > > > > > > > We're at the wrong end of the pipe to do that, I'm pretty sure that > > > > what you are describing simply won't work. > > > > > > In a way, you're on the right end of the pipe because the system > > > that does your traffic shaping is part of the general network, viewed > > > from the machines behind the shaper. > > > > > > Dropping the packets means that the sending side, at least if we're > > > talking TCP, will throttle its sending rate. But, depending on the > > > distance in hops to the sender, it may take up to a few seconds for > > > this to kick in. So I guess that's why it doesn't work for your > > > VoIP case - the senders don't notice fast enough that they should > > > slow down. > > > > that's because you don't limit the bkbits.net to a fixed rate. If you > > want to give priorities, it won't work well because it takes time to be > > effective, but if you rate limit hard both ways it has to work, unless > > you're under syn-flood ;) The downside is that you will waste bandwith > > (i.e. you will hurt the bkbits.net service even when you don't use > > voip), but it will work. > > How about giving something to voip as a hard limit and then using some shaper > to give it more if needed. it may take a few secs to rate limit the rest, the old tcp connections will keep sending packets that will get rejected when voip kicks in, that's the problem described in the email I answered to with the solution (i.e. limit bkbits.net hard). I'm pretty sure that rate limting bkbits.net hard (like we do with my brother's computer when he generates heavy traffic) will let voip to work flawlessy, but it has the downside of limiting the bandwith available to bkbits users even when nobody place phone calls. Upgrading the current line and using the software rate limiting with the solution I proposed, or adding a new line dedicated to bkbits.net, wouldn't be much different in practice (if it was me I'd choose it only in function of what's more cost effective, after double checking that my algorithm infact works fine for higher bandwidth too, I only tested it on 640kbit). Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 1:39 ` Andrea Arcangeli 2003-08-31 2:18 ` Jeff Sipek @ 2003-08-31 2:56 ` Larry McVoy 2003-08-31 13:15 ` Alan Cox 2003-09-02 6:55 ` Henning P. Schmiedehausen 2003-08-31 13:47 ` Pascal Schmidt 2 siblings, 2 replies; 71+ messages in thread From: Larry McVoy @ 2003-08-31 2:56 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: Pascal Schmidt, Larry McVoy, linux-kernel On Sun, Aug 31, 2003 at 03:39:28AM +0200, Andrea Arcangeli wrote: > > Dropping the packets means that the sending side, at least if we're > > talking TCP, will throttle its sending rate. But, depending on the > > distance in hops to the sender, it may take up to a few seconds for > > this to kick in. So I guess that's why it doesn't work for your > > VoIP case - the senders don't notice fast enough that they should > > slow down. > > that's because you don't limit the bkbits.net to a fixed rate. We not only limited bkbits.net to a fixed rate, we limited all TCP based traffic to a fixed rate and it still didn't work. I'm pretty convinced we can't solve the problem at our end. Maybe we can but I'm voting for throwing another T1 at the problem, we'll try working with the ISP suggested solution of trunking them and rate limiting at their end and if that doesn't work then we'll split them and use one for bkbits.net and other bitmover related TCP traffic and use the other one for phones. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 2:56 ` Larry McVoy @ 2003-08-31 13:15 ` Alan Cox 2003-08-31 14:45 ` Andrea Arcangeli 2003-09-02 6:55 ` Henning P. Schmiedehausen 1 sibling, 1 reply; 71+ messages in thread From: Alan Cox @ 2003-08-31 13:15 UTC (permalink / raw) To: Larry McVoy; +Cc: Andrea Arcangeli, Pascal Schmidt, Linux Kernel Mailing List On Sul, 2003-08-31 at 03:56, Larry McVoy wrote: > I'm pretty convinced we can't solve the problem at our end. Maybe we can For bursts of traffic you can't. > but I'm voting for throwing another T1 at the problem, we'll try working or switch to ATM ;) > with the ISP suggested solution of trunking them and rate limiting at > their end and if that doesn't work then we'll split them and use one for > bkbits.net and other bitmover related TCP traffic and use the other one > for phones. Fractioning the line is also doable but less flexible with some kit. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 13:15 ` Alan Cox @ 2003-08-31 14:45 ` Andrea Arcangeli 2003-08-31 15:31 ` Alan Cox 2003-08-31 15:38 ` Jörn Engel 0 siblings, 2 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 14:45 UTC (permalink / raw) To: Alan Cox; +Cc: Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote: > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote: > > I'm pretty convinced we can't solve the problem at our end. Maybe we can > > For bursts of traffic you can't. what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower than what the link can provide (very common here during the day), the isp will have to do the software thing anyways to balance the bandwidth across the different ip. it works flawlessy for me and it's the same problem (I also use streaming services), and they're unusable until I turn the shaping on, I'm sure that if you use the script and you change 1kbyte/sec to everything but voip it'll work fine for you too, since basically everything else won't pass anymore, it will take ages to open an html page and all the bkbits.net users will hang, and the link will be idle 99% of the time, so voip will take it over as much as it can. I don't think it's a matter of "if it works or not", I think it's a matter of how much you're ok to lose in terms of global bandwith for all the other services but voip. the only annoying problem I run into, is that tc is missing a flush operation (like iptables -F) but I never need to tweak it anymore so I don't mind too much. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 14:45 ` Andrea Arcangeli @ 2003-08-31 15:31 ` Alan Cox 2003-08-31 15:43 ` Jörn Engel ` (2 more replies) 2003-08-31 15:38 ` Jörn Engel 1 sibling, 3 replies; 71+ messages in thread From: Alan Cox @ 2003-08-31 15:31 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote: > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote: > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote: > > > I'm pretty convinced we can't solve the problem at our end. Maybe we can > > > > For bursts of traffic you can't. > > what's the difference of rejecting packets in software, or because the > link can't handle them? Assume the guaranteed bandwidth is much lower It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 15:31 ` Alan Cox @ 2003-08-31 15:43 ` Jörn Engel 2003-08-31 15:50 ` Andrea Arcangeli 2003-08-31 15:44 ` Andrea Arcangeli 2003-08-31 16:23 ` Larry McVoy 2 siblings, 1 reply; 71+ messages in thread From: Jörn Engel @ 2003-08-31 15:43 UTC (permalink / raw) To: Alan Cox Cc: Andrea Arcangeli, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, 31 August 2003 16:31:32 +0100, Alan Cox wrote: > On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote: > > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote: > > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote: > > > > I'm pretty convinced we can't solve the problem at our end. Maybe we can > > > > > > For bursts of traffic you can't. > > > > what's the difference of rejecting packets in software, or because the > > link can't handle them? Assume the guaranteed bandwidth is much lower > > It doesn't work when you dont control incoming. As a simple extreme > example if I pingflood you from a fast site then no amount of shaping > your end of the link will help, it has to be shaped at the ISP end. If someone wants to DOS you, he can. Full stop. iirc, Larry was worried about well behaved traffic still doing bad things to his connection. Jörn -- Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. -- Rob Pike ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 15:43 ` Jörn Engel @ 2003-08-31 15:50 ` Andrea Arcangeli 2003-08-31 17:06 ` Florian Weimer 0 siblings, 1 reply; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 15:50 UTC (permalink / raw) To: Jörn Engel Cc: Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 05:43:01PM +0200, Jörn Engel wrote: > On Sun, 31 August 2003 16:31:32 +0100, Alan Cox wrote: > > On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote: > > > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote: > > > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote: > > > > > I'm pretty convinced we can't solve the problem at our end. Maybe we can > > > > > > > > For bursts of traffic you can't. > > > > > > what's the difference of rejecting packets in software, or because the > > > link can't handle them? Assume the guaranteed bandwidth is much lower > > > > It doesn't work when you dont control incoming. As a simple extreme > > example if I pingflood you from a fast site then no amount of shaping > > your end of the link will help, it has to be shaped at the ISP end. > > If someone wants to DOS you, he can. Full stop. yes, that's unfixable at Larry's end, and normaly it's unfixable for the ISP too. > iirc, Larry was worried about well behaved traffic still doing bad > things to his connection. I also understood the problem were the legimate clones/checkouts happening in established state with a single tcp connection generating a burst of traffic. If this is not the case and Larry is under attack when the voip doesn't work, of course nothing can help him, most probably not even his ISP, but I assumed this wasn't the case. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 15:50 ` Andrea Arcangeli @ 2003-08-31 17:06 ` Florian Weimer 2003-08-31 21:21 ` Larry McVoy ` (2 more replies) 0 siblings, 3 replies; 71+ messages in thread From: Florian Weimer @ 2003-08-31 17:06 UTC (permalink / raw) To: Andrea Arcangeli Cc: Jörn Engel, Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List Andrea Arcangeli <andrea@suse.de> writes: > yes, that's unfixable at Larry's end, and normaly it's unfixable for the > ISP too. The ISP can do several things to prioritize production traffic or drop malicious traffic. However, this isn't trivial and requires careful planning, and it's unlikely that anyone who is able to would want to do this for a T1 customer (typically, it requires "unusual" configuration of vital production routers with the fat pipes). >> iirc, Larry was worried about well behaved traffic still doing bad >> things to his connection. In this case, it's a bit easier for the ISP because the tweaking can be done on edge routers (which typically die during DoS attacks as well, so countermeasures cannot be applied there). Especially with low-bandwidth links, it shouldn't be too hard to find a smallish, geeky ISP who is willing to try some fiddling on the edge router to improve performance (at least that's true in Germany, don't know about the U.S.). However, why can't bkbits.net distributed around the world, at least for read access? This would eliminate the choking point once and for all. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 17:06 ` Florian Weimer @ 2003-08-31 21:21 ` Larry McVoy 2003-09-02 7:06 ` Henning P. Schmiedehausen 2003-09-02 21:52 ` Ricky Beam 2 siblings, 0 replies; 71+ messages in thread From: Larry McVoy @ 2003-08-31 21:21 UTC (permalink / raw) To: Andrea Arcangeli, J?rn Engel, Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 07:06:55PM +0200, Florian Weimer wrote: > Andrea Arcangeli <andrea@suse.de> writes: > However, why can't bkbits.net distributed around the world, at least > for read access? It's not just the read access that is the problem but there is absolutely nothing stopping you or anyone else from setting up bkbackup.net as a cache. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 17:06 ` Florian Weimer 2003-08-31 21:21 ` Larry McVoy @ 2003-09-02 7:06 ` Henning P. Schmiedehausen 2003-09-05 8:10 ` Florian Weimer 2003-09-02 21:52 ` Ricky Beam 2 siblings, 1 reply; 71+ messages in thread From: Henning P. Schmiedehausen @ 2003-09-02 7:06 UTC (permalink / raw) To: linux-kernel Florian Weimer <fw@deneb.enyo.de> writes: >do this for a T1 customer (typically, it requires "unusual" >configuration of vital production routers with the fat pipes). You need a shaper connected to the ISP backbone which shapes the outgoing traffic for you and a border router which talks to the T1 (C17xx or C26xx). Normally, if your ISP has some sort of clue, you will also need a bastion router which can handle backbone <-> 100 MBit traffic and does dynamic routing updates (EGP or OSPF) to the ISP backbone (A C26xx or C37xx). This isn't your run-of-the-mill setup but I know of plenty ISPs here that will happily sell you this and also the consulting needed to plan, setup and operate the link. However, you won't get it for $199/month. :-) Regards Henning -- Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/ Java, perl, Solaris, Linux, xSP Consulting, Web Services freelance consultant -- Jakarta Turbine Development -- hero for hire "Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" -- AOL CD when played backwards (User Friendly - 200-10-15) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-02 7:06 ` Henning P. Schmiedehausen @ 2003-09-05 8:10 ` Florian Weimer 2003-09-05 15:35 ` Henning Schmiedehausen 0 siblings, 1 reply; 71+ messages in thread From: Florian Weimer @ 2003-09-05 8:10 UTC (permalink / raw) To: hps; +Cc: linux-kernel "Henning P. Schmiedehausen" <hps@intermeta.de> writes: > Florian Weimer <fw@deneb.enyo.de> writes: > >>do this for a T1 customer (typically, it requires "unusual" >>configuration of vital production routers with the fat pipes). > > You need a shaper connected to the ISP backbone which shapes the > outgoing traffic for you and a border router which talks to the T1 > (C17xx or C26xx). Normally, if your ISP has some sort of clue, you > will also need a bastion router which can handle backbone <-> 100 MBit > traffic and does dynamic routing updates (EGP or OSPF) to the ISP > backbone (A C26xx or C37xx). C37xx can handle a maximum load of 225 kpps (data sheet number, i.e. this value cannot be exceeded even under most favorable conditions), the others handle even less. Such routers are of no help during a DoS attack. Yes, I snipped the DoS context, and your approach would work in a benign environment. 8-) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-05 8:10 ` Florian Weimer @ 2003-09-05 15:35 ` Henning Schmiedehausen 2003-09-05 15:50 ` Florian Weimer ` (2 more replies) 0 siblings, 3 replies; 71+ messages in thread From: Henning Schmiedehausen @ 2003-09-05 15:35 UTC (permalink / raw) To: Florian Weimer; +Cc: Linux Kernel Mailing List On Fri, 2003-09-05 at 10:10, Florian Weimer wrote: > > You need a shaper connected to the ISP backbone which shapes the > > outgoing traffic for you and a border router which talks to the T1 > > (C17xx or C26xx). Normally, if your ISP has some sort of clue, you > > will also need a bastion router which can handle backbone <-> 100 MBit > > traffic and does dynamic routing updates (EGP or OSPF) to the ISP > > backbone (A C26xx or C37xx). > > C37xx can handle a maximum load of 225 kpps (data sheet number, > i.e. this value cannot be exceeded even under most favorable > conditions), the others handle even less. Such routers are of no help > during a DoS attack. > > Yes, I snipped the DoS context, and your approach would work in a > benign environment. 8-) 225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec 100 MBit / 8 bit = 12,5 MBytes / sec So, IMHO even with a small packet saturated 100 MBit link you won't reach 225kpps. AFAIK this was Ciscos intention to publish this number. It basically says "you will have filled your link before you fill our router". I'm pretty sure that your 37xx won't do any routing updates anymore at this point. And if you do _anything_ that forces the packets down the slow path from the routing engine, you're toast anyway. But I'm pretty sure that a C37xx would handle full 100 MBit traffic to a busy website without any problems. In fact, I know that it does. ;-) (We did switch to a C12000 shortly after, mainly because we went Gigabit). Regards Henning -- Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/ Java, perl, Solaris, Linux, xSP Consulting, Web Services freelance consultant -- Jakarta Turbine Development -- hero for hire "Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" -- AOL CD when played backwards (User Friendly - 200-10-15) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-05 15:35 ` Henning Schmiedehausen @ 2003-09-05 15:50 ` Florian Weimer 2003-09-05 16:10 ` P 2003-09-05 16:43 ` Ricky Beam 2 siblings, 0 replies; 71+ messages in thread From: Florian Weimer @ 2003-09-05 15:50 UTC (permalink / raw) To: Henning Schmiedehausen; +Cc: Linux Kernel Mailing List Henning Schmiedehausen <hps@intermeta.de> writes: >> Yes, I snipped the DoS context, and your approach would work in a >> benign environment. 8-) > > 225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec > > 100 MBit / 8 bit = 12,5 MBytes / sec > > So, IMHO even with a small packet saturated 100 MBit link you won't > reach 225kpps. AFAIK this was Ciscos intention to publish this number. > It basically says "you will have filled your link before you fill our > router". Cisco sells Gigabit Ethernet linecards for this router, and the situation is quite different in this case. And with most attacks, it's a futile exercise to try to combat them on a Fast Ethernet link, anyway. 8-( > But I'm pretty sure that a C37xx would handle full 100 MBit traffic to a > busy website without any problems. In fact, I know that it does. ;-) (We > did switch to a C12000 shortly after, mainly because we went Gigabit). Of course it does. After all, web traffic is somewhat different from pure DoS traffic. 8-) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-05 15:35 ` Henning Schmiedehausen 2003-09-05 15:50 ` Florian Weimer @ 2003-09-05 16:10 ` P 2003-09-05 16:43 ` Ricky Beam 2 siblings, 0 replies; 71+ messages in thread From: P @ 2003-09-05 16:10 UTC (permalink / raw) To: Henning Schmiedehausen; +Cc: Florian Weimer, Linux Kernel Mailing List Henning Schmiedehausen wrote: > On Fri, 2003-09-05 at 10:10, Florian Weimer wrote: > >>>You need a shaper connected to the ISP backbone which shapes the >>>outgoing traffic for you and a border router which talks to the T1 >>>(C17xx or C26xx). Normally, if your ISP has some sort of clue, you >>>will also need a bastion router which can handle backbone <-> 100 MBit >>>traffic and does dynamic routing updates (EGP or OSPF) to the ISP >>>backbone (A C26xx or C37xx). >> >>C37xx can handle a maximum load of 225 kpps (data sheet number, >>i.e. this value cannot be exceeded even under most favorable >>conditions), the others handle even less. Such routers are of no help >>during a DoS attack. >> >>Yes, I snipped the DoS context, and your approach would work in a >>benign environment. 8-) > > 225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec > > 100 MBit / 8 bit = 12,5 MBytes / sec > > So, IMHO even with a small packet saturated 100 MBit link you won't > reach 225kpps. AFAIK this was Ciscos intention to publish this number. > It basically says "you will have filled your link before you fill our > router". 100Mb/s LAN @ 64 byte packets is 148500 pps half duplex from my testing => 257Kpps full duplex. Here's some interesting results I've got from the latest intel e100 3.0.0_dev13 driver, showing how the receive rate degrades at higher packet rates. ------------------------ send Kpps recv Kpps ------------------------ 126 126 126.5 118.4 128 115.7 130 102.7 135 99.1 140 90.6 148 88.2 ------------------------ It's not a CPU issue as it can receive at the same rate on another interface. renicing ksoftirqd has no effect. system is PIII 1.2GHz, i815, 2.4.20 NAPI is turned on in the driver. Pádraig. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-05 15:35 ` Henning Schmiedehausen 2003-09-05 15:50 ` Florian Weimer 2003-09-05 16:10 ` P @ 2003-09-05 16:43 ` Ricky Beam 2003-09-07 11:55 ` Henning Schmiedehausen 2 siblings, 1 reply; 71+ messages in thread From: Ricky Beam @ 2003-09-05 16:43 UTC (permalink / raw) To: Henning Schmiedehausen; +Cc: Linux Kernel Mailing List On 5 Sep 2003, Henning Schmiedehausen wrote: >225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec > >100 MBit / 8 bit = 12,5 MBytes / sec > >So, IMHO even with a small packet saturated 100 MBit link you won't >reach 225kpps. AFAIK this was Ciscos intention to publish this number. >It basically says "you will have filled your link before you fill our >router". 64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, Frame relay, ATM, etc. >I'm pretty sure that your 37xx won't do any routing updates anymore at >this point. And if you do _anything_ that forces the packets down the >slow path from the routing engine, you're toast anyway. Sure it can. Yeah, it'll be slow_er_, but not stopped. CEF/PXF doesn't require much CPU to switch a packet. At process switching, the router is rated to a few K pps. Process switching SUCKS. I've seen a 7206/200 _attempt_ to move 150kpps @ 64B each. (misbehaving software and a misconfigured firewall...) The router was chuggin' right along -- discarding UDP traffic like a mad man. From the console, it was working fine. *grin* It's rather hard for telnet to compete (and even harder for ssh.) All those machines are behind a 7401 now. And it doesn't even blink at such things. (That thing's worth every penny we paid for it.) --Ricky ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-05 16:43 ` Ricky Beam @ 2003-09-07 11:55 ` Henning Schmiedehausen 2003-09-07 18:41 ` Ricky Beam 0 siblings, 1 reply; 71+ messages in thread From: Henning Schmiedehausen @ 2003-09-07 11:55 UTC (permalink / raw) To: Ricky Beam; +Cc: Linux Kernel Mailing List On Fri, 2003-09-05 at 18:43, Ricky Beam wrote: On 5 Sep 2003, Henning Schmiedehausen wrote: > >225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec > > > >100 MBit / 8 bit = 12,5 MBytes / sec > > > >So, IMHO even with a small packet saturated 100 MBit link you won't > >reach 225kpps. AFAIK this was Ciscos intention to publish this number. > >It basically says "you will have filled your link before you fill our > >router". > > 64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, > Frame relay, ATM, etc. We were talking 100 MBit Ethernet, weren't we? ;-) Regards Henning ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-07 11:55 ` Henning Schmiedehausen @ 2003-09-07 18:41 ` Ricky Beam 2003-09-08 8:57 ` [COMPLETELY OFF-TOPIC] " Henning Schmiedehausen 0 siblings, 1 reply; 71+ messages in thread From: Ricky Beam @ 2003-09-07 18:41 UTC (permalink / raw) To: Henning Schmiedehausen; +Cc: Linux Kernel Mailing List On 7 Sep 2003, Henning Schmiedehausen wrote: >> 64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, >> Frame relay, ATM, etc. > >We were talking 100 MBit Ethernet, weren't we? ;-) Actually, we're talking about VoIP between remote offices. We already know this is not accomplished via fast ethernet WAN links. The WAN link is, in fact, a T1. T1's do not have minimum frame sizes like ethernet. --Ricky ^ permalink raw reply [flat|nested] 71+ messages in thread
* [COMPLETELY OFF-TOPIC] Re: bandwidth for bkbits.net (good news) 2003-09-07 18:41 ` Ricky Beam @ 2003-09-08 8:57 ` Henning Schmiedehausen 0 siblings, 0 replies; 71+ messages in thread From: Henning Schmiedehausen @ 2003-09-08 8:57 UTC (permalink / raw) To: Ricky Beam; +Cc: Linux Kernel Mailing List That is where the thread started. ;-) As we reached the max kpps on a C37xx point of discussion, we were already on 100 MBit Ethernet. ;-) Let's end this thread. Regards Henning On Sun, 2003-09-07 at 20:41, Ricky Beam wrote: > On 7 Sep 2003, Henning Schmiedehausen wrote: > >> 64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, > >> Frame relay, ATM, etc. > > > >We were talking 100 MBit Ethernet, weren't we? ;-) > > Actually, we're talking about VoIP between remote offices. We already > know this is not accomplished via fast ethernet WAN links. The WAN > link is, in fact, a T1. T1's do not have minimum frame sizes like > ethernet. > > --Ricky -- Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/ Java, perl, Solaris, Linux, xSP Consulting, Web Services freelance consultant -- Jakarta Turbine Development -- hero for hire "Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" -- AOL CD when played backwards (User Friendly - 200-10-15) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 17:06 ` Florian Weimer 2003-08-31 21:21 ` Larry McVoy 2003-09-02 7:06 ` Henning P. Schmiedehausen @ 2003-09-02 21:52 ` Ricky Beam 2003-09-05 8:16 ` Florian Weimer 2 siblings, 1 reply; 71+ messages in thread From: Ricky Beam @ 2003-09-02 21:52 UTC (permalink / raw) To: Florian Weimer; +Cc: Larry McVoy, Linux Kernel Mailing List On Sun, 31 Aug 2003, Florian Weimer wrote: >The ISP can do several things to prioritize production traffic or drop >malicious traffic. However, this isn't trivial and requires careful >planning, and it's unlikely that anyone who is able to would want to >do this for a T1 customer (typically, it requires "unusual" >configuration of vital production routers with the fat pipes). In the cisco world, all it takes is: interface <foo> fair-queue That's it. That's all it takes on the ISPs part to provide a minimal amount of QoS. Cisco's documentation clearly lists how IOS handles the queuing. It's not full policy based traffic shaping with RSVP and all that crap, but it's enough for a many situations. Larry's situation is highly complicated by "the internet". I've never known anyone to be happy with VoIP across the internet. It's hard enough to make it work properly within a single ISP's network _with_ their assistance. Across the wastelands, forget about it. (Why do you think GRIC built their own global network for moving VoIP traffic?) --Ricky ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-02 21:52 ` Ricky Beam @ 2003-09-05 8:16 ` Florian Weimer 0 siblings, 0 replies; 71+ messages in thread From: Florian Weimer @ 2003-09-05 8:16 UTC (permalink / raw) To: Ricky Beam; +Cc: Linux Kernel Mailing List Ricky Beam <jfbeam@bluetronic.net> writes: > On Sun, 31 Aug 2003, Florian Weimer wrote: >>The ISP can do several things to prioritize production traffic or drop >>malicious traffic. However, this isn't trivial and requires careful >>planning, and it's unlikely that anyone who is able to would want to >>do this for a T1 customer (typically, it requires "unusual" >>configuration of vital production routers with the fat pipes). > > In the cisco world, all it takes is: > interface <foo> > fair-queue WFQ is already the default for low-bandwidth links (and it obviously can't work on high-end routers). 8-/ ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 15:31 ` Alan Cox 2003-08-31 15:43 ` Jörn Engel @ 2003-08-31 15:44 ` Andrea Arcangeli 2003-08-31 16:22 ` Larry McVoy 2003-09-02 7:01 ` Henning P. Schmiedehausen 2003-08-31 16:23 ` Larry McVoy 2 siblings, 2 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 15:44 UTC (permalink / raw) To: Alan Cox; +Cc: Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 04:31:32PM +0100, Alan Cox wrote: > On Sul, 2003-08-31 at 15:45, Andrea Arcangeli wrote: > > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote: > > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote: > > > > I'm pretty convinced we can't solve the problem at our end. Maybe we can > > > > > > For bursts of traffic you can't. > > > > what's the difference of rejecting packets in software, or because the > > link can't handle them? Assume the guaranteed bandwidth is much lower > > > It doesn't work when you dont control incoming. As a simple extreme > example if I pingflood you from a fast site then no amount of shaping > your end of the link will help, it has to be shaped at the ISP end. sure, that's why I said it won't work with synflood. I just doubt the ping/syn floods distributed denial of services are an high percentage of the traffic passing through on bkbits.net. I though it was legitimate traffic, and I assume bitkeeer is somehow efficient in handling the transfer, by using a single tcp connection for the whole transfer of the data, just like pserver/cvs-ssh do. For example if bitkeeper would open a new tcp connection for each file (similar to cvsps -p -g w/o the --cvs-direct option that I asked for), it would be much harder to shape that traffic. But I understood the traffic that hurts is all in established state for several seconds, so it should be technically possible to stop it to around 1kbyte/sec globally to give an huge margin to voip. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 15:44 ` Andrea Arcangeli @ 2003-08-31 16:22 ` Larry McVoy 2003-08-31 16:33 ` Andrea Arcangeli 2003-09-02 7:01 ` Henning P. Schmiedehausen 1 sibling, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-31 16:22 UTC (permalink / raw) To: Andrea Arcangeli Cc: Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 05:44:50PM +0200, Andrea Arcangeli wrote: > > It doesn't work when you dont control incoming. As a simple extreme > > example if I pingflood you from a fast site then no amount of shaping > > your end of the link will help, it has to be shaped at the ISP end. > > sure, that's why I said it won't work with synflood. Someone syncs w/ bkbits every 19 seconds 24x7. We also run our web server here. All traffic to from bitmover.com/bitkeeper.com/bkbits.net goes through that T1 line. You guys who are saying it can work are thinking (a) one connection of long duration (think about all the web hits on bkbits.net, those are all short and new TCP connections) and (b) that a little settling time is OK. There is a reason that the phone networks don't work like IP networks. The bandwidth is allocated whether you use it or not and your phone works. Doing optimistic allocation and then backing off means that the phones don't work. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 16:22 ` Larry McVoy @ 2003-08-31 16:33 ` Andrea Arcangeli 2003-08-31 16:48 ` Larry McVoy 0 siblings, 1 reply; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 16:33 UTC (permalink / raw) To: Larry McVoy, Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 09:22:43AM -0700, Larry McVoy wrote: > On Sun, Aug 31, 2003 at 05:44:50PM +0200, Andrea Arcangeli wrote: > > > It doesn't work when you dont control incoming. As a simple extreme > > > example if I pingflood you from a fast site then no amount of shaping > > > your end of the link will help, it has to be shaped at the ISP end. > > > > sure, that's why I said it won't work with synflood. > > Someone syncs w/ bkbits every 19 seconds 24x7. We also run our web server 1 syn every 19 seconds is nothing. > You guys who are saying it can work are thinking (a) one connection of > long duration (think about all the web hits on bkbits.net, those are all it doesn't need to be long duration, just longer than a syn or a ping. If it goes in established for a few packets is should be enough to throttle it just fine. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 16:33 ` Andrea Arcangeli @ 2003-08-31 16:48 ` Larry McVoy 2003-08-31 17:06 ` Andrea Arcangeli 0 siblings, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-31 16:48 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 06:33:50PM +0200, Andrea Arcangeli wrote: > On Sun, Aug 31, 2003 at 09:22:43AM -0700, Larry McVoy wrote: > > On Sun, Aug 31, 2003 at 05:44:50PM +0200, Andrea Arcangeli wrote: > > > > It doesn't work when you dont control incoming. As a simple extreme > > > > example if I pingflood you from a fast site then no amount of shaping > > > > your end of the link will help, it has to be shaped at the ISP end. > > > > > > sure, that's why I said it won't work with synflood. > > > > Someone syncs w/ bkbits every 19 seconds 24x7. We also run our web server > > 1 syn every 19 seconds is nothing. A sync != one connection. And that doesn't include the auto backup that we do to another server, doesn't include all the HTTP traffic, doesn't include our web site traffic. > > You guys who are saying it can work are thinking (a) one connection of > > long duration (think about all the web hits on bkbits.net, those are all > > it doesn't need to be long duration, just longer than a syn or a ping. > If it goes in established for a few packets is should be enough to > throttle it just fine. You are welcome to *demonstrate* something that works but telling me that it works when we've tried what you said to try isn't very compelling. I know this doesn't work from both theory and practice. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 16:48 ` Larry McVoy @ 2003-08-31 17:06 ` Andrea Arcangeli 2003-08-31 21:18 ` Larry McVoy 0 siblings, 1 reply; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 17:06 UTC (permalink / raw) To: Larry McVoy, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 09:48:02AM -0700, Larry McVoy wrote: > works when we've tried what you said to try isn't very compelling. I know > this doesn't work from both theory and practice. Post your configure scripts so we can point you what you did wrong. I already posted a perfectly working configuration that I depend totally on, and it never failed once to throttle the traffic. If you don't care to try it, at least post whatever you tried so far. Any other claim or email that doesn't include scripts will go to /dev/null. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 17:06 ` Andrea Arcangeli @ 2003-08-31 21:18 ` Larry McVoy 2003-08-31 22:49 ` Andrea Arcangeli 0 siblings, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-31 21:18 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 07:06:33PM +0200, Andrea Arcangeli wrote: > On Sun, Aug 31, 2003 at 09:48:02AM -0700, Larry McVoy wrote: > > works when we've tried what you said to try isn't very compelling. I know > > this doesn't work from both theory and practice. > > Post your configure scripts so we can point you what you did wrong. They are Cisco configuration, it won't do you much good. All the traffic goes in/out through a Cisco 2610, we have a full T1 and we clamped all TCP traffic at .75Mbit. Still didn't help even though we verified that it was indeed clamping down on the traffic. I'm not sure why you are arguing this, if you have a fat pipe feeding into a small pipe and you are trying to throttle at far end of the small pipe isn't it obvious that you can't make that work? It's not the packets we send, it's the packets you send. And all the flow control stuff is per connection, not per pipe. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 21:18 ` Larry McVoy @ 2003-08-31 22:49 ` Andrea Arcangeli 2003-08-31 22:52 ` Alan Cox ` (2 more replies) 0 siblings, 3 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 22:49 UTC (permalink / raw) To: Larry McVoy, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 02:18:55PM -0700, Larry McVoy wrote: > On Sun, Aug 31, 2003 at 07:06:33PM +0200, Andrea Arcangeli wrote: > > On Sun, Aug 31, 2003 at 09:48:02AM -0700, Larry McVoy wrote: > > > works when we've tried what you said to try isn't very compelling. I know > > > this doesn't work from both theory and practice. > > > > Post your configure scripts so we can point you what you did wrong. > > They are Cisco configuration, it won't do you much good. All the traffic I don't trust anything but the sourcecode I can read, so please try again with linux. > I'm not sure why you are arguing this, if you have a fat pipe feeding into you never tried with linux, how can you claim you know it doesn't work in practice? The fact is that you never tried it, while we used it all the time. > a small pipe and you are trying to throttle at far end of the small pipe > isn't it obvious that you can't make that work? It's not the packets we > send, it's the packets you send. And all the flow control stuff is per it's tcp, it's trivial to rate limit the packets we send, as far as you go into established somehow. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 22:49 ` Andrea Arcangeli @ 2003-08-31 22:52 ` Alan Cox 2003-08-31 22:58 ` Larry McVoy ` (2 more replies) 2003-08-31 22:56 ` Larry McVoy 2003-09-02 7:11 ` Henning P. Schmiedehausen 2 siblings, 3 replies; 71+ messages in thread From: Alan Cox @ 2003-08-31 22:52 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sul, 2003-08-31 at 23:49, Andrea Arcangeli wrote: > > I'm not sure why you are arguing this, if you have a fat pipe feeding into > > you never tried with linux, how can you claim you know it doesn't work > in practice? The fact is that you never tried it, while we used it all > the time. How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic. Since he doesn't appear to control the ISP end (either directly or via RSVP) he's lost. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 22:52 ` Alan Cox @ 2003-08-31 22:58 ` Larry McVoy 2003-08-31 23:02 ` Andrea Arcangeli 2003-08-31 23:39 ` Roman Zippel 2 siblings, 0 replies; 71+ messages in thread From: Larry McVoy @ 2003-08-31 22:58 UTC (permalink / raw) To: Alan Cox Cc: Andrea Arcangeli, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote: > On Sul, 2003-08-31 at 23:49, Andrea Arcangeli wrote: > > > I'm not sure why you are arguing this, if you have a fat pipe feeding into > > > > you never tried with linux, how can you claim you know it doesn't work > > in practice? The fact is that you never tried it, while we used it all > > the time. > > How about because any undergraduate can do the mathematical proof its > not possible. Unless he controls the ISP end of the link random bursts > of traffic, pingfloods, anything not respecting requests to slow down > will lose voice traffic. Whoops, if I had read this I wouldn't have sent the other reply. What Alan said is what I've been trying to say. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 22:52 ` Alan Cox 2003-08-31 22:58 ` Larry McVoy @ 2003-08-31 23:02 ` Andrea Arcangeli 2003-08-31 23:07 ` Larry McVoy 2003-08-31 23:39 ` Roman Zippel 2 siblings, 1 reply; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 23:02 UTC (permalink / raw) To: Alan Cox Cc: Larry McVoy, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote: > On Sul, 2003-08-31 at 23:49, Andrea Arcangeli wrote: > > > I'm not sure why you are arguing this, if you have a fat pipe feeding into > > > > you never tried with linux, how can you claim you know it doesn't work > > in practice? The fact is that you never tried it, while we used it all > > the time. > > How about because any undergraduate can do the mathematical proof its > not possible. Unless he controls the ISP end of the link random bursts > of traffic, pingfloods, anything not respecting requests to slow down > will lose voice traffic. they are legitimate tcp connections, not udp or icmp. I'm not saying you can control pingfloods or udp floods or syn floods. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 23:02 ` Andrea Arcangeli @ 2003-08-31 23:07 ` Larry McVoy 2003-08-31 23:22 ` Andrea Arcangeli 0 siblings, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-31 23:07 UTC (permalink / raw) To: Andrea Arcangeli Cc: Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 01:02:19AM +0200, Andrea Arcangeli wrote: > On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote: > > How about because any undergraduate can do the mathematical proof its > > not possible. Unless he controls the ISP end of the link random bursts > > of traffic, pingfloods, anything not respecting requests to slow down > > will lose voice traffic. > > they are legitimate tcp connections, not udp or icmp. I'm not saying you > can control pingfloods or udp floods or syn floods. I'll try this one more time and then I'm giving up because as far as I can tell you haven't tried this with a busy server, I think you said you did shaping on your home machine or something. Hardly the same thing. How about a series of tiny HTTP requests? All 1.0, no connection reuse. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 23:07 ` Larry McVoy @ 2003-08-31 23:22 ` Andrea Arcangeli 2003-09-01 0:12 ` Larry McVoy 2003-09-01 11:43 ` Alan Cox 0 siblings, 2 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 23:22 UTC (permalink / raw) To: Larry McVoy, Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 04:07:28PM -0700, Larry McVoy wrote: > On Mon, Sep 01, 2003 at 01:02:19AM +0200, Andrea Arcangeli wrote: > > On Sun, Aug 31, 2003 at 11:52:39PM +0100, Alan Cox wrote: > > > How about because any undergraduate can do the mathematical proof its > > > not possible. Unless he controls the ISP end of the link random bursts > > > of traffic, pingfloods, anything not respecting requests to slow down > > > will lose voice traffic. > > > > they are legitimate tcp connections, not udp or icmp. I'm not saying you > > can control pingfloods or udp floods or syn floods. > > I'll try this one more time and then I'm giving up because as far as I > can tell you haven't tried this with a busy server, I think you said you > did shaping on your home machine or something. Hardly the same thing. the only difference I believe is that the connections are originated by my end, but that only changes the syn packet that normally accounts for a non significant amount of the bandwidth. And while this is located in a house, this isn't what I would call an home user usage. > How about a series of tiny HTTP requests? All 1.0, no connection reuse. on the long run, the way I recommend you to act, is to have a script that turns the traffic shaping on on demand before/after using voip, so you won't hurt the bkbits.net userbase when you're not at the phone. So you will fix the unprofessional thing during conf calls completely and the community should be ok with reasonably short slowdowns while you're at the phone (if only voip would run under linux too you could automate it). Just try for a test with my script changing it to give 1kbyte/sec both ways (not 64k/16k) to everything but voip. I think I already posted all the needed information you need to make it work. You will need to rework the iptables rules carefully though, to include all the traffic but voip. Hope this helps you too (I couldn't live without it), Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 23:22 ` Andrea Arcangeli @ 2003-09-01 0:12 ` Larry McVoy 2003-09-01 0:19 ` Jamie Lokier 2003-09-01 11:43 ` Alan Cox 1 sibling, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-09-01 0:12 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 01:22:24AM +0200, Andrea Arcangeli wrote: > on the long run, the way I recommend you to act, is to have a script > that turns the traffic shaping on on demand before/after using voip That's a cool idea but unfortunately we are using the 3com NBX stuff which is based on Wind River and we don't have any way to know that a voice conversation is going on. I suppose we could build a way based on snooping the network though. Have to think about that. By the way, I'd love to replace the 3com stuff with something based on Linux. I strongly suspect that the phones could do better than they are doing and we're getting the short end of the stick by going with some closed answer. But all the open answers don't come close to offering what 3com does. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 0:12 ` Larry McVoy @ 2003-09-01 0:19 ` Jamie Lokier 2003-09-01 0:33 ` Larry McVoy 0 siblings, 1 reply; 71+ messages in thread From: Jamie Lokier @ 2003-09-01 0:19 UTC (permalink / raw) To: Larry McVoy, Andrea Arcangeli, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List Larry McVoy wrote: > But all the open answers don't come close to offering what 3com does. Have you looked at what Asterisk and Bayonne do? If so, is an important feature missing which you use? Cheers, -- Jamie ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 0:19 ` Jamie Lokier @ 2003-09-01 0:33 ` Larry McVoy 0 siblings, 0 replies; 71+ messages in thread From: Larry McVoy @ 2003-09-01 0:33 UTC (permalink / raw) To: Jamie Lokier Cc: Larry McVoy, Andrea Arcangeli, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 01:19:40AM +0100, Jamie Lokier wrote: > Larry McVoy wrote: > > But all the open answers don't come close to offering what 3com does. > > Have you looked at what Asterisk and Bayonne do? > If so, is an important feature missing which you use? Hunt groups. I want incoming sales and support calls to go to group of people. All the "virtual PBX" solutions do a lame hunt where they ring the first guy 3 times, then the next guy 3 times, etc. The 3com rings everyone at the same time. Believe it or not, if you guys call for support I want someone to pick up the phone *right now*. If you or our commercial customers are having nasty BK problems I don't want you to wait while the stupid phone system goes hunting around looking for someone who isn't out to lunch or whatever. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 23:22 ` Andrea Arcangeli 2003-09-01 0:12 ` Larry McVoy @ 2003-09-01 11:43 ` Alan Cox 2003-09-01 16:13 ` Andrea Arcangeli 1 sibling, 1 reply; 71+ messages in thread From: Alan Cox @ 2003-09-01 11:43 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Llu, 2003-09-01 at 00:22, Andrea Arcangeli wrote: > the only difference I believe is that the connections are originated by > my end, but that only changes the syn packet that normally accounts for > a non significant amount of the bandwidth. And while this is located in > a house, this isn't what I would call an home user usage. Each ACK that has caused previous delays generally opens up a 64K window so you get bursts of data incoming. A sequence of acks can cause the incoming pipe to fill enough to screw up voip very easily. Window bending stuff does help and certainly the packeteer boxes use it to good effect. I've never tried that on Linux but even then Im dubious that just cutting the routes down to an 8K window would help ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 11:43 ` Alan Cox @ 2003-09-01 16:13 ` Andrea Arcangeli 2003-09-01 16:24 ` Larry McVoy 2003-09-01 16:28 ` Alan Cox 0 siblings, 2 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-09-01 16:13 UTC (permalink / raw) To: Alan Cox Cc: Larry McVoy, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 12:43:56PM +0100, Alan Cox wrote: > On Llu, 2003-09-01 at 00:22, Andrea Arcangeli wrote: > > the only difference I believe is that the connections are originated by > > my end, but that only changes the syn packet that normally accounts for > > a non significant amount of the bandwidth. And while this is located in > > a house, this isn't what I would call an home user usage. > > Each ACK that has caused previous delays generally opens up a 64K window > so you get bursts of data incoming. A sequence of acks can cause the the congestion avoidance shouldn't allow what you say. It sends a few packets immediatly (cwnd starts > 1 recently), and that's why non-keepalive connections are bad, but after that the congestion window will remain low if we drop the packets. > incoming pipe to fill enough to screw up voip very easily. > > Window bending stuff does help and certainly the packeteer boxes use it > to good effect. I've never tried that on Linux but even then Im dubious > that just cutting the routes down to an 8K window would help The only object of my posts was to try to correct the misinformation that was spreading from Larry's posts that implied you had to buy a speparate connection to shape the bitkeeper connections doing clones of the tree. That's what I and everybody else understood. Now apparently he was wrong and the hurting factor aren't the bitkeeper clients cloning the tree but the webserver. So I'm glad to have reached my object. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 16:13 ` Andrea Arcangeli @ 2003-09-01 16:24 ` Larry McVoy 2003-09-01 16:28 ` Alan Cox 1 sibling, 0 replies; 71+ messages in thread From: Larry McVoy @ 2003-09-01 16:24 UTC (permalink / raw) To: Andrea Arcangeli Cc: Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 06:13:16PM +0200, Andrea Arcangeli wrote: > The only object of my posts was to try to correct the misinformation > that was spreading from Larry's posts that implied you had to buy a > speparate connection to shape the bitkeeper connections doing clones of > the tree. That's what I and everybody else understood. Now apparently he > was wrong and the hurting factor aren't the bitkeeper clients cloning > the tree but the webserver. So I'm glad to have reached my object. I don't know what axe you are trying to grind but that's nonsense. Yes, I said when you start doing a clone the phones don't work but I never said "the only problem we have is that an out going clone causes problems". That's clearly not true, there are incoming clones, pushes, outgoing pulls, all sorts of HTTP traffic, etc. The problem is that you heard clone and you assumed that clone was the ONLY source of traffic. That's your problem, not mine, so don't go accusing me of spreading misinformation because you weren't thinking or listening, please. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 16:13 ` Andrea Arcangeli 2003-09-01 16:24 ` Larry McVoy @ 2003-09-01 16:28 ` Alan Cox 2003-09-01 16:38 ` Andrea Arcangeli 1 sibling, 1 reply; 71+ messages in thread From: Alan Cox @ 2003-09-01 16:28 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Llu, 2003-09-01 at 17:13, Andrea Arcangeli wrote: > > Each ACK that has caused previous delays generally opens up a 64K window > > so you get bursts of data incoming. A sequence of acks can cause the > > the congestion avoidance shouldn't allow what you say. It sends a few > packets immediatly (cwnd starts > 1 recently), and that's why > non-keepalive connections are bad, but after that the congestion window > will remain low if we drop the packets. You may trigger fast retransmit patterns. Thats why you have to bend the window. ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 16:28 ` Alan Cox @ 2003-09-01 16:38 ` Andrea Arcangeli 0 siblings, 0 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-09-01 16:38 UTC (permalink / raw) To: Alan Cox Cc: Larry McVoy, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 05:28:31PM +0100, Alan Cox wrote: > On Llu, 2003-09-01 at 17:13, Andrea Arcangeli wrote: > > > Each ACK that has caused previous delays generally opens up a 64K window > > > so you get bursts of data incoming. A sequence of acks can cause the > > > > the congestion avoidance shouldn't allow what you say. It sends a few > > packets immediatly (cwnd starts > 1 recently), and that's why > > non-keepalive connections are bad, but after that the congestion window > > will remain low if we drop the packets. > > You may trigger fast retransmit patterns. Thats why you have to bend the > window. fast restransmit are good to trigger, they don't arrive in a flood like if we had a huge cwnd, if we can send packets out of the network, they're right to be sent. During congestions the outgoing acks will be rejected too (it's limiting both ways). there is no difference between artificial congestion generated by a shaper and a true congestion. If tcp is correct it has to slowdown immediatly when it notices congestion. I also don't see anything specific to a busy http server non support keepalive in this ack-fast-retransmit matter. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 22:52 ` Alan Cox 2003-08-31 22:58 ` Larry McVoy 2003-08-31 23:02 ` Andrea Arcangeli @ 2003-08-31 23:39 ` Roman Zippel 2003-09-01 0:09 ` Larry McVoy 2 siblings, 1 reply; 71+ messages in thread From: Roman Zippel @ 2003-08-31 23:39 UTC (permalink / raw) To: Alan Cox Cc: Andrea Arcangeli, Larry McVoy, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List Hi, On Sun, 31 Aug 2003, Alan Cox wrote: > How about because any undergraduate can do the mathematical proof its > not possible. Unless he controls the ISP end of the link random bursts > of traffic, pingfloods, anything not respecting requests to slow down > will lose voice traffic. At first Larry wasn't talking about incoming bursts: "We do VOIP phones and when you guys clone a repo our phones don't work". bye, Roman ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 23:39 ` Roman Zippel @ 2003-09-01 0:09 ` Larry McVoy 2003-09-01 0:20 ` Larry McVoy 0 siblings, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-09-01 0:09 UTC (permalink / raw) To: Roman Zippel Cc: Alan Cox, Andrea Arcangeli, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 01:39:56AM +0200, Roman Zippel wrote: > Hi, > > On Sun, 31 Aug 2003, Alan Cox wrote: > > > How about because any undergraduate can do the mathematical proof its > > not possible. Unless he controls the ISP end of the link random bursts > > of traffic, pingfloods, anything not respecting requests to slow down > > will lose voice traffic. > > At first Larry wasn't talking about incoming bursts: "We do VOIP phones > and when you guys clone a repo our phones don't work". Hey, let me make something clear in case it isn't. This isn't your problem, you have every right to clone away as fast as you want. So I am definitely *not* asking you to ease up on bkbits.net. In fact, I'd be unhappy if you did, we get a fair amount of stress testing of new releases on bkbits.net. So have at it. But getting back to the problem space, it's incoming traffic, outgoing traffic, all the web hits (Google crawls the web interface to bkbits for example, even though we've tried to turn that off through robots.txt people have linkages into specific csets and then google will follow those down and up), etc. I'd love for shaping to be something that would help here but I agree with Alan and others who say that this isn't something we can control at our end. The good news is that I'm 99% sure we have a deal for 2 T1 lines at $500/each and we're currently paying $800/month for one T1 so I view this as a slight increase in dollars for double the bandwidth. The ISP is willing to do the shaping at their end, where it will help, if that works that means that everyone wins, you guys get more bandwidth. If it doesn't work then you guys get a dedicated T1 line for bkbits which is more than what you have now. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 0:09 ` Larry McVoy @ 2003-09-01 0:20 ` Larry McVoy 0 siblings, 0 replies; 71+ messages in thread From: Larry McVoy @ 2003-09-01 0:20 UTC (permalink / raw) To: Roman Zippel, Alan Cox, Andrea Arcangeli, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 05:09:08PM -0700, Larry McVoy wrote: > On Mon, Sep 01, 2003 at 01:39:56AM +0200, Roman Zippel wrote: > > At first Larry wasn't talking about incoming bursts: "We do VOIP phones > > and when you guys clone a repo our phones don't work". > > Hey, let me make something clear in case it isn't. This isn't your problem, > you have every right to clone away as fast as you want. I forgot to add that once we have this sorted out we'll do two other things that you'll like: a) give you BK URL's that don't change (the current URL's are unstable, they are based on revisions and revision numbers are unstable) b) make every changeset (or range of changesets) be something you can grab as a regular diff -Nur style patch. So all the BK users can post to the kernel list and include a URL that you all can wget and there is the patch. No need to use BK at all unless you want to, the data is right there as a patch. All I'm trying to do is to underscore the point that none of you should "be nice" and not beat up on bkbits.net. It's a service, we get at least some benefit from you using the service so bang on it all you want. We'll solve the bandwidth problems. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 22:49 ` Andrea Arcangeli 2003-08-31 22:52 ` Alan Cox @ 2003-08-31 22:56 ` Larry McVoy 2003-08-31 23:13 ` Andrea Arcangeli 2003-09-02 7:11 ` Henning P. Schmiedehausen 2 siblings, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-31 22:56 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 12:49:38AM +0200, Andrea Arcangeli wrote: > > They are Cisco configuration, it won't do you much good. All the traffic > > I don't trust anything but the sourcecode I can read, so please try > again with linux. We did that as well. We ran wondershaper on bkbits.net AND capped the Cisco. > > I'm not sure why you are arguing this, if you have a fat pipe feeding into > > you never tried with linux, how can you claim you know it doesn't work > in practice? The fact is that you never tried it, while we used it all > the time. Not for VOIP with a busy server you don't. > > a small pipe and you are trying to throttle at far end of the small pipe > > isn't it obvious that you can't make that work? It's not the packets we > > send, it's the packets you send. And all the flow control stuff is per > > it's tcp, it's trivial to rate limit the packets we send, as far as you > go into established somehow. Let's see. We've tried Linux. It didn't work. We tried again w/ the Cisco. Didn't work. Tried with both. Didn't work. Stopped hacking and starting thinking and realized it's impossible to make it work at just one end, especially at the wrong end. Several people who I trust (like Alan for example) pointed it that the theory matches the practice, it can't work. Yet you keep insisting it will work. Why? What is the theory that says you can keep the other end of the T1 line from being congested when you don't have control over that router? And that router has several 100Mbit links all of which can point at my 1.5Mbit link. Explain to me how anything you do can make that work. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 22:56 ` Larry McVoy @ 2003-08-31 23:13 ` Andrea Arcangeli 2003-09-01 0:18 ` Jamie Lokier 0 siblings, 1 reply; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 23:13 UTC (permalink / raw) To: Larry McVoy, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 03:56:39PM -0700, Larry McVoy wrote: > Yet you keep insisting it will work. Why? What is the theory that says > you can keep the other end of the T1 line from being congested when you > don't have control over that router? And that router has several 100Mbit it's absolutely trivial, your end only needs to drop 99% of your outgoing acks and their incoming packets for every connection but voip while you are at the phone, you won't kill the connections but everybody but your voip will work. the exponential backoff and sstrash on the other and will rate limit everything immediatly. you will definitely control the other end. Anyways, I was only trying to help, I know how it works, it works perfectly for me, and the reason everything makes perfect sense is that this directory exists in Alan's tree too: 2.4.22ac1/net/sched/ All assuming you're not under attack, if that's the case you should contact your ISP since that's unfixable on your end. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 23:13 ` Andrea Arcangeli @ 2003-09-01 0:18 ` Jamie Lokier 2003-09-01 0:25 ` Larry McVoy 2003-09-01 0:28 ` Andrea Arcangeli 0 siblings, 2 replies; 71+ messages in thread From: Jamie Lokier @ 2003-09-01 0:18 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List Andrea Arcangeli wrote: > On Sun, Aug 31, 2003 at 03:56:39PM -0700, Larry McVoy wrote: > > Yet you keep insisting it will work. Why? What is the theory that says > > you can keep the other end of the T1 line from being congested when you > > don't have control over that router? And that router has several 100Mbit > > it's absolutely trivial, your end only needs to drop 99% of your > outgoing acks and their incoming packets for every connection but voip > while you are at the phone, you won't kill the connections but everybody > but your voip will work. the exponential backoff and sstrash on the > other and will rate limit everything immediatly. Let's work it out. We assume 99% means drop virtually everything: Every 19 seconds on average, 24x7, a new HTTP connection. Rate is not uniform throughout the day. Let's take a wild guess and say it is 10 times higher at peak times. That's one connection every 1.9 seconds. Let's assume you drop 99% of outgoing ACKs. Then all connecting remote clients will retry their SYNs until they get a connection or a timeout. Default tcp_syn_retries (assuming Linux clients) is 5. That's one SYN every 0.38 seconds. -> bad but not awful. Plus existing connections. Let's pretend each connection take 100 seconds. That's 100/1.9 or 52 concurrent connections, roughly. Each of those will retry with an ACK if it has any pending ACK or data. When you start using the phone, that's 100 ACKs total, approximately, with exponential backoff. --> bad but not awful. These calculations are horrendously inaccurate, but should be ok within a couple of orders of magnitude. So, near-total annihilation of bkbits.net when Larry or any of his team are on the phone should work. You can either integrate the phone system with netfilter so it is automatic when a customer calls. Trivial ;) Or disable bkbits.net during Larry's working day. ;) -- Jamie ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 0:18 ` Jamie Lokier @ 2003-09-01 0:25 ` Larry McVoy 2003-09-01 0:28 ` Andrea Arcangeli 1 sibling, 0 replies; 71+ messages in thread From: Larry McVoy @ 2003-09-01 0:25 UTC (permalink / raw) To: Jamie Lokier Cc: Andrea Arcangeli, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 01:18:19AM +0100, Jamie Lokier wrote: > Every 19 seconds on average, 24x7, a new HTTP connection. Wrong. Every 19 seconds is a new BK connection. The HTTP connections are at least an order of magnitude more frequent. And increasing, more and more people are becoming aware that the source is there and you can point to it. > That's one connection every 1.9 seconds. Ha. Don't I wish. Peak connection rates are a lot higher than that. > Trivial ;) Or disable bkbits.net during Larry's working day. ;) It's not my working day or the other people who work locally, all our phones are on a 100Mbit switched net to the 3com before it goes out on POTS. It's the remote people. And half our company is currently remote. I could write a book on how well remote does and doesn't work. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 0:18 ` Jamie Lokier 2003-09-01 0:25 ` Larry McVoy @ 2003-09-01 0:28 ` Andrea Arcangeli 2003-09-01 0:50 ` Jamie Lokier 1 sibling, 1 reply; 71+ messages in thread From: Andrea Arcangeli @ 2003-09-01 0:28 UTC (permalink / raw) To: Jamie Lokier Cc: Larry McVoy, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 01:18:19AM +0100, Jamie Lokier wrote: > So, near-total annihilation of bkbits.net when Larry or any of his > team are on the phone should work. You can either integrate the phone this is *exactly* my point. Of course near-total annihilation of bkbits.net may not be acceptable to the community, but still it should work as a proof of concept. Depending on the connect/sec of the http server (not bkbits, for the largest part of the conversation I couldn't know about the http server, Larry only mentioned the bkbits.net clone until recently), the "reservation" margin will have to change: the less connect/sec the smaller margin Larry will need to reserve, the more connect/sec the bigger marging will be necessary. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 0:28 ` Andrea Arcangeli @ 2003-09-01 0:50 ` Jamie Lokier 2003-09-01 1:10 ` Andrea Arcangeli 0 siblings, 1 reply; 71+ messages in thread From: Jamie Lokier @ 2003-09-01 0:50 UTC (permalink / raw) To: Andrea Arcangeli Cc: Larry McVoy, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List Andrea Arcangeli wrote: > Depending on the connect/sec of the http server (not bkbits, for the > largest part of the conversation I couldn't know about the http server, > Larry only mentioned the bkbits.net clone until recently), the > "reservation" margin will have to change: the less connect/sec the > smaller margin Larry will need to reserve, the more connect/sec the > bigger marging will be necessary. Hi Andrea, Above a certain connection rate, no amount of margin is enough. The connections are a SYN flood. You need to take into account the peak rate, which varies from second to second, due to simple statistics plus the tendency of the net to make traffic burstier than it started. Above a certain number of packets/sec, competing VoIP traffic degrades - a little added latency is equivalent to dropping with VoIP. That translates to dropouts in the audio. Acceptable for hackers talking over a flakey line, but not corporate telephony. At Larry's server, inbound connection rate resembles a low-level SYN flood, which is enough to poke holes in VoIP latency. Of course the real problem is lack of end to end QoS between Larry's nodes. Two T1s is just a workaround :) -- Jamie ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 0:50 ` Jamie Lokier @ 2003-09-01 1:10 ` Andrea Arcangeli 2003-09-01 1:33 ` Larry McVoy 0 siblings, 1 reply; 71+ messages in thread From: Andrea Arcangeli @ 2003-09-01 1:10 UTC (permalink / raw) To: Jamie Lokier Cc: Larry McVoy, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 01:50:41AM +0100, Jamie Lokier wrote: > Andrea Arcangeli wrote: > > Depending on the connect/sec of the http server (not bkbits, for the > > largest part of the conversation I couldn't know about the http server, > > Larry only mentioned the bkbits.net clone until recently), the > > "reservation" margin will have to change: the less connect/sec the > > smaller margin Larry will need to reserve, the more connect/sec the > > bigger marging will be necessary. > > Hi Andrea, > > Above a certain connection rate, no amount of margin is enough. The > connections are a SYN flood. > > You need to take into account the peak rate, which varies from second > to second, due to simple statistics plus the tendency of the net to > make traffic burstier than it started. > > Above a certain number of packets/sec, competing VoIP traffic degrades > - a little added latency is equivalent to dropping with VoIP. That > translates to dropouts in the audio. Acceptable for hackers talking > over a flakey line, but not corporate telephony. > > At Larry's server, inbound connection rate resembles a low-level SYN > flood, which is enough to poke holes in VoIP latency. Yes. if you check my very first email in this thread, you will see I said "it has to work, unless you're under syn-flood ;)". this is obvious, I'm not arguing about that. From Larry's description of the problem it couldn't be a syn flood, but it was a bkbits.net clone thing or checkout or whatever non syn intensive. now apparently bkbits.net has nothing to do with it, and it's all about the http server. Then of course the syn overhead may become noticeable but things have changed totally from the original description of the problem (the bkbits.net clone thing). however keep in mind you will somehow throttle the number of syns too, unless every single syn arrives to the webserver from a different user (unlikely). Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-09-01 1:10 ` Andrea Arcangeli @ 2003-09-01 1:33 ` Larry McVoy 0 siblings, 0 replies; 71+ messages in thread From: Larry McVoy @ 2003-09-01 1:33 UTC (permalink / raw) To: Andrea Arcangeli Cc: Jamie Lokier, Larry McVoy, Alan Cox, Pascal Schmidt, Linux Kernel Mailing List On Mon, Sep 01, 2003 at 03:10:55AM +0200, Andrea Arcangeli wrote: > now apparently bkbits.net has nothing to do with it, and it's all about > the http server. BK _is_ an http server. The same daemon you talk to for clones is the HTTP server. That's one and the same machine. > however keep in mind you will somehow throttle the number of syns too, > unless every single syn arrives to the webserver from a different user > (unlikely). That's exactly the situation that any busy server has. Which is why I kept saying "tell me how you made this work for a busy server". "Busy server" by definition in this context at least means a server that is getting lots of connection requests from lots of different users. Why that isn't obvious to you I don't understand. This is bkbits.net. Yeah, it's not slashdot or anything but there are something like 400 branches of the Linux kernel on it and that's ignoring all other projects. It's the web server which provides insight into the Linux kernel source base, of course it is busy. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 22:49 ` Andrea Arcangeli 2003-08-31 22:52 ` Alan Cox 2003-08-31 22:56 ` Larry McVoy @ 2003-09-02 7:11 ` Henning P. Schmiedehausen 2 siblings, 0 replies; 71+ messages in thread From: Henning P. Schmiedehausen @ 2003-09-02 7:11 UTC (permalink / raw) To: linux-kernel Andrea Arcangeli <andrea@suse.de> writes: >> They are Cisco configuration, it won't do you much good. All the traffic >I don't trust anything but the sourcecode I can read, so please try >again with linux. Andrea, you have no clue at all. Please stop this. You might be terrific at VM and linux kernel internals but you suck at network topology and serious internet traffic routing. You want to try rate limiting a DDoS attack from the wrong side of a small pipe. You can't do this. Not with Linux, with BSD, with a C64 or a Cisco. People have been trying this on the internet since 1995 and know what they're talking about. This is not a case of "I have no clue, so I didn't know that it was impossible and I did it". There are people out there that got Ph.D.'s for proving that it's impossible to do what you suggest and other people that got rich building devices that can do this (from the right end of the pipe). Regards Henning -- Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/ Java, perl, Solaris, Linux, xSP Consulting, Web Services freelance consultant -- Jakarta Turbine Development -- hero for hire "Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" -- AOL CD when played backwards (User Friendly - 200-10-15) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 15:44 ` Andrea Arcangeli 2003-08-31 16:22 ` Larry McVoy @ 2003-09-02 7:01 ` Henning P. Schmiedehausen 1 sibling, 0 replies; 71+ messages in thread From: Henning P. Schmiedehausen @ 2003-09-02 7:01 UTC (permalink / raw) To: linux-kernel Andrea Arcangeli <andrea@suse.de> writes: >> It doesn't work when you dont control incoming. As a simple extreme >> example if I pingflood you from a fast site then no amount of shaping >> your end of the link will help, it has to be shaped at the ISP end. >sure, that's why I said it won't work with synflood. I just doubt the >ping/syn floods distributed denial of services are an high percentage of The traffic on sites like www.microsoft.com, www.kernel.org or I'd say even www.bkbits.net isn't distinguishable from a DDoS Synflood on a regular day. A bazillion of different IP addresses from all over the net trying to establish a TCP connection at the same time. And the other sides are dog-slow modem/isdn/dsl connections which means high latency, low bandwith, many retransmits and still they're legal connections that you want to serve. Forget it. You can't compare your homegrown ISDN/DSL line with the traffic of a serious internet service. If you try, you get burned. Regards Henning -- Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/ Java, perl, Solaris, Linux, xSP Consulting, Web Services freelance consultant -- Jakarta Turbine Development -- hero for hire "Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" -- AOL CD when played backwards (User Friendly - 200-10-15) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 15:31 ` Alan Cox 2003-08-31 15:43 ` Jörn Engel 2003-08-31 15:44 ` Andrea Arcangeli @ 2003-08-31 16:23 ` Larry McVoy 2003-08-31 16:36 ` Andrea Arcangeli 2 siblings, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-31 16:23 UTC (permalink / raw) To: Alan Cox Cc: Andrea Arcangeli, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 04:31:32PM +0100, Alan Cox wrote: > > what's the difference of rejecting packets in software, or because the > > link can't handle them? Assume the guaranteed bandwidth is much lower > > It doesn't work when you dont control incoming. As a simple extreme > example if I pingflood you from a fast site then no amount of shaping > your end of the link will help, it has to be shaped at the ISP end. HTTP traffic is enough to simulate this, the connections are all small, short lived, and there are a lot of them. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 16:23 ` Larry McVoy @ 2003-08-31 16:36 ` Andrea Arcangeli 0 siblings, 0 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 16:36 UTC (permalink / raw) To: Larry McVoy, Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, Aug 31, 2003 at 09:23:37AM -0700, Larry McVoy wrote: > On Sun, Aug 31, 2003 at 04:31:32PM +0100, Alan Cox wrote: > > > what's the difference of rejecting packets in software, or because the > > > link can't handle them? Assume the guaranteed bandwidth is much lower > > > > It doesn't work when you dont control incoming. As a simple extreme > > example if I pingflood you from a fast site then no amount of shaping > > your end of the link will help, it has to be shaped at the ISP end. > > HTTP traffic is enough to simulate this, the connections are all small, > short lived, and there are a lot of them. it's much harder to throttle http, but it should work too. you may need bigger margin due the higher percentage of unthrottable packets like syns. Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 14:45 ` Andrea Arcangeli 2003-08-31 15:31 ` Alan Cox @ 2003-08-31 15:38 ` Jörn Engel 1 sibling, 0 replies; 71+ messages in thread From: Jörn Engel @ 2003-08-31 15:38 UTC (permalink / raw) To: Andrea Arcangeli Cc: Alan Cox, Larry McVoy, Pascal Schmidt, Linux Kernel Mailing List On Sun, 31 August 2003 16:45:05 +0200, Andrea Arcangeli wrote: > On Sun, Aug 31, 2003 at 02:15:12PM +0100, Alan Cox wrote: > > On Sul, 2003-08-31 at 03:56, Larry McVoy wrote: > > > I'm pretty convinced we can't solve the problem at our end. Maybe we can > > > > For bursts of traffic you can't. > > what's the difference of rejecting packets in software, or because the > link can't handle them? Nothing, packets are always rejected in software. :) > it works flawlessy for me and it's the same problem (I also use > streaming services), and they're unusable until I turn the shaping on, > I'm sure that if you use the script and you change 1kbyte/sec to > everything but voip it'll work fine for you too, since basically > everything else won't pass anymore, it will take ages to open an html > page and all the bkbits.net users will hang, and the link will be idle > 99% of the time, so voip will take it over as much as it can. I don't > think it's a matter of "if it works or not", I think it's a matter of > how much you're ok to lose in terms of global bandwith for all the other > services but voip. Yes, that should work for well behaved traffic. You have to leave more breathing room compared to your provider doing the traffic shaping for you, but it is also more flexible than one line for each VOIP and the rest. Still, I don't like the idea of throwing away those packets that have made it through the bottleneck. :) Jörn -- The strong give up and move away, while the weak give up and stay. -- unknown ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 2:56 ` Larry McVoy 2003-08-31 13:15 ` Alan Cox @ 2003-09-02 6:55 ` Henning P. Schmiedehausen 1 sibling, 0 replies; 71+ messages in thread From: Henning P. Schmiedehausen @ 2003-09-02 6:55 UTC (permalink / raw) To: linux-kernel Larry McVoy <lm@bitmover.com> writes: >I'm pretty convinced we can't solve the problem at our end. Maybe we can You're right. You can't no matter what all the "network experts" around here say. It's either placing a shaper at the ISP or getting a second line. I still don't understand why you don't simply co-lo the bkbits.net box. There should be plenty of housing centers in the Bay Area within a 30 minute drive of your office. ;-) Regards Henning -- Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/ Java, perl, Solaris, Linux, xSP Consulting, Web Services freelance consultant -- Jakarta Turbine Development -- hero for hire "Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" -- AOL CD when played backwards (User Friendly - 200-10-15) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 1:39 ` Andrea Arcangeli 2003-08-31 2:18 ` Jeff Sipek 2003-08-31 2:56 ` Larry McVoy @ 2003-08-31 13:47 ` Pascal Schmidt 2003-08-31 14:51 ` Andrea Arcangeli 2 siblings, 1 reply; 71+ messages in thread From: Pascal Schmidt @ 2003-08-31 13:47 UTC (permalink / raw) To: Andrea Arcangeli; +Cc: linux-kernel On Sun, 31 Aug 2003, Andrea Arcangeli wrote: > This is what I use normally to limit my brother downloads, and it works > fine for me (though I don't often place calls through adsl myself, it's > basically useless since people only uses cellphones here, and last time > I chekced voip wasn't free for cellphones with my isp). > > this is the script: Mine is similar, though I use tc filters instead of firewalling rules (my machine is on 192.168.2.0/24, rest of the house is on 192.168.3.0/24) I'm using the imq device on a 2.2 kernel to have all traffic go through that device for shaping: #!/bin/bash # traffic shaping startup script # # (c) 2003 Pascal Schmidt <der.eremit@email.de> . /etc/rc.d/rc.functions PATH=/bin:/sbin:/usr/bin:/usr/sbin case "$1" in start) echo -n "Setting up traffic shaper: " # remove other qdisc if any tc qdisc del dev imq root &> /dev/null # use HTB as root tc qdisc add dev imq root handle 1:0 htb default 3 # complete LAN tc class add dev imq parent 1:0 classid 1:1 htb \ rate 10mbit burst 15k # ISDN connection, split up into two "halves" tc class add dev imq parent 1:1 classid 1:2 htb \ rate 64kbit burst 15k tc class add dev imq parent 1:2 classid 1:21 htb \ rate 40kbit ceil 64kbit burst 15k tc class add dev imq parent 1:2 classid 1:22 htb \ rate 24kbit ceil 59kbit burst 15k # rest of LAN bandwidth for internal traffic tc class add dev imq parent 1:1 classid 1:3 htb \ rate 9mbit ceil 10mbit burst 15k # use SFQ on leaf classes tc qdisc add dev imq parent 1:21 handle 21:0 sfq perturb 10 tc qdisc add dev imq parent 1:22 handle 22:0 sfq perturb 10 tc qdisc add dev imq parent 1:3 handle 3:0 sfq perturb 10 # LAN to LAN traffic goes to fast class tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \ match ip src 192.168.0.0/16 \ match ip dst 192.168.0.0/16 flowid 1:3 # rest of traffic to LAN #1 (must be from Internet) # goes to first ISDN class (half ISDN channel) tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \ match ip dst 192.168.2.0/24 flowid 1:21 # rest of traffic to LAN #2 (must be from Internet) # goes to second ISDN class (half ISDN channel) tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \ match ip dst 192.168.3.0/24 flowid 1:22 check_status # bring up intermediate queue interface echo -n "Bringing up intermediate queue: " ifconfig imq up check_status ;; stop) # bring down intermediate queue interface echo -n "Taking down intermediate queue: " ifconfig imq down check_status # remove root qdisc echo -n "Disabling traffic shaper: " tc qdisc del dev imq root check_status ;; status) tc -d -s class show dev imq ;; restart) $0 stop $0 start ;; *) echo "Usage: $0 {start|stop|status|restart}" exit 1 esac -- Ciao, Pascal ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 13:47 ` Pascal Schmidt @ 2003-08-31 14:51 ` Andrea Arcangeli 0 siblings, 0 replies; 71+ messages in thread From: Andrea Arcangeli @ 2003-08-31 14:51 UTC (permalink / raw) To: Pascal Schmidt; +Cc: linux-kernel On Sun, Aug 31, 2003 at 03:47:17PM +0200, Pascal Schmidt wrote: > On Sun, 31 Aug 2003, Andrea Arcangeli wrote: > > > This is what I use normally to limit my brother downloads, and it works > > fine for me (though I don't often place calls through adsl myself, it's > > basically useless since people only uses cellphones here, and last time > > I chekced voip wasn't free for cellphones with my isp). > > > > this is the script: > > Mine is similar, though I use tc filters instead of firewalling > rules (my machine is on 192.168.2.0/24, rest of the house is > on 192.168.3.0/24) I'm using the imq device on a 2.2 kernel to > have all traffic go through that device for shaping: the main reason I choosen to mark the packets through netfilter is that I don't find `tc` very intuitive (the most annoying thing is that it misses a flush command), so it'll be quicker to add future tweaks this way (like adding more ips or port/ip combinations into the "shaped" channel) thanks, Andrea ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-31 1:05 ` Pascal Schmidt 2003-08-31 1:39 ` Andrea Arcangeli @ 2003-09-02 6:53 ` Henning P. Schmiedehausen 1 sibling, 0 replies; 71+ messages in thread From: Henning P. Schmiedehausen @ 2003-09-02 6:53 UTC (permalink / raw) To: linux-kernel Pascal Schmidt <der.eremit@email.de> writes: >In a way, you're on the right end of the pipe because the system >that does your traffic shaping is part of the general network, viewed >from the machines behind the shaper. Stop argueing. You can shape only outgoing packets, which means that the shaper must have a big uplink pipe and then shape that into a thin downlink pipe. In Larrys' case, the shaper must be on the ISP network before the T1. Once the packets hit the T1 (or your ISDN link), you've already lost. No matter how much shaping, dropping, non-acking you're doing on your side of the link. Regards Henning -- Dipl.-Inf. (Univ.) Henning P. Schmiedehausen INTERMETA GmbH hps@intermeta.de +49 9131 50 654 0 http://www.intermeta.de/ Java, perl, Solaris, Linux, xSP Consulting, Web Services freelance consultant -- Jakarta Turbine Development -- hero for hire "Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" -- AOL CD when played backwards (User Friendly - 200-10-15) ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-30 22:58 ` bandwidth for bkbits.net (good news) Pascal Schmidt 2003-08-30 23:07 ` Larry McVoy @ 2003-08-31 6:30 ` Jörn Engel 1 sibling, 0 replies; 71+ messages in thread From: Jörn Engel @ 2003-08-31 6:30 UTC (permalink / raw) To: Pascal Schmidt; +Cc: linux-kernel On Sun, 31 August 2003 00:58:33 +0200, Pascal Schmidt wrote: > > All you have to do is drop the incoming packets if they exceed > a certain bandwidth. That will stop the corresponding ack automatically > since your TCP stack won't even see the packet. You've cut away the part where I explained that you don't want to throw away the incoming packet. Yes, the stupid approach works, but it is still stupid. > I'm doing this on my ISDN line to limit the rest of the house to one > third of the bandwidth even if they're all busy downloading tons of > warez. I'm paying, I want the bandwidth when I need it. They can get > it all when there's no traffic for my machine. > > No problem with the HTB queueing discipline. But latency just went boom. On your side, your packets will come out quickly because the queue is short. But on your ISP's side, there is a single queue for both your and the warez' traffic. Your packets will get thrown away just as much, as theirs. > Yes, latency is a problem if you want to saturate your bandwidth. It is > easy to guarantee some bandwith for latency critical stuff - just > don't give out that piece of bandwith to something else. Of course, > then most of the time that piece is wasted... This works if your latency requirements are moderate or the pipe is big. Assume 5ms and ISDN and you have a window of 400 Bytes roughly. Each time you happen to receive 400 Bytes of packets at the same time, you hear a pause. A single large packet is more than that. Oops! For Larry's T1 line, the numbers naturally go up, but it still doesn't take a huge amount of packets. > and it doesn't help with problems somewhere along the net. Compared to ISDN or T1, the net usually has big pipes and people tend to care about low latency, so that problem doesn't even exist compared with the receiving end. Jörn -- Simplicity is prerequisite for reliability. -- Edsger W. Dijkstra ^ permalink raw reply [flat|nested] 71+ messages in thread
[parent not found: <qn1b.2Pr.29@gated-at.bofh.it>]
[parent not found: <qoTh.5mt.11@gated-at.bofh.it>]
[parent not found: <rdje.1sH.11@gated-at.bofh.it>]
* Re: bandwidth for bkbits.net (good news) [not found] ` <rdje.1sH.11@gated-at.bofh.it> @ 2003-09-02 17:49 ` Pascal Schmidt 0 siblings, 0 replies; 71+ messages in thread From: Pascal Schmidt @ 2003-09-02 17:49 UTC (permalink / raw) To: hps; +Cc: linux-kernel On Tue, 02 Sep 2003 09:00:20 +0200, you wrote in linux.kernel: > Stop argueing. You can shape only outgoing packets, which means that the > shaper must have a big uplink pipe and then shape that into a thin downlink > pipe. In Larrys' case, the shaper must be on the ISP network before the T1. For Larry's case, you're right. Nothing you can do to shape lots of incoming TCP connections. However, if you're facing a few long-lived TCP connections, you can shape them by dropping incoming packets (that is, shaping the outgoing packets from the router/shaper to the LAN). The sending side will adjust its rate after at most a few seconds. That works perfectly for the case of reserving bandwidth for critical services in the face of ongoing large downloads. That's something a lot of people want and need, so it's not all that worseless per se. -- Ciao, Pascal - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ On Tue, 02 Sep 2003 09:00:20 +0200, you wrote in linux.kernel: > Stop argueing. You can shape only outgoing packets, which means that the > shaper must have a big uplink pipe and then shape that into a thin downlink > pipe. In Larrys' case, the shaper must be on the ISP network before the T1. For Larry's case, you're right. Nothing you can do to shape lots of incoming TCP connections. However, if you're facing a few long-lived TCP connections, you can shape them by dropping incoming packets (that is, shaping the outgoing packets from the router/shaper to the LAN). The sending side will adjust its rate after at most a few seconds. That works perfectly for the case of reserving bandwidth for critical services in the face of ongoing large downloads. That's something a lot of people want and need, so it's not all that worseless per se. -- Ciao, Pascal ^ permalink raw reply [flat|nested] 71+ messages in thread
* bandwidth for bkbits.net (good news) @ 2003-08-30 1:29 Larry McVoy 2003-08-30 15:03 ` Larry McVoy 0 siblings, 1 reply; 71+ messages in thread From: Larry McVoy @ 2003-08-30 1:29 UTC (permalink / raw) To: linux-kernel FYI - we're working with several vendors to try and get more bandwidth for bkbits.net. We think we have a line on a deal where we can get a full T1 just for bkbits.net for about $500/month. If we get that then we'll turn on the patch server feature so you can hit any URL and get a regular diff -Nur style patch for that changeset (or range of changesets). We've avoiding turning on that feature in the past because we share the T1 line that bkbits.net lives on with all the rest of bitmover and we are partialy a distributed company. We do VOIP phones and when you guys clone a repo our phones don't work - that makes us look bad during a sales call. I'm not complaining, we get nice stress testing from bkbits so you should hammer on it all you want but I'd like it if we could really encourage that more. Turning on a patch server should do the trick. ETA on this is a month. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-30 1:29 Larry McVoy @ 2003-08-30 15:03 ` Larry McVoy 2003-08-30 16:10 ` Jörn Engel 2003-08-31 10:13 ` Toon van der Pas 0 siblings, 2 replies; 71+ messages in thread From: Larry McVoy @ 2003-08-30 15:03 UTC (permalink / raw) To: linux-kernel On Fri, Aug 29, 2003 at 06:29:49PM -0700, Larry McVoy wrote: > We've avoiding turning on that feature in the past because we share the > T1 line that bkbits.net lives on with all the rest of bitmover and we are > partialy a distributed company. We do VOIP phones and when you guys clone > a repo our phones don't work - that makes us look bad during a sales call. Many people have sent me mail saying that we should be using traffic shaping to fix this problem. We are using it and we can't seem to make it work. Our theory is that we have a network like ----- [ ISP ] ====== internet ======== [ ISP ] ---- wherein "-" means our skinny T1 or DSL and "=" means some fat internet connection on the backbone. We can shape all we want on our ends but if the internet is blasting us then our skinny pipe gets full and our shaping doesn't work. We really need to have the ISP do the shaping so they can squelch the traffic before it gets to our pipe. If there is someone out there who (a) is running VOIP over the public net to a pile of different end points (T1 on both ends tends to work, T1 to DSL or cable modem tends to get harder) and (b) has figured out traffic shaping that works I'd love to know about it. But just saying QoS/wondershaper doesn't help much (though the thought is appreciated), we've tried that already. Thanks. -- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-30 15:03 ` Larry McVoy @ 2003-08-30 16:10 ` Jörn Engel 2003-08-31 10:13 ` Toon van der Pas 1 sibling, 0 replies; 71+ messages in thread From: Jörn Engel @ 2003-08-30 16:10 UTC (permalink / raw) To: Larry McVoy, linux-kernel On Sat, 30 August 2003 08:03:11 -0700, Larry McVoy wrote: > > Many people have sent me mail saying that we should be using traffic > shaping to fix this problem. We are using it and we can't seem to make > it work. Our theory is that we have a network like > > ----- [ ISP ] ====== internet ======== [ ISP ] ---- > > wherein "-" means our skinny T1 or DSL and "=" means some fat internet > connection on the backbone. > > We can shape all we want on our ends but if the internet is blasting us > then our skinny pipe gets full and our shaping doesn't work. We really > need to have the ISP do the shaping so they can squelch the traffic > before it gets to our pipe. > > If there is someone out there who (a) is running VOIP over the public net > to a pile of different end points (T1 on both ends tends to work, T1 to DSL > or cable modem tends to get harder) and (b) has figured out traffic shaping > that works I'd love to know about it. By principle, you can only control your side of the wire. Unless your ISP does some decent shaping for you or allows you to place a machine at the other end to do it, you are at the internets mercy. If people with enough bandwidth want to DOS you, they can. For well behaved traffic you have some limited control over the incoming traffic through your responses, so QoS should work in theory, but setting it up is still very close to black magic. > But just saying QoS/wondershaper doesn't help much (though the thought is > appreciated), we've tried that already. Wondershaper was magically configured once and works for everyone that has similar needs like the original magician. Plus, this magician had a simple task, as the sending side is the bottleneck and that is the side he has control over. So unless you have a regular DSL line... What you really need might not even exist in the kernel yet. The last couple of times I've tried setting it up and read the code, things were just not adequate. ---<deeper technical stuff follows>--- In order to control incoming traffic, it is easiest to look at tcp. udp can work similarly, but it doesn't have to. To throttle the stream of tcp packets, you could simply through away the acks and the sending side will reduce it's speed. So you have to measure one stream and control a related but different one. Maybe this is possible, not sure. Second, you usually don't want to through away the acks, as packets would be retransmitted then. This reduces the effective bandwidth and limited bandwidth was the problem in the first place. So we have to delay them enough to slow things down, but not beyond the timeout. Third, there is the problem transition from continuous streams to discrete packets, when bandwidth is low. It doesn't take a huge amount of large packets to create enough latency for your sad VOIP. And fourth, the possibility of resonance frequency. If measurement and/or shaping intervals are too long and match nicely to the travel time to one of your peers, you are back at point three. Overall, it is possible to do some decent traffic shaping, but it is far from being simple. And it might even be completely impossible, at least in your case, with the code that is in the kernel today, be it Linux or some BSD. Jörn PS: If anyone can prove how stupid I am and how simple QoS is, please do! Seriously! -- Public Domain - Free as in Beer General Public - Free as in Speech BSD License - Free as in Enterprise Shared Source - Free as in "Work will make you..." ^ permalink raw reply [flat|nested] 71+ messages in thread
* Re: bandwidth for bkbits.net (good news) 2003-08-30 15:03 ` Larry McVoy 2003-08-30 16:10 ` Jörn Engel @ 2003-08-31 10:13 ` Toon van der Pas 1 sibling, 0 replies; 71+ messages in thread From: Toon van der Pas @ 2003-08-31 10:13 UTC (permalink / raw) To: linux-kernel On Sat, Aug 30, 2003 at 08:03:11AM -0700, Larry McVoy wrote: > On Fri, Aug 29, 2003 at 06:29:49PM -0700, Larry McVoy wrote: > > > We've avoiding turning on that feature in the past because we > > share the T1 line that bkbits.net lives on with all the rest of > > bitmover and we are partialy a distributed company. We do VOIP > > phones and when you guys clone a repo our phones don't work - > > that makes us look bad during a sales call. > > Many people have sent me mail saying that we should be using > traffic shaping to fix this problem. We are using it and we can't > seem to make it work. Our theory is that we have a network like > > ----- [ ISP ] ====== internet ======== [ ISP ] ---- > > wherein "-" means our skinny T1 or DSL and "=" means some fat > internet connection on the backbone. > > We can shape all we want on our ends but if the internet is > blasting us then our skinny pipe gets full and our shaping doesn't > work. We really need to have the ISP do the shaping so they can > squelch the traffic before it gets to our pipe. > > If there is someone out there who (a) is running VOIP over the > public net to a pile of different end points (T1 on both ends tends > to work, T1 to DSL or cable modem tends to get harder) and (b) has > figured out traffic shaping that works I'd love to know about it. We are using a Packetshaper box from Packeteer (www.packeteer.com). It solves the problem you are describing (shaping the incoming traffic) by ajusting the windowsize in the outgoing acknowledgements, if I'm not mistaken. It has done miracles for us. Could Linux do this? (Sorry for the plug. I don't have stocks in the company, nor am I involved with them in any other way. We are just a happy customer.) > But just saying QoS/wondershaper doesn't help much (though the > thought is appreciated), we've tried that already. > > Thanks. > -- > --- > Larry McVoy lm at bitmover.com http://www.bitmover.com/lm Regards, Toon. ^ permalink raw reply [flat|nested] 71+ messages in thread
end of thread, other threads:[~2003-09-08 8:57 UTC | newest] Thread overview: 71+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <q2SH.AA.3@gated-at.bofh.it> [not found] ` <qfwI.15D.27@gated-at.bofh.it> [not found] ` <qgCn.2y4.11@gated-at.bofh.it> 2003-08-30 22:58 ` bandwidth for bkbits.net (good news) Pascal Schmidt 2003-08-30 23:07 ` Larry McVoy 2003-08-31 1:05 ` Pascal Schmidt 2003-08-31 1:39 ` Andrea Arcangeli 2003-08-31 2:18 ` Jeff Sipek 2003-08-31 2:42 ` Andrea Arcangeli 2003-08-31 2:56 ` Larry McVoy 2003-08-31 13:15 ` Alan Cox 2003-08-31 14:45 ` Andrea Arcangeli 2003-08-31 15:31 ` Alan Cox 2003-08-31 15:43 ` Jörn Engel 2003-08-31 15:50 ` Andrea Arcangeli 2003-08-31 17:06 ` Florian Weimer 2003-08-31 21:21 ` Larry McVoy 2003-09-02 7:06 ` Henning P. Schmiedehausen 2003-09-05 8:10 ` Florian Weimer 2003-09-05 15:35 ` Henning Schmiedehausen 2003-09-05 15:50 ` Florian Weimer 2003-09-05 16:10 ` P 2003-09-05 16:43 ` Ricky Beam 2003-09-07 11:55 ` Henning Schmiedehausen 2003-09-07 18:41 ` Ricky Beam 2003-09-08 8:57 ` [COMPLETELY OFF-TOPIC] " Henning Schmiedehausen 2003-09-02 21:52 ` Ricky Beam 2003-09-05 8:16 ` Florian Weimer 2003-08-31 15:44 ` Andrea Arcangeli 2003-08-31 16:22 ` Larry McVoy 2003-08-31 16:33 ` Andrea Arcangeli 2003-08-31 16:48 ` Larry McVoy 2003-08-31 17:06 ` Andrea Arcangeli 2003-08-31 21:18 ` Larry McVoy 2003-08-31 22:49 ` Andrea Arcangeli 2003-08-31 22:52 ` Alan Cox 2003-08-31 22:58 ` Larry McVoy 2003-08-31 23:02 ` Andrea Arcangeli 2003-08-31 23:07 ` Larry McVoy 2003-08-31 23:22 ` Andrea Arcangeli 2003-09-01 0:12 ` Larry McVoy 2003-09-01 0:19 ` Jamie Lokier 2003-09-01 0:33 ` Larry McVoy 2003-09-01 11:43 ` Alan Cox 2003-09-01 16:13 ` Andrea Arcangeli 2003-09-01 16:24 ` Larry McVoy 2003-09-01 16:28 ` Alan Cox 2003-09-01 16:38 ` Andrea Arcangeli 2003-08-31 23:39 ` Roman Zippel 2003-09-01 0:09 ` Larry McVoy 2003-09-01 0:20 ` Larry McVoy 2003-08-31 22:56 ` Larry McVoy 2003-08-31 23:13 ` Andrea Arcangeli 2003-09-01 0:18 ` Jamie Lokier 2003-09-01 0:25 ` Larry McVoy 2003-09-01 0:28 ` Andrea Arcangeli 2003-09-01 0:50 ` Jamie Lokier 2003-09-01 1:10 ` Andrea Arcangeli 2003-09-01 1:33 ` Larry McVoy 2003-09-02 7:11 ` Henning P. Schmiedehausen 2003-09-02 7:01 ` Henning P. Schmiedehausen 2003-08-31 16:23 ` Larry McVoy 2003-08-31 16:36 ` Andrea Arcangeli 2003-08-31 15:38 ` Jörn Engel 2003-09-02 6:55 ` Henning P. Schmiedehausen 2003-08-31 13:47 ` Pascal Schmidt 2003-08-31 14:51 ` Andrea Arcangeli 2003-09-02 6:53 ` Henning P. Schmiedehausen 2003-08-31 6:30 ` Jörn Engel [not found] <qn1b.2Pr.29@gated-at.bofh.it> [not found] ` <qoTh.5mt.11@gated-at.bofh.it> [not found] ` <rdje.1sH.11@gated-at.bofh.it> 2003-09-02 17:49 ` Pascal Schmidt 2003-08-30 1:29 Larry McVoy 2003-08-30 15:03 ` Larry McVoy 2003-08-30 16:10 ` Jörn Engel 2003-08-31 10:13 ` Toon van der Pas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).