How can I test my tc script?

How can I test my tc script?

[Optimum Wireless Services]

Hello.

I recently wrote a script that adds a new rule for an ip address each
time a new user is added to our network. I've noticed my tc rules work
ok but, when there are over 20 users connected simultaneously our
internet speed tends to slow down.

Our network has about 120 users in total not all of these get connected
at the same time. I've tried to assign each user 512 burting up to 768
for download speed and 128 bursting up to 256 for upload. I do the
bursting to compensate 20% loss. Users have been complaining about the
slow internet lately but, don't know where is my bottleneck causing the
problem. We have 15Mb of total download bandwidth for the entire
network, don't know if thats enough but, can't get more than that yet.

I would like to know how can I run some sort of benchmark test on a
network that would simulate 10,20,50,100 users connected to see if my gw
is handling the traffic well.

Here's a snip of what the script looks like:

#upload
$tc qdisc del dev eth0 root
$tc qdisc add dev eth0 root handle 1: htb r2q 1

$tc class add dev eth0 parent 1:0  classid 1:1 htb rate 2700kbit ceil
3105kbit

# this is generated for each user:
$iptables -t mangle -A PREROUTING -s 172.16.100.11 -j MARK --set-mark
1000
$iptables -t mangle -A PREROUTING -s 172.16.100.11 -j RETURN
$tc class   add dev eth0 parent 1:10    classid 1:1000 htb rate 156kbit
ceil 256kbit prio 3
$tc filter  add dev eth0 parent 1:0     protocol ip prio 3 handle 1000
fw classid 1:1000
$tc qdisc   add dev eth0 parent 1:1000  handle 500: sfq perturb 2

################ 
Now Download interface

$tc qdisc del dev eth1 root
$tc qdisc add dev eth1 root handle 2: htb r2q 1

$tc class add dev eth1 parent 2:0  classid 2:1 htb rate 15500kbit ceil
15525kbit

#for each user
$iptables -t mangle -A POSTROUTING -d 172.16.100.11 -j MARK --set-mark
1009
$iptables -t mangle -A POSTROUTING -d 172.16.100.11 -j RETURN
$tc class   add dev eth1 parent 2:17    classid 2:1009 htb rate 739kbit
ceil 850kbit prio 3
$tc filter  add dev eth1 parent 2:0     protocol ip prio 3 handle 1009
fw classid 2:1009
$tc qdisc   add dev eth1 parent 2:1009  handle 507: sfq perturb 2


Are these rules ok?

Thanks in advanced for your help and time.

Re: How can I test my tc script?

[Marek Kierdelewicz]

>Hello.

Hi,

>I recently wrote a script that adds a new rule for an ip address each
>time a new user is added to our network. I've noticed my tc rules work
>...
>Our network has about 120 users in total not all of these get connected
>Are these rules ok?

If you have such linear ruleset (iptables marking+tc filter) for 120
users then it's will not work well. If my theory is right, check top
when there are more users logged in. You'll probably see hi cpu usage
in "si"/"hi" (software/hardware interrupt) fraction.

You can easily optimize you ruleset by using:
- tc u32 hashing filters [1] instead of iptables marking and fw tc
  filters;
- shape upload on ifb device [2] in ingress before nat, so you can use
  tc u32 hashing filters too;

With such setup Core2 duo 3GHz + dual port intel nic can easily
(68% peak cpu usage) route symmetric 400mbit of traffic, doing per user
shaping, nat and access control for >2k users.


[1] http://lartc.org/howto/lartc.adv-filter.hashing.html
[2] http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb

Best regards,
Marek Kierdelewicz

Re: How can I test my tc script?

[Optimum Wireless Services]

Thanks for replying.

I thought I could optimize these ruleset but, didn't have an idea how
to. Now, you gave me an idea on how to accomplish it. I'll read the two
links you sent to see how I can do this.

I hope you guys don't mind if I have questions about it later on.

Thanks.


On Thu, 2011-02-03 at 12:01 +0100, Marek Kierdelewicz wrote:
> >Hello.
> 
> Hi,
> 
> >I recently wrote a script that adds a new rule for an ip address each
> >time a new user is added to our network. I've noticed my tc rules work
> >...
> >Our network has about 120 users in total not all of these get connected
> >Are these rules ok?
> 
> If you have such linear ruleset (iptables marking+tc filter) for 120
> users then it's will not work well. If my theory is right, check top
> when there are more users logged in. You'll probably see hi cpu usage
> in "si"/"hi" (software/hardware interrupt) fraction.
> 
> You can easily optimize you ruleset by using:
> - tc u32 hashing filters [1] instead of iptables marking and fw tc
>   filters;
> - shape upload on ifb device [2] in ingress before nat, so you can use
>   tc u32 hashing filters too;
> 
> With such setup Core2 duo 3GHz + dual port intel nic can easily
> (68% peak cpu usage) route symmetric 400mbit of traffic, doing per user
> shaping, nat and access control for >2k users.
> 
> 
> [1] http://lartc.org/howto/lartc.adv-filter.hashing.html
> [2] http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb
> 
> Best regards,
> Marek Kierdelewicz

Re: How can I test my tc script?

[Marek Kierdelewicz]

Hi,

>I hope you guys don't mind if I have questions about it later on.

Don't hasitate to ask.

Best regards,
Marek Kierdelewicz

Re: How can I test my tc script?

[Optimum Wireless Services]

On Thu, 2011-02-03 at 12:01 +0100, Marek Kierdelewicz wrote:
> >Hello.
> 
> Hi,
> 
> >I recently wrote a script that adds a new rule for an ip address each
> >time a new user is added to our network. I've noticed my tc rules work
> >...
> >Our network has about 120 users in total not all of these get connected
> >Are these rules ok?
> 
> If you have such linear ruleset (iptables marking+tc filter) for 120
> users then it's will not work well. If my theory is right, check top
> when there are more users logged in. You'll probably see hi cpu usage
> in "si"/"hi" (software/hardware interrupt) fraction.
> 
> You can easily optimize you ruleset by using:
> - tc u32 hashing filters [1] instead of iptables marking and fw tc
>   filters;
> - shape upload on ifb device [2] in ingress before nat, so you can use
>   tc u32 hashing filters too;
> 
> With such setup Core2 duo 3GHz + dual port intel nic can easily
> (68% peak cpu usage) route symmetric 400mbit of traffic, doing per user
> shaping, nat and access control for >2k users.
> 
> 
> [1] http://lartc.org/howto/lartc.adv-filter.hashing.html

Based on the example in the above link, the authour creates 256 chains,
which I don't understand. This example also mentions that the ip
addresses are in the "basic", "medium", etc... plan and each of these
ips are in different rules.

My network is on 172.16.0.0 so, that means I need to take the same
aproach to create rules like this:

tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
  172.16.0.0 classid 1:1
tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
  172.16.1.0 classid 2:1
..........
tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
  172.16.100.0 classid 99:1

??? 

so that means I would have to write: 254x254 = 64516 rules ? that
doesn't seem correct.

Please correct me and clear my mind... (I guess I have to do a lot of
reading)


> [2] http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb
> 
> Best regards,
> Marek Kierdelewicz

Re: How can I test my tc script?

[Andrew Beverley]

On Wed, 2011-02-02 at 22:17 -0400, Optimum Wireless Services wrote:
> I would like to know how can I run some sort of benchmark test on a
> network that would simulate 10,20,50,100 users connected to see if my gw
> is handling the traffic well.
> 

I suggest using tc-viewer to see what your qdisc is actually doing
during peak use:

tc-viewer --iface=eth1 --zero --counter=1 --unit=kbit

> Here's a snip of what the script looks like:
> 
> ################ 
> Now Download interface
> 
> $tc qdisc del dev eth1 root
> $tc qdisc add dev eth1 root handle 2: htb r2q 1
> 
> $tc class add dev eth1 parent 2:0  classid 2:1 htb rate 15500kbit ceil
> 15525kbit
> 

Does the sum of all the leaf classes below add up to the root rate
above? From the HTB manual: "The rate supplied for a parent should be
the sum of the rates of its children."

This isn't necessarily your problem, but you should definitely make sure
you have it right.

> #for each user
> $iptables -t mangle -A POSTROUTING -d 172.16.100.11 -j MARK --set-mark
> 1009
> $iptables -t mangle -A POSTROUTING -d 172.16.100.11 -j RETURN
> $tc class   add dev eth1 parent 2:17    classid 2:1009 htb rate 739kbit
> ceil 850kbit prio 3

This doesn't look right to me. I see no parent 2:17 ?

> $tc filter  add dev eth1 parent 2:0     protocol ip prio 3 handle 1009
> fw classid 2:1009
> $tc qdisc   add dev eth1 parent 2:1009  handle 507: sfq perturb 2
> 
> 
> Are these rules ok?
> 

I suggest using tc-vewer as above. You'll then be able to see how much
traffic is going into each leaf class and get an idea if your rules are
working. You will also be able to compare the data rate in your qdisc
with the rate on your interface - you might even find that not all
traffic is going into the qdisc.

Andy

Re: How can I test my tc script?

[Optimum Wireless Services]

On Thu, 2011-02-03 at 12:01 +0100, Marek Kierdelewicz wrote:
> >Hello.
> 
> Hi,
> 
> >I recently wrote a script that adds a new rule for an ip address each
> >time a new user is added to our network. I've noticed my tc rules work
> >...
> >Our network has about 120 users in total not all of these get connected
> >Are these rules ok?
> 
> If you have such linear ruleset (iptables marking+tc filter) for 120
> users then it's will not work well. If my theory is right, check top
> when there are more users logged in. You'll probably see hi cpu usage
> in "si"/"hi" (software/hardware interrupt) fraction.
> 
> You can easily optimize you ruleset by using:
> - tc u32 hashing filters [1] instead of iptables marking and fw tc
>   filters;
> - shape upload on ifb device [2] in ingress before nat, so you can use
>   tc u32 hashing filters too;
> 
> With such setup Core2 duo 3GHz + dual port intel nic can easily
> (

> 68% peak cpu usage) route symmetric 400mbit of traffic, doing per user
> shaping, nat and access control for >2k users.
> 
> 
> [1] http://lartc.org/howto/lartc.adv-filter.hashing.html
> [2] http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb
> 

After reading the lartc and the linuxfoundation.org documentation about
the u32 hashing filters it leaves me very confused. I don't get how I
would write my ruleset for all the users I have.

Would I have to create a table for each ip and then create a filter to
match? for ie:

tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
  172.16.100.1 classid 1:1

tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
        match ip src 172.16.100.1 flowid 1:1


tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 800:: \
        match ip src 172.16.100.1/16 \
        hashkey mask 0x000000ff at 12 \
        link 2:


This method is somewhat complicated but, according to the documentation
"very worth it". 

Can someone please explain it a little better, thanks.



> Best regards,
> Marek Kierdelewicz
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Michele Petrazzo - Unipex]

Marek Kierdelewicz ha scritto:
>> Hello.
>
> Hi,
>
>> I recently wrote a script that adds a new rule for an ip address
>> each time a new user is added to our network. I've noticed my tc
>> rules work ... Our network has about 120 users in total not all of
>>  these get connected Are these rules ok?
>
> If you have such linear ruleset (iptables marking+tc filter) for 120
> users then it's will not work well. If my theory is right, check top
> when there are more users logged in. You'll probably see hi cpu usage
> in "si"/"hi" (software/hardware interrupt) fraction.
>

I, are you sure about the big difference?
I agree with you about tc u32 filters are better than iptables +
marking, but like now I have about 100mbit 1k+ users with
mangle+classify and tc+htb+egress and I have a load at about 0.2.
My cpu are xeon 2.6 quad.

In my experience, instead, I see that nic can make big differences about
throughput

Michele

Re: How can I test my tc script?

[Optimum Wireless Services]

Ok. I'm trying to learn how to work with the u32 selector, I would like
to know if I'm doing this correctly:

Lets say this is my download ruleset:

$tc qdisc del dev eth0 root
$tc qdisc add dev eth0 root handle 1: htb r2q 1

# this is the parent class
$tc class add dev eth0 parent 1:0  classid 1:1 htb rate 3000kbit ceil
3000kbit



## I would like to assign users 256kbit for download speed, so the
following rule is a leaf of the parent 1:0 (correct me if wrong, please)

$tc class   add dev eth0 parent 1:100    classid 1:1 htb rate 256kbit
ceil 256kbit prio 3

## Now I should create a filter for the ip address:

$tc filter add dev eth0 parent 1:100 protocol ip u32 match ip dst
172.16.100.1/32 classid 1:1

## Now for the qdisc
$tc qdisc add dev eth0 parent 1:100 handle 500: sfq perturb 2

Are the parent values correct on the filter,class, and qdisc?

Is this correct? Am I in the right path?

Thanks in advanced for taking your time to help me.





On Fri, 2011-02-04 at 09:45 -0400, Optimum Wireless Services wrote:
> On Thu, 2011-02-03 at 12:01 +0100, Marek Kierdelewicz wrote:
> > >Hello.
> > 
> > Hi,
> > 
> > >I recently wrote a script that adds a new rule for an ip address each
> > >time a new user is added to our network. I've noticed my tc rules work
> > >...
> > >Our network has about 120 users in total not all of these get connected
> > >Are these rules ok?
> > 
> > If you have such linear ruleset (iptables marking+tc filter) for 120
> > users then it's will not work well. If my theory is right, check top
> > when there are more users logged in. You'll probably see hi cpu usage
> > in "si"/"hi" (software/hardware interrupt) fraction.
> > 
> > You can easily optimize you ruleset by using:
> > - tc u32 hashing filters [1] instead of iptables marking and fw tc
> >   filters;
> > - shape upload on ifb device [2] in ingress before nat, so you can use
> >   tc u32 hashing filters too;
> > 
> > With such setup Core2 duo 3GHz + dual port intel nic can easily
> > (
> 
> > 68% peak cpu usage) route symmetric 400mbit of traffic, doing per user
> > shaping, nat and access control for >2k users.
> > 
> > 
> > [1] http://lartc.org/howto/lartc.adv-filter.hashing.html
> > [2] http://www.linuxfoundation.org/collaborate/workgroups/networking/ifb
> > 
> 
> After reading the lartc and the linuxfoundation.org documentation about
> the u32 hashing filters it leaves me very confused. I don't get how I
> would write my ruleset for all the users I have.
> 
> Would I have to create a table for each ip and then create a filter to
> match? for ie:
> 
> tc filter add dev eth1 parent 1:0 protocol ip prio 100 match ip src \
>   172.16.100.1 classid 1:1
> 
> tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 2:7b: \
>         match ip src 172.16.100.1 flowid 1:1
> 
> 
> tc filter add dev eth1 protocol ip parent 1:0 prio 5 u32 ht 800:: \
>         match ip src 172.16.100.1/16 \
>         hashkey mask 0x000000ff at 12 \
>         link 2:
> 
> 
> This method is somewhat complicated but, according to the documentation
> "very worth it". 
> 
> Can someone please explain it a little better, thanks.
> 
> 
> 
> > Best regards,
> > Marek Kierdelewicz
> > --
> > To unsubscribe from this list: send the line "unsubscribe netfilter" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Optimum Wireless Services]

Hi, is me once again.

I tried this for a test box and the download speed worked correctly:

$tc qdisc del dev eth0 root
$tc class del dev eth0

$tc qdisc add dev eth0 root handle 1: htb r2q 1

$tc class add dev eth0 parent 1:0  classid 1:1 htb rate 3000kbit ceil
3000kbit

$tc class add dev eth0 parent 1:1 classid 1:100 htb rate 256kbit ceil
256kbit prio 3

$tc filter add dev eth0 parent 1: protocol ip u32 match ip dst
172.16.254.1/32 classid 1:100

$tc qdisc add dev eth0 parent 1:100 handle 100: sfq perturb 2


So, are these ruleset more convenient than the one I currently have?






On Fri, 2011-02-04 at 12:49 -0400, Optimum Wireless Services wrote:
> $tc qdisc del dev eth0 root
> $tc qdisc add dev eth0 root handle 1: htb r2q 1
> 
> # this is the parent class
> $tc class add dev eth0 parent 1:0  classid 1:1 htb rate 3000kbit ceil
> 3000kbit
> 
> 
> 
> ## I would like to assign users 256kbit for download speed, so the
> following rule is a leaf of the parent 1:0 (correct me if wrong,
> please)
> 
> $tc class   add dev eth0 parent 1:100    classid 1:1 htb rate 256kbit
> ceil 256kbit prio 3
> 
> ## Now I should create a filter for the ip address:
> 
> $tc filter add dev eth0 parent 1:100 protocol ip u32 match ip dst
> 172.16.100.1/32 classid 1:1
> 
> ## Now for the qdisc
> $tc qdisc add dev eth0 parent 1:100 handle 500: sfq perturb 2
> 
> Are the parent values correct on the filter,class, and qdisc?
> 
> Is this correct? Am I in the right path?
> 
> Thanks in advanced for taking your time to help me.

Re: How can I test my tc script?

[Andrew Beverley]

On Fri, 2011-02-04 at 15:22 +0100, Michele Petrazzo - Unipex wrote:
> Marek Kierdelewicz ha scritto:
> >> Hello.
> >
> > Hi,
> >
> >> I recently wrote a script that adds a new rule for an ip address
> >> each time a new user is added to our network. I've noticed my tc
> >> rules work ... Our network has about 120 users in total not all of
> >>  these get connected Are these rules ok?
> >
> > If you have such linear ruleset (iptables marking+tc filter) for 120
> > users then it's will not work well. If my theory is right, check top
> > when there are more users logged in. You'll probably see hi cpu usage
> > in "si"/"hi" (software/hardware interrupt) fraction.
> >
> 
> I, are you sure about the big difference?
> I agree with you about tc u32 filters are better than iptables +
> marking, but like now I have about 100mbit 1k+ users with
> mangle+classify and tc+htb+egress and I have a load at about 0.2.
> My cpu are xeon 2.6 quad.
> 

I agree. My CPU load is very low for similar setups.

I think the problems that are being experienced are errors in the way
that HTB is being used (as per my last post). Although you might be able
to fix your problems by changing your scripts entirely as above, you
might want to try fixing your original scripts instead.

Andy

Re: How can I test my tc script?

[Optimum Wireless Services]

On Thu, 2011-02-03 at 19:38 +0000, Andrew Beverley wrote:
> On Wed, 2011-02-02 at 22:17 -0400, Optimum Wireless Services wrote:
> > I would like to know how can I run some sort of benchmark test on a
> > network that would simulate 10,20,50,100 users connected to see if my gw
> > is handling the traffic well.
> > 
> 
> I suggest using tc-viewer to see what your qdisc is actually doing
> during peak use:
> 
> tc-viewer --iface=eth1 --zero --counter=1 --unit=kbit
> 
> > Here's a snip of what the script looks like:
> > 
> > ################ 
> > Now Download interface
> > 
> > $tc qdisc del dev eth1 root
> > $tc qdisc add dev eth1 root handle 2: htb r2q 1
> > 
> > $tc class add dev eth1 parent 2:0  classid 2:1 htb rate 15500kbit ceil
> > 15525kbit
> > 
> 
> Does the sum of all the leaf classes below add up to the root rate
> above? From the HTB manual: "The rate supplied for a parent should be
> the sum of the rates of its children."
> 

Hi Andy.

The sum of my users' classes or leafs rate would not be equal to the
parent's. They will never add up to the total root class since I will be
adding rules every once in a while so, if my root total BW is 15Mbps and
I have 100 users at 512kbps each then, that could be a problem. 

Is there something else to do instead?

Re: How can I test my tc script?

[Optimum Wireless Services]

I apologize for sending so many emails regarding the same topic but, I'm
trying to get this right.

I finally (I think) changed my previous script to write different
filters:

## Download
## Parent ID: 2, Associated with iface: eth1 -- LAN Interface.
$tc qdisc del dev eth1 root
$tc qdisc add dev eth1 root handle 2: htb r2q 1

$tc class add dev eth1 parent 2:0  classid 2:1 htb rate 15000kbit ceil
15000kbit

##  User Name
$tc class   add dev eth1 parent 2:11    classid 2:1001 htb rate 739kbit
ceil 739kbit prio 3
$tc filter  add dev eth1 parent 2:0     protocol ip u32 match ip dst
172.16.100.11/32 classid 2:1001
$tc qdisc   add dev eth1 parent 2:1001  handle 501: sfq perturb 2

etc....

All users have basically the same format.

Now, for the upload I cant get it shape with squid running in the
middle. If I remove squid then, traffic shaping works ok.

Andy recommended to use ifb but, I don't know where to get the download
from. Is supposed to be on kernel 2.26.20 and up and I'm running a
kernel that "qualifies" but, cant really get ifb to do anything.

Well, time to go to bed. Please correct me if I have something wrong
with the rule set.

Thanks.



On Fri, 2011-02-04 at 22:44 -0400, Optimum Wireless Services wrote:
> On Thu, 2011-02-03 at 19:38 +0000, Andrew Beverley wrote:
> > On Wed, 2011-02-02 at 22:17 -0400, Optimum Wireless Services wrote:
> > > I would like to know how can I run some sort of benchmark test on a
> > > network that would simulate 10,20,50,100 users connected to see if my gw
> > > is handling the traffic well.
> > > 
> > 
> > I suggest using tc-viewer to see what your qdisc is actually doing
> > during peak use:
> > 
> > tc-viewer --iface=eth1 --zero --counter=1 --unit=kbit
> > 
> > > Here's a snip of what the script looks like:
> > > 
> > > ################ 
> > > Now Download interface
> > > 
> > > $tc qdisc del dev eth1 root
> > > $tc qdisc add dev eth1 root handle 2: htb r2q 1
> > > 
> > > $tc class add dev eth1 parent 2:0  classid 2:1 htb rate 15500kbit ceil
> > > 15525kbit
> > > 
> > 
> > Does the sum of all the leaf classes below add up to the root rate
> > above? From the HTB manual: "The rate supplied for a parent should be
> > the sum of the rates of its children."
> > 
> 
> Hi Andy.
> 
> The sum of my users' classes or leafs rate would not be equal to the
> parent's. They will never add up to the total root class since I will be
> adding rules every once in a while so, if my root total BW is 15Mbps and
> I have 100 users at 512kbps each then, that could be a problem. 
> 
> Is there something else to do instead?
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Andrew Beverley]

On Sat, 2011-02-05 at 02:07 -0400, Optimum Wireless Services wrote:
> Now, for the upload I cant get it shape with squid running in the
> middle. If I remove squid then, traffic shaping works ok.
> 

If you are doing egress shaping on the internet facing interface for the
upload (which I assume you are - you can't use HTB on ingress), then it
will not work for Squid. That's because Squid's traffic leaving the
server is coming from the server's IP address, not from the client's IP
address (Squid is generating the traffic, not the client). Therefore,
your filter will never match the traffic. Normal traffic from a client
is forwarded and retains its source IP address.

One way to shape Squid traffic per client source IP address would be to
mark the packets that are coming in from the local network (as you were
doing before) and then use Squid's mark preservation feature to retain
the packet's mark. However, you can't use U32 classifiers - you'll have
to go back to marking packets.

> Andy recommended to use ifb but, I don't know where to get the download
> from. Is supposed to be on kernel 2.26.20 and up and I'm running a
> kernel that "qualifies" but, cant really get ifb to do anything.

No need for IFB in your case. That's only if you want to shape multiple
interfaces within one qdisc.

Andy

Re: How can I test my tc script?

[Andrew Beverley]

On Fri, 2011-02-04 at 15:57 -0400, Optimum Wireless Services wrote:
> Hi, is me once again.
> 
> I tried this for a test box and the download speed worked correctly:
> 
> $tc qdisc del dev eth0 root
> $tc class del dev eth0
> 
> $tc qdisc add dev eth0 root handle 1: htb r2q 1
> 
> $tc class add dev eth0 parent 1:0  classid 1:1 htb rate 3000kbit ceil
> 3000kbit
> 
> $tc class add dev eth0 parent 1:1 classid 1:100 htb rate 256kbit ceil
> 256kbit prio 3
> 
> $tc filter add dev eth0 parent 1: protocol ip u32 match ip dst
> 172.16.254.1/32 classid 1:100
> 
> $tc qdisc add dev eth0 parent 1:100 handle 100: sfq perturb 2
> 
> 
> So, are these ruleset more convenient than the one I currently have?
> 

They are more efficient, yes, and in some ways more convenient, but you
lose some of the flexibility you had with marking packets.

Horses for courses really.

Andy

Re: How can I test my tc script?

[Andrew Beverley]

On Fri, 2011-02-04 at 22:44 -0400, Optimum Wireless Services wrote:
> On Thu, 2011-02-03 at 19:38 +0000, Andrew Beverley wrote:
> > On Wed, 2011-02-02 at 22:17 -0400, Optimum Wireless Services wrote:
> > > I would like to know how can I run some sort of benchmark test on a
> > > network that would simulate 10,20,50,100 users connected to see if my gw
> > > is handling the traffic well.
> > > 
> > 
> > I suggest using tc-viewer to see what your qdisc is actually doing
> > during peak use:
> > 
> > tc-viewer --iface=eth1 --zero --counter=1 --unit=kbit
> > 
> > > Here's a snip of what the script looks like:
> > > 
> > > ################ 
> > > Now Download interface
> > > 
> > > $tc qdisc del dev eth1 root
> > > $tc qdisc add dev eth1 root handle 2: htb r2q 1
> > > 
> > > $tc class add dev eth1 parent 2:0  classid 2:1 htb rate 15500kbit ceil
> > > 15525kbit
> > > 
> > 
> > Does the sum of all the leaf classes below add up to the root rate
> > above? From the HTB manual: "The rate supplied for a parent should be
> > the sum of the rates of its children."
> > 
> 
> Hi Andy.
> 
> The sum of my users' classes or leafs rate would not be equal to the
> parent's. They will never add up to the total root class since I will be
> adding rules every once in a while so, if my root total BW is 15Mbps and
> I have 100 users at 512kbps each then, that could be a problem. 
> 

Hmmm, well HTB is known to misbehave if they don't all add up correctly,
which could explain your problems. Why don't you *try* it with them all
adding up correctly, just temporarily for your current number of users,
to see if this is the problem.

> Is there something else to do instead?
> 

You could have a look at another classful qdisc such as HFSC or CBQ.
However, I've never played with them, so they may not be of any more
help.

Andy

Re: How can I test my tc script?

[Marek Kierdelewicz]

> Hi,

Hi

>I, are you sure about the big difference?
>I agree with you about tc u32 filters are better than iptables +
>marking, but like now I have about 100mbit 1k+ users with
>mangle+classify and tc+htb+egress and I have a load at about 0.2.
>My cpu are xeon 2.6 quad.

Do you have individual iptables rule doing marking for each user (1k+
rules in mangle)?

Best regards,
Marek Kierdelewicz

Re: How can I test my tc script?

[Andrew Beverley]

On Sat, 2011-02-05 at 15:58 +0100, Marek Kierdelewicz wrote:
> > Hi,
> 
> Hi
> 
> >I, are you sure about the big difference?
> >I agree with you about tc u32 filters are better than iptables +
> >marking, but like now I have about 100mbit 1k+ users with
> >mangle+classify and tc+htb+egress and I have a load at about 0.2.
> >My cpu are xeon 2.6 quad.
> 
> Do you have individual iptables rule doing marking for each user (1k+
> rules in mangle)?
> 

I have to admit that I don't mark per user. I have lots of users, but I
instead mark the type of traffic and shape on that.

Optimum Wireless Services - what is your aim? Do you really want to
limit bandwidth per user, or do you instead want to provide each user
with a fair share of bandwidth and fast internet access? If it is the
latter, you are better to not mark per user, but instead mark the
traffic type and use tc's hash functionality to share the bandwidth
between client IP address, rather than the default which is per
connection.

Andy

good tutorial

[J Webster]

Does anyone have a tutorial recommendation for tc?
I tried the existing tutorials on LARTC and they were all lacking 
examples - also the pages look like they were written 10 years ago.
I want to limit bandwidth per ip address on my server.

Re: How can I test my tc script?

[Marek Kierdelewicz]

Hi,

>I have to admit that I don't mark per user. I have lots of users, but I
>instead mark the type of traffic and shape on that.

That's the reason for good performance in your case.

Best regards,
Marek Kierdelewicz

Re: good tutorial

[Andrew Beverley]

On Sat, 2011-02-05 at 17:48 +0100, J Webster wrote:
> Does anyone have a tutorial recommendation for tc?
> I tried the existing tutorials on LARTC and they were all lacking 
> examples - also the pages look like they were written 10 years ago.
> I want to limit bandwidth per ip address on my server.

If you check the archives there's a discussion going on right now about
this...

You can use a classful qdisc such as HTB, create a leaf for each IP
address, then filter into that leaf using the U32 filters.

Something like this (untested - I've only copied from the list
archives):


tc qdisc add dev eth0 root handle 1: htb r2q 1
tc class add dev eth0 parent 1:0  classid 1:1 htb \
	rate 3000kbit ceil 3000kbit

tc class add dev eth0 parent 1:1 classid 1:100 htb \
	rate 1500kbit ceil 1500kbit prio 3
tc filter add dev eth0 parent 1: protocol ip u32 \
	match ip dst 172.16.254.1/32 classid 1:100
tc qdisc add dev eth0 parent 1:100 handle 100: sfq perturb 2

tc class add dev eth0 parent 1:1 classid 1:101 htb \
	rate 1500kbit ceil 1500kbit prio 3
tc filter add dev eth0 parent 1: protocol ip u32 \
	match ip dst 172.16.254.2/32 classid 1:101
tc qdisc add dev eth0 parent 1:101 handle 101: sfq perturb 2

...


You'll need a leaf class for each IP address, and all the leaf classes
should add up to the parent.

By the way - please start a new email when starting a new thread. If you
reply to a previous email and change the subject then your email will
still appear in the original thread in the message archives.

Andy

Re: How can I test my tc script?

[Marek Kierdelewicz]

Hi,

>$tc qdisc add dev eth0 root handle 1: htb r2q 1

You can add "default XX" to define direct class as 1:XX. Any traffic
that doesn't get classified by specific filters will end up in class
1:XX. If you wonder why XX is prefixed by "1:", it's because of the
qdisc handle.

># this is the parent class
>$tc class add dev eth0 parent 1:0  classid 1:1 htb rate 3000kbit ceil
>3000kbit

The class you created here has classid of 1:1 and is attached to qdisc
itself.

>$tc class   add dev eth0 parent 1:100    classid 1:1 htb rate 256kbit
>ceil 256kbit prio 3

1) parent should be 1:1, that's classid of the class you created
earlier,
2) classid should be 1:100

that's probably what you wanted:
$tc class   add dev eth0 parent 1:1 classid 1:100 htb rate 256kbit
ceil 256kbit prio 3

>## Now I should create a filter for the ip address:
>$tc filter add dev eth0 parent 1:100 protocol ip u32 match ip dst
>172.16.100.1/32 classid 1:1

1) parent should be set to 1:1, that's the class you attach filter to
2) classid should be 1:100, that's the class, traffic should be
directed to

>## Now for the qdisc
>$tc qdisc add dev eth0 parent 1:100 handle 500: sfq perturb 2

that's ok

Best regards,
Marek Kierdelewicz

Re: How can I test my tc script?

[Marek Kierdelewicz]

>Hi, is me once again.

Hi,

>I tried this for a test box and the download speed worked correctly:
>...

I see you already nailed it ;-).

>So, are these ruleset more convenient than the one I currently have?

Simple u32 won't scale any better then ipt marking + tc fw filter. Real
performance gain is in use of hashing filters.

Best regards,
Marek Kierdelewicz

Re: good tutorial

[J Webster]

Thanks.
Out of interest will this only limit connects incoming to the server and 
not outgoing?
For example, a user connects via VPN to the server, their connection 
should be limited to 2Mbps (3in the example below) but they are 
connecting to say www.youtube.com. I do not want the connection to 
youtube.com to be restricted as maybe 10 users might be accessing 
youtube at the same time.
So the server can have an unlimited outgoing connection to youtube but 
when it passes on the connection to the client (much like a proxy server 
might do), they should only have 2Mbps.

Now the reason I asked for a tutorial somewhere is I went through the 
LARTC tutorial (which just looks like a blank white document) and my 
understanding of leaf, HTB, U32, qdisc, etc is kind of lacking :)

On 05/02/11 20:22, Andrew Beverley wrote:
> On Sat, 2011-02-05 at 17:48 +0100, J Webster wrote:
>> Does anyone have a tutorial recommendation for tc?
>> I tried the existing tutorials on LARTC and they were all lacking
>> examples - also the pages look like they were written 10 years ago.
>> I want to limit bandwidth per ip address on my server.
> If you check the archives there's a discussion going on right now about
> this...
>
> You can use a classful qdisc such as HTB, create a leaf for each IP
> address, then filter into that leaf using the U32 filters.
>
> Something like this (untested - I've only copied from the list
> archives):
>
>
> tc qdisc add dev eth0 root handle 1: htb r2q 1
> tc class add dev eth0 parent 1:0  classid 1:1 htb \
> 	rate 3000kbit ceil 3000kbit
>
> tc class add dev eth0 parent 1:1 classid 1:100 htb \
> 	rate 1500kbit ceil 1500kbit prio 3
> tc filter add dev eth0 parent 1: protocol ip u32 \
> 	match ip dst 172.16.254.1/32 classid 1:100
> tc qdisc add dev eth0 parent 1:100 handle 100: sfq perturb 2
>
> tc class add dev eth0 parent 1:1 classid 1:101 htb \
> 	rate 1500kbit ceil 1500kbit prio 3
> tc filter add dev eth0 parent 1: protocol ip u32 \
> 	match ip dst 172.16.254.2/32 classid 1:101
> tc qdisc add dev eth0 parent 1:101 handle 101: sfq perturb 2
>
> ...
>
>
> You'll need a leaf class for each IP address, and all the leaf classes
> should add up to the parent.
>
> By the way - please start a new email when starting a new thread. If you
> reply to a previous email and change the subject then your email will
> still appear in the original thread in the message archives.
>
> Andy
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: good tutorial

[Bob Miller]

> 
> Now the reason I asked for a tutorial somewhere is I went through the 
> LARTC tutorial (which just looks like a blank white document) and my 
> understanding of leaf, HTB, U32, qdisc, etc is kind of lacking :)
> 

I have implemented shaping on a couple of occasions, I found using tcng
to generate the tc script greatly simplified the whole question of
qdisc, htb, etc, for me.  I note that I have never seen anyone mention
tcng on this list, and it causes me to wonder if there is a reason it is
not commonly used?

Bob Miller
334-7117/660-5315
http://computerisms.ca
bob@computerisms.ca
Network, Internet, Server,
and Open Source Solutions

Re: How can I test my tc script?

[Marek Kierdelewicz]

Hi,

>This method is somewhat complicated but, according to the documentation
>"very worth it". 
>Can someone please explain it a little better, thanks.

Sure... here goes the example:

We want to share 100mbit connection. Internet access is connected to
the router on eth0 and lan in on eth1. We do egress shaping of download
traffic on eth1.

---------------------complete script------------------------------
tc qdisc add dev eth1 root handle 1: htb default 1
tc class add dev eth1 parent 1: classid 1:1 htb rate 100Mbit
tc filter add dev eth1 parent 1:1 prio 11 handle 2: protocol ip \
u32 divisor 256 
tc filter add dev eth1 protocol ip parent 1:1 prio 11 u32 match \
ip dst 0.0.0.0/0 hashkey mask 0x000000ff at 16 link 2:

tc class add dev eth1 parent 1:1 classid 1:400 htb rate 5mbit
tc qdisc add dev eth1 parent 1:400 handle 400: sfq perturb 1
tc filter add dev eth1 protocol ip prio 11 u32 ht 2:46: match ip \
dst 10.5.0.70 flowid 1:400

tc class add dev eth1 parent 1:1 classid 1:401 htb rate 5mbit
tc qdisc add de eth1 parent 1:401 handle 401: sfq perturb 1
tc filter add de0 eth1 protocol ip prio 11 u32 ht 2:a: match ip \
dst 10.5.0.10 flowid 1:401
---------------------------------------------------------------------

Now lets go line by line.

>tc qdisc add dev eth1 root handle 1: htb default 1

add htb qdisc to eth1, direct all unclassified traffic to 1:1 by default

>tc class add dev eth1 parent 1: classid 1:1 htb rate 100Mbit

add main class 1:1 with rate 100mbit; we don't specify ceil, it's
automaticaly set to rate if ommited

>tc filter add dev eth1 parent 1:1 prio 11 handle 2: protocol ip \
>u32 divisor 256 

add hash table with 256 entries (maximal size), table is located at "2:"

>tc filter add dev eth1 protocol ip parent 1:1 prio 11 u32 match \
>ip dst 0.0.0.0/0 hashkey mask 0x000000ff at 16 link 2:

this filter defines how information in ip packet translates to entry in
hash table; 

"u32 match ip dst 0.0.0.0/0" - matches any ip traffic;

"hashkey mask 0x000000ff at 16" - number 16 defines location in ip
header [1] and it's destination IP addres; mask 0x000000ff defines that
we're interested in last octet of the destination ip address; example,
if the packet is directed to 10.0.0.192, then te mask will yild value
192; note: mask returns values in range of 0-255 - it coresponds with
divisor setting in previous rule;

"link 2:" - direct packets to table 2: (added in previous rule);

[1]
http://www.siongboon.com/projects/2006-03-06_serial_communication/IP-Header-v4.png

>tc class add dev eth1 parent 1:1 classid 1:400 htb rate 5mbit

add class fo this client, let if be 1:400;

>tc qdisc add dev eth1 parent 1:400 handle 400: sfq perturb 1

attach sfq qdisc to client's class

>tc filter add dev eth1 protocol ip prio 11 u32 ht 2:46: match ip \
>dst 10.5.0.70 flowid 1:400

add filter that directs traffic to 10.5.0.70 to class 1:400;
interesting parts:

"ht 2:46:" - use hash table 2:, 46 is the last octet of clients ip
address converted from decimal to hexadecimal notation; decimal 70 is
equal 0x46 hexadecimal;

>tc class add dev eth1 parent 1:1 classid 1:401 htb rate 5mbit

add class of second client, let if be 1:401;

>tc qdisc add dev eth1 parent 1:401 handle 401: sfq perturb 1

... sfq

>tc filter add dev eth1 protocol ip prio 11 u32 ht 2:a: match ip \
>dst 10.5.0.10 flowid 1:401

add filter that directs traffic to 10.5.0.10 to class 1:401;
interesting parts:

"ht 2:a:" - use hash table 2:, a is the last octet of clients ip
address converted from decimal to hexadecimal notation; decimal 10 is
equal 0xa hexadecimal;



I hope it'll clear things a bit.

Best regards,
Marek Kierdelewicz

Re: good tutorial

[Andrew Beverley]

On Sat, 2011-02-05 at 20:33 +0100, J Webster wrote:
> Thanks.
> Out of interest will this only limit connects incoming to the server and 
> not outgoing?

It will limit the *outgoing* traffic on whichever interface you attach
it to, but from your perspective this may actually result in traffic
incoming to the server (which I guess is what you want).

So if you attach to your LAN interface, and you actually mean "will this
only limit traffic being downloaded from the internet", then the answer
is yes.

> For example, a user connects via VPN to the server, their connection 
> should be limited to 2Mbps (3in the example below) but they are 
> connecting to say www.youtube.com. I do not want the connection to 
> youtube.com to be restricted as maybe 10 users might be accessing 
> youtube at the same time.

It won't in the example. However, even if you were doing egress shaping
on the internet interface, you'd have to specifically specify youtube's
IP address for the above to happen anyway.

> So the server can have an unlimited outgoing connection to youtube but 
> when it passes on the connection to the client (much like a proxy server 
> might do), they should only have 2Mbps.

Sort of - when you say "outgoing" connection to youtube, I assume you
mean traffic "incoming" from youtube to the server. So if you egress
shape on the LAN interface, then there will be no restrictions for the
overall traffic coming the internet, but the client will be limited
depending on your filters. 

Remember that traffic goes both ways - you'll need to egress shape on
the internet interface if you want to also limit the traffic to the
internet (ie uploads).

> Now the reason I asked for a tutorial somewhere is I went through the 
> LARTC tutorial (which just looks like a blank white document) and my 
> understanding of leaf, HTB, U32, qdisc, etc is kind of lacking :)

Well I found this to be pretty good:

http://www.opalsoft.net/qos/DS.htm

Andy

Re: good tutorial

[Andrew Beverley]

On Sat, 2011-02-05 at 11:51 -0800, Bob Miller wrote:
> > 
> > Now the reason I asked for a tutorial somewhere is I went through the 
> > LARTC tutorial (which just looks like a blank white document) and my 
> > understanding of leaf, HTB, U32, qdisc, etc is kind of lacking :)
> > 
> 
> I have implemented shaping on a couple of occasions, I found using tcng
> to generate the tc script greatly simplified the whole question of
> qdisc, htb, etc, for me.  I note that I have never seen anyone mention
> tcng on this list, and it causes me to wonder if there is a reason it is
> not commonly used?

I seem to remember I looked at it, but preferred the flexibility of
writing rules myself. Good point though - for the needs of most it looks
to be useful. Is it still maintained though? The Sourceforge site
doesn't look like it's been touched for a few years (not that that
necessarily matters of course)...

Andy

Re: How can I test my tc script?

[Marek Kierdelewicz]

And here's a bit more complex example for shaping upload with tc u32
hashing filters and IFB.

As in download example, we want to share 100mbit connection. Internet
access is connected to the router on eth0 and lan in on eth1. We do
ingress shaping of upload traffic on ifb0 after ingress eth1 traffic
is redirected to the device.

---------------------complete script------------------------------
tc qdisc add dev ifb0 root handle 1: htb default 1
tc class add dev ifb0 parent 1: classid 1:1 htb rate 100Mbit
tc filter add dev ifb0 parent 1:1 prio 11 handle 2: protocol ip \
u32 divisor 256
tc filter add dev ifb0 protocol ip parent 1:1 prio 11 u32 match \
ip dst 0.0.0.0/0 hashkey mask 0x000000ff at 12 link 2:

tc class add dev ifb0 parent 1:1 classid 1:400 htb rate 5mbit
tc qdisc add dev ifb0 parent 1:400 handle 400: sfq perturb 1
tc filter add dev ifb0 protocol ip prio 11 u32 ht 2:46: match ip \
src 10.5.0.70 flowid 1:400

tc class add dev ifb0 parent 1:1 classid 1:401 htb rate 5mbit
tc qdisc add dev ifb0 parent 1:401 handle 401: sfq perturb 1
tc filter add dev ifb0 protocol ip prio 11 u32 ht 2:a: match ip \
src 10.5.0.10 flowid 1:401

tc qdisc add dev eth1 ingress
tc filter add dev eth1 protocol ip parent ffff: prio 4 u32 match \
ip dst 0.0.0.0/0 flowid :1 action mirred egress redirect dev ifb0
---------------------------------------------------------------------

Now lets go line by line.

>tc qdisc add dev eth1 root handle 1: htb default 1
>tc class add dev eth1 parent 1: classid 1:1 htb rate 100Mbit
>tc filter add dev eth1 parent 1:1 prio 11 handle 2: protocol ip \
>u32 divisor 256 

analogous to download example

>tc filter add dev eth1 protocol ip parent 1:1 prio 11 u32 match \
>ip dst 0.0.0.0/0 hashkey mask 0x000000ff at 12 link 2:

"at 12" instead of 16, because we want to hash on source ip address

>tc class add dev eth1 parent 1:1 classid 1:400 htb rate 5mbit
>tc qdisc add dev eth1 parent 1:400 handle 400: sfq perturb 1

analogous to download example

>tc filter add dev eth1 protocol ip prio 11 u32 ht 2:46: match ip \
>src 10.5.0.70 flowid 1:400

compared to download shaping, dst is changed to src

>tc class add dev ifb0 parent 1:1 classid 1:401 htb rate 5mbit
>tc qdisc add dev ifb0 parent 1:401 handle 401: sfq perturb 1
>tc filter add dev ifb0 protocol ip prio 11 u32 ht 2:a: match ip \
>src 10.5.0.10 flowid 1:401

analogous to previous client

>tc qdisc add dev eth1 ingress

add ingress qdisc to eth1 interface

>tc filter add dev eth1 protocol ip parent ffff: prio 4 u32 match \
>ip dst 0.0.0.0/0 flowid :1 action mirred egress redirect dev ifb0

redirect any ip traffic from ingress qdisc on device eth1 to ifb0;
note! redirection to ifb doesn't interfere with routing/firewalling/nat;

Best regards,
Marek Kierdelewicz
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Optimum Wireless Services]

Thank you all for replying and sorry for the late reply.

On Sat, 2011-02-05 at 16:37 +0000, Andrew Beverley wrote:
> On Sat, 2011-02-05 at 15:58 +0100, Marek Kierdelewicz wrote:
> > > Hi,
> > 
> > Hi
> > 
> > >I, are you sure about the big difference?
> > >I agree with you about tc u32 filters are better than iptables +
> > >marking, but like now I have about 100mbit 1k+ users with
> > >mangle+classify and tc+htb+egress and I have a load at about 0.2.
> > >My cpu are xeon 2.6 quad.
> > 
> > Do you have individual iptables rule doing marking for each user (1k+
> > rules in mangle)?
> > 
> 
> I have to admit that I don't mark per user. I have lots of users, but I
> instead mark the type of traffic and shape on that.
> 
> Optimum Wireless Services - what is your aim? Do you really want to
> limit bandwidth per user, or do you instead want to provide each user
> with a fair share of bandwidth and fast internet access? If it is the
> latter, you are better to not mark per user, but instead mark the
> traffic type and use tc's hash functionality to share the bandwidth
> between client IP address, rather than the default which is per
> connection.
> 

What I would love to do is offer "fast internet access" and at the same
time sell packages to users: 512/128, 768/256, 1024/512, etc... If an
user has a specific package, I would like this user to have a nice
internet experience with low pings, and little lagging and not
'interfere' with the other users (might be impossible). 

So, what I'm trying to do is not have issues with one user hogging our
internet line.

Dont know how a fair share bandwidth might work on our network since I
have some users downloading p2p all day long. 

I've noticed that at peak times (7pm - 11pm) our internet service is so
slow I'm embarrassed to even say. 

Also, another thing I noticed (this one not time specific) when running
iptraf is that my external interface measures almost our max/total
bandwidth of 15Mbps and our internal interface is only at 4Mbps. So,
that tells me that not much is passing through squid or there are some
big time downloading going on.

So, to tell you the truth I don't know which way to go if limit bw per
user or what. I'm just trying to keep our network running as smooth as
possible.


> Andy
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Optimum Wireless Services]

On Sat, 2011-02-05 at 12:27 +0000, Andrew Beverley wrote:
> On Sat, 2011-02-05 at 02:07 -0400, Optimum Wireless Services wrote:
> > Now, for the upload I cant get it shape with squid running in the
> > middle. If I remove squid then, traffic shaping works ok.
> > 
> 
> If you are doing egress shaping on the internet facing interface for the
> upload (which I assume you are - you can't use HTB on ingress), then it
> will not work for Squid. That's because Squid's traffic leaving the
> server is coming from the server's IP address, not from the client's IP
> address (Squid is generating the traffic, not the client). Therefore,
> your filter will never match the traffic. Normal traffic from a client
> is forwarded and retains its source IP address.
> 

Do you think I should go back to this method?

I have 120 users now but, that number should be increasing. Which method
do you recommend?



> One way to shape Squid traffic per client source IP address would be to
> mark the packets that are coming in from the local network (as you were
> doing before) and then use Squid's mark preservation feature to retain
> the packet's mark. However, you can't use U32 classifiers - you'll have
> to go back to marking packets.
> 
> > Andy recommended to use ifb but, I don't know where to get the download
> > from. Is supposed to be on kernel 2.26.20 and up and I'm running a
> > kernel that "qualifies" but, cant really get ifb to do anything.
> 
> No need for IFB in your case. That's only if you want to shape multiple
> interfaces within one qdisc.
> 
> Andy
> 
> 

Re: How can I test my tc script?

[Optimum Wireless Services]

On Sat, 2011-02-05 at 12:35 +0000, Andrew Beverley wrote:
> On Fri, 2011-02-04 at 22:44 -0400, Optimum Wireless Services wrote:
> > On Thu, 2011-02-03 at 19:38 +0000, Andrew Beverley wrote:
> > > On Wed, 2011-02-02 at 22:17 -0400, Optimum Wireless Services wrote:
> > > > I would like to know how can I run some sort of benchmark test on a
> > > > network that would simulate 10,20,50,100 users connected to see if my gw
> > > > is handling the traffic well.
> > > > 
> > > 
> > > I suggest using tc-viewer to see what your qdisc is actually doing
> > > during peak use:
> > > 
> > > tc-viewer --iface=eth1 --zero --counter=1 --unit=kbit
> > > 
> > > > Here's a snip of what the script looks like:
> > > > 
> > > > ################ 
> > > > Now Download interface
> > > > 
> > > > $tc qdisc del dev eth1 root
> > > > $tc qdisc add dev eth1 root handle 2: htb r2q 1
> > > > 
> > > > $tc class add dev eth1 parent 2:0  classid 2:1 htb rate 15500kbit ceil
> > > > 15525kbit
> > > > 
> > > 
> > > Does the sum of all the leaf classes below add up to the root rate
> > > above? From the HTB manual: "The rate supplied for a parent should be
> > > the sum of the rates of its children."
> > > 
> > 
> > Hi Andy.
> > 
> > The sum of my users' classes or leafs rate would not be equal to the
> > parent's. They will never add up to the total root class since I will be
> > adding rules every once in a while so, if my root total BW is 15Mbps and
> > I have 100 users at 512kbps each then, that could be a problem. 
> > 
> 
> Hmmm, well HTB is known to misbehave if they don't all add up correctly,
> which could explain your problems. Why don't you *try* it with them all
> adding up correctly, just temporarily for your current number of users,
> to see if this is the problem.
> 

Just wondering, I have a total of 15Mbps for our total bandwidth. If I
devide that into 512kbps then I can only have about 29 users at that
rate. Don't know if I'm doing the math correctly but, how many users
should a 15Mbps line should serve without problems?

> > Is there something else to do instead?
> > 
> 
> You could have a look at another classful qdisc such as HFSC or CBQ.
> However, I've never played with them, so they may not be of any more
> help.
> 
> Andy
> 
> 

Re: How can I test my tc script?

[Optimum Wireless Services]

This IS very complex.

I'm going to have to print this and sit to take a closer look at it.

I would like to thank you all for replying to my post you have being
very helpful.



On Sun, 2011-02-06 at 01:05 +0100, Marek Kierdelewicz wrote:
> And here's a bit more complex example for shaping upload with tc u32
> hashing filters and IFB.
> 
> As in download example, we want to share 100mbit connection. Internet
> access is connected to the router on eth0 and lan in on eth1. We do
> ingress shaping of upload traffic on ifb0 after ingress eth1 traffic
> is redirected to the device.
> 
> ---------------------complete script------------------------------
> tc qdisc add dev ifb0 root handle 1: htb default 1
> tc class add dev ifb0 parent 1: classid 1:1 htb rate 100Mbit
> tc filter add dev ifb0 parent 1:1 prio 11 handle 2: protocol ip \
> u32 divisor 256
> tc filter add dev ifb0 protocol ip parent 1:1 prio 11 u32 match \
> ip dst 0.0.0.0/0 hashkey mask 0x000000ff at 12 link 2:
> 
> tc class add dev ifb0 parent 1:1 classid 1:400 htb rate 5mbit
> tc qdisc add dev ifb0 parent 1:400 handle 400: sfq perturb 1
> tc filter add dev ifb0 protocol ip prio 11 u32 ht 2:46: match ip \
> src 10.5.0.70 flowid 1:400
> 
> tc class add dev ifb0 parent 1:1 classid 1:401 htb rate 5mbit
> tc qdisc add dev ifb0 parent 1:401 handle 401: sfq perturb 1
> tc filter add dev ifb0 protocol ip prio 11 u32 ht 2:a: match ip \
> src 10.5.0.10 flowid 1:401
> 
> tc qdisc add dev eth1 ingress
> tc filter add dev eth1 protocol ip parent ffff: prio 4 u32 match \
> ip dst 0.0.0.0/0 flowid :1 action mirred egress redirect dev ifb0
> ---------------------------------------------------------------------
> 
> Now lets go line by line.
> 
> >tc qdisc add dev eth1 root handle 1: htb default 1
> >tc class add dev eth1 parent 1: classid 1:1 htb rate 100Mbit
> >tc filter add dev eth1 parent 1:1 prio 11 handle 2: protocol ip \
> >u32 divisor 256 
> 
> analogous to download example
> 
> >tc filter add dev eth1 protocol ip parent 1:1 prio 11 u32 match \
> >ip dst 0.0.0.0/0 hashkey mask 0x000000ff at 12 link 2:
> 
> "at 12" instead of 16, because we want to hash on source ip address
> 
> >tc class add dev eth1 parent 1:1 classid 1:400 htb rate 5mbit
> >tc qdisc add dev eth1 parent 1:400 handle 400: sfq perturb 1
> 
> analogous to download example
> 
> >tc filter add dev eth1 protocol ip prio 11 u32 ht 2:46: match ip \
> >src 10.5.0.70 flowid 1:400
> 
> compared to download shaping, dst is changed to src
> 
> >tc class add dev ifb0 parent 1:1 classid 1:401 htb rate 5mbit
> >tc qdisc add dev ifb0 parent 1:401 handle 401: sfq perturb 1
> >tc filter add dev ifb0 protocol ip prio 11 u32 ht 2:a: match ip \
> >src 10.5.0.10 flowid 1:401
> 
> analogous to previous client
> 
> >tc qdisc add dev eth1 ingress
> 
> add ingress qdisc to eth1 interface
> 
> >tc filter add dev eth1 protocol ip parent ffff: prio 4 u32 match \
> >ip dst 0.0.0.0/0 flowid :1 action mirred egress redirect dev ifb0
> 
> redirect any ip traffic from ingress qdisc on device eth1 to ifb0;
> note! redirection to ifb doesn't interfere with routing/firewalling/nat;
> 
> Best regards,
> Marek Kierdelewicz
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Andrew Beverley]

> Just wondering, I have a total of 15Mbps for our total bandwidth. If I
> devide that into 512kbps then I can only have about 29 users at that
> rate.

You could use HTB's "ceil" parameter for each leaf class to specify the
max rate per user - that value can be anything up to and including the
maximum rate of the root qdisc. It's the total of all the "rate"
parameters of the leaf classes that should add up to the root's total.

>  Don't know if I'm doing the math correctly but, how many users
> should a 15Mbps line should serve without problems?

Well obviously it depends on what your users are doing, but I think
15Mbps should be fine for 2-300 users. I have a 4Mbps ADSL line shared
between 150 users. It's not the fastest, but it works for general use,
because bulk traffic gets classified at a lower priority than web
browsing.

Andy

Re: How can I test my tc script?

[Andrew Beverley]

On Sat, 2011-02-05 at 22:12 -0400, Optimum Wireless Services wrote:
> Thank you all for replying and sorry for the late reply.

No problem.

> > Optimum Wireless Services - what is your aim? Do you really want to
> > limit bandwidth per user, or do you instead want to provide each user
> > with a fair share of bandwidth and fast internet access? If it is the
> > latter, you are better to not mark per user, but instead mark the
> > traffic type and use tc's hash functionality to share the bandwidth
> > between client IP address, rather than the default which is per
> > connection.
> > 
> 
> What I would love to do is offer "fast internet access" and at the same
> time sell packages to users: 512/128, 768/256, 1024/512, etc...

Well it is probably possible to achieve this, but I'd be tempted to say
that it's a little ambitious in your circumstances, given that you are
limited to a 15Mbps line.

>  If an
> user has a specific package, I would like this user to have a nice
> internet experience with low pings, and little lagging and not
> 'interfere' with the other users (might be impossible). 
> 
> So, what I'm trying to do is not have issues with one user hogging our
> internet line.

Okay... well I'd recommend forgetting the different speeds per user to
keep things simpler, and instead concentrate on classifying your traffic
to offer *everyone* a fast experience.

> 
> Dont know how a fair share bandwidth might work on our network since I
> have some users downloading p2p all day long. 

There's various ways of classifying P2P: OpenDPI and L7-filter being 2
of them. However, I personally just use connlimit and ipset to look for
lots of connection to high port numbers. It probably wrongly classifies
some traffic, but I haven't had any complaints so far :)

> 
> I've noticed that at peak times (7pm - 11pm) our internet service is so
> slow I'm embarrassed to even say. 

There's your answer - you need to classify traffic. Even if you limit
everyone to, say, 1Mbps, you only need more than 15 users for it to slow
down again.

> Also, another thing I noticed (this one not time specific) when running
> iptraf is that my external interface measures almost our max/total
> bandwidth of 15Mbps and our internal interface is only at 4Mbps. So,
> that tells me that not much is passing through squid or there are some
> big time downloading going on.

Or that your tc setup is wrong... Did you try tc-viewer?

> 
> So, to tell you the truth I don't know which way to go if limit bw per
> user or what. I'm just trying to keep our network running as smooth as
> possible.

Okay, well I've used a solution based on Jesper Dangaard Brouer's
excellent ADSL-Optimizer project. I classify traffic as: low-latency (eg
SSH), general (web, email), bulk (large downloads), other, and P2P. Each
of these are marked accordingly and then filtered into HTB.

I've started to document my stuff at
http://www.andybev.com/index.php/Fair_traffic_shaping_an_ADSL_line_for_a_local_network_using_Linux
but haven't finished yet. If you give me a couple of weeks I'll try and
get it updated.

Andy

Re: How can I test my tc script?

[Andrew Beverley]

On Sat, 2011-02-05 at 22:14 -0400, Optimum Wireless Services wrote:
> On Sat, 2011-02-05 at 12:27 +0000, Andrew Beverley wrote:
> > On Sat, 2011-02-05 at 02:07 -0400, Optimum Wireless Services wrote:
> > > Now, for the upload I cant get it shape with squid running in the
> > > middle. If I remove squid then, traffic shaping works ok.
> > > 
> > 
> > If you are doing egress shaping on the internet facing interface for the
> > upload (which I assume you are - you can't use HTB on ingress), then it
> > will not work for Squid. That's because Squid's traffic leaving the
> > server is coming from the server's IP address, not from the client's IP
> > address (Squid is generating the traffic, not the client). Therefore,
> > your filter will never match the traffic. Normal traffic from a client
> > is forwarded and retains its source IP address.
> > 
> 
> Do you think I should go back to this method?
> 
> I have 120 users now but, that number should be increasing. Which method
> do you recommend?

I think that you should forget the per-user stuff as per my other email
and classify traffic. Just my personal opinion - I'd be interested to
hear what others think.

Andy

Re: How can I test my tc script?

[Optimum Wireless Services]

On Sun, 2011-02-06 at 15:54 +0000, Andrew Beverley wrote:
> > Just wondering, I have a total of 15Mbps for our total bandwidth. If I
> > devide that into 512kbps then I can only have about 29 users at that
> > rate.
> 
> You could use HTB's "ceil" parameter for each leaf class to specify the
> max rate per user - that value can be anything up to and including the
> maximum rate of the root qdisc. It's the total of all the "rate"
> parameters of the leaf classes that should add up to the root's total.
> 

If I have users on different packages then how would I accomplish that?

> >  Don't know if I'm doing the math correctly but, how many users
> > should a 15Mbps line should serve without problems?
> 
> Well obviously it depends on what your users are doing, but I think
> 15Mbps should be fine for 2-300 users. I have a 4Mbps ADSL line shared
> between 150 users. It's not the fastest, but it works for general use,
> because bulk traffic gets classified at a lower priority than web
> browsing.
> 

I don't think we have a problem if all users use their connection for
general use. But, I can't tell users what to do and not do, if you know
what I mean. As mentioned earlier, I have user downloading p2p all day
long, there are others using voip, there are others that use it for
gaming, others chatting, etc... So, I thought if I limit users'
bandwidth to a certain amount that I could just have them use whatever
they're assigned and they can do whatever they want with that amount of
bandwidth.

> Andy
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Optimum Wireless Services]

On Sun, 2011-02-06 at 16:15 +0000, Andrew Beverley wrote:
> On Sat, 2011-02-05 at 22:14 -0400, Optimum Wireless Services wrote:
> > On Sat, 2011-02-05 at 12:27 +0000, Andrew Beverley wrote:
> > > On Sat, 2011-02-05 at 02:07 -0400, Optimum Wireless Services wrote:
> > > > Now, for the upload I cant get it shape with squid running in the
> > > > middle. If I remove squid then, traffic shaping works ok.
> > > > 
> > > 
> > > If you are doing egress shaping on the internet facing interface for the
> > > upload (which I assume you are - you can't use HTB on ingress), then it
> > > will not work for Squid. That's because Squid's traffic leaving the
> > > server is coming from the server's IP address, not from the client's IP
> > > address (Squid is generating the traffic, not the client). Therefore,
> > > your filter will never match the traffic. Normal traffic from a client
> > > is forwarded and retains its source IP address.
> > > 
> > 
> > Do you think I should go back to this method?
> > 
> > I have 120 users now but, that number should be increasing. Which method
> > do you recommend?
> 
> I think that you should forget the per-user stuff as per my other email
> and classify traffic. Just my personal opinion - I'd be interested to
> hear what others think.

I just need a way that would allow users to experience a reliable
connection that would always be 15% +- the contracted bandwidth.

This turned to be more complicated than what I first thought. 

After adding the new ruleset, I added this little script to try to
prioritize some traffic, just installed it so, don't know yet how would
traffic behave:

#!/bin/bash
#set -v
iptables='sudo iptables'
tc='sudo tc'

# This script should run before tcshaper.sh
# It will try to improve traffic: low latency, fast up, fast down.

# upload
$tc class add dev eth0 parent 1:1 classid 1:7 htb rate 3000kbit ceil
3000kbit prio 1
$tc class add dev eth0 parent 1:1 classid 1:8 htb rate 3000kbit ceil
3000kbit prio 2
#$tc class add dev eth0 parent 1:1 classid 1:9 htb rate 100kbit ceil
150kbit prio 3

$tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 7 fw
classid 1:7
$tc filter add dev eth0 parent 1:0 protocol ip prio 2 handle 8 fw
classid 1:8
#$tc filter add dev eth0 parent 1:0 protocol ip prio 3 handle 9 fw
classid 1:9

$tc qdisc add dev eth0 parent 1:7 handle 110: sfq perturb 10
$tc qdisc add dev eth0 parent 1:8 handle 120: sfq perturb 10
#$tc qdisc add dev eth0 parent 1:9 handle 130: sfq perturb 1

# FW Marks
# 7 = prio 1, 8 = prio 2, 9 = prio 3

############ Set the iptables rules ###################################
$iptables -A PREROUTING -t mangle -p tcp --sport 22     -j TOS --set-tos
Minimize-Delay

$iptables -A PREROUTING -t mangle -p tcp --sport 22     -j TOS --set-tos
Minimize-Delay
$iptables -A PREROUTING -t mangle -p tcp --dport 22     -j TOS --set-tos
Minimize-Delay
$iptables -A PREROUTING -t mangle -p udp --sport 53     -j TOS --set-tos
Minimize-Delay
$iptables -A PREROUTING -t mangle -p udp --dport 53     -j TOS --set-tos
Minimize-Delay
$iptables -A PREROUTING -t mangle -p icmp               -j TOS --set-tos
Minimize-Delay

$iptables -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK
SYN -j MARK --set-mark 7
$iptables -t mangle -A PREROUTING -p tcp -m tcp --tcp-flags SYN,RST,ACK
SYN -j RETURN

$iptables -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j MARK
--set-mark 7
$iptables -t mangle -A PREROUTING -m tos --tos Minimize-Delay -j RETURN

#$iptables -t mangle -A PREROUTING -p udp --dport 53 -j MARK  --set-mark
8
#$iptables -t mangle -A PREROUTING -p udp --dport 53 -j RETURN

########################################################################################
# download
$tc class add dev eth1 parent 2:1 classid 2:7 htb rate 5000kbit ceil
5000kbit prio 1
$tc class add dev eth1 parent 2:1 classid 2:8 htb rate 5000kbit ceil
5500kbit prio 2

$tc filter add dev eth1 parent 2:0 protocol ip prio 1 handle 7 fw
classid 2:7

$tc filter add dev eth1 parent 2:0 protocol ip prio 2 handle 8 fw
classid 2:8

$tc qdisc add dev eth1 parent 2:7 handle 110: sfq perturb 10
$tc qdisc add dev eth1 parent 2:8 handle 111: sfq perturb 10


> 
> Andy
> 
> 

Re: How can I test my tc script?

[Marek Kierdelewicz]

>> I have 120 users now but, that number should be increasing. Which
>> method do you recommend?
>I think that you should forget the per-user stuff as per my other email
>and classify traffic. Just my personal opinion - I'd be interested to
>hear what others think.

Well. It all depends on the bussiness constrains. Per-user shaping is
sometimes necessary while it's definitely more complex and
resource-consuming. My personal opinion - with >100 users shaping rules
should be script-generated instead of written by hand.

Best regards,
Marek Kierdelewicz

Re: How can I test my tc script?

[Andrew Beverley]

On Sun, 2011-02-06 at 12:35 -0400, Optimum Wireless Services wrote:
> > 
> > I think that you should forget the per-user stuff as per my other email
> > and classify traffic. Just my personal opinion - I'd be interested to
> > hear what others think.
> 
> I just need a way that would allow users to experience a reliable
> connection that would always be 15% +- the contracted bandwidth.
> 

You could buy some more upstream bandwidth ;-)

Have you tried the scripts that Marek posted? That's your answer to
limit per user.

> This turned to be more complicated than what I first thought. 
> 
> After adding the new ruleset, I added this little script to try to
> prioritize some traffic, just installed it so, don't know yet how would
> traffic behave:

<snip>

Herein lies your problem. If you want to prioritise traffic *and* limit
per user, then it starts to get complicated. The rules that you posted
will indeed rate limit different types of traffic, but you would need to
do these in conjunction with your other HTB rules for the different
users.

So you would need separate classes for users and traffic type. They'd
have to be parents/children of each other, but which way round and how
you would do it I do not know.

All I'm saying is that you need to take this one step at a time. Getting
a magic solution that does everything is not going to happen straight
away (unless somebody tells me otherwise).

So, try the per user stuff on its own first. Maybe try the traffic
prioritisation on its own as well. Check the results and take it from
there.

BTW - I don't think the traffic prioritisation scripts you posted will
have much affect anyway: they are only really looking at DNS lookups and
SSH, which isn't going to solve your problems.

Andy

Re: How can I test my tc script?

[Andrew Beverley]

On Sun, 2011-02-06 at 12:28 -0400, Optimum Wireless Services wrote:
> On Sun, 2011-02-06 at 15:54 +0000, Andrew Beverley wrote:
> > > Just wondering, I have a total of 15Mbps for our total bandwidth. If I
> > > devide that into 512kbps then I can only have about 29 users at that
> > > rate.
> > 
> > You could use HTB's "ceil" parameter for each leaf class to specify the
> > max rate per user - that value can be anything up to and including the
> > maximum rate of the root qdisc. It's the total of all the "rate"
> > parameters of the leaf classes that should add up to the root's total.
> > 
> 
> If I have users on different packages then how would I accomplish that?
> 

Something like:

tc qdisc add dev ... root handle 1: htb
tc class add dev ... parent 1: classid 1:1 htb rate 2000kbit
...
tc class add dev .... htb rate 200kbit ceil 2000kbit prio 1
tc class add dev .... htb rate 200kbit ceil 2000kbit prio 1
tc class add dev .... htb rate 200kbit ceil 1000kbit prio 2
tc class add dev .... htb rate 200kbit ceil 1000kbit prio 2
tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
tc class add dev .... htb rate 200kbit ceil 500kbit prio 3

Note that all the rates add up to the root, but that each leaf class has
the ceil set at the top limit.

> > >  Don't know if I'm doing the math correctly but, how many users
> > > should a 15Mbps line should serve without problems?
> > 
> > Well obviously it depends on what your users are doing, but I think
> > 15Mbps should be fine for 2-300 users. I have a 4Mbps ADSL line shared
> > between 150 users. It's not the fastest, but it works for general use,
> > because bulk traffic gets classified at a lower priority than web
> > browsing.
> > 
> 
> I don't think we have a problem if all users use their connection for
> general use. But, I can't tell users what to do and not do, if you know
> what I mean. As mentioned earlier, I have user downloading p2p all day
> long, there are others using voip, there are others that use it for
> gaming, others chatting, etc... So, I thought if I limit users'
> bandwidth to a certain amount that I could just have them use whatever
> they're assigned and they can do whatever they want with that amount of
> bandwidth.

Try it, but you may find that your total aggregate bandwidth is not
enough to do it this way.

Andy

Re: How can I test my tc script?

[Optimum Wireless Services]

Thanks again for replying.

On Sun, 2011-02-06 at 17:02 +0000, Andrew Beverley wrote:
> On Sun, 2011-02-06 at 12:35 -0400, Optimum Wireless Services wrote:
> > > 
> > > I think that you should forget the per-user stuff as per my other email
> > > and classify traffic. Just my personal opinion - I'd be interested to
> > > hear what others think.
> > 
> > I just need a way that would allow users to experience a reliable
> > connection that would always be 15% +- the contracted bandwidth.
> > 
> 
> You could buy some more upstream bandwidth ;-)

I wish we could buy more bandwidth but, is extremely expensive where I'm
at (Dominican Republic).

> 
> Have you tried the scripts that Marek posted? That's your answer to
> limit per user.

Havent tried that script yet, will be doing so, later tonight or
tomorrow.

> 
> > This turned to be more complicated than what I first thought. 
> > 
> > After adding the new ruleset, I added this little script to try to
> > prioritize some traffic, just installed it so, don't know yet how would
> > traffic behave:
> 
> <snip>
> 
> Herein lies your problem. If you want to prioritise traffic *and* limit
> per user, then it starts to get complicated. The rules that you posted
> will indeed rate limit different types of traffic, but you would need to
> do these in conjunction with your other HTB rules for the different
> users.
> 

I just added those prioritising rules so, I will be removing them
immediately. Is hard to really notice how the ruleset are working since
there isn't an utility that can graph or show the actual traffic. I was
looking for tc-viewer but, can't find a link where to download it from.

> So you would need separate classes for users and traffic type. They'd
> have to be parents/children of each other, but which way round and how
> you would do it I do not know.
> 

I figured I added the prio rules to the same parent rule I already had.
Thought that was the correct way to do it. So, I should create another
parent rule for ie; 3, and add the children under that to do traffic
prioritizing.


> All I'm saying is that you need to take this one step at a time. Getting
> a magic solution that does everything is not going to happen straight
> away (unless somebody tells me otherwise).

I agree with you.

> 
> So, try the per user stuff on its own first. Maybe try the traffic
> prioritisation on its own as well. Check the results and take it from
> there.
> 
> BTW - I don't think the traffic prioritisation scripts you posted will
> have much affect anyway: they are only really looking at DNS lookups and
> SSH, which isn't going to solve your problems.
> 

I wanted to speed up dns lookups, icmps, SYN ACK flags, and ssh, for now
and later on add voip since I have an asterisk server on our network as
well.

I will remove the prioritising script and leave the last created ruleset
and see how things behave.

> Andy
> 
> 

Re: How can I test my tc script?

[Optimum Wireless Services]

On Sun, 2011-02-06 at 17:44 +0100, Marek Kierdelewicz wrote:
> >> I have 120 users now but, that number should be increasing. Which
> >> method do you recommend?
> >I think that you should forget the per-user stuff as per my other email
> >and classify traffic. Just my personal opinion - I'd be interested to
> >hear what others think.
> 
> Well. It all depends on the bussiness constrains. Per-user shaping is
> sometimes necessary while it's definitely more complex and
> resource-consuming. My personal opinion - with >100 users shaping rules
> should be script-generated instead of written by hand.
> 

I don't know which is the best solution for a WISP, since it would be
adding and deleting customers regularly. Per-user shaping might be
necessary for most ISPs, I think, because they sell different plans for
different necesities.

I think batch rules should be script-generated and deleted as customers
come and go.


> Best regards,
> Marek Kierdelewicz

Re: How can I test my tc script?

[Optimum Wireless Services]

On Sun, 2011-02-06 at 17:11 +0000, Andrew Beverley wrote:
> On Sun, 2011-02-06 at 12:28 -0400, Optimum Wireless Services wrote:
> > On Sun, 2011-02-06 at 15:54 +0000, Andrew Beverley wrote:
> > > > Just wondering, I have a total of 15Mbps for our total bandwidth. If I
> > > > devide that into 512kbps then I can only have about 29 users at that
> > > > rate.
> > > 
> > > You could use HTB's "ceil" parameter for each leaf class to specify the
> > > max rate per user - that value can be anything up to and including the
> > > maximum rate of the root qdisc. It's the total of all the "rate"
> > > parameters of the leaf classes that should add up to the root's total.
> > > 
> > 
> > If I have users on different packages then how would I accomplish that?
> > 
> 
> Something like:
> 
> tc qdisc add dev ... root handle 1: htb
> tc class add dev ... parent 1: classid 1:1 htb rate 2000kbit
> ...
> tc class add dev .... htb rate 200kbit ceil 2000kbit prio 1
> tc class add dev .... htb rate 200kbit ceil 2000kbit prio 1
> tc class add dev .... htb rate 200kbit ceil 1000kbit prio 2
> tc class add dev .... htb rate 200kbit ceil 1000kbit prio 2
> tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> 
> Note that all the rates add up to the root, but that each leaf class has
> the ceil set at the top limit.

How would the filtering be done here?

Filter by type of service and attach to one of the leafs?

> 
> > > >  Don't know if I'm doing the math correctly but, how many users
> > > > should a 15Mbps line should serve without problems?
> > > 
> > > Well obviously it depends on what your users are doing, but I think
> > > 15Mbps should be fine for 2-300 users. I have a 4Mbps ADSL line shared
> > > between 150 users. It's not the fastest, but it works for general use,
> > > because bulk traffic gets classified at a lower priority than web
> > > browsing.
> > > 
> > 
> > I don't think we have a problem if all users use their connection for
> > general use. But, I can't tell users what to do and not do, if you know
> > what I mean. As mentioned earlier, I have user downloading p2p all day
> > long, there are others using voip, there are others that use it for
> > gaming, others chatting, etc... So, I thought if I limit users'
> > bandwidth to a certain amount that I could just have them use whatever
> > they're assigned and they can do whatever they want with that amount of
> > bandwidth.
> 
> Try it, but you may find that your total aggregate bandwidth is not
> enough to do it this way.
> 
> Andy
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I test my tc script?

[Michele Petrazzo - Unipex]

Marek Kierdelewicz wrote:
>> Hi,
>
> Hi
>
>> I, are you sure about the big difference? I agree with you about tc
>> u32 filters are better than iptables + marking, but like now I have
>> about 100mbit 1k+ users with mangle+classify and tc+htb+egress and
>> I have a load at about 0.2. My cpu are xeon 2.6 quad.
>
> Do you have individual iptables rule doing marking for each user
> (1k+ rules in mangle)?
>

Yes. One rule for mark and one for return. One for classify and one for
return. The first ones inside prerouting and the latter in forward.

srv:~# iptables -nvL -t mangle|wc -l
7193

Like now, I have load: 0.04  ;)

> Best regards, Marek Kierdelewicz

Ciao,
Michele

Re: How can I test my tc script?

[Andrew Beverley]

On Sun, 2011-02-06 at 19:27 -0400, Optimum Wireless Services wrote:
> I just added those prioritising rules so, I will be removing them
> immediately. Is hard to really notice how the ruleset are working since
> there isn't an utility that can graph or show the actual traffic. I was
> looking for tc-viewer but, can't find a link where to download it from.

http://files.andybev.com/web-portal/tc-viewer

tc-viewer --iface=eth0 --zero --counter=1 --unit=kbit

> > So you would need separate classes for users and traffic type. They'd
> > have to be parents/children of each other, but which way round and how
> > you would do it I do not know.
> > 
> 
> I figured I added the prio rules to the same parent rule I already had.
> Thought that was the correct way to do it. So, I should create another
> parent rule for ie; 3, and add the children under that to do traffic
> prioritizing.

Something like that, but you'll have problems sharing traffic between
the different classes (classes for different IP addresses and traffic
for different priorities). That's why I say that I don't know how to do
it myself.

> > So, try the per user stuff on its own first. Maybe try the traffic
> > prioritisation on its own as well. Check the results and take it from
> > there.
> > 
> > BTW - I don't think the traffic prioritisation scripts you posted will
> > have much affect anyway: they are only really looking at DNS lookups and
> > SSH, which isn't going to solve your problems.
> > 
> 
> I wanted to speed up dns lookups, icmps, SYN ACK flags, and ssh, for now
> and later on add voip since I have an asterisk server on our network as
> well.
> 
> I will remove the prioritising script and leave the last created ruleset
> and see how things behave.

If you give me a few days I'll send you through my traffic
prioritisation script. It won't give you the per-user limit, but I can
almost guarantee that it will give everyone faster access, so you can at
least use it for a comparison to see what is possible.

Andy

Re: How can I test my tc script?

[Andrew Beverley]

On Sun, 2011-02-06 at 19:31 -0400, Optimum Wireless Services wrote:
> On Sun, 2011-02-06 at 17:44 +0100, Marek Kierdelewicz wrote:
> > >> I have 120 users now but, that number should be increasing. Which
> > >> method do you recommend?
> > >I think that you should forget the per-user stuff as per my other email
> > >and classify traffic. Just my personal opinion - I'd be interested to
> > >hear what others think.
> > 
> > Well. It all depends on the bussiness constrains. Per-user shaping is
> > sometimes necessary while it's definitely more complex and
> > resource-consuming. My personal opinion - with >100 users shaping rules
> > should be script-generated instead of written by hand.
> > 
> 
> I don't know which is the best solution for a WISP, since it would be
> adding and deleting customers regularly. Per-user shaping might be
> necessary for most ISPs, I think, because they sell different plans for
> different necesities.
> 
> I think batch rules should be script-generated and deleted as customers
> come and go.

You could batch create, say, 200 leaf classes. Any that were unused
would just have their bandwidth shared between other customers, just as
if somebody turned their computer off.

Andy

Re: How can I test my tc script?

[Andrew Beverley]

On Sun, 2011-02-06 at 19:33 -0400, Optimum Wireless Services wrote:
> On Sun, 2011-02-06 at 17:11 +0000, Andrew Beverley wrote:
> > On Sun, 2011-02-06 at 12:28 -0400, Optimum Wireless Services wrote:
> > > On Sun, 2011-02-06 at 15:54 +0000, Andrew Beverley wrote:
> > > > > Just wondering, I have a total of 15Mbps for our total bandwidth. If I
> > > > > devide that into 512kbps then I can only have about 29 users at that
> > > > > rate.
> > > > 
> > > > You could use HTB's "ceil" parameter for each leaf class to specify the
> > > > max rate per user - that value can be anything up to and including the
> > > > maximum rate of the root qdisc. It's the total of all the "rate"
> > > > parameters of the leaf classes that should add up to the root's total.
> > > > 
> > > 
> > > If I have users on different packages then how would I accomplish that?
> > > 
> > 
> > Something like:
> > 
> > tc qdisc add dev ... root handle 1: htb
> > tc class add dev ... parent 1: classid 1:1 htb rate 2000kbit
> > ...
> > tc class add dev .... htb rate 200kbit ceil 2000kbit prio 1
> > tc class add dev .... htb rate 200kbit ceil 2000kbit prio 1
> > tc class add dev .... htb rate 200kbit ceil 1000kbit prio 2
> > tc class add dev .... htb rate 200kbit ceil 1000kbit prio 2
> > tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> > tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> > tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> > tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> > tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> > tc class add dev .... htb rate 200kbit ceil 500kbit prio 3
> > 
> > Note that all the rates add up to the root, but that each leaf class has
> > the ceil set at the top limit.
> 
> How would the filtering be done here?

In the same way as Marek's original script. Just replace his HTB leaf
class rules with ones similar to the above.

You could also use something like:

tc filter add dev eth0 parent ... protocol ip handle ... flow \
	hash keys nfct-dst divisor 1024

to balance loads between client IP address rather than connection stream
(which may help a lot if you get it correct).

Andy

> 
> Filter by type of service and attach to one of the leafs?
> 

Filter by IP hash as per Marek's email.

Andy

Re: How can I test my tc script?

[Optimum Wireless Services]

On Tue, 2011-02-08 at 00:56 +0000, Andrew Beverley wrote:
> On Sun, 2011-02-06 at 19:27 -0400, Optimum Wireless Services wrote:
> > I just added those prioritising rules so, I will be removing them
> > immediately. Is hard to really notice how the ruleset are working since
> > there isn't an utility that can graph or show the actual traffic. I was
> > looking for tc-viewer but, can't find a link where to download it from.
> 
> http://files.andybev.com/web-portal/tc-viewer
> 
> tc-viewer --iface=eth0 --zero --counter=1 --unit=kbit
> 

Great! Thanks.


> > > So you would need separate classes for users and traffic type. They'd
> > > have to be parents/children of each other, but which way round and how
> > > you would do it I do not know.
> > > 
> > 
> > I figured I added the prio rules to the same parent rule I already had.
> > Thought that was the correct way to do it. So, I should create another
> > parent rule for ie; 3, and add the children under that to do traffic
> > prioritizing.
> 
> Something like that, but you'll have problems sharing traffic between
> the different classes (classes for different IP addresses and traffic
> for different priorities). That's why I say that I don't know how to do
> it myself.
> 
> > > So, try the per user stuff on its own first. Maybe try the traffic
> > > prioritisation on its own as well. Check the results and take it from
> > > there.
> > > 
> > > BTW - I don't think the traffic prioritisation scripts you posted will
> > > have much affect anyway: they are only really looking at DNS lookups and
> > > SSH, which isn't going to solve your problems.
> > > 
> > 
> > I wanted to speed up dns lookups, icmps, SYN ACK flags, and ssh, for now
> > and later on add voip since I have an asterisk server on our network as
> > well.
> > 
> > I will remove the prioritising script and leave the last created ruleset
> > and see how things behave.
> 
> If you give me a few days I'll send you through my traffic
> prioritisation script. It won't give you the per-user limit, but I can
> almost guarantee that it will give everyone faster access, so you can at
> least use it for a comparison to see what is possible.
> 
Thanks.

I will defenately test it but, I wouldn't like my users to get used to a
higher speed than they're paying for in case I decide to remove it. If
it works, great. 

How many users do you have on the network you're running the script?

> Andy
> 
> 

Re: How can I test my tc script?

[Optimum Wireless Services]

On Tue, 2011-02-08 at 00:56 +0000, Andrew Beverley wrote:
> tc-viewer --iface=eth0 --zero --counter=1 --unit=kbit

I ran tc-viewer but, some of the values doesn't make sense or do not
match:

 2:1085               <  739000bit -  739000bit >      0.0 kbit/s  (
0pps)
 2:1087               <  739000bit -  739000bit >    403.0 kbit/s
( 36pps)
 2:1089               <  739000bit -  739000bit >      0.0 kbit/s  (
0pps)
 2:1091               <  739000bit -  739000bit >      0.0 kbit/s  (
0pps)
 2:1093               <  739000bit -  739000bit >      0.0 kbit/s  (
0pps)
 2:1095               <  739000bit -  739000bit >    941.4 kbit/s
( 82pps)

How come a leaf that's only allowing 739kbit is allowing 941kbit/s ?

Re: How can I test my tc script?

[Andrew Beverley]

On Mon, 2011-02-07 at 22:11 -0400, Optimum Wireless Services wrote:
> > If you give me a few days I'll send you through my traffic
> > prioritisation script. It won't give you the per-user limit, but I can
> > almost guarantee that it will give everyone faster access, so you can at
> > least use it for a comparison to see what is possible.
> > 
> Thanks.
> 
> I will defenately test it but, I wouldn't like my users to get used to a
> higher speed than they're paying for in case I decide to remove it. If
> it works, great. 
> 
> How many users do you have on the network you're running the script?

Maximum signed-up I don't know, but about 100 concurrently actually
using it at peak periods.

Re: How can I test my tc script?

[Andrew Beverley]

On Mon, 2011-02-07 at 22:20 -0400, Optimum Wireless Services wrote:
> On Tue, 2011-02-08 at 00:56 +0000, Andrew Beverley wrote:
> > tc-viewer --iface=eth0 --zero --counter=1 --unit=kbit
> 
> I ran tc-viewer but, some of the values doesn't make sense or do not
> match:
> 
>  2:1085               <  739000bit -  739000bit >      0.0 kbit/s  (
> 0pps)
>  2:1087               <  739000bit -  739000bit >    403.0 kbit/s
> ( 36pps)
>  2:1089               <  739000bit -  739000bit >      0.0 kbit/s  (
> 0pps)
>  2:1091               <  739000bit -  739000bit >      0.0 kbit/s  (
> 0pps)
>  2:1093               <  739000bit -  739000bit >      0.0 kbit/s  (
> 0pps)
>  2:1095               <  739000bit -  739000bit >    941.4 kbit/s
> ( 82pps)
> 
> How come a leaf that's only allowing 739kbit is allowing 941kbit/s ?
> 

I don't know - I've not seen that before. Maybe this is one of those
"strange things" that happens when your leaf classes don't add up to the
root...?

Andy