HTB limiting working for single big download but not for many small ones

HTB limiting working for single big download but not for many small ones

[Dennis Jacobfeuerborn]

Hi,
I've run into problems trying to limit the overall download bandwidth for a
webserver. This is the code I'm using:

tc qdisc add dev eth0 root handle 1: htb default 30
tc class add dev eth0 parent 1: classid 1:1 htb rate 200mbit ceil 200mbit
tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
0xffff flowid 1:1

The problem is that when I test this with a single connection downloading a
large file I can see the download speed according to the specified limit
(in this case 200mbit).
When I use this code on a production system that receives several thousand
small downloads from an nginx server however I see the following:

# tc -s -d class show dev eth0
class htb 1:1 root prio 0 quantum 200000 rate 200000Kbit ceil 200000Kbit
burst 101600b/8 mpu 0b overhead 0b cburst 101600b/8 mpu 0b overhead 0b level 0
 Sent 246544581544 bytes 55302789 pkt (dropped 1614680, overlimits 0
requeues 0)
 rate 636728Kbit 16228pps backlog 0b 229p requeues 0
 lended: 55302560 borrowed: 0 giants: 51516908
 tokens: -3729 ctokens: -3729

The rate is over 600mbit so I'm not sure why the traffic isn't limited to
the 200mbit specified. Is there something else I need to take into account
here to make this work?

Regards,
  Dennis

Re: HTB limiting working for single big download but not for many small ones

[Dennis Jacobfeuerborn]

On 02/07/2013 06:57 PM, Dennis Jacobfeuerborn wrote:
> Hi,
> I've run into problems trying to limit the overall download bandwidth for a
> webserver. This is the code I'm using:
> 
> tc qdisc add dev eth0 root handle 1: htb default 30
> tc class add dev eth0 parent 1: classid 1:1 htb rate 200mbit ceil 200mbit
> tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
> 0xffff flowid 1:1
> 
> The problem is that when I test this with a single connection downloading a
> large file I can see the download speed according to the specified limit
> (in this case 200mbit).
> When I use this code on a production system that receives several thousand
> small downloads from an nginx server however I see the following:
> 
> # tc -s -d class show dev eth0
> class htb 1:1 root prio 0 quantum 200000 rate 200000Kbit ceil 200000Kbit
> burst 101600b/8 mpu 0b overhead 0b cburst 101600b/8 mpu 0b overhead 0b level 0
>  Sent 246544581544 bytes 55302789 pkt (dropped 1614680, overlimits 0
> requeues 0)
>  rate 636728Kbit 16228pps backlog 0b 229p requeues 0
>  lended: 55302560 borrowed: 0 giants: 51516908
>  tokens: -3729 ctokens: -3729
> 
> The rate is over 600mbit so I'm not sure why the traffic isn't limited to
> the 200mbit specified. Is there something else I need to take into account
> here to make this work?

I've now experimented with burst/cburst but either I end up with a rate way
below 200mbit or easily exceeding 600mbit. Apparently the actual 200mbit
limit is just ignored.

Regards,
  Dennis

Re: HTB limiting working for single big download but not for many small ones

[Dennis Jacobfeuerborn]

On 02/08/2013 12:07 PM, Dennis Jacobfeuerborn wrote:
> On 02/07/2013 06:57 PM, Dennis Jacobfeuerborn wrote:
>> Hi,
>> I've run into problems trying to limit the overall download bandwidth for a
>> webserver. This is the code I'm using:
>>
>> tc qdisc add dev eth0 root handle 1: htb default 30
>> tc class add dev eth0 parent 1: classid 1:1 htb rate 200mbit ceil 200mbit
>> tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80
>> 0xffff flowid 1:1
>>
>> The problem is that when I test this with a single connection downloading a
>> large file I can see the download speed according to the specified limit
>> (in this case 200mbit).
>> When I use this code on a production system that receives several thousand
>> small downloads from an nginx server however I see the following:
>>
>> # tc -s -d class show dev eth0
>> class htb 1:1 root prio 0 quantum 200000 rate 200000Kbit ceil 200000Kbit
>> burst 101600b/8 mpu 0b overhead 0b cburst 101600b/8 mpu 0b overhead 0b level 0
>>  Sent 246544581544 bytes 55302789 pkt (dropped 1614680, overlimits 0
>> requeues 0)
>>  rate 636728Kbit 16228pps backlog 0b 229p requeues 0
>>  lended: 55302560 borrowed: 0 giants: 51516908
>>  tokens: -3729 ctokens: -3729
>>
>> The rate is over 600mbit so I'm not sure why the traffic isn't limited to
>> the 200mbit specified. Is there something else I need to take into account
>> here to make this work?
> 
> I've now experimented with burst/cburst but either I end up with a rate way
> below 200mbit or easily exceeding 600mbit. Apparently the actual 200mbit
> limit is just ignored.

So after further digging I finally stumbled upon the solution. The reason
this wasn't working is that I'm running this in a xen guest and apparently
the tx checksum offloading screws things up. I removed the burst/cburst
options again and instead issued a "ethtool -K eth0 tx off" and now the
traffic is limited at the expected 200mbit.

Regards,
  Dennis