Re: HFSC not working as expected

[Alan Goodman <notifications@yescomputersolutions.com>]

Hi Andy/all,

Thanks for your further useful input.

On 06/07/14 21:42, Andy Furniss wrote:
> Wrong link - but that's data throughput which is < ip throughput which
> is < atm level throughput. Assuming you are using stab when setting the
> 17100kbit 14mbit date throughput is only a bit below expected.
>
> I assume your mtu is 1492, also assuming you have default linux settings
> of tcptimestamps on (costs 12 bytes), so with tcp + ip headers there is
> only 1440 bytes data /packet each of which after allowing for ppp/aal5
> overheads will probably use 32 cells = 1696 bytes.
>
> 1440 / 1696 = 0.85 * 17.1 = 14.5.
>
> I am not sure what overhead you should add with stab for your pppoe as
> tc already sees eth as ip + 14 - maybe adding 40 is too much and you are
> getting 33 cells per packet.

Sorry regarding the link related mistake.  What you would have seen if I 
had correctly linked you was that the connection without traffic shaping 
is managing around 16.4mbit.  With traffic shaping, upper limit set to 
17100kbit and stab overhead 40 I only see 14.5 ish mbit - which means we 
are likely being overly conservative?

My download shaping is being completed on the outbound leg from the 
server...  Traffic flows ADSL -> router in bridge mode -> eth0 -> ppp0 
-> eth1 switch client device.  My download shaping occurs on the eth1 
device.

Quickly relating to mtu... By default CentOS has CLAMPMSS enabled on the 
pppoe connection.  This is set to 1412 bytes.  MTU of the connection is  
1492 according to ifconfig.

> Well, it's quite a fast link, maybe 25 is too short - I would test IIRC
> 128 is default for sfq.

I have experimented with loads of sfq limit settings now and really it 
seems to make little difference so I've decided to set this at 128 
(default) for now.

> Thinking more about it, there could be other reasons that you got the
> latency you saw.
>
> As I said I don't know HFSC, but I notice on both your setups you give
> very little bandwidth to "syn ack rst", I assume ack here means you
> classified by length to get empty (s)acks as almost every packet has ack
> set. Personally I would give those < prio than time critical and you
> should be aware on a highly asymmetric 20:1 adsl line they can eat a
> fair bit of your upstream (2 cells each, 1 for every 2 incoming best
> case, 1 per incoming in recovery after loss).

I feel thats a very valid point.  I have decided to roll class 10 and 
class 11 together for the time being which should mean time critical + 
syn/ack etc get roughly 50% of upload capacity.

I've been playing around with my worst case bittorent scenario some 
more.  Whilst troubleshooting I decided to set ul 15000kbit on the 
download class 1:13 (which the torrent hits).  With the torrent using 
around 200 flows I Immediately saw latency in the priority queue within 
acceptable limits.  So I thought bingo perhaps I set my class 1:2 upper 
limit too high overall?  So I reduced 17100kbit to 15000kbit, adjusted 
sc rates so that total was 15000kbit and deleted the upper limit on 
class 1:13 and reloaded the rules.  Unfortunately this behaved exactly 
like 17100kbit upper limit latency wise.  But I dont understand why this 
is?  Could this be the crux of my issues - some hfsc misunderstandings?

While I had the ul 15000kbit set on the bulk class I also played around 
with getting traffic to hit the 'interactive' 1:12 class.  I found this 
caused similar overall behaviour to when I didnt limit the bulk class at 
all - roundtrip hitting over 100+ms.  Its as though it doesnt like 
hitting the branch linkshare/upper limit?

Below is my current script:

#QoS for Upload

tc qdisc del dev ppp0 root

tc qdisc add dev ppp0 stab mtu 1492 overhead 40 linklayer atm root 
handle 1:0 hfsc default 14

tc class add dev ppp0 parent 1:0 classid 1:1 hfsc sc rate 90mbit
tc class add dev ppp0 parent 1:1 classid 1:2 hfsc sc rate 1100kbit ul 
rate 1100kbit

tc class add dev ppp0 parent 1:2 classid 1:11 hfsc sc umax 1412b dmax 
20ms rate 495kbit # Time critical
tc class add dev ppp0 parent 1:2 classid 1:12 hfsc sc rate 300kbit 
#Interactive
tc class add dev ppp0 parent 1:2 classid 1:13 hfsc sc rate 305kbit #bulk
tc class add dev ppp0 parent 1:1 classid 1:14 hfsc sc rate 90mbit

tc qdisc add dev ppp0 parent 1:11 handle 11: sfq perturb 10
tc qdisc add dev ppp0 parent 1:12 handle 12: sfq perturb 10
tc qdisc add dev ppp0 parent 1:13 handle 13: sfq perturb 10
tc qdisc add dev ppp0 parent 1:14 handle 14: pfifo

tc filter add dev ppp0 parent 1:0 protocol ip prio 2 handle 11 fw flowid 
1:11
tc filter add dev ppp0 parent 1:0 protocol ip prio 3 handle 12 fw flowid 
1:12
tc filter add dev ppp0 parent 1:0 protocol ip prio 4 handle 13 fw flowid 
1:13

#QoS for Download

tc qdisc del dev eth1 root

tc qdisc add dev eth1 stab overhead 40 linklayer atm root handle 1:0 
hfsc default 14

tc class add dev eth1 parent 1:0 classid 1:1 hfsc sc rate 90mbit
tc class add dev eth1 parent 1:1 classid 1:2 hfsc sc rate 17100kbit ul 
rate 17100kbit

tc class add dev eth1 parent 1:2 classid 1:11 hfsc sc umax 1412b dmax 
20ms rate 1545kbit #Time critical
tc class add dev eth1 parent 1:2 classid 1:12 hfsc sc rate 4955kbit 
#Interactive
tc class add dev eth1 parent 1:2 classid 1:13 hfsc sc rate 10600kbit #ul 
rate 15000kbit #Bulk
tc class add dev eth1 parent 1:1 classid 1:14 hfsc sc rate 90mbit

tc qdisc add dev eth1 parent 1:11 handle 11: sfq perturb 10
tc qdisc add dev eth1 parent 1:12 handle 12: sfq perturb 10
tc qdisc add dev eth1 parent 1:13 handle 13: sfq perturb 10
tc qdisc add dev eth1 parent 1:14 handle 14: pfifo


tc filter add dev eth1 parent 1:0 protocol ip prio 2 handle 11 fw flowid 
1:11
tc filter add dev eth1 parent 1:0 protocol ip prio 3 handle 12 fw flowid 
1:12
tc filter add dev eth1 parent 1:0 protocol ip prio 4 handle 13 fw flowid 
1:13

Quick note regarding class 1:14...  Class 1:14 is only getting traffic 
which I dont mark in iptables.  My iptables ruleset guarantees that all 
traffic destined for the internet leaving via ppp0 and all traffic 
destined for a machine inside the NAT leaving via eth1 which hasnt 
already been marked as more important gets marked 13.  Therefore traffic 
not marked must be source localsubnet destination local subnet.  Hope 
this makes sense!

Alan