bond + tc regression ?

bond + tc regression ?

[Vladimir Ivashchenko]

Hi,

I have a traffic policing setup running on Linux, serving about 800 mbps
of traffic. Due to the traffic growth I decided to employ network
interface bonding to scale over a single GigE.

The Sun X4150 server has 2xIntel E5450 QuadCore CPUs and a total of four
built-in e1000e interfaces, which I grouped into two bond interfaces.

With kernel 2.6.23.1, everything works fine, but the system locked up
after a few days.

With kernel 2.6.28.7/2.6.29.1, I get 10-20% packet loss. I get packet loss as
soon as I put a classful qdisc, even prio, without even having any
classes or filters. TC prio statistics report lots of drops, around 10k
per sec. With exactly the same setup on 2.6.23, the number of drops is
only 50 per sec.

On both kernels, the system is running with at least 70% idle CPU.
The network interrupts are distributed accross the cores.

I thought it was a e1000e driver issue, but tweaking e1000e ring buffers
didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs,
I tried running on a different server with bnx cards, I tried disabling
NO_HZ and HRTICK, but still I have the same problem.

However, if I don't utilize bond, but just apply rules on normal ethX
interfaces, there is no packet loss with 2.6.28/29. 

So, the problem appears only when I use 2.6.28/29 + bond + classful tc
combination. 

Any ideas ?

-- 
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211

Re: bond + tc regression ?

[Denys Fedoryschenko]

Can you show example of rules you are putting?
Probably i can find mistakes and give correct example and i will explain maybe 
why it is happened.


On Tuesday 05 May 2009 18:45:58 Vladimir Ivashchenko wrote:
> Hi,
>
> I have a traffic policing setup running on Linux, serving about 800 mbps
> of traffic. Due to the traffic growth I decided to employ network
> interface bonding to scale over a single GigE.
>
> The Sun X4150 server has 2xIntel E5450 QuadCore CPUs and a total of four
> built-in e1000e interfaces, which I grouped into two bond interfaces.
>
> With kernel 2.6.23.1, everything works fine, but the system locked up
> after a few days.
>
> With kernel 2.6.28.7/2.6.29.1, I get 10-20% packet loss. I get packet loss
> as soon as I put a classful qdisc, even prio, without even having any
> classes or filters. TC prio statistics report lots of drops, around 10k per
> sec. With exactly the same setup on 2.6.23, the number of drops is only 50
> per sec.
>
> On both kernels, the system is running with at least 70% idle CPU.
> The network interrupts are distributed accross the cores.
>
> I thought it was a e1000e driver issue, but tweaking e1000e ring buffers
> didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs,
> I tried running on a different server with bnx cards, I tried disabling
> NO_HZ and HRTICK, but still I have the same problem.
>
> However, if I don't utilize bond, but just apply rules on normal ethX
> interfaces, there is no packet loss with 2.6.28/29.
>
> So, the problem appears only when I use 2.6.28/29 + bond + classful tc
> combination.
>
> Any ideas ?

Re: bond + tc regression ?

[Eric Dumazet]

Vladimir Ivashchenko a écrit :
> Hi,
> 
> I have a traffic policing setup running on Linux, serving about 800 mbps
> of traffic. Due to the traffic growth I decided to employ network
> interface bonding to scale over a single GigE.
> 
> The Sun X4150 server has 2xIntel E5450 QuadCore CPUs and a total of four
> built-in e1000e interfaces, which I grouped into two bond interfaces.
> 
> With kernel 2.6.23.1, everything works fine, but the system locked up
> after a few days.
> 
> With kernel 2.6.28.7/2.6.29.1, I get 10-20% packet loss. I get packet loss as
> soon as I put a classful qdisc, even prio, without even having any
> classes or filters. TC prio statistics report lots of drops, around 10k
> per sec. With exactly the same setup on 2.6.23, the number of drops is
> only 50 per sec.
> 
> On both kernels, the system is running with at least 70% idle CPU.
> The network interrupts are distributed accross the cores.

You should not distribute interrupts, but bound a NIC to one CPU
> 
> I thought it was a e1000e driver issue, but tweaking e1000e ring buffers
> didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs,
> I tried running on a different server with bnx cards, I tried disabling
> NO_HZ and HRTICK, but still I have the same problem.
> 
> However, if I don't utilize bond, but just apply rules on normal ethX
> interfaces, there is no packet loss with 2.6.28/29. 
> 
> So, the problem appears only when I use 2.6.28/29 + bond + classful tc
> combination. 
> 
> Any ideas ?
> 

Yes, we need much more information :)
Is it a forwarding setup only ?

cat /proc/interrupts
cat /proc/net/bonding/bond0
cat /proc/net/bonding/bond1
tc -s -d qdisc
mpstat -P ALL 10
ifconfig -a

and so on ...

Re: bond + tc regression ?

[Vladimir Ivashchenko]

> > On both kernels, the system is running with at least 70% idle CPU.
> > The network interrupts are distributed accross the cores.
> 
> You should not distribute interrupts, but bound a NIC to one CPU

Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct.
The defaults are wrong?

I have tried with IRQs bound to one CPU per NIC. Same result.

> > I thought it was a e1000e driver issue, but tweaking e1000e ring buffers
> > didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs,
> > I tried running on a different server with bnx cards, I tried disabling
> > NO_HZ and HRTICK, but still I have the same problem.
> > 
> > However, if I don't utilize bond, but just apply rules on normal ethX
> > interfaces, there is no packet loss with 2.6.28/29. 
> > 
> > So, the problem appears only when I use 2.6.28/29 + bond + classful tc
> > combination. 
> > 
> > Any ideas ?
> > 
> 
> Yes, we need much more information :)
> Is it a forwarding setup only ?

Yes, the server is doing nothing else but forwarding, no iptables.

> cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
  0:        130          0          0          0          0          0          0          0   IO-APIC-edge      timer
  1:          2          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  3:          0          0          0          1          0          1          0          0   IO-APIC-edge
  4:          0          0          1          0          0          0          1          0   IO-APIC-edge
  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 12:          4          0          0          0          0          0          0          0   IO-APIC-edge      i8042
 14:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
 15:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
 17:      30901      31910      31446      30655      31618      30550      31543      30958   IO-APIC-fasteoi   aacraid
 20:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
 21:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb5, ahci
 22:     298387     297642     295508     294368     295533     295430     295275     296036   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
 23:      10868      10926      10980      10738      10939      10615      10761      10909   IO-APIC-fasteoi   uhci_hcd:usb3
 57: 1486251823 1486835830 1486677250 1487105983 1488000303 1485941815 1487728317 1486624997   PCI-MSI-edge      eth0
 58: 1510676329 1509708161 1510347202 1509969755 1508599471 1511220118 1509094578 1509727616   PCI-MSI-edge      eth1
 59: 1482578890 1483618556 1482963700 1483164528 1484561615 1482130645 1484116749 1483557717   PCI-MSI-edge      eth2
 60: 1507341647 1506685822 1506862759 1506612818 1505689367 1507559672 1505911622 1506940613   PCI-MSI-edge      eth3
NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
LOC: 1020533656 1020535165 1020533613 1020534967 1020535173 1020534409 1020534985 1020534220   Local timer interrupts
RES:      18605      21215      15957      18637      22429      19493      16649      15589   Rescheduling interrupts
CAL:        160        214        186        185        199        205        190        180   Function call interrupts
TLB:     259515     264126     309016     312222     263163     265601     306189     305430   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
ERR:          0
MIS:          0

> tc -s -d qdisc

For test sake, I just put "tc qdisc add dev $IFACE root handle 1: prio" and no filters at all. 
I get the same with HTB "tc qdisc add dev $IFACE root handle 1: htb default 99" and no subclasses.

qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 13287736273644 bytes 1263672018 pkt (dropped 0, overlimits 0 requeues 2928480094)
 rate 0bit 0pps backlog 0b 0p requeues 2928480094
qdisc pfifo_fast 0: dev eth1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 40064376195000 bytes 1747026586 pkt (dropped 0, overlimits 0 requeues 463621814)
 rate 0bit 0pps backlog 0b 0p requeues 463621814
qdisc pfifo_fast 0: dev eth2 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 13350145517965 bytes 1350897201 pkt (dropped 0, overlimits 0 requeues 2930879507)
 rate 0bit 0pps backlog 0b 0p requeues 2930879507
qdisc pfifo_fast 0: dev eth3 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 40193456126884 bytes 1950653764 pkt (dropped 0, overlimits 0 requeues 465511120)
 rate 0bit 0pps backlog 0b 0p requeues 465511120
qdisc prio 1: dev bond0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 985164834 bytes 2720991 pkt (dropped 241834, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 1: dev bond1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 2347118738 bytes 3089171 pkt (dropped 304601, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

** Drops on bond0/bond1 are increasing by approximately 5000 per second:

qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 13287874353796 bytes 1264050808 pkt (dropped 0, overlimits 0 requeues 2928520779)
 rate 0bit 0pps backlog 0b 0p requeues 2928520779
qdisc pfifo_fast 0: dev eth1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 40064706826018 bytes 1747459793 pkt (dropped 0, overlimits 0 requeues 463669610)
 rate 0bit 0pps backlog 0b 0p requeues 463669610
qdisc pfifo_fast 0: dev eth2 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 13350283202695 bytes 1351277761 pkt (dropped 0, overlimits 0 requeues 2930918488)
 rate 0bit 0pps backlog 0b 0p requeues 2930918488
qdisc pfifo_fast 0: dev eth3 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 40193784868074 bytes 1951084029 pkt (dropped 0, overlimits 0 requeues 465558015)
 rate 0bit 0pps backlog 0b 0p requeues 465558015
qdisc prio 1: dev bond0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 1260929539 bytes 3480340 pkt (dropped 311145, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0
qdisc prio 1: dev bond1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 3006490946 bytes 3952643 pkt (dropped 396850, overlimits 0 requeues 0)
 rate 0bit 0pps backlog 0b 0p requeues 0

With same setup on 2.6.23, drops are increasing only by 50/sec or so.

As soon as I do "tc qdisc del dev $IFACE root", packet loss stops.

> cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 80
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 17
        Partner Key: 4
        Partner Mac Address: 00:19:e7:b2:07:80

Slave Interface: eth0
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:cc
Aggregator ID: 1

Slave Interface: eth2
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:ce
Aggregator ID: 1

> cat /proc/net/bonding/bond1

Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 80
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 2
        Actor Key: 17
        Partner Key: 5
        Partner Mac Address: 00:19:e7:b2:07:80

Slave Interface: eth1
MII Status: up
Link Failure Count: 1
Permanent HW addr: 00:1b:24:bd:e9:cd
Aggregator ID: 2

Slave Interface: eth3
MII Status: up
Link Failure Count: 2
Permanent HW addr: 00:1b:24:bd:e9:cf
Aggregator ID: 2


> mpstat -P ALL 10

08:04:36 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
08:04:46 PM  all    0.00    0.00    0.01    0.00    0.00    1.05    0.00   98.94  70525.73
08:04:46 PM    0    0.00    0.00    0.00    0.00    0.00    0.70    0.00   99.30   7814.41
08:04:46 PM    1    0.00    0.00    0.00    0.00    0.00    2.10    0.00   97.90   7814.41
08:04:46 PM    2    0.00    0.00    0.00    0.00    0.00    0.20    0.00   99.80   7814.41
08:04:46 PM    3    0.00    0.00    0.10    0.00    0.00    1.30    0.00   98.60   7814.51
08:04:46 PM    4    0.00    0.00    0.00    0.00    0.00    0.50    0.00   99.50   7814.41
08:04:46 PM    5    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7814.41
08:04:46 PM    6    0.00    0.00    0.00    0.00    0.00    0.60    0.00   99.40   7814.41
08:04:46 PM    7    0.00    0.00    0.10    0.00    0.00    0.90    0.00   99.00   7814.51
08:04:46 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00

08:04:46 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
08:04:56 PM  all    0.00    0.00    0.01    0.00    0.00    1.49    0.00   98.50  66429.30
08:04:56 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00   7303.50
08:04:56 PM    1    0.00    0.00    0.00    0.00    0.00    1.60    0.00   98.40   7303.50
08:04:56 PM    2    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    3    0.00    0.00    0.00    0.00    0.00    3.20    0.00   96.80   7303.40
08:04:56 PM    4    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7303.60
08:04:56 PM    5    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    6    0.00    0.00    0.10    0.00    0.00    1.80    0.00   98.10   7303.50
08:04:56 PM    7    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
08:04:56 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00

> ifconfig -a

bond0     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
          inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
          TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)

bond1     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CD
          inet addr:xxx.xxx.70.156  Bcast:xxx.xxx.70.159  Mask:255.255.255.248
          inet6 addr: fe80::21b:24ff:febd:e9cd/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:239471641 errors:0 dropped:344 overruns:0 frame:0
          TX packets:3704083902 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2488754745 (2.3 GiB)  TX bytes:2685275089 (2.5 GiB)

eth0      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:2235085582 errors:0 dropped:353786 overruns:0 frame:0
          TX packets:1266449269 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3768096439 (3.5 GiB)  TX bytes:113363829 (108.1 MiB)
          Memory:fc6e0000-fc700000

eth1      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CD
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:4228974804 errors:0 dropped:344 overruns:0 frame:0
          TX packets:1750216649 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3350270261 (3.1 GiB)  TX bytes:3358220645 (3.1 GiB)
          Memory:fc6c0000-fc6e0000

eth2      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:2495958020 errors:0 dropped:37464 overruns:0 frame:0
          TX packets:1353707165 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:442055526 (421.5 MiB)  TX bytes:2406943933 (2.2 GiB)
          Memory:fcde0000-fce00000

eth3      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CD
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:305464222 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1953867360 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3433479245 (3.1 GiB)  TX bytes:3622113909 (3.3 GiB)
          Memory:fcd80000-fcda0000

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:53537 errors:0 dropped:0 overruns:0 frame:0
          TX packets:53537 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:431006433 (411.0 MiB)  TX bytes:431006433 (411.0 MiB)


NOTE: ifconfig drops on bond0/bond1 are *NOT* increasing. These drops are there from before.

-- 
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com

Re: bond + tc regression ?

[Eric Dumazet]

Vladimir Ivashchenko a écrit :
>>> On both kernels, the system is running with at least 70% idle CPU.
>>> The network interrupts are distributed accross the cores.
>> You should not distribute interrupts, but bound a NIC to one CPU
> 
> Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct.
> The defaults are wrong?

Yes they are, at least for forwarding setups.

> 
> I have tried with IRQs bound to one CPU per NIC. Same result.

Did you check "grep eth /proc/interrupts" that your affinities setup 
were indeed taken into account ?

You should use same CPU for eth0 and eth2 (bond0),

and another CPU for eth1 and eth3 (bond1)

check how your cpus are setup 

egrep 'physical id|core id|processor' /proc/cpuinfo

Because you might play and find best combo


If you use 2.6.29, apply following patch to get better system accounting,
to check if your cpu are saturated or not by hard/soft irqs

--- linux-2.6.29/kernel/sched.c.orig    2009-05-05 20:46:49.000000000 +0200
+++ linux-2.6.29/kernel/sched.c 2009-05-05 20:47:19.000000000 +0200
@@ -4290,7 +4290,7 @@

        if (user_tick)
                account_user_time(p, one_jiffy, one_jiffy_scaled);
-       else if (p != rq->idle)
+       else if ((p != rq->idle) || (irq_count() != HARDIRQ_OFFSET))
                account_system_time(p, HARDIRQ_OFFSET, one_jiffy,
                                    one_jiffy_scaled);
        else



> 
>>> I thought it was a e1000e driver issue, but tweaking e1000e ring buffers
>>> didn't help. I tried using e1000 on 2.6.28 by adding necessary PCI IDs,
>>> I tried running on a different server with bnx cards, I tried disabling
>>> NO_HZ and HRTICK, but still I have the same problem.
>>>
>>> However, if I don't utilize bond, but just apply rules on normal ethX
>>> interfaces, there is no packet loss with 2.6.28/29. 
>>>
>>> So, the problem appears only when I use 2.6.28/29 + bond + classful tc
>>> combination. 
>>>
>>> Any ideas ?
>>>
>> Yes, we need much more information :)
>> Is it a forwarding setup only ?
> 
> Yes, the server is doing nothing else but forwarding, no iptables.
> 
>> cat /proc/interrupts
> 
>            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7
>   0:        130          0          0          0          0          0          0          0   IO-APIC-edge      timer
>   1:          2          0          0          0          0          0          0          0   IO-APIC-edge      i8042
>   3:          0          0          0          1          0          1          0          0   IO-APIC-edge
>   4:          0          0          1          0          0          0          1          0   IO-APIC-edge
>   9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
>  12:          4          0          0          0          0          0          0          0   IO-APIC-edge      i8042
>  14:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
>  15:          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
>  17:      30901      31910      31446      30655      31618      30550      31543      30958   IO-APIC-fasteoi   aacraid
>  20:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
>  21:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb5, ahci
>  22:     298387     297642     295508     294368     295533     295430     295275     296036   IO-APIC-fasteoi   ehci_hcd:usb1, uhci_hcd:usb2
>  23:      10868      10926      10980      10738      10939      10615      10761      10909   IO-APIC-fasteoi   uhci_hcd:usb3
>  57: 1486251823 1486835830 1486677250 1487105983 1488000303 1485941815 1487728317 1486624997   PCI-MSI-edge      eth0
>  58: 1510676329 1509708161 1510347202 1509969755 1508599471 1511220118 1509094578 1509727616   PCI-MSI-edge      eth1
>  59: 1482578890 1483618556 1482963700 1483164528 1484561615 1482130645 1484116749 1483557717   PCI-MSI-edge      eth2
>  60: 1507341647 1506685822 1506862759 1506612818 1505689367 1507559672 1505911622 1506940613   PCI-MSI-edge      eth3
> NMI:          0          0          0          0          0          0          0          0   Non-maskable interrupts
> LOC: 1020533656 1020535165 1020533613 1020534967 1020535173 1020534409 1020534985 1020534220   Local timer interrupts
> RES:      18605      21215      15957      18637      22429      19493      16649      15589   Rescheduling interrupts
> CAL:        160        214        186        185        199        205        190        180   Function call interrupts
> TLB:     259515     264126     309016     312222     263163     265601     306189     305430   TLB shootdowns
> TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
> SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
> ERR:          0
> MIS:          0
> 
>> tc -s -d qdisc
> 
> For test sake, I just put "tc qdisc add dev $IFACE root handle 1: prio" and no filters at all. 
> I get the same with HTB "tc qdisc add dev $IFACE root handle 1: htb default 99" and no subclasses.
> 
> qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 13287736273644 bytes 1263672018 pkt (dropped 0, overlimits 0 requeues 2928480094)
>  rate 0bit 0pps backlog 0b 0p requeues 2928480094
> qdisc pfifo_fast 0: dev eth1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 40064376195000 bytes 1747026586 pkt (dropped 0, overlimits 0 requeues 463621814)
>  rate 0bit 0pps backlog 0b 0p requeues 463621814
> qdisc pfifo_fast 0: dev eth2 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 13350145517965 bytes 1350897201 pkt (dropped 0, overlimits 0 requeues 2930879507)
>  rate 0bit 0pps backlog 0b 0p requeues 2930879507
> qdisc pfifo_fast 0: dev eth3 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 40193456126884 bytes 1950653764 pkt (dropped 0, overlimits 0 requeues 465511120)
>  rate 0bit 0pps backlog 0b 0p requeues 465511120
> qdisc prio 1: dev bond0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 985164834 bytes 2720991 pkt (dropped 241834, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> qdisc prio 1: dev bond1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 2347118738 bytes 3089171 pkt (dropped 304601, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> 
> ** Drops on bond0/bond1 are increasing by approximately 5000 per second:
> 
> qdisc pfifo_fast 0: dev eth0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 13287874353796 bytes 1264050808 pkt (dropped 0, overlimits 0 requeues 2928520779)
>  rate 0bit 0pps backlog 0b 0p requeues 2928520779
> qdisc pfifo_fast 0: dev eth1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 40064706826018 bytes 1747459793 pkt (dropped 0, overlimits 0 requeues 463669610)
>  rate 0bit 0pps backlog 0b 0p requeues 463669610
> qdisc pfifo_fast 0: dev eth2 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 13350283202695 bytes 1351277761 pkt (dropped 0, overlimits 0 requeues 2930918488)
>  rate 0bit 0pps backlog 0b 0p requeues 2930918488
> qdisc pfifo_fast 0: dev eth3 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 40193784868074 bytes 1951084029 pkt (dropped 0, overlimits 0 requeues 465558015)
>  rate 0bit 0pps backlog 0b 0p requeues 465558015
> qdisc prio 1: dev bond0 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 1260929539 bytes 3480340 pkt (dropped 311145, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> qdisc prio 1: dev bond1 root bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
>  Sent 3006490946 bytes 3952643 pkt (dropped 396850, overlimits 0 requeues 0)
>  rate 0bit 0pps backlog 0b 0p requeues 0
> 
> With same setup on 2.6.23, drops are increasing only by 50/sec or so.
> 
> As soon as I do "tc qdisc del dev $IFACE root", packet loss stops.
> 
>> cat /proc/net/bonding/bond0
> 
> Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
> 
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> Transmit Hash Policy: layer3+4 (1)
> MII Status: up
> MII Polling Interval (ms): 80
> Up Delay (ms): 0
> Down Delay (ms): 0
> 
> 802.3ad info
> LACP rate: slow
> Aggregator selection policy (ad_select): stable
> Active Aggregator Info:
>         Aggregator ID: 1
>         Number of ports: 2
>         Actor Key: 17
>         Partner Key: 4
>         Partner Mac Address: 00:19:e7:b2:07:80
> 
> Slave Interface: eth0
> MII Status: up
> Link Failure Count: 1
> Permanent HW addr: 00:1b:24:bd:e9:cc
> Aggregator ID: 1
> 
> Slave Interface: eth2
> MII Status: up
> Link Failure Count: 1
> Permanent HW addr: 00:1b:24:bd:e9:ce
> Aggregator ID: 1
> 
>> cat /proc/net/bonding/bond1
> 
> Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)
> 
> Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> Transmit Hash Policy: layer3+4 (1)
> MII Status: up
> MII Polling Interval (ms): 80
> Up Delay (ms): 0
> Down Delay (ms): 0
> 
> 802.3ad info
> LACP rate: slow
> Aggregator selection policy (ad_select): stable
> Active Aggregator Info:
>         Aggregator ID: 2
>         Number of ports: 2
>         Actor Key: 17
>         Partner Key: 5
>         Partner Mac Address: 00:19:e7:b2:07:80
> 
> Slave Interface: eth1
> MII Status: up
> Link Failure Count: 1
> Permanent HW addr: 00:1b:24:bd:e9:cd
> Aggregator ID: 2
> 
> Slave Interface: eth3
> MII Status: up
> Link Failure Count: 2
> Permanent HW addr: 00:1b:24:bd:e9:cf
> Aggregator ID: 2
> 
> 
>> mpstat -P ALL 10
> 
> 08:04:36 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
> 08:04:46 PM  all    0.00    0.00    0.01    0.00    0.00    1.05    0.00   98.94  70525.73
> 08:04:46 PM    0    0.00    0.00    0.00    0.00    0.00    0.70    0.00   99.30   7814.41
> 08:04:46 PM    1    0.00    0.00    0.00    0.00    0.00    2.10    0.00   97.90   7814.41
> 08:04:46 PM    2    0.00    0.00    0.00    0.00    0.00    0.20    0.00   99.80   7814.41
> 08:04:46 PM    3    0.00    0.00    0.10    0.00    0.00    1.30    0.00   98.60   7814.51
> 08:04:46 PM    4    0.00    0.00    0.00    0.00    0.00    0.50    0.00   99.50   7814.41
> 08:04:46 PM    5    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7814.41
> 08:04:46 PM    6    0.00    0.00    0.00    0.00    0.00    0.60    0.00   99.40   7814.41
> 08:04:46 PM    7    0.00    0.00    0.10    0.00    0.00    0.90    0.00   99.00   7814.51
> 08:04:46 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
> 
> 08:04:46 PM  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
> 08:04:56 PM  all    0.00    0.00    0.01    0.00    0.00    1.49    0.00   98.50  66429.30
> 08:04:56 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00   7303.50
> 08:04:56 PM    1    0.00    0.00    0.00    0.00    0.00    1.60    0.00   98.40   7303.50
> 08:04:56 PM    2    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
> 08:04:56 PM    3    0.00    0.00    0.00    0.00    0.00    3.20    0.00   96.80   7303.40
> 08:04:56 PM    4    0.00    0.00    0.00    0.00    0.00    1.90    0.00   98.10   7303.60
> 08:04:56 PM    5    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
> 08:04:56 PM    6    0.00    0.00    0.10    0.00    0.00    1.80    0.00   98.10   7303.50
> 08:04:56 PM    7    0.00    0.00    0.00    0.00    0.00    1.20    0.00   98.80   7303.50
> 08:04:56 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00      0.00
> 
>> ifconfig -a
> 
> bond0     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
>           inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
>           inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
>           TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)
> 
> bond1     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CD
>           inet addr:xxx.xxx.70.156  Bcast:xxx.xxx.70.159  Mask:255.255.255.248
>           inet6 addr: fe80::21b:24ff:febd:e9cd/64 Scope:Link
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:239471641 errors:0 dropped:344 overruns:0 frame:0
>           TX packets:3704083902 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:2488754745 (2.3 GiB)  TX bytes:2685275089 (2.5 GiB)
> 
> eth0      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:2235085582 errors:0 dropped:353786 overruns:0 frame:0
>           TX packets:1266449269 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:3768096439 (3.5 GiB)  TX bytes:113363829 (108.1 MiB)
>           Memory:fc6e0000-fc700000
> 
> eth1      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CD
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:4228974804 errors:0 dropped:344 overruns:0 frame:0
>           TX packets:1750216649 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:3350270261 (3.1 GiB)  TX bytes:3358220645 (3.1 GiB)
>           Memory:fc6c0000-fc6e0000
> 
> eth2      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:2495958020 errors:0 dropped:37464 overruns:0 frame:0
>           TX packets:1353707165 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:442055526 (421.5 MiB)  TX bytes:2406943933 (2.2 GiB)
>           Memory:fcde0000-fce00000
> 
> eth3      Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CD
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:305464222 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:1953867360 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:3433479245 (3.1 GiB)  TX bytes:3622113909 (3.3 GiB)
>           Memory:fcd80000-fcda0000
> 
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:53537 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:53537 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:431006433 (411.0 MiB)  TX bytes:431006433 (411.0 MiB)
> 
> 
> NOTE: ifconfig drops on bond0/bond1 are *NOT* increasing. These drops are there from before.
> 

Re: bond + tc regression ?

[Vladimir Ivashchenko]

On Tue, May 05, 2009 at 08:50:26PM +0200, Eric Dumazet wrote:

> > I have tried with IRQs bound to one CPU per NIC. Same result.
> 
> Did you check "grep eth /proc/interrupts" that your affinities setup 
> were indeed taken into account ?
> 
> You should use same CPU for eth0 and eth2 (bond0),
> 
> and another CPU for eth1 and eth3 (bond1)

Ok, the best result is when assign all IRQs to the same CPU. Zero drops.

When I bind slaves of bond interfaces to the same CPU, I start to get 
some drops, but much less than before. I didn't play with combinations.

My problem is, after applying your accounting patch below, one of my 
HTB servers reports only 30-40% CPU idle on one of the cores. That won't 
take me for very long, load balancing across cores is needed.

Is there any way at least to balance individual NICs on per core basis?

-- 
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com

Re: bond + tc regression ?

[Stephen Hemminger]

On Wed, 6 May 2009 02:50:08 +0300
Vladimir Ivashchenko <hazard@francoudi.com> wrote:

> On Tue, May 05, 2009 at 08:50:26PM +0200, Eric Dumazet wrote:
> 
> > > I have tried with IRQs bound to one CPU per NIC. Same result.
> > 
> > Did you check "grep eth /proc/interrupts" that your affinities setup 
> > were indeed taken into account ?
> > 
> > You should use same CPU for eth0 and eth2 (bond0),
> > 
> > and another CPU for eth1 and eth3 (bond1)
> 
> Ok, the best result is when assign all IRQs to the same CPU. Zero drops.
> 
> When I bind slaves of bond interfaces to the same CPU, I start to get 
> some drops, but much less than before. I didn't play with combinations.
> 
> My problem is, after applying your accounting patch below, one of my 
> HTB servers reports only 30-40% CPU idle on one of the cores. That won't 
> take me for very long, load balancing across cores is needed.
> 
> Is there any way at least to balance individual NICs on per core basis?
> 

The user level irqbalance program is a good place to start:
  http://www.irqbalance.org/
But it doesn't yet no how to handle multi-queue devices, and it seems
to not handle NUMA (like SMP Nehalam) perfectly.

Re: bond + tc regression ?

[Eric Dumazet]

Vladimir Ivashchenko a écrit :
> On Tue, May 05, 2009 at 08:50:26PM +0200, Eric Dumazet wrote:
> 
>>> I have tried with IRQs bound to one CPU per NIC. Same result.
>> Did you check "grep eth /proc/interrupts" that your affinities setup 
>> were indeed taken into account ?
>>
>> You should use same CPU for eth0 and eth2 (bond0),
>>
>> and another CPU for eth1 and eth3 (bond1)
> 
> Ok, the best result is when assign all IRQs to the same CPU. Zero drops.
> 
> When I bind slaves of bond interfaces to the same CPU, I start to get 
> some drops, but much less than before. I didn't play with combinations.
> 
> My problem is, after applying your accounting patch below, one of my 
> HTB servers reports only 30-40% CPU idle on one of the cores. That won't 
> take me for very long, load balancing across cores is needed.
> 
> Is there any way at least to balance individual NICs on per core basis?
> 

Problem of this setup is you have four NICS, but two logical devices (bond0
& bond1) and a central HTB thing. This essentialy makes flows go through the same
locks (some rwlocks guarding bonding driver, and others guarding HTB structures).

Also when a cpu receives a frame on ethX, it has to forward it on ethY, and
another lock guards access to TX queue of ethY device. If another cpus receives
a frame on ethZ and want to forward it to ethY device, this other cpu will
need same locks and everything slowdown.

I am pretty sure you could get good results choosing two cpus sharing same L2
cache. L2 on your cpu is 6MB. Another point would be to carefuly choose size
of RX rings on ethX devices. You could try to *reduce* them so that number
of inflight skb is small enough that everything fits in this 6MB cache.

Problem is not really CPU power, but RAM bandwidth. Having two cores instead of one
attached to one central memory bank wont increase ram bandwidth, but reduce it.

And making several cores compete for locks on this ram only slows down processing.

Only choice we have is to change bonding so that this driver uses RCU instead
of rwlocks, but it is probably a complex task. Multiple cpus accessing
bonding structures could share memory structures without dirtying them
and ping-pong cache lines.

Ah, I forgot about one patch that could help your setup too (if using more than one
cpu on NIC irqs of course), queued for 2.6.31

(commit 6a321cb370ad3db4ba6e405e638b3a42c41089b0)

You could post oprofile results to help us finding other hot spots.


[PATCH] net: netif_tx_queue_stopped too expensive

netif_tx_queue_stopped(txq) is most of the time false.

Yet its cost is very expensive on SMP.

static inline int netif_tx_queue_stopped(const struct netdev_queue *dev_queue)
{
	return test_bit(__QUEUE_STATE_XOFF, &dev_queue->state);
}

I saw this on oprofile hunting and bnx2 driver bnx2_tx_int().

We probably should split "struct netdev_queue" in two parts, one
being read mostly.

__netif_tx_lock() touches _xmit_lock & xmit_lock_owner, these
deserve a separate cache line.

Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>


diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 2e7783f..1caaebb 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -447,12 +447,18 @@ enum netdev_queue_state_t
 };
 
 struct netdev_queue {
+/*
+ * read mostly part
+ */
 	struct net_device	*dev;
 	struct Qdisc		*qdisc;
 	unsigned long		state;
-	spinlock_t		_xmit_lock;
-	int			xmit_lock_owner;
 	struct Qdisc		*qdisc_sleeping;
+/*
+ * write mostly part
+ */
+	spinlock_t		_xmit_lock ____cacheline_aligned_in_smp;
+	int			xmit_lock_owner;
 } ____cacheline_aligned_in_smp;
 
 

Re: bond + tc regression ?

[Jarek Poplawski]

On 05-05-2009 19:41, Vladimir Ivashchenko wrote:
>>> On both kernels, the system is running with at least 70% idle CPU.
>>> The network interrupts are distributed accross the cores.
>> You should not distribute interrupts, but bound a NIC to one CPU
> 
> Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct.
> The defaults are wrong?
> 
...
>> ifconfig -a
> 
> bond0     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
>           inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
>           inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
>           TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)

Could you try e.g.: ifconfig bond0 txqueuelen 1000
before tc qdisc add?

Jarek P.

Re: bond + tc regression ?

[Ingo Molnar]

* Eric Dumazet <dada1@cosmosbay.com> wrote:

> Vladimir Ivashchenko a écrit :
> >>> On both kernels, the system is running with at least 70% idle CPU.
> >>> The network interrupts are distributed accross the cores.
> >> You should not distribute interrupts, but bound a NIC to one CPU
> > 
> > Kernels 2.6.28 and 2.6.29 do this by default, so I thought its correct.
> > The defaults are wrong?
> 
> Yes they are, at least for forwarding setups.
> 
> > 
> > I have tried with IRQs bound to one CPU per NIC. Same result.
> 
> Did you check "grep eth /proc/interrupts" that your affinities setup 
> were indeed taken into account ?
> 
> You should use same CPU for eth0 and eth2 (bond0),
> 
> and another CPU for eth1 and eth3 (bond1)
> 
> check how your cpus are setup 
> 
> egrep 'physical id|core id|processor' /proc/cpuinfo
> 
> Because you might play and find best combo
> 
> 
> If you use 2.6.29, apply following patch to get better system accounting,
> to check if your cpu are saturated or not by hard/soft irqs
> 
> --- linux-2.6.29/kernel/sched.c.orig    2009-05-05 20:46:49.000000000 +0200
> +++ linux-2.6.29/kernel/sched.c 2009-05-05 20:47:19.000000000 +0200
> @@ -4290,7 +4290,7 @@
> 
>         if (user_tick)
>                 account_user_time(p, one_jiffy, one_jiffy_scaled);
> -       else if (p != rq->idle)
> +       else if ((p != rq->idle) || (irq_count() != HARDIRQ_OFFSET))
>                 account_system_time(p, HARDIRQ_OFFSET, one_jiffy,
>                                     one_jiffy_scaled);
>         else

Note, your scheduler fix is upstream now in Linus's tree, as:

  f5f293a: sched: account system time properly

"git cherry-pick f5f293a" will apply it to a .29 basis.

	Ingo

Re: bond + tc regression ?

[Vladimir Ivashchenko]

On Wed, May 06, 2009 at 05:36:08AM +0200, Eric Dumazet wrote:

> > Is there any way at least to balance individual NICs on per core basis?
> > 
> 
> Problem of this setup is you have four NICS, but two logical devices (bond0
> & bond1) and a central HTB thing. This essentialy makes flows go through the same
> locks (some rwlocks guarding bonding driver, and others guarding HTB structures).
> 
> Also when a cpu receives a frame on ethX, it has to forward it on ethY, and
> another lock guards access to TX queue of ethY device. If another cpus receives
> a frame on ethZ and want to forward it to ethY device, this other cpu will
> need same locks and everything slowdown.
> 
> I am pretty sure you could get good results choosing two cpus sharing same L2
> cache. L2 on your cpu is 6MB. Another point would be to carefuly choose size
> of RX rings on ethX devices. You could try to *reduce* them so that number
> of inflight skb is small enough that everything fits in this 6MB cache.
> 
> Problem is not really CPU power, but RAM bandwidth. Having two cores instead of one
> attached to one central memory bank wont increase ram bandwidth, but reduce it.

Thanks for the detailed explanation.

On the particular server I reported, I worked around the problem by getting rid of classes 
and switching to ingress policers.

However, I have one central box doing HTB, small amount of classes, but 850 mbps of
traffic. The CPU is dual-core 5160 @ 3 Ghz. With 2.6.29 + bond I'm experiencing strange problems 
with HTB, under high load borrowing doesn't seem to work properly. This box has two 
BNX2 and two E1000 NICs, and for some reason I cannot force BNX2 to sit on a single IRQ -
even though I put only one CPU into smp_affinity, it keeps balancing on both. So I cannot
figure out if its related to IRQ balancing or not.

[root@tshape3 tshaper]# cat /proc/irq/63/smp_affinity
01
[root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
 63:   44610754   95469129   PCI-MSI-edge      eth0
[root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
 63:   44614125   95472512   PCI-MSI-edge      eth0

lspci -v:

03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
        Subsystem: Hewlett-Packard Company NC373i Integrated Multifunction Gigabit Server Adapter
        Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 63
        Memory at f8000000 (64-bit, non-prefetchable) [size=32M]
        [virtual] Expansion ROM at 88200000 [disabled] [size=2K]
        Capabilities: [40] PCI-X non-bridge device
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data <?>
        Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
        Kernel driver in use: bnx2
        Kernel modules: bnx2


Any ideas on how to force it on a single CPU ?

Thanks for the new patch, I will try it and let you know.

-- 
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com

Re: bond + tc regression ?

[Vladimir Ivashchenko]

On Wed, May 06, 2009 at 06:10:10AM +0000, Jarek Poplawski wrote:

> >> ifconfig -a
> > 
> > bond0     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
> >           inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
> >           inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
> >           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
> >           RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
> >           TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
> >           collisions:0 txqueuelen:0
> >           RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)
> 
> Could you try e.g.: ifconfig bond0 txqueuelen 1000
> before tc qdisc add?

The drops on ifconfig are not increasing - these numbers are there from some tests made
before.

-- 
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com

Re: bond + tc regression ?

[Eric Dumazet]

Vladimir Ivashchenko a écrit :
> On Wed, May 06, 2009 at 05:36:08AM +0200, Eric Dumazet wrote:
> 
>>> Is there any way at least to balance individual NICs on per core basis?
>>>
>> Problem of this setup is you have four NICS, but two logical devices (bond0
>> & bond1) and a central HTB thing. This essentialy makes flows go through the same
>> locks (some rwlocks guarding bonding driver, and others guarding HTB structures).
>>
>> Also when a cpu receives a frame on ethX, it has to forward it on ethY, and
>> another lock guards access to TX queue of ethY device. If another cpus receives
>> a frame on ethZ and want to forward it to ethY device, this other cpu will
>> need same locks and everything slowdown.
>>
>> I am pretty sure you could get good results choosing two cpus sharing same L2
>> cache. L2 on your cpu is 6MB. Another point would be to carefuly choose size
>> of RX rings on ethX devices. You could try to *reduce* them so that number
>> of inflight skb is small enough that everything fits in this 6MB cache.
>>
>> Problem is not really CPU power, but RAM bandwidth. Having two cores instead of one
>> attached to one central memory bank wont increase ram bandwidth, but reduce it.
> 
> Thanks for the detailed explanation.
> 
> On the particular server I reported, I worked around the problem by getting rid of classes 
> and switching to ingress policers.
> 
> However, I have one central box doing HTB, small amount of classes, but 850 mbps of
> traffic. The CPU is dual-core 5160 @ 3 Ghz. With 2.6.29 + bond I'm experiencing strange problems 
> with HTB, under high load borrowing doesn't seem to work properly. This box has two 
> BNX2 and two E1000 NICs, and for some reason I cannot force BNX2 to sit on a single IRQ -
> even though I put only one CPU into smp_affinity, it keeps balancing on both. So I cannot
> figure out if its related to IRQ balancing or not.
> 
> [root@tshape3 tshaper]# cat /proc/irq/63/smp_affinity
> 01
> [root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
>  63:   44610754   95469129   PCI-MSI-edge      eth0
> [root@tshape3 tshaper]# cat /proc/interrupts | grep eth0
>  63:   44614125   95472512   PCI-MSI-edge      eth0
> 
> lspci -v:
> 
> 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
>         Subsystem: Hewlett-Packard Company NC373i Integrated Multifunction Gigabit Server Adapter
>         Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 63
>         Memory at f8000000 (64-bit, non-prefetchable) [size=32M]
>         [virtual] Expansion ROM at 88200000 [disabled] [size=2K]
>         Capabilities: [40] PCI-X non-bridge device
>         Capabilities: [48] Power Management version 2
>         Capabilities: [50] Vital Product Data <?>
>         Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable+
>         Kernel driver in use: bnx2
>         Kernel modules: bnx2
> 
> 
> Any ideas on how to force it on a single CPU ?
> 
> Thanks for the new patch, I will try it and let you know.
> 

Yes, its doable but tricky with bnx2, this is a known problem on recent kernels as well.


You must do for example (to bind on CPU 0)

echo 1 >/proc/irq/default_smp_affinity

ifconfig eth1 down
# IRQ of eth1 handled by CPU0 only
echo 1 >/proc/irq/34/smp_affinity
ifconfig eth1 up

ifconfig eth0 down
# IRQ of eth0 handled by CPU0 only
echo 1 >/proc/irq/36/smp_affinity
ifconfig eth0 up


One thing to consider too is the BIOS option you might have, labeled "Adjacent Sector Prefetch"

This basically tells your cpu to use 128 bytes cache lines, instead of 64

In your forwarding worload, I believe this extra prefetch can slowdown your machine.

Re: bond + tc regression ?

[Jarek Poplawski]

On Wed, May 06, 2009 at 01:36:16PM +0300, Vladimir Ivashchenko wrote:
> On Wed, May 06, 2009 at 06:10:10AM +0000, Jarek Poplawski wrote:
> 
> > >> ifconfig -a
> > > 
> > > bond0     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
> > >           inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
> > >           inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
> > >           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
> > >           RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
> > >           TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
> > >           collisions:0 txqueuelen:0
> > >           RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)
> > 
> > Could you try e.g.: ifconfig bond0 txqueuelen 1000
> > before tc qdisc add?
> 
> The drops on ifconfig are not increasing - these numbers are there from some tests made
> before.
> 

I'm not sure what do you mean? IMHO you don't use qdiscs properly,
so any TX problems end with drops. (Older kernel versions could mask
this problem with requeuing.)

Jarek P.

Re: bond + tc regression ?

[Denys Fedoryschenko]

On Wednesday 06 May 2009 13:41:25 Eric Dumazet wrote:
> You must do for example (to bind on CPU 0)
>
> echo 1 >/proc/irq/default_smp_affinity
>
> ifconfig eth1 down
> # IRQ of eth1 handled by CPU0 only
> echo 1 >/proc/irq/34/smp_affinity
> ifconfig eth1 up
>
> ifconfig eth0 down
> # IRQ of eth0 handled by CPU0 only
> echo 1 >/proc/irq/36/smp_affinity
> ifconfig eth0 up
I think better to use some method over ethtool, that will cause reset.
WHen you do down - you will loose default route, beware of that
>
>
> One thing to consider too is the BIOS option you might have, labeled
> "Adjacent Sector Prefetch"
>
> This basically tells your cpu to use 128 bytes cache lines, instead of 64
>
> In your forwarding worload, I believe this extra prefetch can slowdown your
> machine.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: bond + tc regression ?

[Vladimir Ivashchenko]

On Wed, May 06, 2009 at 10:48:08AM +0000, Jarek Poplawski wrote:

> > > >> ifconfig -a
> > > > 
> > > > bond0     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
> > > >           inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
> > > >           inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
> > > >           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
> > > >           RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
> > > >           TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
> > > >           collisions:0 txqueuelen:0
> > > >           RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)
> > > 
> > > Could you try e.g.: ifconfig bond0 txqueuelen 1000
> > > before tc qdisc add?
> > 
> > The drops on ifconfig are not increasing - these numbers are there from some tests made
> > before.
> > 
> 
> I'm not sure what do you mean? IMHO you don't use qdiscs properly,
> so any TX problems end with drops. (Older kernel versions could mask
> this problem with requeuing.)

Apologies, my bad, I misread what you wrote.

txqueuelen 1000 fixes the qdisc drops. I didn't notice that bond interfaces have it set to 0 by default.

As suggested by Eric earlier, the drops also disappear if I bind each NIC to a single CPU. That was the 
default on older kernels and perhaps that's why the issue came up only on 2.6.28.

-- 
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com

Re: bond + tc regression ?

[Patrick McHardy]

Vladimir Ivashchenko wrote:
> On Wed, May 06, 2009 at 10:48:08AM +0000, Jarek Poplawski wrote:
> 
>>>>>> ifconfig -a
>>>>> bond0     Link encap:Ethernet  HWaddr 00:1B:24:BD:E9:CC
>>>>>           inet addr:xxx.xxx.135.44  Bcast:xxx.xxx.135.47  Mask:255.255.255.248
>>>>>           inet6 addr: fe80::21b:24ff:febd:e9cc/64 Scope:Link
>>>>>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>>>>>           RX packets:436076190 errors:0 dropped:391250 overruns:0 frame:0
>>>>>           TX packets:2620156321 errors:0 dropped:0 overruns:0 carrier:0
>>>>>           collisions:0 txqueuelen:0
>>>>>           RX bytes:4210046233 (3.9 GiB)  TX bytes:2520272242 (2.3 GiB)
>>>> Could you try e.g.: ifconfig bond0 txqueuelen 1000
>>>> before tc qdisc add?
>>> The drops on ifconfig are not increasing - these numbers are there from some tests made
>>> before.
>>>
>> I'm not sure what do you mean? IMHO you don't use qdiscs properly,
>> so any TX problems end with drops. (Older kernel versions could mask
>> this problem with requeuing.)
> 
> Apologies, my bad, I misread what you wrote.
> 
> txqueuelen 1000 fixes the qdisc drops. I didn't notice that bond interfaces have it set to 0 by default.

The fifos use a queue length of 1 when the tx_queue_len is zero (which
was added as a workaround for similar problems a long time ago). Perhaps
we should instead refuse these broken configurations or at least print
a warning.

Re: bond + tc regression ?

[Vladimir Ivashchenko]

On Wed, 2009-05-06 at 05:36 +0200, Eric Dumazet wrote:

> Ah, I forgot about one patch that could help your setup too (if using more than one
> cpu on NIC irqs of course), queued for 2.6.31

I have tried the patch. Didn't make a noticeable difference. Under 850
mbps HTB+sfq load, 2.6.29.1, four NICs / two bond ifaces, IRQ balancing,
the dual-core server has only 25% idle on each CPU.

What's interesting, the same 850mbps load, identical machine, but with
only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle.
2.5x overhead.

> (commit 6a321cb370ad3db4ba6e405e638b3a42c41089b0)
> 
> You could post oprofile results to help us finding other hot spots.
> 
> 
> [PATCH] net: netif_tx_queue_stopped too expensive
> 
> netif_tx_queue_stopped(txq) is most of the time false.
> 
> Yet its cost is very expensive on SMP.
> 
> static inline int netif_tx_queue_stopped(const struct netdev_queue *dev_queue)
> {
> 	return test_bit(__QUEUE_STATE_XOFF, &dev_queue->state);
> }
> 
> I saw this on oprofile hunting and bnx2 driver bnx2_tx_int().
> 
> We probably should split "struct netdev_queue" in two parts, one
> being read mostly.
> 
> __netif_tx_lock() touches _xmit_lock & xmit_lock_owner, these
> deserve a separate cache line.
> 
> Signed-off-by: Eric Dumazet <dada1@cosmosbay.com>
> 
> 
> diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
> index 2e7783f..1caaebb 100644
> --- a/include/linux/netdevice.h
> +++ b/include/linux/netdevice.h
> @@ -447,12 +447,18 @@ enum netdev_queue_state_t
>  };
>  
>  struct netdev_queue {
> +/*
> + * read mostly part
> + */
>  	struct net_device	*dev;
>  	struct Qdisc		*qdisc;
>  	unsigned long		state;
> -	spinlock_t		_xmit_lock;
> -	int			xmit_lock_owner;
>  	struct Qdisc		*qdisc_sleeping;
> +/*
> + * write mostly part
> + */
> +	spinlock_t		_xmit_lock ____cacheline_aligned_in_smp;
> +	int			xmit_lock_owner;
>  } ____cacheline_aligned_in_smp;
>  
> 
-- 
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211

Re: bond + tc regression ?

[Denys Fedoryschenko]

On Wednesday 06 May 2009 21:45:18 Vladimir Ivashchenko wrote:
> On Wed, 2009-05-06 at 05:36 +0200, Eric Dumazet wrote:
> > Ah, I forgot about one patch that could help your setup too (if using
> > more than one cpu on NIC irqs of course), queued for 2.6.31
>
> I have tried the patch. Didn't make a noticeable difference. Under 850
> mbps HTB+sfq load, 2.6.29.1, four NICs / two bond ifaces, IRQ balancing,
> the dual-core server has only 25% idle on each CPU.
>
> What's interesting, the same 850mbps load, identical machine, but with
> only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle.
> 2.5x overhead.

Probably oprofile can sched some light on this.
On my own experience IRQ balancing hurt performance a lot, because of cache 
misses.

Re: bond + tc regression ?

[Vladimir Ivashchenko]

On Wed, May 06, 2009 at 10:30:04PM +0300, Denys Fedoryschenko wrote:

> > What's interesting, the same 850mbps load, identical machine, but with
> > only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle.
> > 2.5x overhead.
> 
> Probably oprofile can sched some light on this.
> On my own experience IRQ balancing hurt performance a lot, because of cache 
> misses.

This is a dual-core machine, isn't cache shared between the cores?

Without IRQ balancing, one of the cores goes around 10% idle and HTB doesn't do
its job properly. Actually, in my experience HTB stops working properly after
idle goes below 35%.

I'll try gathering some stats using oprofile.

-- 
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com

Re: bond + tc regression ?

[Denys Fedoryschenko]

On Wednesday 06 May 2009 23:47:59 Vladimir Ivashchenko wrote:
> On Wed, May 06, 2009 at 10:30:04PM +0300, Denys Fedoryschenko wrote:
> > > What's interesting, the same 850mbps load, identical machine, but with
> > > only two NICs and no bond, HTB+esfq, kernel 2.6.21.2 => 60% CPU idle.
> > > 2.5x overhead.
> >
> > Probably oprofile can sched some light on this.
> > On my own experience IRQ balancing hurt performance a lot, because of
> > cache misses.
>
> This is a dual-core machine, isn't cache shared between the cores?
>
> Without IRQ balancing, one of the cores goes around 10% idle and HTB
> doesn't do its job properly. Actually, in my experience HTB stops working
> properly after idle goes below 35%.
It seems they should. No idea, more experienced guys should know more.

Can you show me please
cat /proc/net/psched
If it is highres working, try to add in HTB script, first line

HZ=1000
to set environment variable. Because if clock resolution high, burst 
calculation going crazy on high speeds.
Maybe it will help.

Also without irq balance, did you try to assign interface to cpu by 
smp_affinity? (/proc/irq/NN/smp_affinity)

And still i think best thing is oprofile. It can show "hot" places in code, 
who is spending cpu cycles.

>
> I'll try gathering some stats using oprofile.

Re: bond + tc regression ?

[Vladimir Ivashchenko]

> > Without IRQ balancing, one of the cores goes around 10% idle and HTB
> > doesn't do its job properly. Actually, in my experience HTB stops working
> > properly after idle goes below 35%.
> It seems they should. No idea, more experienced guys should know more.
> 
> Can you show me please
> cat /proc/net/psched
> If it is highres working, try to add in HTB script, first line
> 
> HZ=1000
> to set environment variable. Because if clock resolution high, burst 
> calculation going crazy on high speeds.
> Maybe it will help.

Wow, instead of 98425b burst, its calculating 970203b. 

Exporting HZ=1000 doesn't help. However, even if I recompile the kernel
to 1000 Hz and the burst is calculated correctly, for some reason HTB on
2.6.29 is still worse at rate control than 2.6.21.

With 2.6.21, ceil of 775 mbits, burst 99425b -> actual rate 825 mbits.
With 2.6.29, same ceil/burst -> actual rate 890 mbits.

Moreover, after I stop the traffic *COMPLETELY* on 2.6.29, actual rate
reported by htb goes ballistic and stays at 1100mbits. Then it drops
back to expected value after a minute or so.

> Also without irq balance, did you try to assign interface to cpu by 
> smp_affinity? (/proc/irq/NN/smp_affinity)

Yes, I did, didn't make any difference.

> And still i think best thing is oprofile. It can show "hot" places in code, 
> who is spending cpu cycles.

For some reason I get a hard freeze when I start oprofile daemon, even
without traffic. Never used oprofile before, so I'm not sure if I'm
doing something wrong ... I'm starting it just with --vmlinux parameter
and nothing else. I use vanilla 2.6.29 and oprofile from FC8.

-- 
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211

Re: bond + tc regression ?

[Denys Fedoryschenko]

On Friday 08 May 2009 23:46:11 Vladimir Ivashchenko wrote:
> > > Without IRQ balancing, one of the cores goes around 10% idle and HTB
> > > doesn't do its job properly. Actually, in my experience HTB stops
> > > working properly after idle goes below 35%.
> >
> > It seems they should. No idea, more experienced guys should know more.
> >
> > Can you show me please
> > cat /proc/net/psched
> > If it is highres working, try to add in HTB script, first line
> >
> > HZ=1000
> > to set environment variable. Because if clock resolution high, burst
> > calculation going crazy on high speeds.
> > Maybe it will help.
>
> Wow, instead of 98425b burst, its calculating 970203b.
Kind of strange burst, something wrong there. For 1000HZ and 1 Gbit it should 
be 126375b. You value is for 8Gbit/s.
What version of iproute2 you are using ( tc -V )?

>
> Exporting HZ=1000 doesn't help. However, even if I recompile the kernel
> to 1000 Hz and the burst is calculated correctly, for some reason HTB on
> 2.6.29 is still worse at rate control than 2.6.21.
>
> With 2.6.21, ceil of 775 mbits, burst 99425b -> actual rate 825 mbits.
> With 2.6.29, same ceil/burst -> actual rate 890 mbits.
It depends also if there is child classes, what is bursts set for them, and 
what is ceil/burst set for them.

>
> Moreover, after I stop the traffic *COMPLETELY* on 2.6.29, actual rate
> reported by htb goes ballistic and stays at 1100mbits. Then it drops
> back to expected value after a minute or so.
It is average bandwidth for some period, it is not realtime value. 

>
> > Also without irq balance, did you try to assign interface to cpu by
> > smp_affinity? (/proc/irq/NN/smp_affinity)
>
> Yes, I did, didn't make any difference.
What is a clock source?
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
Timer resolution?
cat /proc/net/psched

>
> > And still i think best thing is oprofile. It can show "hot" places in
> > code, who is spending cpu cycles.
>
> For some reason I get a hard freeze when I start oprofile daemon, even
> without traffic. Never used oprofile before, so I'm not sure if I'm
> doing something wrong ... I'm starting it just with --vmlinux parameter
> and nothing else. I use vanilla 2.6.29 and oprofile from FC8.

Re: bond + tc regression ?

[Vladimir Ivashchenko]

> > Wow, instead of 98425b burst, its calculating 970203b.
> Kind of strange burst, something wrong there. For 1000HZ and 1 Gbit it should 
> be 126375b. You value is for 8Gbit/s.
> What version of iproute2 you are using ( tc -V )?

That was iproute2-ss080725, I think it is confused by tickless mode.
With iproute2-ss090324 I'm getting an opposite: 1589b :)

> >
> > With 2.6.21, ceil of 775 mbits, burst 99425b -> actual rate 825 mbits.
> > With 2.6.29, same ceil/burst -> actual rate 890 mbits.
> It depends also if there is child classes, what is bursts set for them, and 
> what is ceil/burst set for them.

All child classes have smaller bursts than the parent. However, there are two 
sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I
don't know HTB internals, perhaps these two classes make the parent class 
overstretch itself.

By the way, I experience the same "overstretching" with hfsc. In any case, 
I prefer HTB because it reports statistics of parent classes, unlike hfsc.

> > Moreover, after I stop the traffic *COMPLETELY* on 2.6.29, actual rate
> > reported by htb goes ballistic and stays at 1100mbits. Then it drops
> > back to expected value after a minute or so.
> It is average bandwidth for some period, it is not realtime value. 

But why it would it jump from 850mbits to 1200mbits *AFTER* I remove all
the traffic ?

> > Yes, I did, didn't make any difference.
> What is a clock source?
> cat /sys/devices/system/clocksource/clocksource0/current_clocksource

tsc

> Timer resolution?
> cat /proc/net/psched

With tickless kernel:

000003e8 00000400 000f4240 3b9aca00

-- 
Best Regards
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel, Cyprus - www.prime-tel.com

Re: bond + tc regression ?

[Denys Fedoryschenko]

On Saturday 09 May 2009 01:07:27 Vladimir Ivashchenko wrote:
> > > Wow, instead of 98425b burst, its calculating 970203b.
> >
> > Kind of strange burst, something wrong there. For 1000HZ and 1 Gbit it
> > should be 126375b. You value is for 8Gbit/s.
> > What version of iproute2 you are using ( tc -V )?
>
> That was iproute2-ss080725, I think it is confused by tickless mode.
> With iproute2-ss090324 I'm getting an opposite: 1589b :)
And it is too low. Thats why i set HZ=1000
>
>
> All child classes have smaller bursts than the parent. However, there are
> two sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I
> don't know HTB internals, perhaps these two classes make the parent class
> overstretch itself.
As i remember important to keep sum of child rates lower or equal parent rate.
Sure ceil of childs must not exceed ceil of parent.
Sometimes i had mess, when i tried to play with quantum value. After all that 
i switched to HFSC which works for me flawlessly. Maybe we should give more 
attention to HTB problem with high speeds and help kernel developers spot 
problem, if there is any.

>
> By the way, I experience the same "overstretching" with hfsc. In any case,
> I prefer HTB because it reports statistics of parent classes, unlike hfsc.
Sometimes it happen when some offloading enabled on devices.
Check ethtool -k device

I think everything except rx/tx checksumming have to be off, at least for 
test.

Disable them by "ethtool -K device tso off " for example.


>
> But why it would it jump from 850mbits to 1200mbits *AFTER* I remove all
> the traffic ?
>
Well, i dont know how it is doing averaging, even maybe for 1 minute. 
I dont like it at all, and thats why i prefer HFSC. But HTB work very well in 
some setups

Re: bond + tc regression ?

[Vladimir Ivashchenko]

[-- Attachment #1: Type: text/plain, Size: 1948 bytes --]


> > All child classes have smaller bursts than the parent. However, there are
> > two sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I
> > don't know HTB internals, perhaps these two classes make the parent class
> > overstretch itself.
> As i remember important to keep sum of child rates lower or equal parent rate.
> Sure ceil of childs must not exceed ceil of parent.
> Sometimes i had mess, when i tried to play with quantum value. After all that 
> i switched to HFSC which works for me flawlessly. Maybe we should give more 
> attention to HTB problem with high speeds and help kernel developers spot 
> problem, if there is any.

In case of HFSC my problem is even worse. With 775mbit ceiling
configured it is passing over 900mbit in reality. Moreover not having
statistics for parent classes makes it difficult to troubleshoot :( I'm
100% sure that it is 900 mbps, I see this on the switch.

Attached is "tc -s -d class show dev bond0" output.

To calculate total traffic rate:

$ cat hfsc-stat.txt | grep rate | grep Kbit | sed 's/Kbit//' | awk
'{ a=a+$2; } END { print a; }'
906955

Did I misconfigure something ?... How can hfsc go above 775mbit when
everything goes via class 1:2 with 775mbit rate & ul ?

> > By the way, I experience the same "overstretching" with hfsc. In any case,
> > I prefer HTB because it reports statistics of parent classes, unlike hfsc.
> Sometimes it happen when some offloading enabled on devices.
> Check ethtool -k device

Offload parameters for eth0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp segmentation offload: off
udp fragmentation offload: off


> I think everything except rx/tx checksumming have to be off, at least for 
> test.
> 
> Disable them by "ethtool -K device tso off " for example.

Doesn't help.

-- 
Best Regards,
Vladimir Ivashchenko
Chief Technology Officer
PrimeTel PLC, Cyprus - www.prime-tel.com
Tel: +357 25 100100 Fax: +357 2210 2211


[-- Attachment #2: hfsc-stat.txt --]
[-- Type: text/plain, Size: 18482 bytes --]

class hfsc 1:99 parent 1: leaf 99: sc m1 0bit d 15.0ms m2 542500Kbit 
 Sent 20385133 bytes 192619 pkt (dropped 7070, overlimits 0 requeues 0) 
 rate 402936bit 432pps backlog 0b 0p requeues 0 
 period 192581 work 20385133 bytes rtwork 20353003 bytes level 0 

class hfsc 1:5005 parent 1:97 leaf 5005: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 2948666978 bytes 4359774 pkt (dropped 50623, overlimits 0 requeues 0) 
 rate 55946Kbit 10338pps backlog 0b 7p requeues 0 
 period 1246140 work 2945839727 bytes rtwork 332352472 bytes level 0 

class hfsc 1:98 parent 1:2 leaf 98: sc m1 0bit d 500.0ms m2 43400Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 4644738789 bytes 5853965 pkt (dropped 165294, overlimits 0 requeues 0) 
 rate 85749Kbit 13526pps backlog 0b 0p requeues 0 
 period 1968169 work 4633993957 bytes rtwork 2322556251 bytes level 0 

class hfsc 1:100 parent 1:2 sc m1 0bit d 5.0ms m2 15000Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 290192 work 76779316 bytes level 1 

class hfsc 1:10 parent 1:2 sc m1 0bit d 1.0ms m2 542500Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 464809 work 877481554 bytes level 1 

class hfsc 1:5004 parent 1:97 leaf 5004: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 3181480387 bytes 4586738 pkt (dropped 50789, overlimits 0 requeues 0) 
 rate 60049Kbit 10765pps backlog 0b 9p requeues 0 
 period 1258266 work 3178564479 bytes rtwork 332600104 bytes level 0 

class hfsc 1:5006 parent 1:97 leaf 5006: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 842333834 bytes 1339029 pkt (dropped 9194, overlimits 0 requeues 0) 
 rate 18154Kbit 3349pps backlog 0b 2p requeues 0 
 period 806244 work 841676259 bytes rtwork 332340905 bytes level 0 

class hfsc 1:3e9 parent 1:200 leaf 3e9: sc m1 0bit d 0us m2 256000bit ul m1 0bit d 0us m2 100000Kbit 
 Sent 23292 bytes 308 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 488bit 1pps backlog 0b 0p requeues 0 
 period 298 work 23292 bytes rtwork 21371 bytes level 0 

class hfsc 1:5001 parent 1:97 leaf 5001: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 2541073789 bytes 3768515 pkt (dropped 39084, overlimits 0 requeues 0) 
 rate 52514Kbit 9201pps backlog 0b 4p requeues 0 
 period 1221187 work 2538919598 bytes rtwork 332543703 bytes level 0 

class hfsc 1:3e8 parent 1:200 leaf 3e8: sc m1 0bit d 0us m2 512000bit ul m1 0bit d 0us m2 100000Kbit 
 Sent 2140 bytes 34 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 32bit 0pps backlog 0b 0p requeues 0 
 period 34 work 2140 bytes rtwork 2140 bytes level 0 

class hfsc 1:3eb parent 1:300 leaf 3eb: sc m1 0bit d 0us m2 64000bit ul m1 0bit d 0us m2 100000Kbit 
 Sent 206 bytes 3 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 3 work 206 bytes rtwork 206 bytes level 0 

class hfsc 1:5003 parent 1:97 leaf 5003: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 2245687618 bytes 3498522 pkt (dropped 35135, overlimits 0 requeues 0) 
 rate 42763Kbit 8415pps backlog 0b 5p requeues 0 
 period 1192583 work 2244028709 bytes rtwork 332304111 bytes level 0 

class hfsc 1:3ea parent 1:300 leaf 3ea: sc m1 0bit d 0us m2 1024Kbit ul m1 0bit d 0us m2 100000Kbit 
 Sent 54214 bytes 442 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 16bit 0pps backlog 0b 0p requeues 0 
 period 413 work 54214 bytes rtwork 48427 bytes level 0 

class hfsc 1:5002 parent 1:97 leaf 5002: sc m1 0bit d 200.0ms m2 6103Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 2915382517 bytes 4002191 pkt (dropped 35677, overlimits 0 requeues 0) 
 rate 54029Kbit 9447pps backlog 0b 7p requeues 0 
 period 1219564 work 2912952580 bytes rtwork 332338435 bytes level 0 

class hfsc 1:97 parent 1:2 sc m1 0bit d 200.0ms m2 97650Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 1853176 work 14661981352 bytes level 1 

class hfsc 1: root 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 0 level 3 

class hfsc 1:4004 parent 1:79 leaf 4004: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 4401715817 bytes 4690294 pkt (dropped 32975, overlimits 0 requeues 0) 
 rate 84643Kbit 11103pps backlog 0b 0p requeues 0 
 period 3255986 work 4402143323 bytes rtwork 1368797472 bytes level 0 

class hfsc 1:4005 parent 1:79 leaf 4005: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 4858823203 bytes 5510336 pkt (dropped 45336, overlimits 0 requeues 0) 
 rate 90141Kbit 12773pps backlog 0b 0p requeues 0 
 period 3648819 work 4859323215 bytes rtwork 1370067779 bytes level 0 

class hfsc 1:2 parent 1: sc m1 0bit d 0us m2 775000Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 6902 work 48071507639 bytes level 2 

class hfsc 1:4006 parent 1:79 leaf 4006: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 2215296226 bytes 2405111 pkt (dropped 16771, overlimits 0 requeues 0) 
 rate 41477Kbit 5802pps backlog 0b 0p requeues 0 
 period 2036400 work 2217428341 bytes rtwork 1369522295 bytes level 0 

class hfsc 1:c9 parent 1:200 leaf c9: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit 
 Sent 240026752 bytes 543130 pkt (dropped 7429, overlimits 0 requeues 0) 
 rate 4201Kbit 1193pps backlog 0b 0p requeues 0 
 period 307816 work 240015088 bytes rtwork 1763247 bytes level 0 

class hfsc 1:4001 parent 1:79 leaf 4001: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 4432353076 bytes 4969719 pkt (dropped 31037, overlimits 0 requeues 0) 
 rate 79306Kbit 10880pps backlog 0b 0p requeues 0 
 period 3452541 work 4431357910 bytes rtwork 1370028629 bytes level 0 

class hfsc 1:4002 parent 1:79 leaf 4002: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 4446019233 bytes 4948736 pkt (dropped 42181, overlimits 0 requeues 0) 
 rate 87046Kbit 11613pps backlog 0b 0p requeues 0 
 period 3427533 work 4448206779 bytes rtwork 1370171471 bytes level 0 

class hfsc 1:4003 parent 1:79 leaf 4003: sc m1 0bit d 50.0ms m2 25090Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 3759018918 bytes 4336293 pkt (dropped 23579, overlimits 0 requeues 0) 
 rate 75258Kbit 10430pps backlog 0b 0p requeues 0 
 period 3147915 work 3758684585 bytes rtwork 1368528984 bytes level 0 

class hfsc 1:80 parent 1:2 leaf 80: sc m1 0bit d 50.0ms m2 1000bit ul m1 0bit d 0us m2 542500Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 0 level 0 

class hfsc 1:b parent 1:10 leaf b: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit 
 Sent 878491329 bytes 1005975 pkt (dropped 15849, overlimits 0 requeues 0) 
 rate 14093Kbit 2159pps backlog 0b 0p requeues 0 
 period 464809 work 877481554 bytes rtwork 1891407 bytes level 0 

class hfsc 1:300 parent 1:2 sc m1 0bit d 15.0ms m2 30000Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 742649 work 1854498600 bytes level 1 

class hfsc 1:3999 parent 1:79 leaf 3999: sc m1 0bit d 0us m2 1000bit ul m1 0bit d 0us m2 30432Kbit 
 Sent 1617385819 bytes 1260031 pkt (dropped 110848, overlimits 0 requeues 0) 
 rate 30581Kbit 2977pps backlog 0b 56p requeues 0 
 period 1381 work 1609588187 bytes rtwork 58059 bytes level 0 

class hfsc 1:12d parent 1:300 leaf 12d: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit 
 Sent 1860128386 bytes 2152291 pkt (dropped 51268, overlimits 0 requeues 0) 
 rate 29842Kbit 4647pps backlog 0b 2p requeues 0 
 period 742499 work 1854444180 bytes rtwork 1775294 bytes level 0 

class hfsc 1:79 parent 1:2 sc m1 0bit d 0us m2 401450Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 9008 work 25726746683 bytes level 1 

class hfsc 1:2000 parent 1:79 sc m1 0bit d 0us m2 1000bit ul m1 0bit d 0us m2 401450Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 0 level 0 

class hfsc 1:200 parent 1:2 sc m1 0bit d 10.0ms m2 35000Kbit ul m1 0bit d 0us m2 542500Kbit 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 rate 0bit 0pps backlog 0b 0p requeues 0 
 period 308015 work 240040520 bytes level 1 

class hfsc 1:65 parent 1:100 leaf 65: sc m1 0bit d 0us m2 32000bit ul m1 0bit d 0us m2 100000Kbit 
 Sent 76808971 bytes 457236 pkt (dropped 16755, overlimits 0 requeues 0) 
 rate 1163Kbit 989pps backlog 0b 2p requeues 0 
 period 290193 work 76779316 bytes rtwork 1743830 bytes level 0 

class prio 99:1 parent 99: 
 Sent 1543 bytes 11 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 99:2 parent 99: 
 Sent 19693553 bytes 185805 pkt (dropped 6999, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class prio 99:3 parent 99: 
 Sent 690143 bytes 6804 pkt (dropped 71, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0 
class sfq 3999:1a parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 5942 

class sfq 3999:2d parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot -1279 

class sfq 3999:3e parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 3072 

class sfq 3999:46 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot -1035 

class sfq 3999:ae parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot -871 

class sfq 3999:c4 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 1429 

class sfq 3999:e3 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 388 

class sfq 3999:ee parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot -582 

class sfq 3999:155 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot -1409 

class sfq 3999:17c parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot -128 

class sfq 3999:1c2 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 9048 

class sfq 3999:1ca parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 6p requeues 0 
 allot -1156 

class sfq 3999:217 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot -90 

class sfq 3999:229 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 336 

class sfq 3999:234 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 4p requeues 0 
 allot -1248 

class sfq 3999:236 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot -350 

class sfq 3999:24d parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 2602 

class sfq 3999:259 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot 12595 

class sfq 3999:27e parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 15861 

class sfq 3999:28e parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 578 

class sfq 3999:2af parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 6p requeues 0 
 allot -280 

class sfq 3999:32b parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 109 

class sfq 3999:37e parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 6p requeues 0 
 allot -1355 

class sfq 3999:3ed parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot -1080 

class sfq 3999:400 parent 3999: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot -293 

class sfq 98:9 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 7590 

class sfq 98:88 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 10598 

class sfq 98:10c parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 16677 

class sfq 98:111 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 9084 

class sfq 98:22f parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 3028 

class sfq 98:294 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 12054 

class sfq 98:2f9 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 23634 

class sfq 98:322 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot -28605 

class sfq 98:364 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 6056 

class sfq 98:3b3 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 4546 

class sfq 98:3f9 parent 98: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 30100 

class sfq 5001:39a parent 5001: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 24540 

class sfq 4002:226 parent 4002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 4542 

class sfq 5002:49 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 9084 

class sfq 5002:a7 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 9084 

class sfq 5002:138 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 7570 

class sfq 5002:167 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 4542 

class sfq 5002:19c parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 10606 

class sfq 5002:20b parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 2974 

class sfq 5002:22a parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 12112 

class sfq 5002:256 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 25094 

class sfq 5002:289 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 9084 

class sfq 5002:2f4 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 3028 

class sfq 5002:341 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 7570 

class sfq 5002:351 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 8974 

class sfq 5002:376 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 6056 

class sfq 5002:3a6 parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 3028 

class sfq 5002:3bc parent 5002: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 6064 

class sfq 5003:68 parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot 4542 

class sfq 5003:a7 parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot -24050 

class sfq 5003:14d parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 2056 

class sfq 5003:1f3 parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 6056 

class sfq 5003:27f parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 3036 

class sfq 5003:2e8 parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 28716 

class sfq 5003:311 parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 13572 

class sfq 5003:31a parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 13626 

class sfq 5003:361 parent 5003: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 3048 

class sfq 5004:1e parent 5004: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 10932 

class sfq 5004:101 parent 5004: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 4542 

class sfq 5004:12e parent 5004: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 9084 

class sfq 5004:156 parent 5004: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 19526 

class sfq 5004:179 parent 5004: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 671163409p requeues 0 
 allot 724 

class sfq 5004:1a0 parent 5004: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 19682 

class sfq 5004:34c parent 5004: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot -30965 

class sfq 4005:134 parent 4005: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 1514 

class sfq 12d:47 parent 12d: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 4542 

class sfq 12d:a8 parent 12d: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 3p requeues 0 
 allot 1514 

class sfq 12d:ef parent 12d: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 1514 

class sfq 12d:368 parent 12d: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 3268 

class sfq 65:4e parent 65: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 2p requeues 0 
 allot 1514 

class sfq 65:91 parent 65: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 1514 

class sfq 65:286 parent 65: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 10530 

class sfq c9:357 parent c9: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 1514 

class sfq b:72 parent b: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 1514 

class sfq b:1ab parent b: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 13560 

class sfq b:20b parent b: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 13572 

class sfq b:38e parent b: 
 (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 1p requeues 0 
 allot 1514 

Re: bond + tc regression ?

[Jarek Poplawski]

On 17-05-2009 20:46, Vladimir Ivashchenko wrote:
>>> All child classes have smaller bursts than the parent. However, there are
>>> two sub-classes which have ceil at 70% of parent, e.g. ~500mbit each. I
>>> don't know HTB internals, perhaps these two classes make the parent class
>>> overstretch itself.
>> As i remember important to keep sum of child rates lower or equal parent rate.
>> Sure ceil of childs must not exceed ceil of parent.
>> Sometimes i had mess, when i tried to play with quantum value. After all that 
>> i switched to HFSC which works for me flawlessly. Maybe we should give more 
>> attention to HTB problem with high speeds and help kernel developers spot 
>> problem, if there is any.
> 
> In case of HFSC my problem is even worse. With 775mbit ceiling
> configured it is passing over 900mbit in reality. Moreover not having
> statistics for parent classes makes it difficult to troubleshoot :( I'm
> 100% sure that it is 900 mbps, I see this on the switch.
> 
> Attached is "tc -s -d class show dev bond0" output.
> 
> To calculate total traffic rate:
> 
> $ cat hfsc-stat.txt | grep rate | grep Kbit | sed 's/Kbit//' | awk
> '{ a=a+$2; } END { print a; }'
> 906955
> 
> Did I misconfigure something ?... How can hfsc go above 775mbit when
> everything goes via class 1:2 with 775mbit rate & ul ?

Maybe... It's a lot of checking - it seems test cases could be simpler
to show the real problem. Anyway, it looks like the sum of m2 of 1:2
children is more than 775Mbit.


>>> By the way, I experience the same "overstretching" with hfsc. In any case,
>>> I prefer HTB because it reports statistics of parent classes, unlike hfsc.
>> Sometimes it happen when some offloading enabled on devices.
>> Check ethtool -k device
> 
> Offload parameters for eth0:
> rx-checksumming: on
> tx-checksumming: on
> scatter-gather: on
> tcp segmentation offload: off
> udp fragmentation offload: off

Current versions of ethtool should show "generic segmentation offload"
too.

I hope you've read the nearby thread "HTB accuracy for high speed",
which explains at least partially some problems/bugs, and maybe you'll
try some patches too (at last one of them addresses the problem you've
reported). Anyway, if you don't find hfsc is better for you I'd be
more interested in tracking this on htb test cases yet.

Thanks,
Jarek P.