All of lore.kernel.org
 help / color / mirror / Atom feed
* Strange packet drops with heavy firewalling
@ 2010-04-09  9:56 Benny Amorsen
  2010-04-09 11:47 ` Eric Dumazet
       [not found] ` <q2v40c9f5b21004120116p766df82dj88c6af4e4cad55f@mail.gmail.com>
  0 siblings, 2 replies; 15+ messages in thread
From: Benny Amorsen @ 2010-04-09  9:56 UTC (permalink / raw)
  To: netdev


I have a netfilter-box which is dropping packets. ethtool -S counts
10-20 rx_discards per second on the interface.

The switch does not have flow control enabled; with flow control enabled
the rx_discards turn into tx_on_sent which ultimately cause the same
problem (the load is pretty constant so the switch has to drop the
packets instead).

perf top shows something like:
             5201.00 -  6.7% : _spin_unlock_irqrestore
             4232.00 -  5.5% : finish_task_switch
             3597.00 -  4.6% : tg3_poll	[tg3]
             3257.00 -  4.2% : handle_IRQ_event
             2515.00 -  3.2% : tick_nohz_restart_sched_tick
             1947.00 -  2.5% : nf_ct_tuple_equal
             1927.00 -  2.5% : tg3_start_xmit	[tg3]
             1879.00 -  2.4% : kmem_cache_alloc_node
             1625.00 -  2.1% : tick_nohz_stop_sched_tick
             1619.00 -  2.1% : ipt_do_table
             1595.00 -  2.1% : ip_route_input
             1547.00 -  2.0% : kmem_cache_free
             1474.00 -  1.9% : __alloc_skb
             1424.00 -  1.8% : fget_light
             1391.00 -  1.8% : nf_iterate

The rule set is quite large (more than 4000 rules), but organized so
that each packet only has to traverse a few rules before getting
accepted or rejected.

When the problem started we were using a different server, an old
two-socket 32-bit Xeon with hyperthreading. CPU usage often hit 100% on
one CPU with that server. After replacing the server with a ProLiant
DL160 G5 with a quad-core Xeon (without hyperthreading) the CPU usage
rarely exceeds 10% on any CPU, but the packet loss persists.

We're using the built-in dual Broadcom Corporation NetXtreme BCM5722 Gigabit
Ethernet PCI Express nics, and the kernel is
kernel-2.6.32.9-70.fc12.x86_64 from Fedora. Next step is probably
installing a better ethernet card, perhaps an Intel 82576-based one, so
that we can get multiqueue support.

The traffic is about 300Mbps (twice that if you count both in and out,
like Cisco).


/Benny


^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-09  9:56 Strange packet drops with heavy firewalling Benny Amorsen
@ 2010-04-09 11:47 ` Eric Dumazet
  2010-04-09 12:33   ` Benny Amorsen
       [not found] ` <q2v40c9f5b21004120116p766df82dj88c6af4e4cad55f@mail.gmail.com>
  1 sibling, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2010-04-09 11:47 UTC (permalink / raw)
  To: Benny Amorsen; +Cc: netdev

Le vendredi 09 avril 2010 à 11:56 +0200, Benny Amorsen a écrit :
> I have a netfilter-box which is dropping packets. ethtool -S counts
> 10-20 rx_discards per second on the interface.
> 
> The switch does not have flow control enabled; with flow control enabled
> the rx_discards turn into tx_on_sent which ultimately cause the same
> problem (the load is pretty constant so the switch has to drop the
> packets instead).
> 
> perf top shows something like:
>              5201.00 -  6.7% : _spin_unlock_irqrestore
>              4232.00 -  5.5% : finish_task_switch
>              3597.00 -  4.6% : tg3_poll	[tg3]
>              3257.00 -  4.2% : handle_IRQ_event
>              2515.00 -  3.2% : tick_nohz_restart_sched_tick
>              1947.00 -  2.5% : nf_ct_tuple_equal
>              1927.00 -  2.5% : tg3_start_xmit	[tg3]
>              1879.00 -  2.4% : kmem_cache_alloc_node
>              1625.00 -  2.1% : tick_nohz_stop_sched_tick
>              1619.00 -  2.1% : ipt_do_table
>              1595.00 -  2.1% : ip_route_input
>              1547.00 -  2.0% : kmem_cache_free
>              1474.00 -  1.9% : __alloc_skb
>              1424.00 -  1.8% : fget_light
>              1391.00 -  1.8% : nf_iterate
> 
> The rule set is quite large (more than 4000 rules), but organized so
> that each packet only has to traverse a few rules before getting
> accepted or rejected.
> 
> When the problem started we were using a different server, an old
> two-socket 32-bit Xeon with hyperthreading. CPU usage often hit 100% on
> one CPU with that server. After replacing the server with a ProLiant
> DL160 G5 with a quad-core Xeon (without hyperthreading) the CPU usage
> rarely exceeds 10% on any CPU, but the packet loss persists.
> 

might be micro bursts, check 'ethtool -g eth0' RX parameters (increase
RX ring from 200 to 511 if you want more buffers ?)

> We're using the built-in dual Broadcom Corporation NetXtreme BCM5722 Gigabit
> Ethernet PCI Express nics, and the kernel is
> kernel-2.6.32.9-70.fc12.x86_64 from Fedora. Next step is probably
> installing a better ethernet card, perhaps an Intel 82576-based one, so
> that we can get multiqueue support.
> 

Sure, but before this, could you check

cat /proc/net/softnet_stat
cat /proc/interrupts
(check eth0 IRQS are delivered to one cpu)

grep . /proc/sys/net/ipv4/netfilter/ip_conntrack_*
(might need to increase ip_conntrack_buckets)

ethtool -c eth0
(might change coalesce params to reduce number of irqs)

ethtool -g eth0



^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-09 11:47 ` Eric Dumazet
@ 2010-04-09 12:33   ` Benny Amorsen
  2010-04-09 13:29     ` Eric Dumazet
  0 siblings, 1 reply; 15+ messages in thread
From: Benny Amorsen @ 2010-04-09 12:33 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: netdev

Eric Dumazet <eric.dumazet@gmail.com> writes:

> might be micro bursts, check 'ethtool -g eth0' RX parameters (increase
> RX ring from 200 to 511 if you want more buffers ?)

I tried that already actually. (I didn't expect it to cause traffic
interruption, but it did. Oh well.)

It didn't make a difference, at least not one I could detect from the
number of packet drops and the CPU utilization.

> cat /proc/net/softnet_stat

000002d9 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
42bc8143 00000000 0000024c 00000000 00000000 00000000 00000000 00000000 00000000
0000031b 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
1c5a35e9 00000000 000005f7 00000000 00000000 00000000 00000000 00000000 00000000

I am not quite sure how to interpret that...

> cat /proc/interrupts

  79:       1240 4050590849       1253       1263   PCI-MSI-edge      eth0
  80:         12          9         14 3613521843   PCI-MSI-edge      eth1

> (check eth0 IRQS are delivered to one cpu)

Yes CPU1 handles eth0 and CPU3 handles eth1.

> grep . /proc/sys/net/ipv4/netfilter/ip_conntrack_*

nf_conntrack_acct:1
nf_conntrack_buckets:8192
nf_conntrack_checksum:1
nf_conntrack_count:49311
nf_conntrack_events:1
nf_conntrack_events_retry_timeout:15
nf_conntrack_expect_max:2048
nf_conntrack_generic_timeout:600
nf_conntrack_icmp_timeout:30
nf_conntrack_log_invalid:1
nf_conntrack_max:1048576
nf_conntrack_tcp_be_liberal:0
nf_conntrack_tcp_loose:1
nf_conntrack_tcp_max_retrans:3
nf_conntrack_tcp_timeout_close:10
nf_conntrack_tcp_timeout_close_wait:60
nf_conntrack_tcp_timeout_established:432000
nf_conntrack_tcp_timeout_fin_wait:120
nf_conntrack_tcp_timeout_last_ack:30
nf_conntrack_tcp_timeout_max_retrans:300
nf_conntrack_tcp_timeout_syn_recv:60
nf_conntrack_tcp_timeout_syn_sent:120
nf_conntrack_tcp_timeout_time_wait:120
nf_conntrack_tcp_timeout_unacknowledged:300
nf_conntrack_udp_timeout:30
nf_conntrack_udp_timeout_stream:180

> (might need to increase ip_conntrack_buckets)

You got me there. I had forgotten nf_conntrack.hashsize=1048576
and nf_conntrack.expect_hashsize=32768 on the kernel command line. It
was on the hot standby firewall, but not on the primary one. I will do a
failover to the hot standby sometime during the weekend.

It still isn't possible to change without a reboot, is it?

> ethtool -c eth0
> (might change coalesce params to reduce number of irqs)

Coalesce parameters for eth0:
Adaptive RX: off  TX: off
stats-block-usecs: 0
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 20
rx-frames: 5
rx-usecs-irq: 0
rx-frames-irq: 5

tx-usecs: 72
tx-frames: 53
tx-usecs-irq: 0
tx-frames-irq: 5

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

I played quite a lot with the parameters but it did not seem to make any
difference. I didn't try adaptive though, but the load is fairly static
so it didn't seem appropriate.

> ethtool -g eth0

Ring parameters for eth0:
Pre-set maximums:
RX:		511
RX Mini:	0
RX Jumbo:	0
TX:		511
Current hardware settings:
RX:		200
RX Mini:	0
RX Jumbo:	0
TX:		511

Right now RX is 200, but when it was 511 it didn't seem to make a
difference.

Thank you very much for the help! I will report back whether it was the
hash buckets.


/Benny

^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-09 12:33   ` Benny Amorsen
@ 2010-04-09 13:29     ` Eric Dumazet
  2010-04-12  6:20       ` Benny Amorsen
  0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2010-04-09 13:29 UTC (permalink / raw)
  To: Benny Amorsen; +Cc: netdev

Le vendredi 09 avril 2010 à 14:33 +0200, Benny Amorsen a écrit :

> Thank you very much for the help! I will report back whether it was the
> hash buckets.

OK

You could try :

ethtool -C eth0 tx-usecs 200 tx-frames 100 tx-frames-irq 100
ethtool -C eth1 tx-usecs 200 tx-frames 100 tx-frames-irq 100

(to reduce tx completion irqs)


Before buying multiqueue devices, you also could try net-next-2.6 kernel,
because RPS (Remote Packet Steering) is in.

In your setup, this might help a bit, distribute the packets to all cpus,
with appropriate cache handling.




^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-09 13:29     ` Eric Dumazet
@ 2010-04-12  6:20       ` Benny Amorsen
  0 siblings, 0 replies; 15+ messages in thread
From: Benny Amorsen @ 2010-04-12  6:20 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: netdev

Eric Dumazet <eric.dumazet@gmail.com> writes:

> Le vendredi 09 avril 2010 à 14:33 +0200, Benny Amorsen a écrit :
>
>> Thank you very much for the help! I will report back whether it was the
>> hash buckets.
>
> OK
>
> You could try :
>
> ethtool -C eth0 tx-usecs 200 tx-frames 100 tx-frames-irq 100
> ethtool -C eth1 tx-usecs 200 tx-frames 100 tx-frames-irq 100
>
> (to reduce tx completion irqs)

Alas, even with the hash buckets I still have the same problem. Perhaps
slightly less severe, but it's still there.

I implemented the other changes you suggested as well except for the
ethtool -G. I may try to switch to net-next if I can find an easy way to
make an RPM out of it.

Thank you for the help!

/proc/sys/net/netfilter/nf_conntrack_acct:1
/proc/sys/net/netfilter/nf_conntrack_buckets:1048576
/proc/sys/net/netfilter/nf_conntrack_checksum:1
/proc/sys/net/netfilter/nf_conntrack_count:43430
/proc/sys/net/netfilter/nf_conntrack_events:1
/proc/sys/net/netfilter/nf_conntrack_events_retry_timeout:15
/proc/sys/net/netfilter/nf_conntrack_expect_max:2048
/proc/sys/net/netfilter/nf_conntrack_generic_timeout:600
/proc/sys/net/netfilter/nf_conntrack_icmp_timeout:30
/proc/sys/net/netfilter/nf_conntrack_log_invalid:1
/proc/sys/net/netfilter/nf_conntrack_max:1048576
/proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal:0
/proc/sys/net/netfilter/nf_conntrack_tcp_loose:1
/proc/sys/net/netfilter/nf_conntrack_tcp_max_retrans:3
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close:10
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait:60
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established:432000
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_fin_wait:120
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_last_ack:30
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_max_retrans:300
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_syn_recv:60
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_syn_sent:120
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_time_wait:120
/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_unacknowledged:300
/proc/sys/net/netfilter/nf_conntrack_udp_timeout:30
/proc/sys/net/netfilter/nf_conntrack_udp_timeout_stream:180


/Benny

^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
       [not found] ` <q2v40c9f5b21004120116p766df82dj88c6af4e4cad55f@mail.gmail.com>
@ 2010-04-12 14:44   ` Benny Lyne Amorsen
       [not found]     ` <p2x40c9f5b21004120833jd7a749cak6ea69cebd28f8352@mail.gmail.com>
  0 siblings, 1 reply; 15+ messages in thread
From: Benny Lyne Amorsen @ 2010-04-12 14:44 UTC (permalink / raw)
  To: zhigang gong; +Cc: netdev

man, 12 04 2010 kl. 16:16 +0800, skrev zhigang gong:

> How do you know the per CPU usage data, by oprofile? I'm just a little
> surprised with the result, as it shows your new core is running 10x
> faster than your old core :). 

Well the old server had only two CPU's plus hyperthreading, and the
CPU's were Pentium-4-based. Add a slow memory bus to that and you have a
fairly slow system. It's almost 5 years old, so Moore's law says 2**3
increase in number of transistors...

In about the same time frame Linux has gone from being able to fill
1Gbps ethernet to being able to fill 10Gbps ethernet 

> What's the average packet size?

I asked the switch (I can't find a handy equivalent to ifstat which
counts packets instead of bytes). The 5 minute average packet sizes seem
to vary in the range 450 to 550 bytes.

> If your packet size is 64 bytes, then the pps(packet per second) rate
> should be about 585Kpps. As I know, this value is almost the best
> result when the standard linux kernel is processing the networking
> traffic with a normal 1Gb ethernet card (without multi-queue support)
> on a intel box. If it is the case, to buy a better ethernet card with
> multi-queue support should be a good choice. Otherwise, it may not
> help. 

I am far from that, perhaps 1/10th of that. I do a lot more processing
on at least some of the packets though (the ones starting new flows).


/Benny



^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
       [not found]     ` <p2x40c9f5b21004120833jd7a749cak6ea69cebd28f8352@mail.gmail.com>
@ 2010-04-12 17:06       ` Benny Amorsen
  2010-04-12 23:18         ` Changli Gao
  0 siblings, 1 reply; 15+ messages in thread
From: Benny Amorsen @ 2010-04-12 17:06 UTC (permalink / raw)
  To: zhigang gong; +Cc: netdev

man, 12 04 2010 kl. 23:33 +0800, skrev zhigang gong:

>  
> Now, I agree with Eric's analysis, there may be some bursts, for
> example a burst of a bunch of first packets for different new flows.
> What mode are you using the ethernet driver in? I guess it's NAPI,
> right?

I presume so.

>  And whether your time consumption workload is handled in soft-irq
> context or in a user space process? 

soft-irq, the box is doing pure iptables. The only time it does a little
bit of user-space work is when I use conntrackd, but killing conntrackd
does not affect the packet loss measurably.

I switched to a 82576-based card, and now I get:

             3341.00 -  4.9% : _spin_lock
             2506.00 -  3.7% : irq_entries_start
             2163.00 -  3.2% : _spin_lock_irqsave
             1616.00 -  2.4% : native_read_tsc
             1572.00 -  2.3% : igb_poll	[igb]
             1386.00 -  2.0% : get_partial_node
             1236.00 -  1.8% : igb_clean_tx_irq	[igb]
             1205.00 -  1.8% : igb_xmit_frame_adv	[igb]
             1170.00 -  1.7% : ipt_do_table
             1049.00 -  1.6% : fget_light
             1015.00 -  1.5% : tick_nohz_stop_sched_tick
              967.00 -  1.4% : fput
              945.00 -  1.4% : __slab_free
              919.00 -  1.4% : datagram_poll
              874.00 -  1.3% : dev_queue_xmit

And it seems the packet loss is gone!

# ethtool -S eth0|fgrep drop
     tx_dropped: 0
     rx_queue_drop_packet_count: 0
     dropped_smbus: 0
     rx_queue_0_drops: 0
     rx_queue_1_drops: 0
     rx_queue_2_drops: 0
     rx_queue_3_drops: 0

I'm a bit surprised by this though:
 
  99:         24    1306226          3          2   PCI-MSI-edge      eth1-tx-0
 100:      15735    1648774          3          7   PCI-MSI-edge      eth1-tx-1
 101:          8         11          9    1083022   PCI-MSI-edge      eth1-tx-2
 102:          0          0          0          0   PCI-MSI-edge      eth1-tx-3
 103:         18         15       6131    1095383   PCI-MSI-edge      eth1-rx-0
 104:        217         32      46544    1335325   PCI-MSI-edge      eth1-rx-1
 105:        154    1305595        218         16   PCI-MSI-edge      eth1-rx-2
 106:         17         16       8229    1467509   PCI-MSI-edge      eth1-rx-3
 107:          0          0          1          0   PCI-MSI-edge      eth1
 108:          2         14         15    1003053   PCI-MSI-edge      eth0-tx-0
 109:       8226    1668924        478        487   PCI-MSI-edge      eth0-tx-1
 110:          3    1188874         17         12   PCI-MSI-edge      eth0-tx-2
 111:          0          0          0          0   PCI-MSI-edge      eth0-tx-3
 112:        203        185       5324    1015263   PCI-MSI-edge      eth0-rx-0
 113:       4141    1600793        153        159   PCI-MSI-edge      eth0-rx-1
 114:      16242    1210108        436       3124   PCI-MSI-edge      eth0-rx-2
 115:        267       4173      19471    1321252   PCI-MSI-edge      eth0-rx-3
 116:          0          1          0          0   PCI-MSI-edge      eth0


irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
which to my mind should cause the same problem as before (where CPU1 and
CPU3 was handling all packets). Yet the box clearly works much better
than before.

Anyway, this brings the saga to an end from my point of view. Thank you
very much for looking into this, you and Eric Dumazet have been
invaluable!


/Benny




^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-12 17:06       ` Benny Amorsen
@ 2010-04-12 23:18         ` Changli Gao
  2010-04-13  5:56           ` Eric Dumazet
  2010-04-13 12:33           ` Paweł Staszewski
  0 siblings, 2 replies; 15+ messages in thread
From: Changli Gao @ 2010-04-12 23:18 UTC (permalink / raw)
  To: Benny Amorsen; +Cc: zhigang gong, netdev

On Tue, Apr 13, 2010 at 1:06 AM, Benny Amorsen <benny+usenet@amorsen.dk> wrote:
>
>  99:         24    1306226          3          2   PCI-MSI-edge      eth1-tx-0
>  100:      15735    1648774          3          7   PCI-MSI-edge      eth1-tx-1
>  101:          8         11          9    1083022   PCI-MSI-edge      eth1-tx-2
>  102:          0          0          0          0   PCI-MSI-edge      eth1-tx-3
>  103:         18         15       6131    1095383   PCI-MSI-edge      eth1-rx-0
>  104:        217         32      46544    1335325   PCI-MSI-edge      eth1-rx-1
>  105:        154    1305595        218         16   PCI-MSI-edge      eth1-rx-2
>  106:         17         16       8229    1467509   PCI-MSI-edge      eth1-rx-3
>  107:          0          0          1          0   PCI-MSI-edge      eth1
>  108:          2         14         15    1003053   PCI-MSI-edge      eth0-tx-0
>  109:       8226    1668924        478        487   PCI-MSI-edge      eth0-tx-1
>  110:          3    1188874         17         12   PCI-MSI-edge      eth0-tx-2
>  111:          0          0          0          0   PCI-MSI-edge      eth0-tx-3
>  112:        203        185       5324    1015263   PCI-MSI-edge      eth0-rx-0
>  113:       4141    1600793        153        159   PCI-MSI-edge      eth0-rx-1
>  114:      16242    1210108        436       3124   PCI-MSI-edge      eth0-rx-2
>  115:        267       4173      19471    1321252   PCI-MSI-edge      eth0-rx-3
>  116:          0          1          0          0   PCI-MSI-edge      eth0
>
>
> irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
> which to my mind should cause the same problem as before (where CPU1 and
> CPU3 was handling all packets). Yet the box clearly works much better
> than before.

irqbalanced? I don't think it can work properly. Try RPS in netdev and
linux-next tree, and if cpu load isn't even, try this patch:
http://patchwork.ozlabs.org/patch/49915/ .


-- 
Regards，
Changli Gao(xiaosuo@gmail.com)

^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-12 23:18         ` Changli Gao
@ 2010-04-13  5:56           ` Eric Dumazet
  2010-04-13  7:56             ` Benny Amorsen
  2010-04-13 12:33           ` Paweł Staszewski
  1 sibling, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2010-04-13  5:56 UTC (permalink / raw)
  To: Changli Gao; +Cc: Benny Amorsen, zhigang gong, netdev

Le mardi 13 avril 2010 à 07:18 +0800, Changli Gao a écrit :
> On Tue, Apr 13, 2010 at 1:06 AM, Benny Amorsen <benny+usenet@amorsen.dk> wrote:
> >
> >  99:         24    1306226          3          2   PCI-MSI-edge      eth1-tx-0
> >  100:      15735    1648774          3          7   PCI-MSI-edge      eth1-tx-1
> >  101:          8         11          9    1083022   PCI-MSI-edge      eth1-tx-2
> >  102:          0          0          0          0   PCI-MSI-edge      eth1-tx-3
> >  103:         18         15       6131    1095383   PCI-MSI-edge      eth1-rx-0
> >  104:        217         32      46544    1335325   PCI-MSI-edge      eth1-rx-1
> >  105:        154    1305595        218         16   PCI-MSI-edge      eth1-rx-2
> >  106:         17         16       8229    1467509   PCI-MSI-edge      eth1-rx-3
> >  107:          0          0          1          0   PCI-MSI-edge      eth1
> >  108:          2         14         15    1003053   PCI-MSI-edge      eth0-tx-0
> >  109:       8226    1668924        478        487   PCI-MSI-edge      eth0-tx-1
> >  110:          3    1188874         17         12   PCI-MSI-edge      eth0-tx-2
> >  111:          0          0          0          0   PCI-MSI-edge      eth0-tx-3
> >  112:        203        185       5324    1015263   PCI-MSI-edge      eth0-rx-0
> >  113:       4141    1600793        153        159   PCI-MSI-edge      eth0-rx-1
> >  114:      16242    1210108        436       3124   PCI-MSI-edge      eth0-rx-2
> >  115:        267       4173      19471    1321252   PCI-MSI-edge      eth0-rx-3
> >  116:          0          1          0          0   PCI-MSI-edge      eth0
> >
> >
> > irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
> > which to my mind should cause the same problem as before (where CPU1 and
> > CPU3 was handling all packets). Yet the box clearly works much better
> > than before.
> 
> irqbalanced? I don't think it can work properly. Try RPS in netdev and
> linux-next tree, and if cpu load isn't even, try this patch:
> http://patchwork.ozlabs.org/patch/49915/ .
> 
> 

Dont try RPS on multiqueue devices !

If number of queues matches CPU numbers, it brings nothing but extra
latencies !

Benny, I am not sure your irqbalance is up2date with multiqueue devices,
you might need to disable it and manually irqaffine each interrupt

echo 01 >/proc/irq/100/smp_affinity
echo 02 >/proc/irq/101/smp_affinity
echo 04 >/proc/irq/102/smp_affinity
echo 08 >/proc/irq/103/smp_affinity
echo 10 >/proc/irq/104/smp_affinity
echo 20 >/proc/irq/105/smp_affinity
echo 40 >/proc/irq/106/smp_affinity
echo 80 >/proc/irq/107/smp_affinity

echo 01 >/proc/irq/108/smp_affinity
echo 02 >/proc/irq/109/smp_affinity
echo 04 >/proc/irq/110/smp_affinity
echo 08 >/proc/irq/111/smp_affinity
echo 10 >/proc/irq/112/smp_affinity
echo 20 >/proc/irq/113/smp_affinity
echo 40 >/proc/irq/114/smp_affinity
echo 80 >/proc/irq/115/smp_affinity



^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-13  5:56           ` Eric Dumazet
@ 2010-04-13  7:56             ` Benny Amorsen
  2010-04-15 13:23               ` Benny Amorsen
  0 siblings, 1 reply; 15+ messages in thread
From: Benny Amorsen @ 2010-04-13  7:56 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Changli Gao, zhigang gong, netdev

Eric Dumazet <eric.dumazet@gmail.com> writes:

> Benny, I am not sure your irqbalance is up2date with multiqueue devices,
> you might need to disable it and manually irqaffine each interrupt

True, that would probably help. Irqbalance might just believe that the
load is so low that it isn't worth rebalancing. The CPU's are spending
more than 90% of their time idling.

I'll keep monitoring the server, and if it starts dropping packets again
or load increases I'll check whether irqbalanced does the right thing,
and if not I'll implement your suggestion.

Thank you very much!


/Benny

^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-12 23:18         ` Changli Gao
  2010-04-13  5:56           ` Eric Dumazet
@ 2010-04-13 12:33           ` Paweł Staszewski
  2010-04-13 12:53             ` Eric Dumazet
  1 sibling, 1 reply; 15+ messages in thread
From: Paweł Staszewski @ 2010-04-13 12:33 UTC (permalink / raw)
  To: Changli Gao; +Cc: Benny Amorsen, zhigang gong, netdev

[-- Attachment #1: Type: text/plain, Size: 2348 bytes --]

W dniu 2010-04-13 01:18, Changli Gao pisze:
> On Tue, Apr 13, 2010 at 1:06 AM, Benny Amorsen<benny+usenet@amorsen.dk>  wrote:
>    
>>   99:         24    1306226          3          2   PCI-MSI-edge      eth1-tx-0
>>   100:      15735    1648774          3          7   PCI-MSI-edge      eth1-tx-1
>>   101:          8         11          9    1083022   PCI-MSI-edge      eth1-tx-2
>>   102:          0          0          0          0   PCI-MSI-edge      eth1-tx-3
>>   103:         18         15       6131    1095383   PCI-MSI-edge      eth1-rx-0
>>   104:        217         32      46544    1335325   PCI-MSI-edge      eth1-rx-1
>>   105:        154    1305595        218         16   PCI-MSI-edge      eth1-rx-2
>>   106:         17         16       8229    1467509   PCI-MSI-edge      eth1-rx-3
>>   107:          0          0          1          0   PCI-MSI-edge      eth1
>>   108:          2         14         15    1003053   PCI-MSI-edge      eth0-tx-0
>>   109:       8226    1668924        478        487   PCI-MSI-edge      eth0-tx-1
>>   110:          3    1188874         17         12   PCI-MSI-edge      eth0-tx-2
>>   111:          0          0          0          0   PCI-MSI-edge      eth0-tx-3
>>   112:        203        185       5324    1015263   PCI-MSI-edge      eth0-rx-0
>>   113:       4141    1600793        153        159   PCI-MSI-edge      eth0-rx-1
>>   114:      16242    1210108        436       3124   PCI-MSI-edge      eth0-rx-2
>>   115:        267       4173      19471    1321252   PCI-MSI-edge      eth0-rx-3
>>   116:          0          1          0          0   PCI-MSI-edge      eth0
>>
>>
>> irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
>> which to my mind should cause the same problem as before (where CPU1 and
>> CPU3 was handling all packets). Yet the box clearly works much better
>> than before.
>>      
> irqbalanced? I don't think it can work properly. Try RPS in netdev and
> linux-next tree, and if cpu load isn't even, try this patch:
> http://patchwork.ozlabs.org/patch/49915/ .
>
>
>    
Yes without irqbalance - and with irq affinity set by hand router will 
work much better.

But I don't think that RPS will help him - I make some tests with RPS 
and AFFINITY - results in attached file.
Test router make traffic management (hfsc) for almost 9k users





[-- Attachment #2: RPS_AFFINITY_TEST.txt --]
[-- Type: text/plain, Size: 5028 bytes --]

##############################################################################
eth0 -> CPU0
eth1 -> CPU5
RPS:
echo 00e0 > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo 000e > /sys/class/net/eth0/queues/rx-0/rps_cpus

------------------------------------------------------------------------------
   PerfTop:   85205 irqs/sec  kernel:97.1% [100000 cycles],  (all, 8 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

           214930.00 - 24.5% : _raw_spin_lock
            63844.00 -  7.3% : u32_classify
            48381.00 -  5.5% : e1000_clean
            47754.00 -  5.5% : rb_next
            37222.00 -  4.2% : e1000_intr_msi
            26295.00 -  3.0% : hfsc_enqueue
            17371.00 -  2.0% : rb_erase
            15290.00 -  1.7% : _raw_spin_lock_irqsave
            14958.00 -  1.7% : rb_insert_color
            14439.00 -  1.6% : update_vf
            14384.00 -  1.6% : e1000_xmit_frame
            14356.00 -  1.6% : hfsc_dequeue
            13804.00 -  1.6% : e1000_clean_tx_irq
            13413.00 -  1.5% : ipt_do_table
             9654.00 -  1.1% : ip_route_input

##############################################################################
eth0 -> CPU0
eth1 -> CPU5
NO RPS

------------------------------------------------------------------------------
   PerfTop:   33800 irqs/sec  kernel:96.9% [100000 cycles],  (all, 8 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

            19361.00 - 11.2% : e1000_clean
            16424.00 -  9.5% : rb_next
            13060.00 -  7.5% : e1000_intr_msi
             7293.00 -  4.2% : u32_classify
             6875.00 -  4.0% : ipt_do_table
             5811.00 -  3.4% : _raw_spin_lock
             5754.00 -  3.3% : e1000_xmit_frame
             5671.00 -  3.3% : hfsc_dequeue
             4503.00 -  2.6% : __alloc_skb
             4156.00 -  2.4% : hfsc_enqueue
             4090.00 -  2.4% : e1000_clean_tx_irq
             3809.00 -  2.2% : e1000_clean_rx_irq
             3424.00 -  2.0% : update_vf
             3028.00 -  1.7% : rb_erase
             2714.00 -  1.6% : ip_route_input

##############################################################################
eth0 -> CPU0,CPU1,CPU2,CPU4 -> affinity echo 0f > /proc/irq/30/smp_affinity
eth1 -> CPU5,CPU6,CPU7,CPU8 -> affinity echo f0 > /proc/irq/31/smp_affinity
NO RPS
------------------------------------------------------------------------------
   PerfTop:   42362 irqs/sec  kernel:96.0% [100000 cycles],  (all, 8 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

            33815.00 - 10.6% : rb_next
            21357.00 -  6.7% : u32_classify
            14525.00 -  4.6% : _raw_spin_lock
            14346.00 -  4.5% : e1000_clean
            12798.00 -  4.0% : hfsc_enqueue
            10526.00 -  3.3% : ipt_do_table
             9999.00 -  3.1% : hfsc_dequeue
             9976.00 -  3.1% : e1000_intr_msi
             9787.00 -  3.1% : rb_erase
             8259.00 -  2.6% : e1000_xmit_frame
             8015.00 -  2.5% : rb_insert_color
             7948.00 -  2.5% : update_vf
             6868.00 -  2.2% : e1000_clean_tx_irq
             6822.00 -  2.1% : e1000_clean_rx_irq
             6368.00 -  2.0% : __alloc_skb

##############################################################################
eth0 -> CPU0,CPU1,CPU2,CPU4 -> affinity echo 0f > /proc/irq/30/smp_affinity
eth1 -> CPU5,CPU6,CPU7,CPU8 -> affinity echo f0 > /proc/irq/31/smp_affinity
RPS:
echo 0f > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo f0 > /sys/class/net/eth1/queues/rx-0/rps_cpus
------------------------------------------------------------------------------
   PerfTop:   81051 irqs/sec  kernel:96.9% [100000 cycles],  (all, 8 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

           167110.00 - 22.3% : _raw_spin_lock
            58221.00 -  7.8% : u32_classify
            46379.00 -  6.2% : rb_next
            35189.00 -  4.7% : e1000_clean
            25614.00 -  3.4% : e1000_intr_msi
            24094.00 -  3.2% : hfsc_enqueue
            16231.00 -  2.2% : rb_erase
            14298.00 -  1.9% : rb_insert_color
            13751.00 -  1.8% : update_vf
            13712.00 -  1.8% : ipt_do_table
            13588.00 -  1.8% : hfsc_dequeue
            13335.00 -  1.8% : e1000_xmit_frame
            12449.00 -  1.7% : e1000_clean_tx_irq
            11510.00 -  1.5% : net_tx_action
            11428.00 -  1.5% : _raw_spin_lock_irqsave


^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-13 12:33           ` Paweł Staszewski
@ 2010-04-13 12:53             ` Eric Dumazet
  2010-04-13 13:39               ` Paweł Staszewski
  0 siblings, 1 reply; 15+ messages in thread
From: Eric Dumazet @ 2010-04-13 12:53 UTC (permalink / raw)
  To: Paweł Staszewski; +Cc: Changli Gao, Benny Amorsen, zhigang gong, netdev

Le mardi 13 avril 2010 à 14:33 +0200, Paweł Staszewski a écrit :
> W dniu 2010-04-13 01:18, Changli Gao pisze:
> > On Tue, Apr 13, 2010 at 1:06 AM, Benny Amorsen<benny+usenet@amorsen.dk>  wrote:
> >    
> >>   99:         24    1306226          3          2   PCI-MSI-edge      eth1-tx-0
> >>   100:      15735    1648774          3          7   PCI-MSI-edge      eth1-tx-1
> >>   101:          8         11          9    1083022   PCI-MSI-edge      eth1-tx-2
> >>   102:          0          0          0          0   PCI-MSI-edge      eth1-tx-3
> >>   103:         18         15       6131    1095383   PCI-MSI-edge      eth1-rx-0
> >>   104:        217         32      46544    1335325   PCI-MSI-edge      eth1-rx-1
> >>   105:        154    1305595        218         16   PCI-MSI-edge      eth1-rx-2
> >>   106:         17         16       8229    1467509   PCI-MSI-edge      eth1-rx-3
> >>   107:          0          0          1          0   PCI-MSI-edge      eth1
> >>   108:          2         14         15    1003053   PCI-MSI-edge      eth0-tx-0
> >>   109:       8226    1668924        478        487   PCI-MSI-edge      eth0-tx-1
> >>   110:          3    1188874         17         12   PCI-MSI-edge      eth0-tx-2
> >>   111:          0          0          0          0   PCI-MSI-edge      eth0-tx-3
> >>   112:        203        185       5324    1015263   PCI-MSI-edge      eth0-rx-0
> >>   113:       4141    1600793        153        159   PCI-MSI-edge      eth0-rx-1
> >>   114:      16242    1210108        436       3124   PCI-MSI-edge      eth0-rx-2
> >>   115:        267       4173      19471    1321252   PCI-MSI-edge      eth0-rx-3
> >>   116:          0          1          0          0   PCI-MSI-edge      eth0
> >>
> >>
> >> irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
> >> which to my mind should cause the same problem as before (where CPU1 and
> >> CPU3 was handling all packets). Yet the box clearly works much better
> >> than before.
> >>      
> > irqbalanced? I don't think it can work properly. Try RPS in netdev and
> > linux-next tree, and if cpu load isn't even, try this patch:
> > http://patchwork.ozlabs.org/patch/49915/ .
> >
> >
> >    
> Yes without irqbalance - and with irq affinity set by hand router will 
> work much better.
> 
> But I don't think that RPS will help him - I make some tests with RPS 
> and AFFINITY - results in attached file.
> Test router make traffic management (hfsc) for almost 9k users

Thanks for sharing Pawel.

But obviously you are mixing apples and oranges.

 Are you aware that HFSC and other trafic shapers do serialize access to
data structures ? If many cpus try to access these structures in //, you
have a lot of cache line misses. HFSC is a real memory hog :(

Benny do have firewalling (highly parallelized these days, iptables was
well improved in this area), but no traffic control.

Anyway, Benny has now multiqueue devices, and therefore RPS will not
help him. I suggested RPS before his move to multiqueue, and multiqueue
is the most sensible way to improve things, when no central lock is
used. Every cpu can really work in //.




^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-13 12:53             ` Eric Dumazet
@ 2010-04-13 13:39               ` Paweł Staszewski
  0 siblings, 0 replies; 15+ messages in thread
From: Paweł Staszewski @ 2010-04-13 13:39 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Changli Gao, Benny Amorsen, zhigang gong, netdev

W dniu 2010-04-13 14:53, Eric Dumazet pisze:
> Le mardi 13 avril 2010 à 14:33 +0200, Paweł Staszewski a écrit :
>    
>> W dniu 2010-04-13 01:18, Changli Gao pisze:
>>      
>>> On Tue, Apr 13, 2010 at 1:06 AM, Benny Amorsen<benny+usenet@amorsen.dk>   wrote:
>>>
>>>        
>>>>    99:         24    1306226          3          2   PCI-MSI-edge      eth1-tx-0
>>>>    100:      15735    1648774          3          7   PCI-MSI-edge      eth1-tx-1
>>>>    101:          8         11          9    1083022   PCI-MSI-edge      eth1-tx-2
>>>>    102:          0          0          0          0   PCI-MSI-edge      eth1-tx-3
>>>>    103:         18         15       6131    1095383   PCI-MSI-edge      eth1-rx-0
>>>>    104:        217         32      46544    1335325   PCI-MSI-edge      eth1-rx-1
>>>>    105:        154    1305595        218         16   PCI-MSI-edge      eth1-rx-2
>>>>    106:         17         16       8229    1467509   PCI-MSI-edge      eth1-rx-3
>>>>    107:          0          0          1          0   PCI-MSI-edge      eth1
>>>>    108:          2         14         15    1003053   PCI-MSI-edge      eth0-tx-0
>>>>    109:       8226    1668924        478        487   PCI-MSI-edge      eth0-tx-1
>>>>    110:          3    1188874         17         12   PCI-MSI-edge      eth0-tx-2
>>>>    111:          0          0          0          0   PCI-MSI-edge      eth0-tx-3
>>>>    112:        203        185       5324    1015263   PCI-MSI-edge      eth0-rx-0
>>>>    113:       4141    1600793        153        159   PCI-MSI-edge      eth0-rx-1
>>>>    114:      16242    1210108        436       3124   PCI-MSI-edge      eth0-rx-2
>>>>    115:        267       4173      19471    1321252   PCI-MSI-edge      eth0-rx-3
>>>>    116:          0          1          0          0   PCI-MSI-edge      eth0
>>>>
>>>>
>>>> irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
>>>> which to my mind should cause the same problem as before (where CPU1 and
>>>> CPU3 was handling all packets). Yet the box clearly works much better
>>>> than before.
>>>>
>>>>          
>>> irqbalanced? I don't think it can work properly. Try RPS in netdev and
>>> linux-next tree, and if cpu load isn't even, try this patch:
>>> http://patchwork.ozlabs.org/patch/49915/ .
>>>
>>>
>>>
>>>        
>> Yes without irqbalance - and with irq affinity set by hand router will
>> work much better.
>>
>> But I don't think that RPS will help him - I make some tests with RPS
>> and AFFINITY - results in attached file.
>> Test router make traffic management (hfsc) for almost 9k users
>>      
> Thanks for sharing Pawel.
>
> But obviously you are mixing apples and oranges.
>
>   Are you aware that HFSC and other trafic shapers do serialize access to
> data structures ? If many cpus try to access these structures in //, you
> have a lot of cache line misses. HFSC is a real memory hog :(
>
>    
Thanks Eric for explanation why RPS is useless for traffic management 
routers.

> Benny do have firewalling (highly parallelized these days, iptables was
> well improved in this area), but no traffic control.
>
>    
Hmm so maybe better choice for traffic management is use iptables for 
"filter classification" instead of "u32 filters"- something like 
iptables CLASSIFY target

> Anyway, Benny has now multiqueue devices, and therefore RPS will not
> help him. I suggested RPS before his move to multiqueue, and multiqueue
> is the most sensible way to improve things, when no central lock is
> used. Every cpu can really work in //.
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>    


^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-13  7:56             ` Benny Amorsen
@ 2010-04-15 13:23               ` Benny Amorsen
  2010-04-15 13:42                 ` Eric Dumazet
  0 siblings, 1 reply; 15+ messages in thread
From: Benny Amorsen @ 2010-04-15 13:23 UTC (permalink / raw)
  To: Eric Dumazet; +Cc: Changli Gao, zhigang gong, netdev

Benny Amorsen <benny+usenet@amorsen.dk> writes:

> I'll keep monitoring the server, and if it starts dropping packets again
> or load increases I'll check whether irqbalanced does the right thing,
> and if not I'll implement your suggestion.

It did start dropping packets (although very few, a few packets dropped
at once perhaps every ten minutes). Irqbalanced didn't move the
interrupts.

Doing

echo 01 >/proc/irq/99/smp_affinity
echo 02 >/proc/irq/100/smp_affinity
echo 04 >/proc/irq/101/smp_affinity

and so on like Erik Dumazet suggested seems to have helped, but not
entirely solved the problem.

The problem now manifests itself this way in ethtool -S:
     rx_no_buffer_count: 270
     rx_queue_drop_packet_count: 270

I can't be sure that I'm not just getting hit by a 1Gbps traffic spike,
of course, but it is a bit strange that a machine which can do 200Mbps
at 92% idle can't handle subsecond peaks close to 1Gbps...

I wish ifstat could report errors so I could see what the traffic rate
was when the problem occurred...


/Benny

^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: Strange packet drops with heavy firewalling
  2010-04-15 13:23               ` Benny Amorsen
@ 2010-04-15 13:42                 ` Eric Dumazet
  0 siblings, 0 replies; 15+ messages in thread
From: Eric Dumazet @ 2010-04-15 13:42 UTC (permalink / raw)
  To: Benny Amorsen; +Cc: Changli Gao, zhigang gong, netdev

Le jeudi 15 avril 2010 à 15:23 +0200, Benny Amorsen a écrit :
> Benny Amorsen <benny+usenet@amorsen.dk> writes:
> 
> > I'll keep monitoring the server, and if it starts dropping packets again
> > or load increases I'll check whether irqbalanced does the right thing,
> > and if not I'll implement your suggestion.
> 
> It did start dropping packets (although very few, a few packets dropped
> at once perhaps every ten minutes). Irqbalanced didn't move the
> interrupts.
> 
> Doing
> 
> echo 01 >/proc/irq/99/smp_affinity
> echo 02 >/proc/irq/100/smp_affinity
> echo 04 >/proc/irq/101/smp_affinity
> 
> and so on like Erik Dumazet suggested seems to have helped, but not
> entirely solved the problem.
> 
> The problem now manifests itself this way in ethtool -S:
>      rx_no_buffer_count: 270
>      rx_queue_drop_packet_count: 270
> 
> I can't be sure that I'm not just getting hit by a 1Gbps traffic spike,
> of course, but it is a bit strange that a machine which can do 200Mbps
> at 92% idle can't handle subsecond peaks close to 1Gbps...
> 

Even with multiqueue, its quite possible one queue gets more than one
packet per micro second. Time to process a packet might be greater then
1 us even on recent hardware. So bursts of 1000 small packets with same
flow information, hit one queue, one cpu, and fill rx ring.

Loosing these packets is OK, its very likely its an attack :)

> I wish ifstat could report errors so I could see what the traffic rate
> was when the problem occurred...

yes, it could be added I guess.



^ permalink raw reply	[flat|nested] 15+ messages in thread
end of thread, other threads:[~2010-04-15 13:43 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-04-09  9:56 Strange packet drops with heavy firewalling Benny Amorsen
2010-04-09 11:47 ` Eric Dumazet
2010-04-09 12:33   ` Benny Amorsen
2010-04-09 13:29     ` Eric Dumazet
2010-04-12  6:20       ` Benny Amorsen
     [not found] ` <q2v40c9f5b21004120116p766df82dj88c6af4e4cad55f@mail.gmail.com>
2010-04-12 14:44   ` Benny Lyne Amorsen
     [not found]     ` <p2x40c9f5b21004120833jd7a749cak6ea69cebd28f8352@mail.gmail.com>
2010-04-12 17:06       ` Benny Amorsen
2010-04-12 23:18         ` Changli Gao
2010-04-13  5:56           ` Eric Dumazet
2010-04-13  7:56             ` Benny Amorsen
2010-04-15 13:23               ` Benny Amorsen
2010-04-15 13:42                 ` Eric Dumazet
2010-04-13 12:33           ` Paweł Staszewski
2010-04-13 12:53             ` Eric Dumazet
2010-04-13 13:39               ` Paweł Staszewski

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.