From: "Paweł Staszewski" <pstaszewski@itcare.pl>
To: Changli Gao <xiaosuo@gmail.com>
Cc: Benny Amorsen <benny+usenet@amorsen.dk>,
zhigang gong <zhigang.gong@gmail.com>,
netdev@vger.kernel.org
Subject: Re: Strange packet drops with heavy firewalling
Date: Tue, 13 Apr 2010 14:33:42 +0200 [thread overview]
Message-ID: <4BC464A6.9000307@itcare.pl> (raw)
In-Reply-To: <u2y412e6f7f1004121618p6d6eff30q8a45a03faa59a912@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2348 bytes --]
W dniu 2010-04-13 01:18, Changli Gao pisze:
> On Tue, Apr 13, 2010 at 1:06 AM, Benny Amorsen<benny+usenet@amorsen.dk> wrote:
>
>> 99: 24 1306226 3 2 PCI-MSI-edge eth1-tx-0
>> 100: 15735 1648774 3 7 PCI-MSI-edge eth1-tx-1
>> 101: 8 11 9 1083022 PCI-MSI-edge eth1-tx-2
>> 102: 0 0 0 0 PCI-MSI-edge eth1-tx-3
>> 103: 18 15 6131 1095383 PCI-MSI-edge eth1-rx-0
>> 104: 217 32 46544 1335325 PCI-MSI-edge eth1-rx-1
>> 105: 154 1305595 218 16 PCI-MSI-edge eth1-rx-2
>> 106: 17 16 8229 1467509 PCI-MSI-edge eth1-rx-3
>> 107: 0 0 1 0 PCI-MSI-edge eth1
>> 108: 2 14 15 1003053 PCI-MSI-edge eth0-tx-0
>> 109: 8226 1668924 478 487 PCI-MSI-edge eth0-tx-1
>> 110: 3 1188874 17 12 PCI-MSI-edge eth0-tx-2
>> 111: 0 0 0 0 PCI-MSI-edge eth0-tx-3
>> 112: 203 185 5324 1015263 PCI-MSI-edge eth0-rx-0
>> 113: 4141 1600793 153 159 PCI-MSI-edge eth0-rx-1
>> 114: 16242 1210108 436 3124 PCI-MSI-edge eth0-rx-2
>> 115: 267 4173 19471 1321252 PCI-MSI-edge eth0-rx-3
>> 116: 0 1 0 0 PCI-MSI-edge eth0
>>
>>
>> irqbalanced seems to have picked CPU1 and CPU3 for all the interrupts,
>> which to my mind should cause the same problem as before (where CPU1 and
>> CPU3 was handling all packets). Yet the box clearly works much better
>> than before.
>>
> irqbalanced? I don't think it can work properly. Try RPS in netdev and
> linux-next tree, and if cpu load isn't even, try this patch:
> http://patchwork.ozlabs.org/patch/49915/ .
>
>
>
Yes without irqbalance - and with irq affinity set by hand router will
work much better.
But I don't think that RPS will help him - I make some tests with RPS
and AFFINITY - results in attached file.
Test router make traffic management (hfsc) for almost 9k users
[-- Attachment #2: RPS_AFFINITY_TEST.txt --]
[-- Type: text/plain, Size: 5028 bytes --]
##############################################################################
eth0 -> CPU0
eth1 -> CPU5
RPS:
echo 00e0 > /sys/class/net/eth1/queues/rx-0/rps_cpus
echo 000e > /sys/class/net/eth0/queues/rx-0/rps_cpus
------------------------------------------------------------------------------
PerfTop: 85205 irqs/sec kernel:97.1% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
214930.00 - 24.5% : _raw_spin_lock
63844.00 - 7.3% : u32_classify
48381.00 - 5.5% : e1000_clean
47754.00 - 5.5% : rb_next
37222.00 - 4.2% : e1000_intr_msi
26295.00 - 3.0% : hfsc_enqueue
17371.00 - 2.0% : rb_erase
15290.00 - 1.7% : _raw_spin_lock_irqsave
14958.00 - 1.7% : rb_insert_color
14439.00 - 1.6% : update_vf
14384.00 - 1.6% : e1000_xmit_frame
14356.00 - 1.6% : hfsc_dequeue
13804.00 - 1.6% : e1000_clean_tx_irq
13413.00 - 1.5% : ipt_do_table
9654.00 - 1.1% : ip_route_input
##############################################################################
eth0 -> CPU0
eth1 -> CPU5
NO RPS
------------------------------------------------------------------------------
PerfTop: 33800 irqs/sec kernel:96.9% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
19361.00 - 11.2% : e1000_clean
16424.00 - 9.5% : rb_next
13060.00 - 7.5% : e1000_intr_msi
7293.00 - 4.2% : u32_classify
6875.00 - 4.0% : ipt_do_table
5811.00 - 3.4% : _raw_spin_lock
5754.00 - 3.3% : e1000_xmit_frame
5671.00 - 3.3% : hfsc_dequeue
4503.00 - 2.6% : __alloc_skb
4156.00 - 2.4% : hfsc_enqueue
4090.00 - 2.4% : e1000_clean_tx_irq
3809.00 - 2.2% : e1000_clean_rx_irq
3424.00 - 2.0% : update_vf
3028.00 - 1.7% : rb_erase
2714.00 - 1.6% : ip_route_input
##############################################################################
eth0 -> CPU0,CPU1,CPU2,CPU4 -> affinity echo 0f > /proc/irq/30/smp_affinity
eth1 -> CPU5,CPU6,CPU7,CPU8 -> affinity echo f0 > /proc/irq/31/smp_affinity
NO RPS
------------------------------------------------------------------------------
PerfTop: 42362 irqs/sec kernel:96.0% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
33815.00 - 10.6% : rb_next
21357.00 - 6.7% : u32_classify
14525.00 - 4.6% : _raw_spin_lock
14346.00 - 4.5% : e1000_clean
12798.00 - 4.0% : hfsc_enqueue
10526.00 - 3.3% : ipt_do_table
9999.00 - 3.1% : hfsc_dequeue
9976.00 - 3.1% : e1000_intr_msi
9787.00 - 3.1% : rb_erase
8259.00 - 2.6% : e1000_xmit_frame
8015.00 - 2.5% : rb_insert_color
7948.00 - 2.5% : update_vf
6868.00 - 2.2% : e1000_clean_tx_irq
6822.00 - 2.1% : e1000_clean_rx_irq
6368.00 - 2.0% : __alloc_skb
##############################################################################
eth0 -> CPU0,CPU1,CPU2,CPU4 -> affinity echo 0f > /proc/irq/30/smp_affinity
eth1 -> CPU5,CPU6,CPU7,CPU8 -> affinity echo f0 > /proc/irq/31/smp_affinity
RPS:
echo 0f > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo f0 > /sys/class/net/eth1/queues/rx-0/rps_cpus
------------------------------------------------------------------------------
PerfTop: 81051 irqs/sec kernel:96.9% [100000 cycles], (all, 8 CPUs)
------------------------------------------------------------------------------
samples pcnt kernel function
_______ _____ _______________
167110.00 - 22.3% : _raw_spin_lock
58221.00 - 7.8% : u32_classify
46379.00 - 6.2% : rb_next
35189.00 - 4.7% : e1000_clean
25614.00 - 3.4% : e1000_intr_msi
24094.00 - 3.2% : hfsc_enqueue
16231.00 - 2.2% : rb_erase
14298.00 - 1.9% : rb_insert_color
13751.00 - 1.8% : update_vf
13712.00 - 1.8% : ipt_do_table
13588.00 - 1.8% : hfsc_dequeue
13335.00 - 1.8% : e1000_xmit_frame
12449.00 - 1.7% : e1000_clean_tx_irq
11510.00 - 1.5% : net_tx_action
11428.00 - 1.5% : _raw_spin_lock_irqsave
next prev parent reply other threads:[~2010-04-13 12:33 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-04-09 9:56 Strange packet drops with heavy firewalling Benny Amorsen
2010-04-09 11:47 ` Eric Dumazet
2010-04-09 12:33 ` Benny Amorsen
2010-04-09 13:29 ` Eric Dumazet
2010-04-12 6:20 ` Benny Amorsen
[not found] ` <q2v40c9f5b21004120116p766df82dj88c6af4e4cad55f@mail.gmail.com>
2010-04-12 14:44 ` Benny Lyne Amorsen
[not found] ` <p2x40c9f5b21004120833jd7a749cak6ea69cebd28f8352@mail.gmail.com>
2010-04-12 17:06 ` Benny Amorsen
2010-04-12 23:18 ` Changli Gao
2010-04-13 5:56 ` Eric Dumazet
2010-04-13 7:56 ` Benny Amorsen
2010-04-15 13:23 ` Benny Amorsen
2010-04-15 13:42 ` Eric Dumazet
2010-04-13 12:33 ` Paweł Staszewski [this message]
2010-04-13 12:53 ` Eric Dumazet
2010-04-13 13:39 ` Paweł Staszewski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4BC464A6.9000307@itcare.pl \
--to=pstaszewski@itcare.pl \
--cc=benny+usenet@amorsen.dk \
--cc=netdev@vger.kernel.org \
--cc=xiaosuo@gmail.com \
--cc=zhigang.gong@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.