* Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
@ 2017-08-11 17:51 Paweł Staszewski
2017-08-12 12:23 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-11 17:51 UTC (permalink / raw)
To: Linux Kernel Network Developers
Hi
I made some tests for performance comparison.
Tested HW (FORWARDING HOST):
Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Test diagram:
TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
(enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
172.16.0.1 to 172.16.0.255
TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
Settings used for FORWARDING HOST (changed param. was only number of RSS
combined queues + set affinity assignment for them to fit with first
numa node where 2x10G port card is installed)
ixgbe driver used from kernel (in-kernel build - not a module)
#!/bin/sh
ifc='enp216s0f0 enp216s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 4096 tx 1024
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 10
ethtool -L $i combined 16
ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
tx-nocache-copy on ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
ip link set up dev enp216s0f0
ip link set up dev enp216s0f1
ip a a 10.0.0.1/30 dev enp216s0f0
ip link add link enp216s0f1 name vlan1000 type vlan id 1000
ip link set up dev vlan1000
ip a a 10.0.0.5/30 dev vlan1000
ip route add 172.16.0.0/12 via 10.0.0.6
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
#cat /sys/devices/system/node/node1/cpulist
#14-27,42-55
#cat /sys/devices/system/node/node0/cpulist
#0-13,28-41
#################################################
Looks like forwarding performance when using vlans on ixgbe is less that
without vlans for about 30-40% (wondering if this is some vlan
offloading problem and ixgbe)
settings below:
ethtool -k enp216s0f0
Features for enp216s0f0:
Cannot get device udp-fragmentation-offload settings: Operation not
supported
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: on
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: on
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: on
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: on
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off
hw-tc-offload: off
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
Another thing is that forwarding performance does not scale with number
of cores when 7+ cores are reached
perf top:
PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16.32% [kernel] [k] skb_dst_force
16.30% [kernel] [k] dst_release
15.11% [kernel] [k] rt_cache_valid
12.62% [kernel] [k] ipv4_mtu
5.60% [kernel] [k] do_raw_spin_lock
3.03% [kernel] [k] fib_table_lookup
2.70% [kernel] [k] ip_finish_output2
2.10% [kernel] [k] dev_gro_receive
1.89% [kernel] [k] eth_type_trans
1.81% [kernel] [k] ixgbe_poll
1.15% [kernel] [k] ixgbe_xmit_frame_ring
1.06% [kernel] [k] __build_skb
1.04% [kernel] [k] __dev_queue_xmit
0.97% [kernel] [k] ip_rcv
0.78% [kernel] [k] netif_skb_features
0.74% [kernel] [k] ipt_do_table
0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
0.64% [kernel] [k] ip_forward
0.59% [kernel] [k] __netif_receive_skb_core
0.55% [kernel] [k] dev_hard_start_xmit
0.53% [kernel] [k] ip_route_input_rcu
0.53% [kernel] [k] ip_rcv_finish
0.51% [kernel] [k] page_frag_free
0.50% [kernel] [k] kmem_cache_alloc
0.50% [kernel] [k] udp_v4_early_demux
0.44% [kernel] [k] skb_release_data
0.42% [kernel] [k] inet_gro_receive
0.40% [kernel] [k] sch_direct_xmit
0.39% [kernel] [k] __local_bh_enable_ip
0.33% [kernel] [k] netdev_pick_tx
0.33% [kernel] [k] validate_xmit_skb
0.28% [kernel] [k] fib_validate_source
0.27% [kernel] [k] deliver_ptype_list_skb
0.25% [kernel] [k] eth_header
0.23% [kernel] [k] get_dma_ops
0.22% [kernel] [k] skb_network_protocol
0.21% [kernel] [k] ip_output
0.21% [kernel] [k] vlan_dev_hard_start_xmit
0.20% [kernel] [k] ixgbe_alloc_rx_buffers
0.18% [kernel] [k] nf_hook_slow
0.18% [kernel] [k] apic_timer_interrupt
0.18% [kernel] [k] virt_to_head_page
0.18% [kernel] [k] build_skb
0.16% [kernel] [k] swiotlb_map_page
0.16% [kernel] [k] ip_finish_output
0.16% [kernel] [k] udp4_gro_receive
RESULTS:
CSV format - delimeter ";"
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;1;64;1470912;88247040;1470720;85305530
1;1;64;1470912;88285440;1470977;85335110
2;1;64;1470464;88247040;1470402;85290508
3;1;64;1471424;88262400;1471230;85353728
4;1;64;1468736;88166400;1468672;85201652
5;1;64;1470016;88181760;1469949;85234944
6;1;64;1470720;88247040;1470466;85290624
7;1;64;1471232;88277760;1471167;85346246
8;1;64;1469184;88170240;1469249;85216326
9;1;64;1470592;88227840;1470847;85294394
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;2;64;2413120;144802560;2413245;139975924
1;2;64;2415296;144913920;2415356;140098188
2;2;64;2416768;144898560;2416573;140105670
3;2;64;2418176;145056000;2418110;140261806
4;2;64;2416512;144990720;2416509;140172950
5;2;64;2415168;144860160;2414466;140064780
6;2;64;2416960;144983040;2416833;140190930
7;2;64;2413632;144768000;2413568;140001734
8;2;64;2415296;144898560;2414589;140087168
9;2;64;2416576;144963840;2416892;140190930
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;3;64;3419008;205155840;3418882;198239244
1;3;64;3428032;205585920;3427971;198744234
2;3;64;3425472;205536000;3425344;198677260
3;3;64;3425088;205470720;3425156;198603136
4;3;64;3427648;205693440;3426883;198773888
5;3;64;3426880;205670400;3427392;198796044
6;3;64;3429120;205678080;3430140;198848186
7;3;64;3422976;205355520;3423490;198458136
8;3;64;3423168;205336320;3423486;198495372
9;3;64;3424384;205493760;3425538;198617868
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;4;64;4406464;264364800;4405244;255560296
1;4;64;4404672;264349440;4405122;255541504
2;4;64;4402368;264049920;4403326;255188864
3;4;64;4401344;264076800;4400702;255207134
4;4;64;4385536;263074560;4386620;254312716
5;4;64;4386560;263189760;4385404;254379532
6;4;64;4398784;263857920;4399031;255025288
7;4;64;4407232;264445440;4407998;255637900
8;4;64;4413184;264698880;4413758;255875816
9;4;64;4411328;264526080;4411906;255712372
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;5;64;5094464;305871360;5094464;295657262
1;5;64;5090816;305514240;5091201;295274810
2;5;64;5088384;305387520;5089792;295175108
3;5;64;5079296;304869120;5079484;294680368
4;5;64;5092992;305544960;5094207;295349166
5;5;64;5092416;305502720;5093372;295334260
6;5;64;5080896;304896000;5081090;294677004
7;5;64;5085376;305114880;5086401;294933058
8;5;64;5092544;305575680;5092036;295356938
9;5;64;5093056;305652480;5093832;295449506
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;6;64;5705088;342351360;5705784;330965110
1;6;64;5710272;342743040;5707591;331373952
2;6;64;5703424;342182400;5701826;330776552
3;6;64;5708736;342604800;5707963;331147462
4;6;64;5710144;342654720;5712067;331202910
5;6;64;5712064;342777600;5711361;331292288
6;6;64;5710144;342585600;5708607;331144272
7;6;64;5699840;342021120;5697853;330609222
8;6;64;5701184;342124800;5702909;330653592
9;6;64;5711360;342735360;5713283;331247686
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;7;64;6244416;374603520;6243591;362180072
1;7;64;6230912;374016000;6231490;361534126
2;7;64;6244800;374776320;6244866;362224326
3;7;64;6238720;374376960;6238261;361838510
4;7;64;6218816;373079040;6220413;360683962
5;7;64;6224320;373566720;6225086;361017404
6;7;64;6224000;373570560;6221370;360936088
7;7;64;6210048;372741120;6210627;360212654
8;7;64;6231616;374035200;6231537;361445502
9;7;64;6227840;373724160;6228802;361162752
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;8;64;6251840;375144960;6251849;362609678
1;8;64;6250816;375014400;6250881;362547038
2;8;64;6257728;375432960;6257160;362911104
3;8;64;6255552;375325440;6255622;362822074
4;8;64;6243776;374576640;6243270;362120622
5;8;64;6237184;374296320;6237690;361790080
6;8;64;6240960;374415360;6240714;361927366
7;8;64;6222784;373317120;6223746;360854424
8;8;64;6225920;373593600;6227014;361154980
9;8;64;6238528;374304000;6237701;361845238
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;14;64;6486144;389184000;6486135;376236488
1;14;64;6454912;387390720;6454222;374466734
2;14;64;6441152;386480640;6440431;373572780
3;14;64;6450240;386972160;6450870;374070014
4;14;64;6465600;387997440;6467221;375089654
5;14;64;6448384;386860800;6448000;373980230
6;14;64;6452352;387095040;6452148;374168904
7;14;64;6441984;386507520;6443203;373665058
8;14;64;6456704;387340800;6455744;374429092
9;14;64;6464640;387901440;6465218;374949004
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;6939008;416325120;6938696;402411192
1;16;64;6941952;416444160;6941745;402558918
2;16;64;6960576;417584640;6960707;403698718
3;16;64;6940736;416486400;6941820;402503876
4;16;64;6927680;415741440;6927420;401853870
5;16;64;6929792;415687680;6929917;401839196
6;16;64;6950400;416989440;6950661;403026166
7;16;64;6953664;417216000;6953454;403260544
8;16;64;6948480;416851200;6948800;403023266
9;16;64;6924160;415422720;6924092;401542468
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-11 17:51 Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on Paweł Staszewski
@ 2017-08-12 12:23 ` Jesper Dangaard Brouer
2017-08-12 17:27 ` Paweł Staszewski
0 siblings, 1 reply; 37+ messages in thread
From: Jesper Dangaard Brouer @ 2017-08-12 12:23 UTC (permalink / raw)
To: Paweł Staszewski; +Cc: brouer, Linux Kernel Network Developers
On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> Hi
>
> I made some tests for performance comparison.
Thanks for doing this. Feel free to Cc me, if you do more of these
tests (so I don't miss them on the mailing list).
I don't understand stand if you are reporting a potential problem?
It would be good if you can provide a short summary section (of the
issue) in the _start_ of the email, and then provide all this nice data
afterwards, to back your case.
My understanding is, you report:
1. VLANs on ixgbe show a 30-40% slowdown
2. System stopped scaling after 7+ CPUs
> Tested HW (FORWARDING HOST):
>
> Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
Interesting, I've not heard about a Intel CPU called "Gold" before now,
but it does exist:
https://ark.intel.com/products/123541/Intel-Xeon-Gold-6132-Processor-19_25M-Cache-2_60-GHz
> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
This is one of my all time favorite NICs!
> Test diagram:
>
>
> TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
> (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
>
> Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
> 172.16.0.1 to 172.16.0.255
>
> TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
What kind of traffic flow? E.g. distribution, many/few source IPs...
> Settings used for FORWARDING HOST (changed param. was only number of RSS
> combined queues + set affinity assignment for them to fit with first
> numa node where 2x10G port card is installed)
>
> ixgbe driver used from kernel (in-kernel build - not a module)
>
Nice with a script showing you setup, thanks. I would be good if it had
comments, telling why you think this is a needed setup adjustment.
> #!/bin/sh
> ifc='enp216s0f0 enp216s0f1'
> for i in $ifc
> do
> ip link set up dev $i
> ethtool -A $i autoneg off rx off tx off
Good:
Turning off Ethernet flow control, to avoid receiver being the
bottleneck via pause-frames.
> ethtool -G $i rx 4096 tx 1024
You adjust the RX and TX ring queue sizes, this have effects that you
don't realize. Especially for the ixgbe driver, which have a page
recycle trick tied to the RX ring queue size.
> ip link set $i txqueuelen 1000
Setting tx queue len to the default 1000 seems redundant.
> ethtool -C $i rx-usecs 10
Adjusting this also have effects you might not realize. This actually
also affect the page recycle scheme of ixgbe. And can sometimes be
used to solve stalling on DMA TX completions, which could be you issue
here.
> ethtool -L $i combined 16
> ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
> tx-nocache-copy on ntuple on
Here are many setting above.
GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
this result in approx 10% slowdown.
AFAIK "tx-nocache-copy on" was also determined to be a bad option.
The "ntuple on" AFAIK disables the flow-director in the NIC. I though
this would actually help VLAN traffic, but I guess not.
> ethtool -N $i rx-flow-hash udp4 sdfn
Why do you change the NICs flow-hash?
> done
>
> ip link set up dev enp216s0f0
> ip link set up dev enp216s0f1
>
> ip a a 10.0.0.1/30 dev enp216s0f0
>
> ip link add link enp216s0f1 name vlan1000 type vlan id 1000
> ip link set up dev vlan1000
> ip a a 10.0.0.5/30 dev vlan1000
>
>
> ip route add 172.16.0.0/12 via 10.0.0.6
>
> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
> #cat /sys/devices/system/node/node1/cpulist
> #14-27,42-55
> #cat /sys/devices/system/node/node0/cpulist
> #0-13,28-41
Is this a NUMA system?
> #################################################
>
>
> Looks like forwarding performance when using vlans on ixgbe is less that
> without vlans for about 30-40% (wondering if this is some vlan
> offloading problem and ixgbe)
I would see this as a problem/bug that enabling VLANs cost this much.
> settings below:
>
> ethtool -k enp216s0f0
> Features for enp216s0f0:
> Cannot get device udp-fragmentation-offload settings: Operation not
> supported
> rx-checksumming: on
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: on
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp-mangleid-segmentation: on
> tx-tcp6-segmentation: on
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: on
> tx-vlan-offload: on
> ntuple-filters: on
> receive-hashing: on
> highdma: on [fixed]
> rx-vlan-filter: on
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: on
> tx-gre-csum-segmentation: on
> tx-ipxip4-segmentation: on
> tx-ipxip6-segmentation: on
> tx-udp_tnl-segmentation: on
> tx-udp_tnl-csum-segmentation: on
> tx-gso-partial: on
> tx-sctp-segmentation: off [fixed]
> tx-esp-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: on
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off
> hw-tc-offload: off
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> rx-udp_tunnel-port-offload: on
>
>
> Another thing is that forwarding performance does not scale with number
> of cores when 7+ cores are reached
I've seen problems with using Hyper-Threading CPUs. Could it be that
above 7 CPUs you are starting to use sibling-cores ?
> perf top:
>
> PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> 16.32% [kernel] [k] skb_dst_force
> 16.30% [kernel] [k] dst_release
> 15.11% [kernel] [k] rt_cache_valid
> 12.62% [kernel] [k] ipv4_mtu
It seems a little strange that these 4 functions are on the top
> 5.60% [kernel] [k] do_raw_spin_lock
Why is calling/taking this lock? (Use perf call-graph recording).
> 3.03% [kernel] [k] fib_table_lookup
> 2.70% [kernel] [k] ip_finish_output2
> 2.10% [kernel] [k] dev_gro_receive
> 1.89% [kernel] [k] eth_type_trans
> 1.81% [kernel] [k] ixgbe_poll
> 1.15% [kernel] [k] ixgbe_xmit_frame_ring
> 1.06% [kernel] [k] __build_skb
> 1.04% [kernel] [k] __dev_queue_xmit
> 0.97% [kernel] [k] ip_rcv
> 0.78% [kernel] [k] netif_skb_features
> 0.74% [kernel] [k] ipt_do_table
Unloading netfilter modules, will give more performance, but it
semifake to do so.
> 0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
> 0.64% [kernel] [k] ip_forward
> 0.59% [kernel] [k] __netif_receive_skb_core
> 0.55% [kernel] [k] dev_hard_start_xmit
> 0.53% [kernel] [k] ip_route_input_rcu
> 0.53% [kernel] [k] ip_rcv_finish
> 0.51% [kernel] [k] page_frag_free
> 0.50% [kernel] [k] kmem_cache_alloc
> 0.50% [kernel] [k] udp_v4_early_demux
> 0.44% [kernel] [k] skb_release_data
> 0.42% [kernel] [k] inet_gro_receive
> 0.40% [kernel] [k] sch_direct_xmit
> 0.39% [kernel] [k] __local_bh_enable_ip
> 0.33% [kernel] [k] netdev_pick_tx
> 0.33% [kernel] [k] validate_xmit_skb
> 0.28% [kernel] [k] fib_validate_source
> 0.27% [kernel] [k] deliver_ptype_list_skb
> 0.25% [kernel] [k] eth_header
> 0.23% [kernel] [k] get_dma_ops
> 0.22% [kernel] [k] skb_network_protocol
> 0.21% [kernel] [k] ip_output
> 0.21% [kernel] [k] vlan_dev_hard_start_xmit
> 0.20% [kernel] [k] ixgbe_alloc_rx_buffers
> 0.18% [kernel] [k] nf_hook_slow
> 0.18% [kernel] [k] apic_timer_interrupt
> 0.18% [kernel] [k] virt_to_head_page
> 0.18% [kernel] [k] build_skb
> 0.16% [kernel] [k] swiotlb_map_page
> 0.16% [kernel] [k] ip_finish_output
> 0.16% [kernel] [k] udp4_gro_receive
>
>
> RESULTS:
>
> CSV format - delimeter ";"
>
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;1;64;1470912;88247040;1470720;85305530
> 1;1;64;1470912;88285440;1470977;85335110
> 2;1;64;1470464;88247040;1470402;85290508
> 3;1;64;1471424;88262400;1471230;85353728
> 4;1;64;1468736;88166400;1468672;85201652
> 5;1;64;1470016;88181760;1469949;85234944
> 6;1;64;1470720;88247040;1470466;85290624
> 7;1;64;1471232;88277760;1471167;85346246
> 8;1;64;1469184;88170240;1469249;85216326
> 9;1;64;1470592;88227840;1470847;85294394
Single core 1.47Mpps seems a little low, I would expect 2Mpps.
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;2;64;2413120;144802560;2413245;139975924
> 1;2;64;2415296;144913920;2415356;140098188
> 2;2;64;2416768;144898560;2416573;140105670
> 3;2;64;2418176;145056000;2418110;140261806
> 4;2;64;2416512;144990720;2416509;140172950
> 5;2;64;2415168;144860160;2414466;140064780
> 6;2;64;2416960;144983040;2416833;140190930
> 7;2;64;2413632;144768000;2413568;140001734
> 8;2;64;2415296;144898560;2414589;140087168
> 9;2;64;2416576;144963840;2416892;140190930
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;3;64;3419008;205155840;3418882;198239244
> 1;3;64;3428032;205585920;3427971;198744234
> 2;3;64;3425472;205536000;3425344;198677260
> 3;3;64;3425088;205470720;3425156;198603136
> 4;3;64;3427648;205693440;3426883;198773888
> 5;3;64;3426880;205670400;3427392;198796044
> 6;3;64;3429120;205678080;3430140;198848186
> 7;3;64;3422976;205355520;3423490;198458136
> 8;3;64;3423168;205336320;3423486;198495372
> 9;3;64;3424384;205493760;3425538;198617868
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;4;64;4406464;264364800;4405244;255560296
> 1;4;64;4404672;264349440;4405122;255541504
> 2;4;64;4402368;264049920;4403326;255188864
> 3;4;64;4401344;264076800;4400702;255207134
> 4;4;64;4385536;263074560;4386620;254312716
> 5;4;64;4386560;263189760;4385404;254379532
> 6;4;64;4398784;263857920;4399031;255025288
> 7;4;64;4407232;264445440;4407998;255637900
> 8;4;64;4413184;264698880;4413758;255875816
> 9;4;64;4411328;264526080;4411906;255712372
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;5;64;5094464;305871360;5094464;295657262
> 1;5;64;5090816;305514240;5091201;295274810
> 2;5;64;5088384;305387520;5089792;295175108
> 3;5;64;5079296;304869120;5079484;294680368
> 4;5;64;5092992;305544960;5094207;295349166
> 5;5;64;5092416;305502720;5093372;295334260
> 6;5;64;5080896;304896000;5081090;294677004
> 7;5;64;5085376;305114880;5086401;294933058
> 8;5;64;5092544;305575680;5092036;295356938
> 9;5;64;5093056;305652480;5093832;295449506
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;6;64;5705088;342351360;5705784;330965110
> 1;6;64;5710272;342743040;5707591;331373952
> 2;6;64;5703424;342182400;5701826;330776552
> 3;6;64;5708736;342604800;5707963;331147462
> 4;6;64;5710144;342654720;5712067;331202910
> 5;6;64;5712064;342777600;5711361;331292288
> 6;6;64;5710144;342585600;5708607;331144272
> 7;6;64;5699840;342021120;5697853;330609222
> 8;6;64;5701184;342124800;5702909;330653592
> 9;6;64;5711360;342735360;5713283;331247686
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;7;64;6244416;374603520;6243591;362180072
> 1;7;64;6230912;374016000;6231490;361534126
> 2;7;64;6244800;374776320;6244866;362224326
> 3;7;64;6238720;374376960;6238261;361838510
> 4;7;64;6218816;373079040;6220413;360683962
> 5;7;64;6224320;373566720;6225086;361017404
> 6;7;64;6224000;373570560;6221370;360936088
> 7;7;64;6210048;372741120;6210627;360212654
> 8;7;64;6231616;374035200;6231537;361445502
> 9;7;64;6227840;373724160;6228802;361162752
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;8;64;6251840;375144960;6251849;362609678
> 1;8;64;6250816;375014400;6250881;362547038
> 2;8;64;6257728;375432960;6257160;362911104
> 3;8;64;6255552;375325440;6255622;362822074
> 4;8;64;6243776;374576640;6243270;362120622
> 5;8;64;6237184;374296320;6237690;361790080
> 6;8;64;6240960;374415360;6240714;361927366
> 7;8;64;6222784;373317120;6223746;360854424
> 8;8;64;6225920;373593600;6227014;361154980
> 9;8;64;6238528;374304000;6237701;361845238
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;14;64;6486144;389184000;6486135;376236488
> 1;14;64;6454912;387390720;6454222;374466734
> 2;14;64;6441152;386480640;6440431;373572780
> 3;14;64;6450240;386972160;6450870;374070014
> 4;14;64;6465600;387997440;6467221;375089654
> 5;14;64;6448384;386860800;6448000;373980230
> 6;14;64;6452352;387095040;6452148;374168904
> 7;14;64;6441984;386507520;6443203;373665058
> 8;14;64;6456704;387340800;6455744;374429092
> 9;14;64;6464640;387901440;6465218;374949004
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;16;64;6939008;416325120;6938696;402411192
> 1;16;64;6941952;416444160;6941745;402558918
> 2;16;64;6960576;417584640;6960707;403698718
> 3;16;64;6940736;416486400;6941820;402503876
> 4;16;64;6927680;415741440;6927420;401853870
> 5;16;64;6929792;415687680;6929917;401839196
> 6;16;64;6950400;416989440;6950661;403026166
> 7;16;64;6953664;417216000;6953454;403260544
> 8;16;64;6948480;416851200;6948800;403023266
> 9;16;64;6924160;415422720;6924092;401542468
I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
issue/bug. You could be stalling on DMA TX completion being too slow,
but you already increased the interval and increased the TX ring queue
size. You could play with those setting and see if it changes this?
Could you try my napi_monitor tool in:
https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
Also provide the output from:
mpstat -P ALL -u -I SCPU -I SUM 2
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-12 12:23 ` Jesper Dangaard Brouer
@ 2017-08-12 17:27 ` Paweł Staszewski
2017-08-13 16:58 ` Paweł Staszewski
2017-08-14 0:07 ` Alexander Duyck
0 siblings, 2 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-12 17:27 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Linux Kernel Network Developers
[-- Attachment #1: Type: text/plain, Size: 43660 bytes --]
Hi and thanks for reply
W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
> On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> Hi
>>
>> I made some tests for performance comparison.
> Thanks for doing this. Feel free to Cc me, if you do more of these
> tests (so I don't miss them on the mailing list).
>
> I don't understand stand if you are reporting a potential problem?
>
> It would be good if you can provide a short summary section (of the
> issue) in the _start_ of the email, and then provide all this nice data
> afterwards, to back your case.
>
> My understanding is, you report:
>
> 1. VLANs on ixgbe show a 30-40% slowdown
> 2. System stopped scaling after 7+ CPUs
This is not only problem/bug report - but some kind of comparision plus
some toughts about possible problems :)
And can help somebody when searching the net for possible expectations :)
Also - dono better list where are the smartest people that know what is
going in kernel with networking :)
Next time i will place summary on top - sorry :)
>
>> Tested HW (FORWARDING HOST):
>>
>> Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
> Interesting, I've not heard about a Intel CPU called "Gold" before now,
> but it does exist:
> https://ark.intel.com/products/123541/Intel-Xeon-Gold-6132-Processor-19_25M-Cache-2_60-GHz
>
>
>> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
> This is one of my all time favorite NICs!
Yes this is a good NIC - will have connectx-4 2x100G by monday so will
also do some tests
>
>> Test diagram:
>>
>>
>> TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
>> (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
>>
>> Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
>> 172.16.0.1 to 172.16.0.255
>>
>> TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
> What kind of traffic flow? E.g. distribution, many/few source IPs...
Traffic generator is pktgen so udp flows - better paste parameters from
pktgen:
UDP_MIN=9
UDP_MAX=19
pg_set $dev "dst_min 172.16.0.1"
pg_set $dev "dst_max 172.16.0.100"
# Setup random UDP port src range
#pg_set $dev "flag UDPSRC_RND"
pg_set $dev "flag UDPSRC_RND"
pg_set $dev "udp_src_min $UDP_MIN"
pg_set $dev "udp_src_max $UDP_MAX"
>
>
>> Settings used for FORWARDING HOST (changed param. was only number of RSS
>> combined queues + set affinity assignment for them to fit with first
>> numa node where 2x10G port card is installed)
>>
>> ixgbe driver used from kernel (in-kernel build - not a module)
>>
> Nice with a script showing you setup, thanks. I would be good if it had
> comments, telling why you think this is a needed setup adjustment.
>
>> #!/bin/sh
>> ifc='enp216s0f0 enp216s0f1'
>> for i in $ifc
>> do
>> ip link set up dev $i
>> ethtool -A $i autoneg off rx off tx off
> Good:
> Turning off Ethernet flow control, to avoid receiver being the
> bottleneck via pause-frames.
Yes - enabled flow controll is really bad :)
>> ethtool -G $i rx 4096 tx 1024
> You adjust the RX and TX ring queue sizes, this have effects that you
> don't realize. Especially for the ixgbe driver, which have a page
> recycle trick tied to the RX ring queue size.
rx ring 4096 and tx ring 1024
- this is because have best performance then with average packet size
from 64 to 1500 bytes
Can be a little better performance for smaller frames like 64 - with rx
ring set to 1024
below 1 core/1 RSS queue with rx ring set to 1024
0;1;64;1530112;91772160;1529919;88724208
1;1;64;1531584;91872000;1531520;88813196
2;1;64;1531392;91895040;1531262;88831930
3;1;64;1530880;91875840;1531201;88783558
4;1;64;1530688;91829760;1530688;88768826
5;1;64;1530432;91810560;1530624;88764940
6;1;64;1530880;91868160;1530878;88787328
7;1;64;1530496;91845120;1530560;88765114
8;1;64;1530496;91837440;1530687;88772538
9;1;64;1530176;91795200;1530496;88735360
so from 1.47Mpps to 1.53Mpps
But with bigger packets > 200 performance is better when rx is set to 4096
>
>> ip link set $i txqueuelen 1000
> Setting tx queue len to the default 1000 seems redundant.
Yes cause i'm changing this parameter also to see if any impact on
performance we have
>
>> ethtool -C $i rx-usecs 10
> Adjusting this also have effects you might not realize. This actually
> also affect the page recycle scheme of ixgbe. And can sometimes be
> used to solve stalling on DMA TX completions, which could be you issue
> here.
same here - rx-usecs - setting to 10 was kind of compromise to have good
performance with big ans small packet sizes
Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256
(1Core/1RSS queue):
0;1;64;1506304;90424320;1506626;87402868
1;1;64;1505536;90343680;1504830;87321088
2;1;64;1506880;90416640;1507522;87388120
3;1;64;1511040;90700800;1511682;87684864
4;1;64;1511040;90681600;1511102;87662476
5;1;64;1511488;90712320;1511614;87673728
6;1;64;1511296;90700800;1511038;87669900
7;1;64;1513344;90773760;1513280;87751680
8;1;64;1513536;90850560;1513470;87807360
9;1;64;1512128;90696960;1512000;87696000
And rx-usecs set to 1
0;1;64;1533632;92037120;1533504;88954368
1;1;64;1533632;92006400;1533570;88943348
2;1;64;1533504;91994880;1533504;88931980
3;1;64;1532864;91979520;1532674;88902516
4;1;64;1533952;92044800;1534080;88961792
5;1;64;1533888;92048640;1534270;88969100
6;1;64;1533952;92037120;1534082;88969216
7;1;64;1533952;92021760;1534208;88969332
8;1;64;1533056;91983360;1532930;88883724
9;1;64;1533760;92021760;1533886;88946828
rx-useck set to 2
0;1;64;1522432;91334400;1522304;88301056
1;1;64;1521920;91330560;1522496;88286208
2;1;64;1522496;91322880;1522432;88304768
3;1;64;1523456;91422720;1523649;88382762
4;1;64;1527680;91676160;1527424;88601728
5;1;64;1527104;91626240;1526912;88572032
6;1;64;1527424;91641600;1527424;88590592
7;1;64;1526336;91572480;1526912;88523776
8;1;64;1527040;91637760;1526912;88579456
9;1;64;1527040;91595520;1526784;88553472
rx-usecs set to 3
0;1;64;1526272;91549440;1526592;88527488
1;1;64;1526528;91560960;1526272;88516352
2;1;64;1525952;91580160;1525888;88527488
3;1;64;1525504;91511040;1524864;88456960
4;1;64;1526272;91568640;1526208;88494080
5;1;64;1525568;91545600;1525312;88494080
6;1;64;1526144;91584000;1526080;88512640
7;1;64;1525376;91530240;1525376;88482944
8;1;64;1526784;91607040;1526592;88549760
9;1;64;1526208;91560960;1526528;88512640
>
>> ethtool -L $i combined 16
>> ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
>> tx-nocache-copy on ntuple on
> Here are many setting above.
Yes mostly NIC defaults besides the ntuple that is on (for testing some
nfc drop filters - and trying to test also tc-offload )
> GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
> this result in approx 10% slowdown.
Ok lets give a try :)
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on
ntuple on
rx-usecs 10
1 CPU / 1 RSS QUEUE
0;1;64;1609344;96537600;1609279;93327104
1;1;64;1608320;96514560;1608256;93293812
2;1;64;1608000;96487680;1608125;93267770
3;1;64;1608320;96522240;1608576;93297524
4;1;64;1605888;96387840;1606211;93148986
5;1;64;1601472;96072960;1601600;92870644
6;1;64;1602624;96180480;1602243;92959674
7;1;64;1601728;96107520;1602113;92907764
8;1;64;1602176;96122880;1602176;92933806
9;1;64;1603904;96253440;1603777;93045208
A little better performance 1.6Mpps
But wondering if disabling tso will have no performance impact for tcp
traffic ...
Will try to get some pktgen like pktgen-dpdk that can generate also tcp
traffic - to compare this.
>
> AFAIK "tx-nocache-copy on" was also determined to be a bad option.
I set this to on cause i have better performance (a little 10kpps for
this test)
below same test as above with tx-nocache-copy off
0;1;64;1591552;95496960;1591230;92313654
1;1;64;1596224;95738880;1595842;92555066
2;1;64;1595456;95700480;1595201;92521774
3;1;64;1595456;95723520;1595072;92528966
4;1;64;1595136;95692800;1595457;92503040
5;1;64;1594624;95631360;1594496;92473402
6;1;64;1596224;95761920;1595778;92551180
7;1;64;1595200;95700480;1595331;92521542
8;1;64;1595584;95692800;1595457;92521426
9;1;64;1594624;95662080;1594048;92469574
>
> The "ntuple on" AFAIK disables the flow-director in the NIC. I though
> this would actually help VLAN traffic, but I guess not.
yes I enabled this cause was thinking that can help with traffic on vlans
below same test with ntuple off
so all settings for ixgbe:
gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off
ntuple off
rx-usecs 10
rx-flow-hash udp4 sdfn
0;1;64;1611840;96691200;1611905;93460794
1;1;64;1610688;96645120;1610818;93427328
2;1;64;1610752;96668160;1610497;93442176
3;1;64;1610624;96664320;1610817;93427212
4;1;64;1610752;96652800;1610623;93412480
5;1;64;1610048;96614400;1610112;93404940
6;1;64;1611264;96641280;1611390;93427212
7;1;64;1611008;96691200;1610942;93468160
8;1;64;1610048;96652800;1609984;93408652
9;1;64;1611136;96641280;1610690;93434636
Performance is a little better
and now with tx-nocache-copy on
0;1;64;1597248;95834880;1597311;92644096
1;1;64;1597888;95865600;1597824;92677446
2;1;64;1597952;95834880;1597822;92644038
3;1;64;1597568;95877120;1597375;92685044
4;1;64;1597184;95827200;1597314;92629190
5;1;64;1597696;95842560;1597565;92625652
6;1;64;1597312;95834880;1597376;92644038
7;1;64;1597568;95873280;1597634;92647924
8;1;64;1598400;95919360;1598849;92699602
9;1;64;1597824;95873280;1598208;92684928
That is weird - so enabling tx-nocache-copy with disabled ntuple have
bad performance impact - but with enabled ntuple there is no performance
impact
>
>
>> ethtool -N $i rx-flow-hash udp4 sdfn
> Why do you change the NICs flow-hash?
whan used 16 cores / 16 rss queues - there was better load distribution
over all cores when sdfn rx-flow-hash enabled
>
>> done
>>
>> ip link set up dev enp216s0f0
>> ip link set up dev enp216s0f1
>>
>> ip a a 10.0.0.1/30 dev enp216s0f0
>>
>> ip link add link enp216s0f1 name vlan1000 type vlan id 1000
>> ip link set up dev vlan1000
>> ip a a 10.0.0.5/30 dev vlan1000
>>
>>
>> ip route add 172.16.0.0/12 via 10.0.0.6
>>
>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
>> #cat /sys/devices/system/node/node1/cpulist
>> #14-27,42-55
>> #cat /sys/devices/system/node/node0/cpulist
>> #0-13,28-41
> Is this a NUMA system?
This is 2x CPU 6132 - so have two separate pcie access to the nic - need
to check what cpu is assigned to pcie where network card is connected to
have network card on local cpu where all irq's are binded
>
>
>> #################################################
>>
>>
>> Looks like forwarding performance when using vlans on ixgbe is less that
>> without vlans for about 30-40% (wondering if this is some vlan
>> offloading problem and ixgbe)
> I would see this as a problem/bug that enabling VLANs cost this much.
Yes - was thinking that with tx/rx vlan offloading there will be not
much performance impact when vlans used.
>
>> settings below:
>>
>> ethtool -k enp216s0f0
>> Features for enp216s0f0:
>> Cannot get device udp-fragmentation-offload settings: Operation not
>> supported
>> rx-checksumming: on
>> tx-checksumming: on
>> tx-checksum-ipv4: off [fixed]
>> tx-checksum-ip-generic: on
>> tx-checksum-ipv6: off [fixed]
>> tx-checksum-fcoe-crc: off [fixed]
>> tx-checksum-sctp: on
>> scatter-gather: on
>> tx-scatter-gather: on
>> tx-scatter-gather-fraglist: off [fixed]
>> tcp-segmentation-offload: on
>> tx-tcp-segmentation: on
>> tx-tcp-ecn-segmentation: off [fixed]
>> tx-tcp-mangleid-segmentation: on
>> tx-tcp6-segmentation: on
>> udp-fragmentation-offload: off
>> generic-segmentation-offload: off
>> generic-receive-offload: on
>> large-receive-offload: off
>> rx-vlan-offload: on
>> tx-vlan-offload: on
>> ntuple-filters: on
>> receive-hashing: on
>> highdma: on [fixed]
>> rx-vlan-filter: on
>> vlan-challenged: off [fixed]
>> tx-lockless: off [fixed]
>> netns-local: off [fixed]
>> tx-gso-robust: off [fixed]
>> tx-fcoe-segmentation: off [fixed]
>> tx-gre-segmentation: on
>> tx-gre-csum-segmentation: on
>> tx-ipxip4-segmentation: on
>> tx-ipxip6-segmentation: on
>> tx-udp_tnl-segmentation: on
>> tx-udp_tnl-csum-segmentation: on
>> tx-gso-partial: on
>> tx-sctp-segmentation: off [fixed]
>> tx-esp-segmentation: off [fixed]
>> fcoe-mtu: off [fixed]
>> tx-nocache-copy: on
>> loopback: off [fixed]
>> rx-fcs: off [fixed]
>> rx-all: off
>> tx-vlan-stag-hw-insert: off [fixed]
>> rx-vlan-stag-hw-parse: off [fixed]
>> rx-vlan-stag-filter: off [fixed]
>> l2-fwd-offload: off
>> hw-tc-offload: off
>> esp-hw-offload: off [fixed]
>> esp-tx-csum-hw-offload: off [fixed]
>> rx-udp_tunnel-port-offload: on
>>
>>
>> Another thing is that forwarding performance does not scale with number
>> of cores when 7+ cores are reached
> I've seen problems with using Hyper-Threading CPUs. Could it be that
> above 7 CPUs you are starting to use sibling-cores ?
>
Turbostats can help here:
Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI
C1 C2 C1% C2% CPU%c1 CPU%c6 CoreTmp PkgTmp PkgWatt
RAMWatt PKG_% RAM_%
- - - 72 2.27 3188 2600 194844 0
64 69282 0.07 97.83 18.38 79.36 -4 54 123.49
16.08 0.00 0.00
0 0 0 8 0.74 1028 2600 1513 0
32 1462 1.50 97.99 10.92 88.34 47 51 58.34
5.34 0.00 0.00
0 0 28 7 0.67 1015 2600 1255 0
12 1249 0.96 98.61 10.99
0 1 1 7 0.68 1019 2600 1260 0
0 1260 0.00 99.54 8.44 90.88 49
0 1 29 9 0.71 1208 2600 1252 0
0 1253 0.00 99.48 8.41
0 2 2 7 0.67 1019 2600 1261 0
0 1260 0.00 99.54 8.44 90.89 48
0 2 30 7 0.67 1017 2600 1255 0
0 1255 0.00 99.55 8.44
0 3 3 7 0.68 1019 2600 1260 0
0 1259 0.00 99.53 8.46 90.86 -4
0 3 31 7 0.67 1017 2600 1256 0
0 1256 0.00 99.55 8.46
0 4 4 7 0.67 1027 2600 1260 0
0 1260 0.00 99.54 8.43 90.90 -4
0 4 32 7 0.66 1018 2600 1255 0
0 1255 0.00 99.55 8.44
0 5 5 7 0.68 1020 2600 1260 0
0 1257 0.00 99.54 8.44 90.89 50
0 5 33 7 0.68 1019 2600 1255 0
0 1255 0.00 99.55 8.43
0 6 6 7 0.70 1019 2600 1260 0
0 1259 0.00 99.53 8.43 90.87 -4
0 6 34 7 0.70 1019 2600 1255 0
0 1255 0.00 99.54 8.43
0 8 7 7 0.68 1019 2600 1262 0
0 1261 0.00 99.52 8.42 90.90 50
0 8 35 7 0.67 1019 2600 1255 0
0 1255 0.00 99.55 8.43
0 9 8 7 0.68 1019 2600 1260 0
0 1257 0.00 99.54 8.40 90.92 49
0 9 36 7 0.66 1017 2600 1255 0
0 1255 0.00 99.55 8.41
0 10 9 7 0.66 1018 2600 1257 0
0 1257 0.00 99.54 8.40 90.94 -4
0 10 37 7 0.66 1018 2600 1255 0
0 1255 0.00 99.55 8.41
0 11 10 7 0.66 1019 2600 1257 0
0 1259 0.00 99.54 8.56 90.77 -4
0 11 38 7 0.66 1018 2600 1255 0
3 1252 0.19 99.36 8.57
0 12 11 7 0.67 1019 2600 1260 0
0 1260 0.00 99.54 8.44 90.88 -4
0 12 39 7 0.67 1019 2600 1255 0
0 1256 0.00 99.55 8.44
0 13 12 7 0.68 1019 2600 1257 0
4 1254 0.32 99.22 8.67 90.65 -4
0 13 40 7 0.69 1019 2600 1256 0
4 1253 0.24 99.31 8.66
0 14 13 7 0.71 1020 2600 1260 0
0 1259 0.00 99.53 8.41 90.88 -4
0 14 41 7 0.72 1020 2600 1255 0
0 1255 0.00 99.54 8.40
1 0 14 3564 99.19 3594 2600 125472 0
0 0 0.00 0.00 0.81 0.00 54 54 65.15
10.74 0.00 0.00
1 0 42 3 0.07 3701 2600 1255 0
0 1255 0.00 99.95 99.93
1 1 15 11 0.32 3301 2600 1257 0
0 1257 0.00 99.81 26.37 73.31 42
1 1 43 10 0.31 3301 2600 1255 0
0 1255 0.00 99.82 26.38
1 2 16 10 0.31 3301 2600 1257 0
0 1257 0.00 99.81 26.37 73.32 39
1 2 44 10 0.32 3301 2600 1255 0
0 1255 0.00 99.82 26.36
1 3 17 10 0.32 3301 2600 1257 0
0 1257 0.00 99.81 26.40 73.28 39
1 3 45 11 0.32 3301 2600 1255 0
0 1255 0.00 99.81 26.40
1 4 18 10 0.32 3301 2600 1257 0
0 1257 0.00 99.82 26.40 73.28 40
1 4 46 11 0.32 3301 2600 1255 0
0 1255 0.00 99.82 26.40
1 5 19 11 0.33 3301 2600 1257 0
0 1257 0.00 99.81 26.40 73.27 39
1 5 47 11 0.33 3300 2600 1255 0
0 1255 0.00 99.82 26.40
1 6 20 12 0.35 3301 2600 1257 0
0 1257 0.00 99.81 26.38 73.27 42
1 6 48 12 0.36 3301 2600 1255 0
0 1255 0.00 99.81 26.37
1 8 21 11 0.33 3301 2600 1257 0
0 1257 0.00 99.82 26.37 73.29 42
1 8 49 11 0.33 3301 2600 1255 0
0 1255 0.00 99.82 26.38
1 9 22 10 0.32 3300 2600 1257 0
0 1257 0.00 99.82 26.35 73.34 41
1 9 50 10 0.30 3301 2600 1255 0
0 1255 0.00 99.82 26.36
1 10 23 10 0.31 3301 2600 1257 0
0 1257 0.00 99.82 26.37 73.33 41
1 10 51 10 0.31 3301 2600 1255 0
0 1255 0.00 99.82 26.36
1 11 24 10 0.32 3301 2600 1257 0
0 1257 0.00 99.81 26.62 73.06 41
1 11 52 10 0.32 3301 2600 1255 0
4 1251 0.32 99.50 26.62
1 12 25 11 0.33 3301 2600 1257 0
0 1257 0.00 99.81 26.39 73.28 41
1 12 53 11 0.33 3301 2600 1258 0
0 1254 0.00 99.82 26.38
1 13 26 12 0.36 3317 2600 1259 0
0 1258 0.00 99.79 26.41 73.23 39
1 13 54 11 0.34 3301 2600 1255 0
0 1254 0.00 99.82 26.42
1 14 27 12 0.36 3301 2600 1257 0
5 1251 0.24 99.58 26.54 73.10 41
1 14 55 12 0.36 3300 2600 1255 0
0 1254 0.00 99.82 26.54
So it looks like in all tests i'm using core+sibling
But side effect of this is that :
33 * 100.0 = 3300.0 MHz max turbo 28 active cores
33 * 100.0 = 3300.0 MHz max turbo 24 active cores
33 * 100.0 = 3300.0 MHz max turbo 20 active cores
33 * 100.0 = 3300.0 MHz max turbo 14 active cores
34 * 100.0 = 3400.0 MHz max turbo 12 active cores
34 * 100.0 = 3400.0 MHz max turbo 8 active cores
35 * 100.0 = 3500.0 MHz max turbo 4 active cores
37 * 100.0 = 3700.0 MHz max turbo 2 active cores
So more cores = less MHz per core/sibling
>
>> perf top:
>>
>> PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> 16.32% [kernel] [k] skb_dst_force
>> 16.30% [kernel] [k] dst_release
>> 15.11% [kernel] [k] rt_cache_valid
>> 12.62% [kernel] [k] ipv4_mtu
> It seems a little strange that these 4 functions are on the top
Yes dono why there is ipv4_mtu called and taking soo much cycles
>
>> 5.60% [kernel] [k] do_raw_spin_lock
> Why is calling/taking this lock? (Use perf call-graph recording).
can be hard to paste it here:)
attached file
>
>> 3.03% [kernel] [k] fib_table_lookup
>> 2.70% [kernel] [k] ip_finish_output2
>> 2.10% [kernel] [k] dev_gro_receive
>> 1.89% [kernel] [k] eth_type_trans
>> 1.81% [kernel] [k] ixgbe_poll
>> 1.15% [kernel] [k] ixgbe_xmit_frame_ring
>> 1.06% [kernel] [k] __build_skb
>> 1.04% [kernel] [k] __dev_queue_xmit
>> 0.97% [kernel] [k] ip_rcv
>> 0.78% [kernel] [k] netif_skb_features
>> 0.74% [kernel] [k] ipt_do_table
> Unloading netfilter modules, will give more performance, but it
> semifake to do so.
Compiled in kernel - only in filter mode - with ipv4+ipv6 - no other
modules conntrack or other .
>> 0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
>> 0.64% [kernel] [k] ip_forward
>> 0.59% [kernel] [k] __netif_receive_skb_core
>> 0.55% [kernel] [k] dev_hard_start_xmit
>> 0.53% [kernel] [k] ip_route_input_rcu
>> 0.53% [kernel] [k] ip_rcv_finish
>> 0.51% [kernel] [k] page_frag_free
>> 0.50% [kernel] [k] kmem_cache_alloc
>> 0.50% [kernel] [k] udp_v4_early_demux
>> 0.44% [kernel] [k] skb_release_data
>> 0.42% [kernel] [k] inet_gro_receive
>> 0.40% [kernel] [k] sch_direct_xmit
>> 0.39% [kernel] [k] __local_bh_enable_ip
>> 0.33% [kernel] [k] netdev_pick_tx
>> 0.33% [kernel] [k] validate_xmit_skb
>> 0.28% [kernel] [k] fib_validate_source
>> 0.27% [kernel] [k] deliver_ptype_list_skb
>> 0.25% [kernel] [k] eth_header
>> 0.23% [kernel] [k] get_dma_ops
>> 0.22% [kernel] [k] skb_network_protocol
>> 0.21% [kernel] [k] ip_output
>> 0.21% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.20% [kernel] [k] ixgbe_alloc_rx_buffers
>> 0.18% [kernel] [k] nf_hook_slow
>> 0.18% [kernel] [k] apic_timer_interrupt
>> 0.18% [kernel] [k] virt_to_head_page
>> 0.18% [kernel] [k] build_skb
>> 0.16% [kernel] [k] swiotlb_map_page
>> 0.16% [kernel] [k] ip_finish_output
>> 0.16% [kernel] [k] udp4_gro_receive
>>
>>
>> RESULTS:
>>
>> CSV format - delimeter ";"
>>
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;1;64;1470912;88247040;1470720;85305530
>> 1;1;64;1470912;88285440;1470977;85335110
>> 2;1;64;1470464;88247040;1470402;85290508
>> 3;1;64;1471424;88262400;1471230;85353728
>> 4;1;64;1468736;88166400;1468672;85201652
>> 5;1;64;1470016;88181760;1469949;85234944
>> 6;1;64;1470720;88247040;1470466;85290624
>> 7;1;64;1471232;88277760;1471167;85346246
>> 8;1;64;1469184;88170240;1469249;85216326
>> 9;1;64;1470592;88227840;1470847;85294394
> Single core 1.47Mpps seems a little low, I would expect 2Mpps.
>
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;2;64;2413120;144802560;2413245;139975924
>> 1;2;64;2415296;144913920;2415356;140098188
>> 2;2;64;2416768;144898560;2416573;140105670
>> 3;2;64;2418176;145056000;2418110;140261806
>> 4;2;64;2416512;144990720;2416509;140172950
>> 5;2;64;2415168;144860160;2414466;140064780
>> 6;2;64;2416960;144983040;2416833;140190930
>> 7;2;64;2413632;144768000;2413568;140001734
>> 8;2;64;2415296;144898560;2414589;140087168
>> 9;2;64;2416576;144963840;2416892;140190930
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;3;64;3419008;205155840;3418882;198239244
>> 1;3;64;3428032;205585920;3427971;198744234
>> 2;3;64;3425472;205536000;3425344;198677260
>> 3;3;64;3425088;205470720;3425156;198603136
>> 4;3;64;3427648;205693440;3426883;198773888
>> 5;3;64;3426880;205670400;3427392;198796044
>> 6;3;64;3429120;205678080;3430140;198848186
>> 7;3;64;3422976;205355520;3423490;198458136
>> 8;3;64;3423168;205336320;3423486;198495372
>> 9;3;64;3424384;205493760;3425538;198617868
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;4;64;4406464;264364800;4405244;255560296
>> 1;4;64;4404672;264349440;4405122;255541504
>> 2;4;64;4402368;264049920;4403326;255188864
>> 3;4;64;4401344;264076800;4400702;255207134
>> 4;4;64;4385536;263074560;4386620;254312716
>> 5;4;64;4386560;263189760;4385404;254379532
>> 6;4;64;4398784;263857920;4399031;255025288
>> 7;4;64;4407232;264445440;4407998;255637900
>> 8;4;64;4413184;264698880;4413758;255875816
>> 9;4;64;4411328;264526080;4411906;255712372
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;5;64;5094464;305871360;5094464;295657262
>> 1;5;64;5090816;305514240;5091201;295274810
>> 2;5;64;5088384;305387520;5089792;295175108
>> 3;5;64;5079296;304869120;5079484;294680368
>> 4;5;64;5092992;305544960;5094207;295349166
>> 5;5;64;5092416;305502720;5093372;295334260
>> 6;5;64;5080896;304896000;5081090;294677004
>> 7;5;64;5085376;305114880;5086401;294933058
>> 8;5;64;5092544;305575680;5092036;295356938
>> 9;5;64;5093056;305652480;5093832;295449506
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;6;64;5705088;342351360;5705784;330965110
>> 1;6;64;5710272;342743040;5707591;331373952
>> 2;6;64;5703424;342182400;5701826;330776552
>> 3;6;64;5708736;342604800;5707963;331147462
>> 4;6;64;5710144;342654720;5712067;331202910
>> 5;6;64;5712064;342777600;5711361;331292288
>> 6;6;64;5710144;342585600;5708607;331144272
>> 7;6;64;5699840;342021120;5697853;330609222
>> 8;6;64;5701184;342124800;5702909;330653592
>> 9;6;64;5711360;342735360;5713283;331247686
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;7;64;6244416;374603520;6243591;362180072
>> 1;7;64;6230912;374016000;6231490;361534126
>> 2;7;64;6244800;374776320;6244866;362224326
>> 3;7;64;6238720;374376960;6238261;361838510
>> 4;7;64;6218816;373079040;6220413;360683962
>> 5;7;64;6224320;373566720;6225086;361017404
>> 6;7;64;6224000;373570560;6221370;360936088
>> 7;7;64;6210048;372741120;6210627;360212654
>> 8;7;64;6231616;374035200;6231537;361445502
>> 9;7;64;6227840;373724160;6228802;361162752
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;8;64;6251840;375144960;6251849;362609678
>> 1;8;64;6250816;375014400;6250881;362547038
>> 2;8;64;6257728;375432960;6257160;362911104
>> 3;8;64;6255552;375325440;6255622;362822074
>> 4;8;64;6243776;374576640;6243270;362120622
>> 5;8;64;6237184;374296320;6237690;361790080
>> 6;8;64;6240960;374415360;6240714;361927366
>> 7;8;64;6222784;373317120;6223746;360854424
>> 8;8;64;6225920;373593600;6227014;361154980
>> 9;8;64;6238528;374304000;6237701;361845238
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;14;64;6486144;389184000;6486135;376236488
>> 1;14;64;6454912;387390720;6454222;374466734
>> 2;14;64;6441152;386480640;6440431;373572780
>> 3;14;64;6450240;386972160;6450870;374070014
>> 4;14;64;6465600;387997440;6467221;375089654
>> 5;14;64;6448384;386860800;6448000;373980230
>> 6;14;64;6452352;387095040;6452148;374168904
>> 7;14;64;6441984;386507520;6443203;373665058
>> 8;14;64;6456704;387340800;6455744;374429092
>> 9;14;64;6464640;387901440;6465218;374949004
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;16;64;6939008;416325120;6938696;402411192
>> 1;16;64;6941952;416444160;6941745;402558918
>> 2;16;64;6960576;417584640;6960707;403698718
>> 3;16;64;6940736;416486400;6941820;402503876
>> 4;16;64;6927680;415741440;6927420;401853870
>> 5;16;64;6929792;415687680;6929917;401839196
>> 6;16;64;6950400;416989440;6950661;403026166
>> 7;16;64;6953664;417216000;6953454;403260544
>> 8;16;64;6948480;416851200;6948800;403023266
>> 9;16;64;6924160;415422720;6924092;401542468
> I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
> issue/bug. You could be stalling on DMA TX completion being too slow,
> but you already increased the interval and increased the TX ring queue
> size. You could play with those setting and see if it changes this?
>
> Could you try my napi_monitor tool in:
> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
>
> Also provide the output from:
> mpstat -P ALL -u -I SCPU -I SUM 2
with 16 cores / 16 RSS queues
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.00 0.00 0.01 0.00 0.00 28.57 0.00
0.00 0.00 71.42
Average: 0 0.00 0.00 0.04 0.00 0.00 0.08 0.00
0.00 0.00 99.88
Average: 1 0.00 0.00 0.12 0.00 0.00 0.00 0.00
0.00 0.00 99.88
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 11 0.08 0.00 0.04 0.00 0.00 0.00 0.00
0.00 0.00 99.88
Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 14 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 15 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 16 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 17 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 18 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 19 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 20 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 21 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 22 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 23 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 24 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 25 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 26 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 27 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 28 0.00 0.00 0.04 0.00 0.00 0.00 0.00
0.00 0.00 99.96
Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 39 0.04 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 99.96
Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 42 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 43 0.00 0.00 0.00 0.00 0.00 100.00 0.00
0.00 0.00 0.00
Average: 44 0.00 0.00 0.04 0.17 0.00 0.00 0.00
0.00 0.00 99.79
Average: 45 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 46 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 47 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 48 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 49 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 51 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 52 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 54 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: CPU intr/s
Average: all 123596.08
Average: 0 646.38
Average: 1 500.54
Average: 2 511.67
Average: 3 534.25
Average: 4 542.21
Average: 5 531.54
Average: 6 554.58
Average: 7 535.88
Average: 8 544.58
Average: 9 536.42
Average: 10 575.46
Average: 11 601.12
Average: 12 502.08
Average: 13 575.46
Average: 14 5917.92
Average: 15 5949.58
Average: 16 7021.29
Average: 17 7299.71
Average: 18 7391.67
Average: 19 7354.25
Average: 20 7543.42
Average: 21 7354.25
Average: 22 7322.33
Average: 23 7368.71
Average: 24 7429.00
Average: 25 7406.46
Average: 26 7400.67
Average: 27 7447.21
Average: 28 517.00
Average: 29 549.54
Average: 30 529.33
Average: 31 533.83
Average: 32 541.25
Average: 33 541.17
Average: 34 532.50
Average: 35 545.17
Average: 36 528.96
Average: 37 509.92
Average: 38 520.12
Average: 39 523.29
Average: 40 530.75
Average: 41 542.33
Average: 42 5921.71
Average: 43 5949.42
Average: 44 503.04
Average: 45 542.75
Average: 46 582.50
Average: 47 581.71
Average: 48 495.29
Average: 49 524.38
Average: 50 527.92
Average: 51 528.12
Average: 52 456.38
Average: 53 477.00
Average: 54 440.92
Average: 55 568.83
Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
Average: 0 0.00 250.00 0.17 87.00 0.00
0.00 45.46 250.00 0.00 13.75
Average: 1 0.00 233.42 0.00 0.00 0.00
0.00 0.00 249.92 0.00 17.21
Average: 2 0.00 249.04 0.00 0.00 0.00
0.00 0.00 249.96 0.00 12.67
Average: 3 0.00 249.92 0.00 0.00 0.00
0.00 0.00 249.92 0.00 34.42
Average: 4 0.00 248.67 0.17 0.00 0.00
0.00 0.00 249.96 0.00 43.42
Average: 5 0.00 249.46 0.00 0.00 0.00
0.00 0.00 249.92 0.00 32.17
Average: 6 0.00 249.79 0.00 0.00 0.00
0.00 0.00 249.87 0.00 54.92
Average: 7 0.00 240.12 0.00 0.00 0.00
0.00 0.00 249.96 0.00 45.79
Average: 8 0.00 247.42 0.00 0.00 0.00
0.00 0.00 249.92 0.00 47.25
Average: 9 0.00 249.29 0.00 0.00 0.00
0.00 0.00 249.96 0.00 37.17
Average: 10 0.00 248.75 0.00 0.00 0.00
0.00 0.00 249.92 0.00 76.79
Average: 11 0.00 249.29 0.00 0.00 0.00
0.00 42.79 249.83 0.00 59.21
Average: 12 0.00 249.83 0.00 0.00 0.00
0.00 0.00 249.96 0.00 2.29
Average: 13 0.00 249.92 0.00 0.00 0.00
0.00 0.00 249.92 0.00 75.62
Average: 14 0.00 148.21 0.17 5758.04 0.00
0.00 0.00 8.42 0.00 3.08
Average: 15 0.00 148.42 0.46 5789.25 0.00
0.00 0.00 8.33 0.00 3.12
Average: 16 0.00 142.62 0.79 6866.46 0.00
0.00 0.00 8.29 0.00 3.12
Average: 17 0.00 143.17 0.42 7145.00 0.00
0.00 0.00 8.08 0.00 3.04
Average: 18 0.00 153.62 0.42 7226.42 0.00
0.00 0.00 8.04 0.00 3.17
Average: 19 0.00 150.46 0.46 7192.21 0.00
0.00 0.00 8.04 0.00 3.08
Average: 20 0.00 145.21 0.17 7386.50 0.00
0.00 0.00 8.29 0.00 3.25
Average: 21 0.00 150.96 0.46 7191.37 0.00
0.00 0.00 8.25 0.00 3.21
Average: 22 0.00 146.67 0.54 7163.96 0.00
0.00 0.00 8.04 0.00 3.12
Average: 23 0.00 151.38 0.42 7205.75 0.00
0.00 0.00 8.00 0.00 3.17
Average: 24 0.00 153.33 0.17 7264.12 0.00
0.00 0.00 8.08 0.00 3.29
Average: 25 0.00 153.21 0.17 7241.83 0.00
0.00 0.00 7.96 0.00 3.29
Average: 26 0.00 153.96 0.17 7234.88 0.00
0.00 0.00 8.38 0.00 3.29
Average: 27 0.00 151.71 0.79 7283.25 0.00
0.00 0.00 8.04 0.00 3.42
Average: 28 0.00 245.71 0.00 0.00 0.00
0.00 0.00 249.50 0.00 21.79
Average: 29 0.00 233.21 0.00 0.00 0.00
0.00 0.00 249.87 0.00 66.46
Average: 30 0.00 248.92 0.00 0.00 0.00
0.00 0.00 250.00 0.00 30.42
Average: 31 0.00 249.92 0.00 0.00 0.00
0.00 0.00 249.96 0.00 33.96
Average: 32 0.00 248.67 0.00 0.00 0.00
0.00 0.00 249.96 0.00 42.62
Average: 33 0.00 249.46 0.00 0.00 0.00
0.00 0.00 249.92 0.00 41.79
Average: 34 0.00 249.79 0.00 0.00 0.00
0.00 0.00 249.87 0.00 32.83
Average: 35 0.00 240.12 0.00 0.00 0.00
0.00 0.00 249.96 0.00 55.08
Average: 36 0.00 247.42 0.00 0.00 0.00
0.00 0.00 249.96 0.00 31.58
Average: 37 0.00 249.29 0.00 0.00 0.00
0.00 0.00 249.92 0.00 10.71
Average: 38 0.00 248.75 0.00 0.00 0.00
0.00 0.00 249.87 0.00 21.50
Average: 39 0.00 249.50 0.00 0.00 0.00
0.00 0.00 249.83 0.00 23.96
Average: 40 0.00 249.83 0.00 0.00 0.00
0.00 0.00 249.96 0.00 30.96
Average: 41 0.00 249.92 0.00 0.00 0.00
0.00 0.00 249.92 0.00 42.50
Average: 42 0.00 148.38 0.71 5761.00 0.00
0.00 0.00 8.25 0.00 3.38
Average: 43 0.00 147.21 0.50 5790.33 0.00
0.00 0.00 8.00 0.00 3.38
Average: 44 0.00 248.96 0.00 0.00 0.00
0.00 0.00 248.13 0.00 5.96
Average: 45 0.00 249.04 0.00 0.00 0.00
0.00 0.00 248.88 0.00 44.83
Average: 46 0.00 248.96 0.00 0.00 0.00
0.00 0.00 248.58 0.00 84.96
Average: 47 0.00 249.00 0.00 0.00 0.00
0.00 0.00 248.75 0.00 83.96
Average: 48 0.00 249.12 0.00 0.00 0.00
0.00 0.00 132.83 0.00 113.33
Average: 49 0.00 249.12 0.00 0.00 0.00
0.00 0.00 248.62 0.00 26.62
Average: 50 0.00 248.92 0.00 0.00 0.00
0.00 0.00 248.58 0.00 30.42
Average: 51 0.00 249.08 0.00 0.00 0.00
0.00 0.00 248.42 0.00 30.63
Average: 52 0.00 249.21 0.00 0.00 0.00
0.00 0.00 131.96 0.00 75.21
Average: 53 0.00 249.08 0.00 0.00 0.00
0.00 0.00 136.12 0.00 91.79
Average: 54 0.00 249.00 0.00 0.00 0.00
0.00 0.00 136.79 0.00 55.12
Average: 55 0.00 249.04 0.00 0.00 0.00
0.00 0.00 248.71 0.00 71.08
>
[-- Attachment #2: perf-report --]
[-- Type: text/plain, Size: 2361492 bytes --]
=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2017.08.12 19:25:11 =~=~=~=~=~=~=~=~=~=~=~=
\r --6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
lines 86-170\r | |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
lines 171-255\r | | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.68%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
lines 256-340\r |
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.68%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
lines 341-425\r | --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
lines 426-510\r --1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] smpboot_thread_fn
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.68%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
lines 511-595\r |
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
lines 596-680\r | --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] kthread
|
---kthread
lines 681-765\r |
--6.21%--smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
lines 766-850\r |
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] smpboot_thread_fn
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
lines 851-935\r | | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] run_ksoftirqd
|
---run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
lines 936-1020\r |
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
lines 1021-1105\r | |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] kthread
|
---kthread
smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ret_from_fork
|
lines 1106-1190\r ---ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
|
--6.21%--run_ksoftirqd
__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
lines 1191-1275\r | |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
lines 1276-1360\r | |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
lines 1361-1445\r |
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
lines 1446-1530\r | |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] run_ksoftirqd
|
lines 1531-1615\r ---run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.68%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
lines 1616-1700\r | |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
lines 1701-1785\r | | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
lines 1786-1870\r --6.18%--ixgbe_poll
|
--5.63%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.63%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
lines 1871-1955\r | --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
lines 1956-2040\r |
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] smpboot_thread_fn
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
lines 2041-2125\r --5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
lines 2126-2210\r | | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] smpboot_thread_fn
lines 2211-2295\r |
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.63%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
lines 2296-2380\r |--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
lines 2381-2465\r | |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
lines 2466-2550\r |
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
lines 2551-2635\r | --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
lines 2636-2720\r |
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
lines 2721-2805\r | --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
lines 2806-2890\r | --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.68%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
lines 2891-2975\r |
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
lines 2976-3060\r | | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.63%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
lines 3061-3145\r |
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
lines 3146-3230\r | | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.21%--__softirqentry_text_start
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
lines 3231-3315\r |
--5.63%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
lines 3316-3400\r | |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
lines 3401-3485\r |
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.68%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
lines 3486-3570\r | | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
lines 3571-3655\r |
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
lines 3656-3740\r | | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.21% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
lines 3741-3825\r |
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.21% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.19%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
lines 3826-3910\r | | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.21% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] net_rx_action
|
--6.21%--net_rx_action
|
--6.18%--ixgbe_poll
|
--5.63%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.21% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
lines 3911-3995\r |
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
6.21% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
lines 3996-4080\r | |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
6.21% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] smpboot_thread_fn
|
--6.21%--smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.47%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
lines 4081-4165\r 6.21% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] kthread
|
---kthread
smpboot_thread_fn
|
--6.21%--run_ksoftirqd
__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.47%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
smpboot_thread_fn
|
--6.21%--run_ksoftirqd
__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.47%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
lines 4166-4250\r --5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] smpboot_thread_fn
|
--6.21%--smpboot_thread_fn
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
lines 4251-4335\r | | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
6.21% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.21%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
6.21% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
lines 4336-4420\r --5.47%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] run_ksoftirqd
|
---run_ksoftirqd
__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.47%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
lines 4421-4505\r | --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] run_ksoftirqd
|
--6.21%--run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] kthread
lines 4506-4590\r |
---kthread
|
--6.21%--smpboot_thread_fn
run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
lines 4591-4675\r |
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] smpboot_thread_fn
|
---smpboot_thread_fn
run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
lines 4676-4760\r | | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
6.21% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] kthread
|
---kthread
|
--6.21%--smpboot_thread_fn
|
--6.20%--run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
lines 4761-4845\r --6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--6.21%--smpboot_thread_fn
|
--6.20%--run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
lines 4846-4930\r | --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.21% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] smpboot_thread_fn
|
--6.20%--smpboot_thread_fn
|
--6.20%--run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
lines 4931-5015\r | | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.20% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.20% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] run_ksoftirqd
|
---run_ksoftirqd
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
lines 5016-5100\r --5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.20% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __softirqentry_text_start
|
--6.20%--__softirqentry_text_start
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
lines 5101-5185\r | | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.20% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] net_rx_action
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.47%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.20% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] net_rx_action
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
lines 5186-5270\r --5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.20% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] net_rx_action
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
lines 5271-5355\r | | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
6.20% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] net_rx_action
|
--6.20%--net_rx_action
|
--6.17%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.19% 0.10% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_poll
|
--6.09%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
lines 5356-5440\r --5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.19% 0.10% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_poll
|
--6.09%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
lines 5441-5525\r | |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.19% 0.09% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_poll
|
--6.10%--ixgbe_poll
|
--5.68%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.19% 0.10% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_poll
|
--6.09%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
lines 5526-5610\r | --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.19% 0.09% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_poll
|
--6.10%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
6.19% 0.10% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_poll
lines 5611-5695\r |
--6.09%--ixgbe_poll
|
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.19% 0.10% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_poll
|
--6.08%--ixgbe_poll
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
lines 5696-5780\r | | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.18% 0.10% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_poll
|
--6.08%--ixgbe_poll
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
6.18% 0.09% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_poll
|
--6.09%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
lines 5781-5865\r |
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.18% 0.10% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_poll
|
--6.09%--ixgbe_poll
|
--5.65%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
lines 5866-5950\r | | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
6.18% 0.10% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_poll
|
--6.08%--ixgbe_poll
|
--5.63%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
6.17% 0.16% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_poll
|
--6.01%--ixgbe_poll
|
--5.47%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
lines 5951-6035\r | |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.17% 0.17% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_poll
|
--6.00%--ixgbe_poll
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
lines 6036-6120\r --0.82%--rt_cache_valid
6.17% 0.17% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_poll
|
--6.00%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
6.17% 0.17% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_poll
|
--6.00%--ixgbe_poll
|
--5.45%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
lines 6121-6205\r | |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
5.68% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] napi_gro_receive
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.67% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] napi_gro_receive
|
--5.67%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
lines 6206-6290\r --5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.67% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] napi_gro_receive
|
--5.66%--napi_gro_receive
|
--5.63%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
lines 6291-6375\r | |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.66% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] napi_gro_receive
|
--5.66%--napi_gro_receive
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.66% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] napi_gro_receive
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
lines 6376-6460\r | --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.65% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] napi_gro_receive
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.65% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] napi_gro_receive
|
--5.65%--napi_gro_receive
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
lines 6461-6545\r |
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.65% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] napi_gro_receive
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
lines 6546-6630\r | | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.65% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] napi_gro_receive
|
--5.64%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.65% 0.08% ksoftirqd/22 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.57%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
lines 6631-6715\r | |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.64% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] napi_gro_receive
|
--5.63%--napi_gro_receive
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.64% 0.09% ksoftirqd/21 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.56%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
lines 6716-6800\r --5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.64% 0.07% ksoftirqd/18 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.57%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
lines 6801-6885
5.64% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.63%--netif_receive_skb_internal
|
--5.63%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.63% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.62%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.55%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
lines 6886-6970\r | | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.63% 0.07% ksoftirqd/25 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.56%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.63% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.62%--netif_receive_skb_internal
|
--5.62%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
lines 6971-7055\r | |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.63% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] napi_gro_receive
|
--5.62%--napi_gro_receive
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.62% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.62%--netif_receive_skb_internal
|
--5.61%--__netif_receive_skb
|
--5.56%--__netif_receive_skb_core
|
lines 7056-7140\r --5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.62% 0.08% ksoftirqd/27 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.55%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
lines 7141-7225\r |
--1.07%--rt_cache_valid
5.62% 0.08% ksoftirqd/17 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.54%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.62% 0.08% ksoftirqd/24 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.54%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
lines 7226-7310\r | | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.62% 0.08% ksoftirqd/23 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.54%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.61% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
lines 7311-7395\r | |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.61% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.61%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.61% 0.08% ksoftirqd/26 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.54%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
lines 7396-7480\r --5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.61% 0.08% ksoftirqd/19 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.53%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
lines 7481-7565
5.61% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.54%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.61% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
lines 7566-7650\r | | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.60% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.60%--netif_receive_skb_internal
|
--5.60%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.60% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.60%--netif_receive_skb_internal
|
--5.59%--__netif_receive_skb
|
--5.53%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
lines 7651-7735\r | |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.59% 0.08% ksoftirqd/20 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.52%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.58% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.58%--netif_receive_skb_internal
|
--5.57%--__netif_receive_skb
|
--5.51%--__netif_receive_skb_core
|
lines 7736-7820\r --5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.56% 0.07% ksoftirqd/18 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.50%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
lines 7821-7905
5.56% 0.07% ksoftirqd/22 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.50%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.56% 0.06% ksoftirqd/25 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.50%--__netif_receive_skb_core
|
--5.48%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
lines 7906-7990\r --1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.56% 0.07% ksoftirqd/21 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.49%--__netif_receive_skb_core
|
--5.47%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.55% 0.07% ksoftirqd/27 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.48%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
lines 7991-8075\r | | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.54% 0.07% ksoftirqd/17 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.47%--__netif_receive_skb_core
|
--5.46%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.54% 0.07% ksoftirqd/24 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.47%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
lines 8076-8160\r | | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.54% 0.07% ksoftirqd/23 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.47%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.54% 0.07% ksoftirqd/26 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.47%--__netif_receive_skb_core
|
--5.45%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
lines 8161-8245\r | |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.53% 0.07% ksoftirqd/19 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.46%--__netif_receive_skb_core
|
--5.44%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.52% 0.07% ksoftirqd/20 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.44%--__netif_receive_skb_core
|
--5.43%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
lines 8246-8330\r | --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.48% 0.05% ksoftirqd/22 [kernel.vmlinux] [k] ip_rcv
|
--5.43%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.48% 0.05% ksoftirqd/25 [kernel.vmlinux] [k] ip_rcv
|
--5.44%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
lines 8331-8415\r | |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.48% 0.05% ksoftirqd/18 [kernel.vmlinux] [k] ip_rcv
|
--5.43%--ip_rcv
|
--5.41%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.47% 0.05% ksoftirqd/21 [kernel.vmlinux] [k] ip_rcv
|
--5.42%--ip_rcv
|
--5.40%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
lines 8416-8500\r | --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.47% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] napi_gro_receive
|
--5.46%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
lines 8501-8585
5.46% 0.05% ksoftirqd/27 [kernel.vmlinux] [k] ip_rcv
|
--5.41%--ip_rcv
|
--5.39%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.46% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] napi_gro_receive
|
--5.45%--napi_gro_receive
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
lines 8586-8670\r | | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.46% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] kthread
|
---kthread
|
--5.46%--smpboot_thread_fn
|
--5.46%--run_ksoftirqd
|
--5.46%--__softirqentry_text_start
|
--5.46%--net_rx_action
|
--5.43%--ixgbe_poll
|
--4.99%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
5.46% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ret_from_fork
|
---ret_from_fork
kthread
|
--5.46%--smpboot_thread_fn
|
--5.46%--run_ksoftirqd
|
--5.46%--__softirqentry_text_start
|
lines 8671-8755\r --5.46%--net_rx_action
|
--5.43%--ixgbe_poll
|
--4.99%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
5.46% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] smpboot_thread_fn
|
--5.46%--smpboot_thread_fn
|
--5.46%--run_ksoftirqd
|
--5.46%--__softirqentry_text_start
|
--5.46%--net_rx_action
|
--5.43%--ixgbe_poll
|
--4.99%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
lines 8756-8840\r | |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
5.46% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] run_ksoftirqd
|
--5.46%--run_ksoftirqd
|
--5.46%--__softirqentry_text_start
|
--5.46%--net_rx_action
|
--5.43%--ixgbe_poll
|
--4.99%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
lines 8841-8925\r 5.46% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __softirqentry_text_start
|
--5.46%--__softirqentry_text_start
|
--5.46%--net_rx_action
|
--5.43%--ixgbe_poll
|
--4.99%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
5.46% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] net_rx_action
|
--5.46%--net_rx_action
|
--5.43%--ixgbe_poll
|
--4.99%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
lines 8926-9010\r | |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
5.46% 0.05% ksoftirqd/17 [kernel.vmlinux] [k] ip_rcv
|
--5.41%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.45% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] napi_gro_receive
|
--5.44%--napi_gro_receive
|
--5.43%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
lines 9011-9095\r | --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.45% 0.05% ksoftirqd/24 [kernel.vmlinux] [k] ip_rcv
|
--5.40%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.45% 0.05% ksoftirqd/26 [kernel.vmlinux] [k] ip_rcv
|
--5.41%--ip_rcv
|
--5.38%--ip_rcv_finish
lines 9096-9180\r |
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.45% 0.05% ksoftirqd/23 [kernel.vmlinux] [k] ip_rcv
|
--5.40%--ip_rcv
|
--5.38%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.45% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] napi_gro_receive
|
--5.44%--napi_gro_receive
|
lines 9181-9265\r --5.43%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
5.44% 0.05% ksoftirqd/19 [kernel.vmlinux] [k] ip_rcv
|
--5.39%--ip_rcv
|
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
lines 9266-9350\r |
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.44% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.44%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.44% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.43%--netif_receive_skb_internal
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
lines 9351-9435\r | |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.44% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.43%--__netif_receive_skb
|
--5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.44% 0.09% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_poll
|
--5.34%--ixgbe_poll
|
--4.99%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
lines 9436-9520\r |
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
5.43% 0.05% ksoftirqd/20 [kernel.vmlinux] [k] ip_rcv
|
--5.38%--ip_rcv
|
--5.36%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.43% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.42%--__netif_receive_skb
|
lines 9521-9605\r --5.42%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.43% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.42%--netif_receive_skb_internal
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
lines 9606-9690\r | |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.43% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--5.42%--netif_receive_skb_internal
|
--5.42%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
5.42% 0.06% ksoftirqd/15 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.37%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
lines 9691-9775\r | | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.42% 0.05% ksoftirqd/14 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.37%--__netif_receive_skb_core
|
--5.34%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.42% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
lines 9776-9860\r | |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
5.41% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] __netif_receive_skb
|
--5.41%--__netif_receive_skb
|
--5.40%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.41% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.39%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
lines 9861-9945\r | --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.91%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.41% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.39%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.41% 0.05% ksoftirqd/43 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.35%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
lines 9946-10030\r | |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
5.41% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.38%--ip_rcv_finish
|
|--4.04%--ip_forward
| |
| --3.95%--ip_forward_finish
| |
| --3.94%--ip_output
| |
| --3.93%--ip_finish_output
| |
| --3.92%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.41% 0.06% ksoftirqd/42 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--5.35%--__netif_receive_skb_core
|
--5.32%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
lines 10031-10115\r | --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.40% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.38%--ip_rcv_finish
|
|--4.03%--ip_forward
| |
| --3.93%--ip_forward_finish
| |
| --3.93%--ip_output
| |
| --3.91%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.39% 0.03% ksoftirqd/27 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.36%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
lines 10116-10200\r | |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--1.33%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.38% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.36%--ip_rcv_finish
|
|--4.01%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.89%--ip_finish_output
| |
| --3.87%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
5.38% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.36%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.94%--ip_forward_finish
| |
| --3.93%--ip_output
| |
lines 10201-10285\r | --3.92%--ip_finish_output
| |
| --3.90%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.38% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.36%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.89%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.86%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
5.38% 0.03% ksoftirqd/23 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.35%--ip_rcv_finish
|
|--4.00%--ip_forward
| |
| --3.90%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
lines 10286-10370\r | |
| --3.86%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
5.37% 0.02% ksoftirqd/19 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.35%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.36% 0.03% ksoftirqd/20 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.33%--ip_rcv_finish
|
|--3.99%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
lines 10371-10455\r | --3.85%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
5.34% 0.07% ksoftirqd/15 [kernel.vmlinux] [k] ip_rcv
|
--5.28%--ip_rcv
|
--5.26%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.34% 0.07% ksoftirqd/14 [kernel.vmlinux] [k] ip_rcv
|
--5.27%--ip_rcv
|
--5.25%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
lines 10456-10540\r | |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.32% 0.08% ksoftirqd/43 [kernel.vmlinux] [k] ip_rcv
|
--5.24%--ip_rcv
|
--5.22%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
5.32% 0.07% ksoftirqd/42 [kernel.vmlinux] [k] ip_rcv
|
--5.25%--ip_rcv
|
--5.23%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
lines 10541-10625\r | --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.26% 0.04% ksoftirqd/15 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.22%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
| |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.25% 0.04% ksoftirqd/14 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.21%--ip_rcv_finish
|
|--4.07%--ip_forward
| |
| --3.91%--ip_forward_finish
lines 10626-10710\r | |
| --3.90%--ip_output
| |
| --3.88%--ip_finish_output
| |
| --3.85%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.23% 0.04% ksoftirqd/42 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.19%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
| --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
5.23% 0.05% ksoftirqd/43 [kernel.vmlinux] [k] ip_rcv_finish
|
--5.18%--ip_rcv_finish
|
|--4.05%--ip_forward
| |
| --3.89%--ip_forward_finish
| |
lines 10711-10795\r | --3.88%--ip_output
| |
| --3.86%--ip_finish_output
| |
| --3.83%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
4.99% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] napi_gro_receive
|
--4.98%--napi_gro_receive
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
4.96% 0.08% ksoftirqd/16 [kernel.vmlinux] [k] __netif_receive_skb
lines 10796-10880\r |
--4.88%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
4.95% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] netif_receive_skb_internal
|
--4.94%--netif_receive_skb_internal
|
--4.94%--__netif_receive_skb
|
--4.87%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
lines 10881-10965\r | --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
4.88% 0.06% ksoftirqd/16 [kernel.vmlinux] [k] __netif_receive_skb_core
|
--4.82%--__netif_receive_skb_core
|
--4.80%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
4.80% 0.05% ksoftirqd/16 [kernel.vmlinux] [k] ip_rcv
|
--4.75%--ip_rcv
|
--4.73%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
lines 10966-11050\r | |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
4.73% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] ip_rcv_finish
|
--4.71%--ip_rcv_finish
|
|--3.54%--ip_forward
| |
| --3.45%--ip_forward_finish
| |
| --3.44%--ip_output
| |
| --3.43%--ip_finish_output
| |
| --3.41%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
4.09% 0.05% ksoftirqd/15 [kernel.vmlinux] [k] ip_forward
|
--4.03%--ip_forward
|
--3.91%--ip_forward_finish
|
--3.90%--ip_output
|
--3.88%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.69%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.90%--skb_dst_force
4.08% 0.06% ksoftirqd/14 [kernel.vmlinux] [k] ip_forward
|
lines 11051-11135\r --4.03%--ip_forward
|
--3.91%--ip_forward_finish
|
--3.90%--ip_output
|
--3.88%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.94%--skb_dst_drop.isra.77
| | |
| | --0.94%--dst_release
| |
| --0.59%--sch_direct_xmit
|
--0.89%--skb_dst_force
4.07% 0.06% ksoftirqd/42 [kernel.vmlinux] [k] ip_forward
|
--4.01%--ip_forward
|
--3.89%--ip_forward_finish
|
--3.88%--ip_output
|
--3.86%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.71%--neigh_connected_output
|
--2.67%--dev_queue_xmit
|
--2.66%--__dev_queue_xmit
|
|--1.69%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.65%--dev_queue_xmit
| |
| --1.64%--__dev_queue_xmit
| |
| |--0.92%--skb_dst_drop.isra.77
| | |
| | --0.91%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.89%--skb_dst_force
4.07% 0.06% ksoftirqd/43 [kernel.vmlinux] [k] ip_forward
|
--4.01%--ip_forward
|
--3.89%--ip_forward_finish
|
--3.88%--ip_output
|
--3.86%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.71%--dev_hard_start_xmit
lines 11136-11220\r | |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.67%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.87%--skb_dst_force
4.06% 0.06% ksoftirqd/22 [kernel.vmlinux] [k] ip_forward
|
--4.01%--ip_forward
|
--3.95%--ip_forward_finish
|
--3.94%--ip_output
|
--3.93%--ip_finish_output
|
--3.92%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.75%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.02%--skb_dst_force
4.06% 0.06% ksoftirqd/25 [kernel.vmlinux] [k] ip_forward
|
--4.01%--ip_forward
|
--3.95%--ip_forward_finish
|
--3.94%--ip_output
|
--3.93%--ip_finish_output
|
--3.91%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--1.02%--skb_dst_force
4.05% 0.06% ksoftirqd/17 [kernel.vmlinux] [k] ip_forward
|
--3.99%--ip_forward
|
lines 11221-11305\r --3.94%--ip_forward_finish
|
--3.93%--ip_output
|
--3.92%--ip_finish_output
|
--3.90%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.00%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--0.99%--skb_dst_force
4.05% 0.06% ksoftirqd/21 [kernel.vmlinux] [k] ip_forward
|
--3.99%--ip_forward
|
--3.93%--ip_forward_finish
|
--3.93%--ip_output
|
--3.91%--ip_finish_output
|
--3.90%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.74%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.63%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.61%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--1.03%--skb_dst_force
4.04% 0.06% ksoftirqd/19 [kernel.vmlinux] [k] ip_forward
|
--3.98%--ip_forward
|
--3.92%--ip_forward_finish
|
--3.91%--ip_output
|
--3.90%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.77%--neigh_connected_output
|
--2.74%--dev_queue_xmit
|
--2.73%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
lines 11306-11390\r | |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--0.99%--skb_dst_force
4.04% 0.06% ksoftirqd/18 [kernel.vmlinux] [k] ip_forward
|
--3.98%--ip_forward
|
--3.91%--ip_forward_finish
|
--3.91%--ip_output
|
--3.90%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.71%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.60%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.04%--skb_dst_drop.isra.77
| |
| --1.03%--dst_release
|
--1.00%--skb_dst_force
4.03% 0.05% ksoftirqd/26 [kernel.vmlinux] [k] ip_forward
|
--3.98%--ip_forward
|
--3.91%--ip_forward_finish
|
--3.91%--ip_output
|
--3.89%--ip_finish_output
|
--3.87%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--0.99%--skb_dst_force
4.03% 0.06% ksoftirqd/23 [kernel.vmlinux] [k] ip_forward
|
--3.97%--ip_forward
|
--3.90%--ip_forward_finish
|
--3.90%--ip_output
|
lines 11391-11475\r --3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.00%--skb_dst_force
4.02% 0.06% ksoftirqd/27 [kernel.vmlinux] [k] ip_forward
|
--3.97%--ip_forward
|
--3.90%--ip_forward_finish
|
--3.89%--ip_output
|
--3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.82%--neigh_connected_output
|
--2.78%--dev_queue_xmit
|
--2.78%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--1.01%--skb_dst_force
4.02% 0.06% ksoftirqd/24 [kernel.vmlinux] [k] ip_forward
|
--3.96%--ip_forward
|
--3.90%--ip_forward_finish
|
--3.89%--ip_output
|
--3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.58%--__dev_queue_xmit
lines 11476-11560\r | |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--0.99%--skb_dst_force
4.01% 0.06% ksoftirqd/20 [kernel.vmlinux] [k] ip_forward
|
--3.95%--ip_forward
|
--3.89%--ip_forward_finish
|
--3.88%--ip_output
|
--3.86%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.84%--neigh_connected_output
|
--2.81%--dev_queue_xmit
|
--2.80%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.00%--dst_release
| |
| --0.56%--sch_direct_xmit
|
--1.02%--skb_dst_force
3.95% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] ip_forward_finish
|
--3.94%--ip_forward_finish
|
--3.94%--ip_output
|
--3.93%--ip_finish_output
|
--3.91%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--1.02%--skb_dst_force
3.95% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ip_forward_finish
|
--3.94%--ip_forward_finish
|
--3.94%--ip_output
|
--3.93%--ip_finish_output
|
--3.92%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
lines 11561-11645\r --2.75%--dev_queue_xmit
|
--2.75%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.02%--skb_dst_force
3.94% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ip_output
|
--3.93%--ip_output
|
--3.93%--ip_finish_output
|
--3.92%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.75%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.02%--skb_dst_force
3.94% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] ip_output
|
--3.93%--ip_output
|
--3.93%--ip_finish_output
|
--3.91%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--1.02%--skb_dst_force
3.94% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ip_forward_finish
|
--3.93%--ip_forward_finish
|
--3.93%--ip_output
|
--3.92%--ip_finish_output
lines 11646-11730\r |
--3.90%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.00%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.93% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ip_output
|
--3.92%--ip_output
|
--3.92%--ip_finish_output
|
--3.90%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.00%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.93% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] ip_forward_finish
|
--3.93%--ip_forward_finish
|
--3.93%--ip_output
|
--3.91%--ip_finish_output
|
--3.90%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.74%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.63%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.61%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.51%--sch_direct_xmit
|
lines 11731-11815\r --1.03%--skb_dst_force
3.93% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ip_finish_output
|
--3.92%--ip_finish_output
|
--3.92%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.75%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.02%--skb_dst_force
3.93% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] ip_finish_output
|
--3.92%--ip_finish_output
|
--3.91%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--1.02%--skb_dst_force
3.93% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] ip_output
|
--3.91%--ip_output
|
--3.91%--ip_finish_output
|
--3.90%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.74%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.63%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.61%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--1.03%--skb_dst_force
lines 11816-11900
3.92% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ip_forward_finish
|
--3.91%--ip_forward_finish
|
--3.91%--ip_output
|
--3.90%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.77%--neigh_connected_output
|
--2.74%--dev_queue_xmit
|
--2.73%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.92% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ip_finish_output
|
--3.91%--ip_finish_output
|
--3.90%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.00%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.92% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ip_forward_finish
|
--3.91%--ip_forward_finish
|
--3.90%--ip_output
|
--3.88%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.69%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
lines 11901-11985\r | |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.90%--skb_dst_force
3.92% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] ip_forward_finish
|
--3.91%--ip_forward_finish
|
--3.91%--ip_output
|
--3.90%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.71%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.60%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.04%--skb_dst_drop.isra.77
| |
| --1.03%--dst_release
|
--1.00%--skb_dst_force
3.92% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ip_output
|
--3.90%--ip_output
|
--3.90%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.77%--neigh_connected_output
|
--2.74%--dev_queue_xmit
|
--2.73%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.92% 0.88% ksoftirqd/22 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.04%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
| --2.75%--dev_queue_xmit
| |
| --2.75%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
lines 11986-12070\r | | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.02%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--0.88%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
3.91% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] ip_forward_finish
|
--3.91%--ip_forward_finish
|
--3.91%--ip_output
|
--3.89%--ip_finish_output
|
--3.87%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--0.99%--skb_dst_force
3.91% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] ip_finish_output
|
--3.90%--ip_finish_output
|
--3.90%--ip_finish_output2
|
--2.79%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.74%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.63%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.61%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--1.03%--skb_dst_force
lines 12071-12155
3.91% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] ip_output
|
--3.90%--ip_output
|
--3.90%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.71%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.60%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.04%--skb_dst_drop.isra.77
| |
| --1.03%--dst_release
|
--1.00%--skb_dst_force
3.91% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] ip_output
|
--3.89%--ip_output
|
--3.89%--ip_finish_output
|
--3.87%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--0.99%--skb_dst_force
3.91% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] ip_output
|
--3.89%--ip_output
|
--3.88%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.69%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
lines 12156-12240\r --0.90%--skb_dst_force
3.91% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] ip_forward_finish
|
--3.90%--ip_forward_finish
|
--3.90%--ip_output
|
--3.88%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.94%--skb_dst_drop.isra.77
| | |
| | --0.94%--dst_release
| |
| --0.59%--sch_direct_xmit
|
--0.89%--skb_dst_force
3.91% 0.90% ksoftirqd/25 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.01%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --1.02%--skb_dst_force
|
--0.90%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.90%--ip_finish_output
ip_finish_output2
3.90% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] ip_forward_finish
|
--3.90%--ip_forward_finish
|
--3.90%--ip_output
|
--3.88%--ip_finish_output
lines 12241-12325\r |
--3.86%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.00%--skb_dst_force
3.90% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] ip_output
|
--3.89%--ip_output
|
--3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.00%--skb_dst_force
3.90% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] ip_output
|
--3.88%--ip_output
|
--3.88%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.94%--skb_dst_drop.isra.77
| | |
| | --0.94%--dst_release
| |
| --0.59%--sch_direct_xmit
|
--0.89%--skb_dst_force
lines 12326-12410\r 3.90% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ip_finish_output
|
--3.89%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.77%--neigh_connected_output
|
--2.74%--dev_queue_xmit
|
--2.73%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.90% 0.90% ksoftirqd/17 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.00%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.00%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--0.90%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.90%--ip_finish_output
ip_finish_output2
3.90% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] ip_forward_finish
|
--3.89%--ip_forward_finish
|
--3.89%--ip_output
|
--3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.82%--neigh_connected_output
lines 12411-12495\r |
--2.78%--dev_queue_xmit
|
--2.78%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--1.01%--skb_dst_force
3.90% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] ip_forward_finish
|
--3.89%--ip_forward_finish
|
--3.89%--ip_output
|
--3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.58%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.90% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] ip_finish_output
|
--3.89%--ip_finish_output
|
--3.88%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.71%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.60%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.04%--skb_dst_drop.isra.77
| |
| --1.03%--dst_release
|
--1.00%--skb_dst_force
3.90% 0.86% ksoftirqd/21 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.03%--ip_finish_output2
| |
| --2.79%--neigh_connected_output
| |
lines 12496-12580\r | --2.75%--dev_queue_xmit
| |
| --2.74%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.63%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.61%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --1.03%--skb_dst_force
|
--0.86%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.86%--ip_finish_output
ip_finish_output2
3.90% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] ip_output
|
--3.88%--ip_output
|
--3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.58%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.89% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] ip_output
|
--3.88%--ip_output
|
--3.88%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.82%--neigh_connected_output
|
--2.78%--dev_queue_xmit
|
--2.78%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
lines 12581-12665\r | |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--1.01%--skb_dst_force
3.89% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] ip_finish_output
|
--3.88%--ip_finish_output
|
--3.87%--ip_finish_output2
|
--2.75%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--0.99%--skb_dst_force
3.89% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] ip_forward_finish
|
--3.88%--ip_forward_finish
|
--3.88%--ip_output
|
--3.86%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.71%--neigh_connected_output
|
--2.67%--dev_queue_xmit
|
--2.66%--__dev_queue_xmit
|
|--1.69%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.65%--dev_queue_xmit
| |
| --1.64%--__dev_queue_xmit
| |
| |--0.92%--skb_dst_drop.isra.77
| | |
| | --0.91%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.89%--skb_dst_force
3.89% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] ip_forward_finish
|
--3.88%--ip_forward_finish
|
--3.88%--ip_output
|
--3.86%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
lines 12666-12750\r --2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.71%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.67%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.87%--skb_dst_force
3.89% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ip_forward_finish
|
--3.88%--ip_forward_finish
|
--3.88%--ip_output
|
--3.86%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.84%--neigh_connected_output
|
--2.81%--dev_queue_xmit
|
--2.80%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.00%--dst_release
| |
| --0.56%--sch_direct_xmit
|
--1.02%--skb_dst_force
3.89% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] ip_output
|
--3.86%--ip_output
|
--3.86%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.71%--neigh_connected_output
|
--2.67%--dev_queue_xmit
|
--2.66%--__dev_queue_xmit
|
|--1.69%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.65%--dev_queue_xmit
| |
| --1.64%--__dev_queue_xmit
| |
| |--0.92%--skb_dst_drop.isra.77
| | |
| | --0.91%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.89%--skb_dst_force
3.88% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] ip_finish_output
|
--3.87%--ip_finish_output
lines 12751-12835\r |
--3.85%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.69%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.90%--skb_dst_force
3.88% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] ip_finish_output
|
--3.87%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.76%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.00%--skb_dst_force
3.88% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] ip_output
|
--3.86%--ip_output
|
--3.86%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.71%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.67%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.87%--skb_dst_force
3.88% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ip_output
|
lines 12836-12920\r --3.87%--ip_output
|
--3.86%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.84%--neigh_connected_output
|
--2.81%--dev_queue_xmit
|
--2.80%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.00%--dst_release
| |
| --0.56%--sch_direct_xmit
|
--1.02%--skb_dst_force
3.88% 0.86% ksoftirqd/19 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.02%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--0.86%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.86%--ip_finish_output
ip_finish_output2
3.88% 0.89% ksoftirqd/18 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.99%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.71%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
lines 12921-13005\r | | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.60%--dev_queue_xmit
| | |
| | --1.59%--__dev_queue_xmit
| | |
| | --1.04%--skb_dst_drop.isra.77
| | |
| | --1.03%--dst_release
| |
| --1.00%--skb_dst_force
|
--0.89%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
3.88% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] ip_finish_output
|
--3.87%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.74%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.58%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--0.99%--skb_dst_force
3.88% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] ip_finish_output
|
--3.87%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.94%--skb_dst_drop.isra.77
| | |
| | --0.94%--dst_release
| |
| --0.59%--sch_direct_xmit
lines 13006-13090\r |
--0.89%--skb_dst_force
3.88% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] ip_finish_output
|
--3.87%--ip_finish_output
|
--3.86%--ip_finish_output2
|
--2.82%--neigh_connected_output
|
--2.78%--dev_queue_xmit
|
--2.78%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--1.01%--skb_dst_force
3.87% 0.88% ksoftirqd/26 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.99%--ip_finish_output2
| |
| --2.75%--neigh_connected_output
| |
| --2.71%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.63%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | --1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.99%--skb_dst_force
|
--0.88%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.88%--ip_finish_output
ip_finish_output2
3.87% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ip_finish_output
|
--3.86%--ip_finish_output
|
--3.85%--ip_finish_output2
|
--2.84%--neigh_connected_output
|
--2.81%--dev_queue_xmit
|
lines 13091-13175\r --2.80%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.00%--dst_release
| |
| --0.56%--sch_direct_xmit
|
--1.02%--skb_dst_force
3.86% 0.86% ksoftirqd/23 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.01%--ip_finish_output2
| |
| --2.76%--neigh_connected_output
| |
| --2.72%--dev_queue_xmit
| |
| --2.72%--__dev_queue_xmit
| |
| |--1.64%--dev_hard_start_xmit
| | |
| | --1.62%--vlan_dev_hard_start_xmit
| | |
| | --1.61%--dev_queue_xmit
| | |
| | --1.60%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.52%--sch_direct_xmit
| |
| --1.00%--skb_dst_force
|
--0.86%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.86%--ip_finish_output
ip_finish_output2
3.86% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] ip_finish_output
|
--3.85%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.71%--neigh_connected_output
|
--2.67%--dev_queue_xmit
|
--2.66%--__dev_queue_xmit
|
|--1.69%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.65%--dev_queue_xmit
| |
| --1.64%--__dev_queue_xmit
| |
| |--0.92%--skb_dst_drop.isra.77
lines 13176-13260\r | | |
| | --0.91%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.89%--skb_dst_force
3.86% 0.88% ksoftirqd/24 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.98%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.70%--__dev_queue_xmit
| |
| |--1.62%--dev_hard_start_xmit
| | |
| | --1.61%--vlan_dev_hard_start_xmit
| | |
| | --1.59%--dev_queue_xmit
| | |
| | --1.58%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.51%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--0.88%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
3.86% 0.80% ksoftirqd/27 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.06%--ip_finish_output2
| |
| --2.82%--neigh_connected_output
| |
| --2.78%--dev_queue_xmit
| |
| --2.78%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.02%--skb_dst_drop.isra.77
| | | |
| | | --1.01%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --1.01%--skb_dst_force
|
--0.80%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
lines 13261-13345\r napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.80%--ip_finish_output
ip_finish_output2
3.86% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] ip_finish_output
|
--3.84%--ip_finish_output
|
--3.83%--ip_finish_output2
|
--2.72%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.71%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.67%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.87%--skb_dst_force
3.85% 0.86% ksoftirqd/15 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.99%--ip_finish_output2
| |
| --2.74%--neigh_connected_output
| |
| --2.70%--dev_queue_xmit
| |
| --2.69%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.90%--skb_dst_force
|
--0.86%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.86%--ip_finish_output
lines 13346-13430\r ip_finish_output2
3.85% 0.88% ksoftirqd/14 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.97%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.70%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.94%--skb_dst_drop.isra.77
| | | |
| | | --0.94%--dst_release
| | |
| | --0.59%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--0.88%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.88%--ip_finish_output
ip_finish_output2
3.85% 0.75% ksoftirqd/20 [kernel.vmlinux] [k] ip_finish_output2
|
|--3.10%--ip_finish_output2
| |
| --2.84%--neigh_connected_output
| |
| --2.81%--dev_queue_xmit
| |
| --2.80%--__dev_queue_xmit
| |
| |--1.68%--dev_hard_start_xmit
| | |
| | --1.67%--vlan_dev_hard_start_xmit
| | |
| | --1.66%--dev_queue_xmit
| | |
| | --1.65%--__dev_queue_xmit
| | |
| | |--1.01%--skb_dst_drop.isra.77
| | | |
| | | --1.00%--dst_release
| | |
| | --0.56%--sch_direct_xmit
| |
| --1.02%--skb_dst_force
|
--0.75%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
lines 13431-13515\r ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.75%--ip_finish_output
ip_finish_output2
3.83% 0.88% ksoftirqd/42 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.95%--ip_finish_output2
| |
| --2.71%--neigh_connected_output
| |
| --2.67%--dev_queue_xmit
| |
| --2.66%--__dev_queue_xmit
| |
| |--1.69%--dev_hard_start_xmit
| | |
| | --1.68%--vlan_dev_hard_start_xmit
| | |
| | --1.65%--dev_queue_xmit
| | |
| | --1.64%--__dev_queue_xmit
| | |
| | |--0.92%--skb_dst_drop.isra.77
| | | |
| | | --0.91%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.89%--skb_dst_force
|
--0.88%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.87%--ip_finish_output
ip_finish_output2
3.83% 0.87% ksoftirqd/43 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.96%--ip_finish_output2
| |
| --2.72%--neigh_connected_output
| |
| --2.68%--dev_queue_xmit
| |
| --2.67%--__dev_queue_xmit
| |
| |--1.71%--dev_hard_start_xmit
| | |
| | --1.69%--vlan_dev_hard_start_xmit
| | |
| | --1.67%--dev_queue_xmit
| | |
| | --1.66%--__dev_queue_xmit
| | |
| | |--0.93%--skb_dst_drop.isra.77
| | | |
| | | --0.93%--dst_release
| | |
| | --0.60%--sch_direct_xmit
| |
| --0.87%--skb_dst_force
|
--0.87%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
lines 13516-13600\r __softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
|
--0.87%--ip_finish_output
ip_finish_output2
3.56% 0.05% ksoftirqd/16 [kernel.vmlinux] [k] ip_forward
|
--3.50%--ip_forward
|
--3.45%--ip_forward_finish
|
--3.44%--ip_output
|
--3.43%--ip_finish_output
|
--3.41%--ip_finish_output2
|
--2.39%--neigh_connected_output
|
--2.35%--dev_queue_xmit
|
--2.35%--__dev_queue_xmit
|
|--1.42%--dev_hard_start_xmit
| |
| --1.40%--vlan_dev_hard_start_xmit
| |
| --1.39%--dev_queue_xmit
| |
| --1.38%--__dev_queue_xmit
| |
| --0.87%--skb_dst_drop.isra.77
| |
| --0.86%--dst_release
|
--0.85%--skb_dst_force
3.45% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] ip_forward_finish
|
--3.44%--ip_forward_finish
|
--3.44%--ip_output
|
--3.43%--ip_finish_output
|
--3.41%--ip_finish_output2
|
--2.39%--neigh_connected_output
|
--2.35%--dev_queue_xmit
|
--2.35%--__dev_queue_xmit
|
|--1.42%--dev_hard_start_xmit
| |
| --1.40%--vlan_dev_hard_start_xmit
| |
| --1.39%--dev_queue_xmit
| |
| --1.38%--__dev_queue_xmit
| |
| --0.87%--skb_dst_drop.isra.77
| |
| --0.86%--dst_release
|
--0.85%--skb_dst_force
3.44% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] ip_output
|
--3.43%--ip_output
|
--3.43%--ip_finish_output
|
--3.41%--ip_finish_output2
lines 13601-13685\r |
--2.39%--neigh_connected_output
|
--2.35%--dev_queue_xmit
|
--2.35%--__dev_queue_xmit
|
|--1.42%--dev_hard_start_xmit
| |
| --1.40%--vlan_dev_hard_start_xmit
| |
| --1.39%--dev_queue_xmit
| |
| --1.38%--__dev_queue_xmit
| |
| --0.87%--skb_dst_drop.isra.77
| |
| --0.86%--dst_release
|
--0.85%--skb_dst_force
3.43% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] ip_finish_output
|
--3.42%--ip_finish_output
|
--3.41%--ip_finish_output2
|
--2.39%--neigh_connected_output
|
--2.35%--dev_queue_xmit
|
--2.35%--__dev_queue_xmit
|
|--1.42%--dev_hard_start_xmit
| |
| --1.40%--vlan_dev_hard_start_xmit
| |
| --1.39%--dev_queue_xmit
| |
| --1.38%--__dev_queue_xmit
| |
| --0.87%--skb_dst_drop.isra.77
| |
| --0.86%--dst_release
|
--0.85%--skb_dst_force
3.41% 0.81% ksoftirqd/16 [kernel.vmlinux] [k] ip_finish_output2
|
|--2.60%--ip_finish_output2
| |
| --2.39%--neigh_connected_output
| |
| --2.35%--dev_queue_xmit
| |
| --2.35%--__dev_queue_xmit
| |
| |--1.42%--dev_hard_start_xmit
| | |
| | --1.40%--vlan_dev_hard_start_xmit
| | |
| | --1.39%--dev_queue_xmit
| | |
| | --1.38%--__dev_queue_xmit
| | |
| | --0.87%--skb_dst_drop.isra.77
| | |
| | --0.86%--dst_release
| |
| --0.85%--skb_dst_force
|
--0.81%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
lines 13686-13770\r ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
3.04% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] dev_queue_xmit
|
--3.04%--dev_queue_xmit
|
--3.03%--__dev_queue_xmit
|
|--1.82%--dev_hard_start_xmit
| |
| --1.81%--vlan_dev_hard_start_xmit
| |
| --1.80%--dev_queue_xmit
| |
| --1.79%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.09%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--1.11%--skb_dst_force
3.04% 0.06% ksoftirqd/20 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.97%--__dev_queue_xmit
|
|--1.79%--dev_hard_start_xmit
| |
| --1.78%--vlan_dev_hard_start_xmit
| |
| --1.77%--dev_queue_xmit
| |
| --1.76%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.09%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--1.11%--skb_dst_force
3.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] dev_queue_xmit
|
--3.01%--dev_queue_xmit
|
--3.00%--__dev_queue_xmit
|
|--1.82%--dev_hard_start_xmit
| |
| --1.80%--vlan_dev_hard_start_xmit
| |
| --1.79%--dev_queue_xmit
| |
| --1.78%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.09%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--1.10%--skb_dst_force
3.00% 0.06% ksoftirqd/27 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.95%--__dev_queue_xmit
|
|--1.79%--dev_hard_start_xmit
| |
| --1.77%--vlan_dev_hard_start_xmit
| |
| --1.76%--dev_queue_xmit
| |
| --1.75%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.09%--dst_release
| |
lines 13771-13855\r | --0.60%--sch_direct_xmit
|
--1.10%--skb_dst_force
2.99% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.98%--dev_queue_xmit
|
--2.98%--__dev_queue_xmit
|
|--1.80%--dev_hard_start_xmit
| |
| --1.78%--vlan_dev_hard_start_xmit
| |
| --1.77%--dev_queue_xmit
| |
| --1.76%--__dev_queue_xmit
| |
| |--1.11%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.57%--sch_direct_xmit
|
--1.10%--skb_dst_force
2.98% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.98%--dev_queue_xmit
|
--2.97%--__dev_queue_xmit
|
|--1.78%--dev_hard_start_xmit
| |
| --1.76%--vlan_dev_hard_start_xmit
| |
| --1.75%--dev_queue_xmit
| |
| --1.74%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.10%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--1.12%--skb_dst_force
2.98% 0.06% ksoftirqd/22 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.93%--__dev_queue_xmit
|
|--1.77%--dev_hard_start_xmit
| |
| --1.75%--vlan_dev_hard_start_xmit
| |
| --1.74%--dev_queue_xmit
| |
| --1.73%--__dev_queue_xmit
| |
| |--1.11%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.57%--sch_direct_xmit
|
--1.10%--skb_dst_force
2.98% 0.06% ksoftirqd/21 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.92%--__dev_queue_xmit
|
|--1.75%--dev_hard_start_xmit
| |
| --1.73%--vlan_dev_hard_start_xmit
| |
| --1.72%--dev_queue_xmit
| |
| --1.71%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.10%--dst_release
| |
| --0.55%--sch_direct_xmit
lines 13856-13940\r |
--1.12%--skb_dst_force
2.97% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.97%--dev_queue_xmit
|
--2.96%--__dev_queue_xmit
|
|--1.80%--dev_hard_start_xmit
| |
| --1.78%--vlan_dev_hard_start_xmit
| |
| --1.77%--dev_queue_xmit
| |
| --1.75%--__dev_queue_xmit
| |
| |--1.08%--skb_dst_drop.isra.77
| | |
| | --1.08%--dst_release
| |
| --0.59%--sch_direct_xmit
|
--1.08%--skb_dst_force
2.96% 0.06% ksoftirqd/19 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.90%--__dev_queue_xmit
|
|--1.77%--dev_hard_start_xmit
| |
| --1.75%--vlan_dev_hard_start_xmit
| |
| --1.74%--dev_queue_xmit
| |
| --1.73%--__dev_queue_xmit
| |
| |--1.08%--skb_dst_drop.isra.77
| | |
| | --1.08%--dst_release
| |
| --0.59%--sch_direct_xmit
|
--1.08%--skb_dst_force
2.96% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.96%--dev_queue_xmit
|
--2.95%--__dev_queue_xmit
|
|--1.76%--dev_hard_start_xmit
| |
| --1.75%--vlan_dev_hard_start_xmit
| |
| --1.73%--dev_queue_xmit
| |
| --1.73%--__dev_queue_xmit
| |
| |--1.11%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.54%--sch_direct_xmit
|
--1.11%--skb_dst_force
2.96% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.95%--dev_queue_xmit
|
--2.95%--__dev_queue_xmit
|
|--1.78%--dev_hard_start_xmit
| |
| --1.76%--vlan_dev_hard_start_xmit
| |
| --1.75%--dev_queue_xmit
| |
| --1.74%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.09%--dst_release
| |
lines 13941-14025\r | --0.56%--sch_direct_xmit
|
--1.09%--skb_dst_force
2.96% 0.05% ksoftirqd/25 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.90%--__dev_queue_xmit
|
|--1.74%--dev_hard_start_xmit
| |
| --1.72%--vlan_dev_hard_start_xmit
| |
| --1.71%--dev_queue_xmit
| |
| --1.70%--__dev_queue_xmit
| |
| |--1.11%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.54%--sch_direct_xmit
|
--1.11%--skb_dst_force
2.95% 0.05% ksoftirqd/23 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.90%--__dev_queue_xmit
|
|--1.75%--dev_hard_start_xmit
| |
| --1.73%--vlan_dev_hard_start_xmit
| |
| --1.73%--dev_queue_xmit
| |
| --1.71%--__dev_queue_xmit
| |
| |--1.10%--skb_dst_drop.isra.77
| | |
| | --1.09%--dst_release
| |
| --0.56%--sch_direct_xmit
|
--1.09%--skb_dst_force
2.94% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.94%--dev_queue_xmit
|
--2.93%--__dev_queue_xmit
|
|--1.77%--dev_hard_start_xmit
| |
| --1.76%--vlan_dev_hard_start_xmit
| |
| --1.75%--dev_queue_xmit
| |
| --1.74%--__dev_queue_xmit
| |
| |--1.09%--skb_dst_drop.isra.77
| | |
| | --1.08%--dst_release
| |
| --0.57%--sch_direct_xmit
|
--1.08%--skb_dst_force
2.94% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.94%--dev_queue_xmit
|
--2.94%--__dev_queue_xmit
|
|--1.76%--dev_hard_start_xmit
| |
| --1.74%--vlan_dev_hard_start_xmit
| |
| --1.73%--dev_queue_xmit
| |
| --1.72%--__dev_queue_xmit
| |
| |--1.12%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.52%--sch_direct_xmit
lines 14026-14110\r |
--1.09%--skb_dst_force
2.94% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.93%--dev_queue_xmit
|
--2.93%--__dev_queue_xmit
|
|--1.77%--dev_hard_start_xmit
| |
| --1.75%--vlan_dev_hard_start_xmit
| |
| --1.74%--dev_queue_xmit
| |
| --1.73%--__dev_queue_xmit
| |
| |--1.11%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.54%--sch_direct_xmit
|
--1.08%--skb_dst_force
2.94% 0.05% ksoftirqd/17 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.89%--__dev_queue_xmit
|
|--1.74%--dev_hard_start_xmit
| |
| --1.73%--vlan_dev_hard_start_xmit
| |
| --1.72%--dev_queue_xmit
| |
| --1.71%--__dev_queue_xmit
| |
| |--1.09%--skb_dst_drop.isra.77
| | |
| | --1.08%--dst_release
| |
| --0.57%--sch_direct_xmit
|
--1.08%--skb_dst_force
2.94% 0.05% ksoftirqd/18 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.89%--__dev_queue_xmit
|
|--1.73%--dev_hard_start_xmit
| |
| --1.72%--vlan_dev_hard_start_xmit
| |
| --1.71%--dev_queue_xmit
| |
| --1.70%--__dev_queue_xmit
| |
| |--1.12%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.09%--skb_dst_force
2.93% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.93%--dev_queue_xmit
|
--2.93%--__dev_queue_xmit
|
|--1.76%--dev_hard_start_xmit
| |
| --1.74%--vlan_dev_hard_start_xmit
| |
| --1.73%--dev_queue_xmit
| |
| --1.72%--__dev_queue_xmit
| |
| |--1.08%--skb_dst_drop.isra.77
| | |
| | --1.08%--dst_release
| |
| --0.55%--sch_direct_xmit
|
lines 14111-14195\r --1.08%--skb_dst_force
2.93% 0.06% ksoftirqd/26 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.88%--__dev_queue_xmit
|
|--1.74%--dev_hard_start_xmit
| |
| --1.72%--vlan_dev_hard_start_xmit
| |
| --1.71%--dev_queue_xmit
| |
| --1.70%--__dev_queue_xmit
| |
| |--1.11%--skb_dst_drop.isra.77
| | |
| | --1.11%--dst_release
| |
| --0.54%--sch_direct_xmit
|
--1.08%--skb_dst_force
2.93% 0.06% ksoftirqd/24 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.87%--__dev_queue_xmit
|
|--1.73%--dev_hard_start_xmit
| |
| --1.72%--vlan_dev_hard_start_xmit
| |
| --1.70%--dev_queue_xmit
| |
| --1.69%--__dev_queue_xmit
| |
| |--1.08%--skb_dst_drop.isra.77
| | |
| | --1.08%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--1.08%--skb_dst_force
2.93% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.92%--dev_queue_xmit
|
--2.91%--__dev_queue_xmit
|
|--1.84%--dev_hard_start_xmit
| |
| --1.83%--vlan_dev_hard_start_xmit
| |
| --1.80%--dev_queue_xmit
| |
| --1.79%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.65%--sch_direct_xmit
|
--0.97%--skb_dst_force
2.92% 0.08% ksoftirqd/15 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.84%--__dev_queue_xmit
|
|--1.80%--dev_hard_start_xmit
| |
| --1.78%--vlan_dev_hard_start_xmit
| |
| --1.76%--dev_queue_xmit
| |
| --1.75%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.65%--sch_direct_xmit
|
--0.97%--skb_dst_force
2.91% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] dev_queue_xmit
lines 14196-14280\r |
--2.90%--dev_queue_xmit
|
--2.90%--__dev_queue_xmit
|
|--1.84%--dev_hard_start_xmit
| |
| --1.82%--vlan_dev_hard_start_xmit
| |
| --1.80%--dev_queue_xmit
| |
| --1.80%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.64%--sch_direct_xmit
|
--0.97%--skb_dst_force
2.90% 0.08% ksoftirqd/14 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.82%--__dev_queue_xmit
|
|--1.79%--dev_hard_start_xmit
| |
| --1.77%--vlan_dev_hard_start_xmit
| |
| --1.76%--dev_queue_xmit
| |
| --1.75%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.64%--sch_direct_xmit
|
--0.97%--skb_dst_force
2.90% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.90%--dev_queue_xmit
|
--2.89%--__dev_queue_xmit
|
|--1.85%--dev_hard_start_xmit
| |
| --1.83%--vlan_dev_hard_start_xmit
| |
| --1.80%--dev_queue_xmit
| |
| --1.80%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.64%--sch_direct_xmit
|
--0.95%--skb_dst_force
2.90% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.89%--dev_queue_xmit
|
--2.89%--__dev_queue_xmit
|
|--1.83%--dev_hard_start_xmit
| |
| --1.82%--vlan_dev_hard_start_xmit
| |
| --1.79%--dev_queue_xmit
| |
| --1.78%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.65%--sch_direct_xmit
|
--0.97%--skb_dst_force
lines 14281-14365\r 2.90% 0.08% ksoftirqd/43 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.82%--__dev_queue_xmit
|
|--1.81%--dev_hard_start_xmit
| |
| --1.79%--vlan_dev_hard_start_xmit
| |
| --1.77%--dev_queue_xmit
| |
| --1.76%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.64%--sch_direct_xmit
|
--0.95%--skb_dst_force
2.89% 0.07% ksoftirqd/42 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.82%--__dev_queue_xmit
|
|--1.79%--dev_hard_start_xmit
| |
| --1.77%--vlan_dev_hard_start_xmit
| |
| --1.75%--dev_queue_xmit
| |
| --1.74%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.65%--sch_direct_xmit
|
--0.97%--skb_dst_force
2.84% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] neigh_connected_output
|
--2.83%--neigh_connected_output
|
--2.81%--dev_queue_xmit
|
--2.80%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.00%--dst_release
| |
| --0.56%--sch_direct_xmit
|
--1.02%--skb_dst_force
2.83% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] neigh_connected_output
|
--2.81%--neigh_connected_output
|
--2.78%--dev_queue_xmit
|
--2.78%--__dev_queue_xmit
|
|--1.68%--dev_hard_start_xmit
| |
| --1.67%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.55%--sch_direct_xmit
lines 14366-14450\r |
--1.01%--skb_dst_force
2.79% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] neigh_connected_output
|
--2.78%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.75%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.02%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.02%--skb_dst_force
2.79% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] neigh_connected_output
|
--2.77%--neigh_connected_output
|
--2.75%--dev_queue_xmit
|
--2.74%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.63%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.61%--__dev_queue_xmit
| |
| |--1.02%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--1.03%--skb_dst_force
2.78% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] neigh_connected_output
|
--2.76%--neigh_connected_output
|
--2.74%--dev_queue_xmit
|
--2.73%--__dev_queue_xmit
|
|--1.66%--dev_hard_start_xmit
| |
| --1.64%--vlan_dev_hard_start_xmit
| |
| --1.63%--dev_queue_xmit
| |
| --1.62%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.55%--sch_direct_xmit
|
--0.99%--skb_dst_force
2.76% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] neigh_connected_output
|
--2.75%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
lines 14451-14535\r | --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--1.02%--skb_dst_force
2.76% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] neigh_connected_output
|
--2.75%--neigh_connected_output
|
--2.72%--dev_queue_xmit
|
--2.72%--__dev_queue_xmit
|
|--1.64%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.01%--skb_dst_drop.isra.77
| | |
| | --1.01%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--1.00%--skb_dst_force
2.75% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] neigh_connected_output
|
--2.74%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.62%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| --1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--0.99%--skb_dst_force
2.75% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] neigh_connected_output
|
--2.74%--neigh_connected_output
|
--2.71%--dev_queue_xmit
|
--2.71%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.60%--dev_queue_xmit
| |
| --1.59%--__dev_queue_xmit
| |
| --1.04%--skb_dst_drop.isra.77
| |
| --1.03%--dst_release
|
--1.00%--skb_dst_force
2.75% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] neigh_connected_output
|
--2.73%--neigh_connected_output
|
--2.71%--dev_queue_xmit
lines 14536-14620\r |
--2.70%--__dev_queue_xmit
|
|--1.63%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.61%--dev_queue_xmit
| |
| --1.60%--__dev_queue_xmit
| |
| |--1.00%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.52%--sch_direct_xmit
|
--0.99%--skb_dst_force
2.74% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] neigh_connected_output
|
--2.73%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.69%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.65%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.90%--skb_dst_force
2.74% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] neigh_connected_output
|
--2.73%--neigh_connected_output
|
--2.70%--dev_queue_xmit
|
--2.70%--__dev_queue_xmit
|
|--1.62%--dev_hard_start_xmit
| |
| --1.61%--vlan_dev_hard_start_xmit
| |
| --1.59%--dev_queue_xmit
| |
| --1.58%--__dev_queue_xmit
| |
| |--0.99%--skb_dst_drop.isra.77
| | |
| | --0.99%--dst_release
| |
| --0.51%--sch_direct_xmit
|
--0.99%--skb_dst_force
2.72% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] neigh_connected_output
|
--2.71%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.70%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.66%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.94%--skb_dst_drop.isra.77
| | |
lines 14621-14705\r | | --0.94%--dst_release
| |
| --0.59%--sch_direct_xmit
|
--0.89%--skb_dst_force
2.72% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] neigh_connected_output
|
--2.71%--neigh_connected_output
|
--2.68%--dev_queue_xmit
|
--2.67%--__dev_queue_xmit
|
|--1.71%--dev_hard_start_xmit
| |
| --1.69%--vlan_dev_hard_start_xmit
| |
| --1.67%--dev_queue_xmit
| |
| --1.66%--__dev_queue_xmit
| |
| |--0.93%--skb_dst_drop.isra.77
| | |
| | --0.93%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.87%--skb_dst_force
2.71% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] neigh_connected_output
|
--2.70%--neigh_connected_output
|
--2.67%--dev_queue_xmit
|
--2.66%--__dev_queue_xmit
|
|--1.69%--dev_hard_start_xmit
| |
| --1.68%--vlan_dev_hard_start_xmit
| |
| --1.65%--dev_queue_xmit
| |
| --1.64%--__dev_queue_xmit
| |
| |--0.92%--skb_dst_drop.isra.77
| | |
| | --0.91%--dst_release
| |
| --0.60%--sch_direct_xmit
|
--0.89%--skb_dst_force
2.55% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] dev_queue_xmit
|
--2.55%--dev_queue_xmit
|
--2.54%--__dev_queue_xmit
|
|--1.54%--dev_hard_start_xmit
| |
| --1.52%--vlan_dev_hard_start_xmit
| |
| --1.51%--dev_queue_xmit
| |
| --1.50%--__dev_queue_xmit
| |
| --0.94%--skb_dst_drop.isra.77
| |
| --0.94%--dst_release
|
--0.93%--skb_dst_force
2.55% 0.05% ksoftirqd/16 [kernel.vmlinux] [k] __dev_queue_xmit
|
--2.50%--__dev_queue_xmit
|
|--1.51%--dev_hard_start_xmit
| |
| --1.50%--vlan_dev_hard_start_xmit
| |
| --1.49%--dev_queue_xmit
| |
| --1.48%--__dev_queue_xmit
lines 14706-14790\r | |
| --0.94%--skb_dst_drop.isra.77
| |
| --0.94%--dst_release
|
--0.93%--skb_dst_force
2.39% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] neigh_connected_output
|
--2.38%--neigh_connected_output
|
--2.35%--dev_queue_xmit
|
--2.35%--__dev_queue_xmit
|
|--1.42%--dev_hard_start_xmit
| |
| --1.40%--vlan_dev_hard_start_xmit
| |
| --1.39%--dev_queue_xmit
| |
| --1.38%--__dev_queue_xmit
| |
| --0.87%--skb_dst_drop.isra.77
| |
| --0.86%--dst_release
|
--0.85%--skb_dst_force
1.85% 0.04% ksoftirqd/43 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.81%--dev_hard_start_xmit
|
--1.81%--vlan_dev_hard_start_xmit
|
--1.78%--dev_queue_xmit
|
--1.78%--__dev_queue_xmit
|
|--1.01%--skb_dst_drop.isra.77
| |
| --1.01%--dst_release
|
--0.62%--sch_direct_xmit
1.85% 0.05% ksoftirqd/15 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.80%--dev_hard_start_xmit
|
--1.80%--vlan_dev_hard_start_xmit
|
--1.77%--dev_queue_xmit
|
--1.76%--__dev_queue_xmit
|
|--1.01%--skb_dst_drop.isra.77
| |
| --1.01%--dst_release
|
--0.62%--sch_direct_xmit
1.85% 0.04% ksoftirqd/14 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.80%--dev_hard_start_xmit
|
--1.80%--vlan_dev_hard_start_xmit
|
--1.78%--dev_queue_xmit
|
--1.77%--__dev_queue_xmit
|
|--1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--0.62%--sch_direct_xmit
1.84% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.81%--vlan_dev_hard_start_xmit
|
--1.81%--dev_queue_xmit
|
--1.80%--__dev_queue_xmit
|
lines 14791-14875\r |--1.01%--skb_dst_drop.isra.77
| |
| --1.01%--dst_release
|
--0.64%--sch_direct_xmit
1.83% 0.04% ksoftirqd/42 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.79%--dev_hard_start_xmit
|
--1.79%--vlan_dev_hard_start_xmit
|
--1.76%--dev_queue_xmit
|
--1.76%--__dev_queue_xmit
|
|--0.99%--skb_dst_drop.isra.77
| |
| --0.99%--dst_release
|
--0.62%--sch_direct_xmit
1.83% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.81%--vlan_dev_hard_start_xmit
|
--1.80%--dev_queue_xmit
|
--1.79%--__dev_queue_xmit
|
|--1.01%--skb_dst_drop.isra.77
| |
| --1.01%--dst_release
|
--0.65%--sch_direct_xmit
1.83% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.81%--vlan_dev_hard_start_xmit
|
--1.80%--dev_queue_xmit
|
--1.80%--__dev_queue_xmit
|
|--1.02%--skb_dst_drop.isra.77
| |
| --1.02%--dst_release
|
--0.64%--sch_direct_xmit
1.83% 0.03% ksoftirqd/20 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.79%--dev_hard_start_xmit
|
--1.79%--vlan_dev_hard_start_xmit
|
--1.78%--dev_queue_xmit
|
--1.77%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.09%--dst_release
|
--0.59%--sch_direct_xmit
1.82% 0.03% ksoftirqd/27 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.79%--dev_hard_start_xmit
|
--1.79%--vlan_dev_hard_start_xmit
|
--1.77%--dev_queue_xmit
|
--1.76%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.09%--dst_release
|
--0.58%--sch_direct_xmit
1.82% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.79%--vlan_dev_hard_start_xmit
lines 14876-14960\r |
--1.79%--dev_queue_xmit
|
--1.78%--__dev_queue_xmit
|
|--0.99%--skb_dst_drop.isra.77
| |
| --0.99%--dst_release
|
--0.65%--sch_direct_xmit
1.81% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.80%--vlan_dev_hard_start_xmit
|
--1.80%--dev_queue_xmit
|
--1.79%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.09%--dst_release
|
--0.60%--sch_direct_xmit
1.80% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.79%--vlan_dev_hard_start_xmit
|
--1.79%--dev_queue_xmit
|
--1.78%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.09%--dst_release
|
--0.60%--sch_direct_xmit
1.80% 0.03% ksoftirqd/22 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.77%--dev_hard_start_xmit
|
--1.77%--vlan_dev_hard_start_xmit
|
--1.75%--dev_queue_xmit
|
--1.75%--__dev_queue_xmit
|
|--1.11%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.55%--sch_direct_xmit
1.80% 0.03% ksoftirqd/19 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.76%--dev_hard_start_xmit
|
--1.76%--vlan_dev_hard_start_xmit
|
--1.75%--dev_queue_xmit
|
--1.74%--__dev_queue_xmit
|
|--1.08%--skb_dst_drop.isra.77
| |
| --1.08%--dst_release
|
--0.58%--sch_direct_xmit
1.78% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.77%--vlan_dev_hard_start_xmit
|
--1.77%--dev_queue_xmit
|
--1.76%--__dev_queue_xmit
|
|--1.11%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.57%--sch_direct_xmit
lines 14961-15045\r 1.78% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.77%--vlan_dev_hard_start_xmit
|
--1.77%--dev_queue_xmit
|
--1.75%--__dev_queue_xmit
|
|--1.08%--skb_dst_drop.isra.77
| |
| --1.08%--dst_release
|
--0.59%--sch_direct_xmit
1.78% 0.03% ksoftirqd/21 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.75%--dev_hard_start_xmit
|
--1.75%--vlan_dev_hard_start_xmit
|
--1.74%--dev_queue_xmit
|
--1.73%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.10%--dst_release
|
--0.54%--sch_direct_xmit
1.78% 0.03% ksoftirqd/23 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.75%--dev_hard_start_xmit
|
--1.75%--vlan_dev_hard_start_xmit
|
--1.74%--dev_queue_xmit
|
--1.73%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.09%--dst_release
|
--0.54%--sch_direct_xmit
1.77% 0.03% ksoftirqd/17 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.74%--dev_hard_start_xmit
|
--1.74%--vlan_dev_hard_start_xmit
|
--1.73%--dev_queue_xmit
|
--1.72%--__dev_queue_xmit
|
|--1.09%--skb_dst_drop.isra.77
| |
| --1.08%--dst_release
|
--0.55%--sch_direct_xmit
1.77% 0.03% ksoftirqd/26 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.74%--dev_hard_start_xmit
|
--1.74%--vlan_dev_hard_start_xmit
|
--1.73%--dev_queue_xmit
|
--1.72%--__dev_queue_xmit
|
|--1.11%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.52%--sch_direct_xmit
1.77% 0.03% ksoftirqd/25 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.73%--dev_hard_start_xmit
|
--1.73%--vlan_dev_hard_start_xmit
|
--1.72%--dev_queue_xmit
lines 15046-15130\r |
--1.71%--__dev_queue_xmit
|
|--1.11%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.52%--sch_direct_xmit
1.76% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.75%--vlan_dev_hard_start_xmit
|
--1.75%--dev_queue_xmit
|
--1.74%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.10%--dst_release
|
--0.55%--sch_direct_xmit
1.76% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.75%--vlan_dev_hard_start_xmit
|
--1.75%--dev_queue_xmit
|
--1.74%--__dev_queue_xmit
|
|--1.10%--skb_dst_drop.isra.77
| |
| --1.09%--dst_release
|
--0.56%--sch_direct_xmit
1.76% 0.03% ksoftirqd/18 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.73%--dev_hard_start_xmit
|
--1.73%--vlan_dev_hard_start_xmit
|
--1.72%--dev_queue_xmit
|
--1.71%--__dev_queue_xmit
|
|--1.12%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.51%--sch_direct_xmit
1.76% 0.03% ksoftirqd/24 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.73%--dev_hard_start_xmit
|
--1.72%--vlan_dev_hard_start_xmit
|
--1.71%--dev_queue_xmit
|
--1.70%--__dev_queue_xmit
|
|--1.08%--skb_dst_drop.isra.77
| |
| --1.08%--dst_release
|
--0.54%--sch_direct_xmit
1.76% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.75%--vlan_dev_hard_start_xmit
|
--1.75%--dev_queue_xmit
|
--1.74%--__dev_queue_xmit
|
|--1.09%--skb_dst_drop.isra.77
| |
| --1.08%--dst_release
|
--0.57%--sch_direct_xmit
1.75% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
lines 15131-15215\r --1.74%--vlan_dev_hard_start_xmit
|
--1.74%--dev_queue_xmit
|
--1.73%--__dev_queue_xmit
|
|--1.11%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.54%--sch_direct_xmit
1.75% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.74%--vlan_dev_hard_start_xmit
|
--1.74%--dev_queue_xmit
|
--1.73%--__dev_queue_xmit
|
|--1.11%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.54%--sch_direct_xmit
1.74% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.73%--vlan_dev_hard_start_xmit
|
--1.73%--dev_queue_xmit
|
--1.72%--__dev_queue_xmit
|
|--1.08%--skb_dst_drop.isra.77
| |
| --1.08%--dst_release
|
--0.55%--sch_direct_xmit
1.74% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.73%--vlan_dev_hard_start_xmit
|
--1.73%--dev_queue_xmit
|
--1.72%--__dev_queue_xmit
|
|--1.12%--skb_dst_drop.isra.77
| |
| --1.11%--dst_release
|
--0.52%--sch_direct_xmit
1.54% 0.03% ksoftirqd/16 [kernel.vmlinux] [k] dev_hard_start_xmit
|
--1.51%--dev_hard_start_xmit
|
--1.51%--vlan_dev_hard_start_xmit
|
--1.49%--dev_queue_xmit
|
--1.49%--__dev_queue_xmit
|
--0.94%--skb_dst_drop.isra.77
|
--0.94%--dst_release
1.52% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] vlan_dev_hard_start_xmit
|
--1.51%--vlan_dev_hard_start_xmit
|
--1.51%--dev_queue_xmit
|
--1.50%--__dev_queue_xmit
|
--0.94%--skb_dst_drop.isra.77
|
--0.94%--dst_release
1.33% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.33%--ip_route_input_noref
|
--1.32%--ip_route_input_rcu
lines 15216-15300\r |
--1.09%--rt_cache_valid
1.33% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
1.33% 0.03% ksoftirqd/18 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.29%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
1.32% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.32%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
1.32% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.31%--ip_route_input_noref
|
--1.31%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
1.32% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
1.31% 0.03% ksoftirqd/27 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.28%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
1.31% 0.03% ksoftirqd/24 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.28%--ip_route_input_rcu
|
--1.09%--rt_cache_valid
1.31% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
1.31% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
1.31% 0.03% ksoftirqd/21 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.28%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
1.31% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.31%--ip_route_input_noref
|
--1.30%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
1.30% 0.04% ksoftirqd/23 [kernel.vmlinux] [k] ip_route_input_rcu
|
lines 15301-15385\r --1.27%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
1.30% 0.03% ksoftirqd/26 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.27%--ip_route_input_rcu
|
--1.07%--rt_cache_valid
1.30% 0.03% ksoftirqd/22 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.27%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
1.30% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.30%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
1.30% 0.03% ksoftirqd/25 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.27%--ip_route_input_rcu
|
--1.08%--rt_cache_valid
1.30% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.29%--ip_route_input_noref
|
--1.28%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
1.30% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
1.29% 0.04% ksoftirqd/20 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.25%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
1.29% 0.04% ksoftirqd/19 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.25%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
1.29% 0.03% ksoftirqd/17 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.25%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
1.14% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.13%--ip_route_input_noref
|
--1.13%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
1.13% 0.03% ksoftirqd/16 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.10%--ip_route_input_rcu
|
--0.92%--rt_cache_valid
1.12% 0.00% swapper [kernel.vmlinux] [k] verify_cpu
|
---verify_cpu
|
--1.08%--start_secondary
|
--1.08%--cpu_startup_entry
lines 15386-15470\r |
--1.07%--do_idle
|
--1.05%--cpuidle_enter
|
--1.03%--cpuidle_enter_state
|
--0.80%--acpi_idle_enter
|
--0.79%--acpi_idle_do_entry
|
--0.79%--acpi_processor_ffh_cstate_enter
1.12% 0.00% swapper [kernel.vmlinux] [k] cpu_startup_entry
|
--1.12%--cpu_startup_entry
|
--1.12%--do_idle
|
--1.10%--cpuidle_enter
|
--1.07%--cpuidle_enter_state
|
--0.83%--acpi_idle_enter
|
--0.82%--acpi_idle_do_entry
|
--0.82%--acpi_processor_ffh_cstate_enter
1.12% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.12%--skb_dst_drop.isra.77
|
--1.11%--dst_release
1.12% 0.00% swapper [kernel.vmlinux] [k] do_idle
|
--1.11%--do_idle
|
--1.10%--cpuidle_enter
|
--1.07%--cpuidle_enter_state
|
--0.83%--acpi_idle_enter
|
--0.82%--acpi_idle_do_entry
|
--0.82%--acpi_processor_ffh_cstate_enter
1.12% 1.12% ksoftirqd/18 [kernel.vmlinux] [k] dst_release
|
--1.12%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.03%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.03%--skb_dst_drop.isra.77
dst_release
1.12% 1.12% ksoftirqd/21 [kernel.vmlinux] [k] skb_dst_force
|
--1.12%--ret_from_fork
kthread
lines 15471-15555\r smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.03%--neigh_connected_output
dev_queue_xmit
|
--1.03%--__dev_queue_xmit
skb_dst_force
1.11% 1.11% ksoftirqd/20 [kernel.vmlinux] [k] skb_dst_force
|
--1.11%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.03%--neigh_connected_output
dev_queue_xmit
|
--1.02%--__dev_queue_xmit
skb_dst_force
1.11% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.11%--skb_dst_drop.isra.77
|
--1.11%--dst_release
1.11% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.11%--skb_dst_drop.isra.77
|
--1.11%--dst_release
1.11% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.11%--skb_dst_drop.isra.77
|
--1.11%--dst_release
1.11% 1.11% ksoftirqd/25 [kernel.vmlinux] [k] dst_release
|
--1.11%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
lines 15556-15640\r ip_output
ip_finish_output
ip_finish_output2
|
--1.02%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.02%--skb_dst_drop.isra.77
dst_release
1.11% 1.11% ksoftirqd/22 [kernel.vmlinux] [k] dst_release
|
--1.11%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.02%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.02%--skb_dst_drop.isra.77
dst_release
1.11% 1.11% ksoftirqd/26 [kernel.vmlinux] [k] dst_release
|
--1.11%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.02%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.02%--skb_dst_drop.isra.77
dst_release
1.11% 1.11% ksoftirqd/25 [kernel.vmlinux] [k] skb_dst_force
|
--1.11%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
lines 15641-15725\r __softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.02%--neigh_connected_output
dev_queue_xmit
|
--1.02%--__dev_queue_xmit
skb_dst_force
1.10% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.10%--skb_dst_drop.isra.77
|
--1.10%--dst_release
1.10% 1.10% ksoftirqd/22 [kernel.vmlinux] [k] skb_dst_force
|
--1.10%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.02%--neigh_connected_output
dev_queue_xmit
|
--1.01%--__dev_queue_xmit
skb_dst_force
1.10% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.10%--ip_route_input_noref
|
--1.09%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
1.10% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.10%--skb_dst_drop.isra.77
|
--1.09%--dst_release
1.10% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.10%--skb_dst_drop.isra.77
|
--1.09%--dst_release
1.10% 0.00% swapper [kernel.vmlinux] [k] cpuidle_enter
|
--1.10%--cpuidle_enter
|
--1.07%--cpuidle_enter_state
|
--0.83%--acpi_idle_enter
|
--0.82%--acpi_idle_do_entry
|
--0.82%--acpi_processor_ffh_cstate_enter
lines 15726-15810
1.10% 1.10% ksoftirqd/21 [kernel.vmlinux] [k] dst_release
|
--1.10%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.01%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.01%--skb_dst_drop.isra.77
dst_release
1.10% 1.10% ksoftirqd/27 [kernel.vmlinux] [k] skb_dst_force
|
--1.10%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.01%--neigh_connected_output
dev_queue_xmit
|
--1.01%--__dev_queue_xmit
skb_dst_force
1.10% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.09%--skb_dst_drop.isra.77
|
--1.09%--dst_release
1.10% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
1.09% 1.09% ksoftirqd/18 [kernel.vmlinux] [k] skb_dst_force
|
--1.09%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
lines 15811-15895\r __netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.00%--neigh_connected_output
dev_queue_xmit
|
--1.00%--__dev_queue_xmit
skb_dst_force
1.09% 1.09% ksoftirqd/27 [kernel.vmlinux] [k] dst_release
|
--1.09%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.01%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.01%--skb_dst_drop.isra.77
dst_release
1.09% 1.09% ksoftirqd/23 [kernel.vmlinux] [k] dst_release
|
--1.09%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.01%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.00%--skb_dst_drop.isra.77
dst_release
1.09% 1.09% ksoftirqd/20 [kernel.vmlinux] [k] dst_release
|
--1.09%--ret_from_fork
kthread
smpboot_thread_fn
lines 15896-15980\r run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.00%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--1.00%--skb_dst_drop.isra.77
dst_release
1.09% 1.09% ksoftirqd/23 [kernel.vmlinux] [k] skb_dst_force
|
--1.09%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--1.00%--neigh_connected_output
dev_queue_xmit
|
--1.00%--__dev_queue_xmit
skb_dst_force
1.09% 1.09% ksoftirqd/18 [kernel.vmlinux] [k] rt_cache_valid
|
--1.09%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.09%--ip_route_input_rcu
rt_cache_valid
1.09% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.09%--ip_route_input_noref
|
--1.08%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
1.09% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.08%--skb_dst_drop.isra.77
lines 15981-16065\r |
--1.08%--dst_release
1.09% 1.09% ksoftirqd/24 [kernel.vmlinux] [k] rt_cache_valid
|
--1.09%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.09%--ip_route_input_rcu
rt_cache_valid
1.09% 0.05% ksoftirqd/15 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.04%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
1.09% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ip_route_input_noref
|
--1.08%--ip_route_input_noref
|
--1.07%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
1.08% 1.08% ksoftirqd/22 [kernel.vmlinux] [k] rt_cache_valid
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.08%--ip_route_input_rcu
rt_cache_valid
1.08% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.08%--skb_dst_drop.isra.77
|
--1.08%--dst_release
1.08% 1.08% ksoftirqd/17 [kernel.vmlinux] [k] dst_release
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.99%--neigh_connected_output
lines 16066-16150\r dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.99%--skb_dst_drop.isra.77
dst_release
1.08% 0.04% ksoftirqd/14 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.04%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
1.08% 1.08% ksoftirqd/17 [kernel.vmlinux] [k] skb_dst_force
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.99%--neigh_connected_output
dev_queue_xmit
|
--0.99%--__dev_queue_xmit
skb_dst_force
1.08% 1.08% ksoftirqd/24 [kernel.vmlinux] [k] dst_release
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.99%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.99%--skb_dst_drop.isra.77
dst_release
1.08% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.08%--skb_dst_drop.isra.77
|
--1.08%--dst_release
1.08% 1.08% ksoftirqd/25 [kernel.vmlinux] [k] rt_cache_valid
|
--1.08%--ret_from_fork
kthread
lines 16151-16235\r smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.08%--ip_route_input_rcu
rt_cache_valid
1.08% 1.08% ksoftirqd/19 [kernel.vmlinux] [k] skb_dst_force
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.99%--neigh_connected_output
dev_queue_xmit
|
--0.99%--__dev_queue_xmit
skb_dst_force
1.08% 0.04% ksoftirqd/42 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.04%--ip_route_input_rcu
|
--0.82%--rt_cache_valid
1.08% 1.08% ksoftirqd/23 [kernel.vmlinux] [k] rt_cache_valid
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.07%--ip_route_input_rcu
rt_cache_valid
1.08% 1.08% ksoftirqd/24 [kernel.vmlinux] [k] skb_dst_force
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
lines 16236-16320\r ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.99%--neigh_connected_output
dev_queue_xmit
|
--0.99%--__dev_queue_xmit
skb_dst_force
1.08% 1.08% ksoftirqd/26 [kernel.vmlinux] [k] skb_dst_force
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.99%--neigh_connected_output
dev_queue_xmit
|
--0.99%--__dev_queue_xmit
skb_dst_force
1.08% 1.08% ksoftirqd/19 [kernel.vmlinux] [k] dst_release
|
--1.08%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.99%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.99%--skb_dst_drop.isra.77
dst_release
1.08% 0.00% swapper [kernel.vmlinux] [k] start_secondary
|
---start_secondary
|
--1.08%--cpu_startup_entry
|
--1.07%--do_idle
|
--1.05%--cpuidle_enter
|
--1.03%--cpuidle_enter_state
|
--0.80%--acpi_idle_enter
|
--0.79%--acpi_idle_do_entry
lines 16321-16405\r |
--0.79%--acpi_processor_ffh_cstate_enter
1.07% 0.05% ksoftirqd/43 [kernel.vmlinux] [k] ip_route_input_rcu
|
--1.03%--ip_route_input_rcu
|
--0.81%--rt_cache_valid
1.07% 1.07% ksoftirqd/27 [kernel.vmlinux] [k] rt_cache_valid
|
--1.07%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.07%--ip_route_input_rcu
rt_cache_valid
1.07% 1.07% ksoftirqd/26 [kernel.vmlinux] [k] rt_cache_valid
|
--1.07%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.07%--ip_route_input_rcu
rt_cache_valid
1.07% 1.07% ksoftirqd/21 [kernel.vmlinux] [k] rt_cache_valid
|
--1.07%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.07%--ip_route_input_rcu
rt_cache_valid
1.07% 0.01% swapper [kernel.vmlinux] [k] cpuidle_enter_state
|
--1.06%--cpuidle_enter_state
|
--0.83%--acpi_idle_enter
|
--0.82%--acpi_idle_do_entry
|
--0.82%--acpi_processor_ffh_cstate_enter
1.06% 1.06% ksoftirqd/17 [kernel.vmlinux] [k] rt_cache_valid
|
--1.06%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
lines 16406-16490\r __softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.05%--ip_route_input_rcu
rt_cache_valid
1.05% 1.05% ksoftirqd/20 [kernel.vmlinux] [k] rt_cache_valid
|
--1.05%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.05%--ip_route_input_rcu
rt_cache_valid
1.05% 1.05% ksoftirqd/19 [kernel.vmlinux] [k] rt_cache_valid
|
--1.05%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--1.05%--ip_route_input_rcu
rt_cache_valid
1.02% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.02%--skb_dst_drop.isra.77
|
--1.02%--dst_release
1.02% 1.02% ksoftirqd/14 [kernel.vmlinux] [k] dst_release
|
--1.02%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.94%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
lines 16491-16575\r vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.93%--skb_dst_drop.isra.77
dst_release
1.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.01%--skb_dst_drop.isra.77
|
--1.01%--dst_release
1.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--1.01%--skb_dst_drop.isra.77
|
--1.01%--dst_release
1.01% 1.01% ksoftirqd/43 [kernel.vmlinux] [k] dst_release
|
--1.01%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.93%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.93%--skb_dst_drop.isra.77
dst_release
1.01% 1.00% ksoftirqd/15 [kernel.vmlinux] [k] dst_release
|
--1.00%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.93%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.92%--skb_dst_drop.isra.77
dst_release
0.99% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
lines 16576-16660\r --0.99%--skb_dst_drop.isra.77
|
--0.99%--dst_release
0.99% 0.99% ksoftirqd/42 [kernel.vmlinux] [k] dst_release
|
--0.99%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.91%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.91%--skb_dst_drop.isra.77
dst_release
0.98% 0.97% ksoftirqd/15 [kernel.vmlinux] [k] skb_dst_force
|
--0.97%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.90%--neigh_connected_output
dev_queue_xmit
|
--0.90%--__dev_queue_xmit
skb_dst_force
0.97% 0.97% ksoftirqd/42 [kernel.vmlinux] [k] skb_dst_force
|
--0.97%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.89%--neigh_connected_output
lines 16661-16745\r dev_queue_xmit
|
--0.89%--__dev_queue_xmit
skb_dst_force
0.97% 0.97% ksoftirqd/14 [kernel.vmlinux] [k] skb_dst_force
|
--0.97%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.89%--neigh_connected_output
dev_queue_xmit
|
--0.89%--__dev_queue_xmit
skb_dst_force
0.95% 0.94% ksoftirqd/43 [kernel.vmlinux] [k] skb_dst_force
|
--0.94%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.87%--neigh_connected_output
dev_queue_xmit
|
--0.87%--__dev_queue_xmit
skb_dst_force
0.94% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_dst_drop.isra.77
|
--0.94%--skb_dst_drop.isra.77
|
--0.94%--dst_release
0.94% 0.94% ksoftirqd/16 [kernel.vmlinux] [k] dst_release
|
--0.94%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
lines 16746-16830\r |
--0.86%--neigh_connected_output
dev_queue_xmit
__dev_queue_xmit
dev_hard_start_xmit
vlan_dev_hard_start_xmit
dev_queue_xmit
__dev_queue_xmit
|
--0.86%--skb_dst_drop.isra.77
dst_release
0.93% 0.93% ksoftirqd/16 [kernel.vmlinux] [k] skb_dst_force
|
--0.93%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_forward
ip_forward_finish
ip_output
ip_finish_output
ip_finish_output2
|
--0.85%--neigh_connected_output
dev_queue_xmit
|
--0.85%--__dev_queue_xmit
skb_dst_force
0.92% 0.92% ksoftirqd/16 [kernel.vmlinux] [k] rt_cache_valid
|
--0.92%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--0.91%--ip_route_input_rcu
rt_cache_valid
0.83% 0.01% swapper [kernel.vmlinux] [k] acpi_idle_enter
|
--0.82%--acpi_idle_enter
|
--0.82%--acpi_idle_do_entry
|
--0.82%--acpi_processor_ffh_cstate_enter
0.82% 0.82% swapper [kernel.vmlinux] [k] acpi_processor_ffh_cstate_enter
|
---verify_cpu
|
--0.79%--start_secondary
cpu_startup_entry
do_idle
cpuidle_enter
cpuidle_enter_state
acpi_idle_enter
|
--0.79%--acpi_idle_do_entry
acpi_processor_ffh_cstate_enter
0.82% 0.82% ksoftirqd/42 [kernel.vmlinux] [k] rt_cache_valid
|
--0.82%--ret_from_fork
kthread
lines 16831-16915\r smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--0.82%--ip_route_input_rcu
rt_cache_valid
0.82% 0.00% swapper [kernel.vmlinux] [k] acpi_idle_do_entry
|
--0.82%--acpi_idle_do_entry
acpi_processor_ffh_cstate_enter
0.82% 0.82% ksoftirqd/14 [kernel.vmlinux] [k] rt_cache_valid
|
--0.82%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--0.82%--ip_route_input_rcu
rt_cache_valid
0.82% 0.81% ksoftirqd/15 [kernel.vmlinux] [k] rt_cache_valid
|
--0.81%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--0.81%--ip_route_input_rcu
rt_cache_valid
0.82% 0.82% ksoftirqd/43 [kernel.vmlinux] [k] rt_cache_valid
|
--0.82%--ret_from_fork
kthread
smpboot_thread_fn
run_ksoftirqd
__softirqentry_text_start
net_rx_action
ixgbe_poll
napi_gro_receive
netif_receive_skb_internal
__netif_receive_skb
__netif_receive_skb_core
ip_rcv
ip_rcv_finish
ip_route_input_noref
|
--0.81%--ip_route_input_rcu
rt_cache_valid
0.65% 0.03% ksoftirqd/15 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.62%--sch_direct_xmit
lines 16916-17000\r 0.65% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.62%--sch_direct_xmit
0.64% 0.03% ksoftirqd/43 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.61%--sch_direct_xmit
0.64% 0.03% ksoftirqd/14 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.62%--sch_direct_xmit
0.60% 0.03% ksoftirqd/20 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.58%--sch_direct_xmit
0.60% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.57%--sch_direct_xmit
0.59% 0.03% ksoftirqd/19 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.57%--sch_direct_xmit
0.57% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.55%--sch_direct_xmit
0.57% 0.03% ksoftirqd/17 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.54%--sch_direct_xmit
0.56% 0.03% ksoftirqd/23 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.53%--sch_direct_xmit
0.55% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.53%--sch_direct_xmit
0.55% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.53%--sch_direct_xmit
0.54% 0.03% ksoftirqd/25 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.51%--sch_direct_xmit
0.54% 0.03% ksoftirqd/26 [kernel.vmlinux] [k] sch_direct_xmit
|
--0.51%--sch_direct_xmit
0.52% 0.03% ksoftirqd/18 [kernel.vmlinux] [k] sch_direct_xmit
0.49% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] sch_direct_xmit
0.40% 0.40% ksoftirqd/27 [kernel.vmlinux] [k] do_raw_spin_lock
0.40% 0.40% ksoftirqd/19 [kernel.vmlinux] [k] do_raw_spin_lock
0.39% 0.39% ksoftirqd/20 [kernel.vmlinux] [k] do_raw_spin_lock
0.38% 0.38% ksoftirqd/22 [kernel.vmlinux] [k] do_raw_spin_lock
0.38% 0.38% ksoftirqd/17 [kernel.vmlinux] [k] do_raw_spin_lock
0.38% 0.38% ksoftirqd/43 [kernel.vmlinux] [k] do_raw_spin_lock
0.38% 0.38% ksoftirqd/23 [kernel.vmlinux] [k] do_raw_spin_lock
0.38% 0.38% ksoftirqd/42 [kernel.vmlinux] [k] do_raw_spin_lock
0.36% 0.36% ksoftirqd/21 [kernel.vmlinux] [k] do_raw_spin_lock
0.36% 0.36% ksoftirqd/24 [kernel.vmlinux] [k] do_raw_spin_lock
0.36% 0.36% ksoftirqd/14 [kernel.vmlinux] [k] do_raw_spin_lock
0.35% 0.35% ksoftirqd/15 [kernel.vmlinux] [k] do_raw_spin_lock
0.35% 0.35% ksoftirqd/25 [kernel.vmlinux] [k] do_raw_spin_lock
0.34% 0.34% ksoftirqd/26 [kernel.vmlinux] [k] do_raw_spin_lock
0.34% 0.34% ksoftirqd/18 [kernel.vmlinux] [k] do_raw_spin_lock
0.31% 0.31% ksoftirqd/16 [kernel.vmlinux] [k] do_raw_spin_lock
0.24% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] neigh_resolve_output
0.24% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] neigh_resolve_output
0.24% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] neigh_resolve_output
0.24% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] neigh_resolve_output
0.24% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] neigh_resolve_output
0.24% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] neigh_resolve_output
0.24% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] neigh_resolve_output
0.23% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] neigh_resolve_output
lines 17001-17085\r 0.22% 0.02% swapper [kernel.vmlinux] [k] apic_timer_interrupt
0.21% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.21% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.20% 0.00% swapper [kernel.vmlinux] [k] smp_apic_timer_interrupt
0.20% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] neigh_resolve_output
0.20% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.20% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.20% 0.00% perf perf [.] 0xffffffffffc5e1b1
0.20% 0.00% perf perf [.] 0xffffffffffc08f7a
0.20% 0.00% perf libc-2.23.so [.] __libc_start_main
0.20% 0.00% perf [unknown] [k] 0x1fae258d4c544155
0.19% 0.00% perf [kernel.vmlinux] [k] entry_SYSCALL_64_fastpath
0.19% 0.19% ksoftirqd/14 [kernel.vmlinux] [k] fib_table_lookup
0.19% 0.19% ksoftirqd/15 [kernel.vmlinux] [k] fib_table_lookup
0.19% 0.19% ksoftirqd/27 [kernel.vmlinux] [k] fib_table_lookup
0.19% 0.19% ksoftirqd/42 [kernel.vmlinux] [k] fib_table_lookup
0.19% 0.19% ksoftirqd/43 [kernel.vmlinux] [k] fib_table_lookup
0.18% 0.00% perf libpthread-2.23.so [.] __write_nocancel
0.18% 0.00% perf [kernel.vmlinux] [k] sys_write
0.18% 0.00% perf [kernel.vmlinux] [k] vfs_write
0.18% 0.18% ksoftirqd/21 [kernel.vmlinux] [k] fib_table_lookup
0.18% 0.13% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.18% 0.00% perf [kernel.vmlinux] [k] __vfs_write
0.18% 0.00% perf [kernel.vmlinux] [k] generic_file_write_iter
0.18% 0.00% perf [kernel.vmlinux] [k] __generic_file_write_iter
0.18% 0.00% perf [kernel.vmlinux] [k] generic_perform_write
0.18% 0.00% perf perf [.] 0xffffffffffc16f5e
0.18% 0.18% ksoftirqd/19 [kernel.vmlinux] [k] fib_table_lookup
0.18% 0.18% ksoftirqd/26 [kernel.vmlinux] [k] fib_table_lookup
0.18% 0.12% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.18% 0.18% ksoftirqd/18 [kernel.vmlinux] [k] fib_table_lookup
0.18% 0.18% ksoftirqd/20 [kernel.vmlinux] [k] fib_table_lookup
0.17% 0.17% ksoftirqd/17 [kernel.vmlinux] [k] fib_table_lookup
0.17% 0.17% ksoftirqd/25 [kernel.vmlinux] [k] fib_table_lookup
0.17% 0.17% ksoftirqd/24 [kernel.vmlinux] [k] fib_table_lookup
0.17% 0.00% perf perf [.] 0xffffffffffc159e7
0.17% 0.11% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.17% 0.17% ksoftirqd/23 [kernel.vmlinux] [k] fib_table_lookup
0.16% 0.16% ksoftirqd/43 [kernel.vmlinux] [k] eth_type_trans
0.16% 0.16% ksoftirqd/22 [kernel.vmlinux] [k] fib_table_lookup
0.16% 0.11% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.16% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] build_skb
0.16% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] build_skb
0.16% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] build_skb
0.16% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] build_skb
0.16% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] napi_consume_skb
0.16% 0.16% ksoftirqd/16 [kernel.vmlinux] [k] fib_table_lookup
0.15% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] napi_consume_skb
0.15% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] napi_consume_skb
0.15% 0.15% ksoftirqd/15 [kernel.vmlinux] [k] eth_type_trans
0.15% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] napi_consume_skb
0.15% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] _kfree_skb_defer
0.15% 0.08% ksoftirqd/42 [kernel.vmlinux] [k] __build_skb
0.15% 0.15% ksoftirqd/14 [kernel.vmlinux] [k] eth_type_trans
0.15% 0.08% ksoftirqd/14 [kernel.vmlinux] [k] __build_skb
0.15% 0.08% ksoftirqd/15 [kernel.vmlinux] [k] __build_skb
0.15% 0.08% ksoftirqd/43 [kernel.vmlinux] [k] __build_skb
0.14% 0.14% ksoftirqd/42 [kernel.vmlinux] [k] eth_type_trans
0.14% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] build_skb
0.14% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] _kfree_skb_defer
0.14% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] _kfree_skb_defer
0.14% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] build_skb
0.14% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] _kfree_skb_defer
0.14% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] build_skb
0.14% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] build_skb
0.14% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] build_skb
0.14% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] build_skb
0.14% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] build_skb
0.14% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] build_skb
0.14% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] fib_validate_source
0.13% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] build_skb
0.13% 0.08% ksoftirqd/20 [kernel.vmlinux] [k] __build_skb
0.13% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.13% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] fib_validate_source
0.13% 0.07% ksoftirqd/19 [kernel.vmlinux] [k] __build_skb
0.13% 0.13% ksoftirqd/20 [kernel.vmlinux] [k] eth_type_trans
0.13% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] build_skb
0.13% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] build_skb
0.13% 0.07% ksoftirqd/18 [kernel.vmlinux] [k] __build_skb
0.13% 0.13% ksoftirqd/26 [kernel.vmlinux] [k] eth_type_trans
0.13% 0.07% ksoftirqd/17 [kernel.vmlinux] [k] __build_skb
0.13% 0.03% ksoftirqd/15 [kernel.vmlinux] [k] validate_xmit_skb
0.13% 0.07% ksoftirqd/25 [kernel.vmlinux] [k] __build_skb
0.13% 0.07% ksoftirqd/27 [kernel.vmlinux] [k] __build_skb
0.12% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] skb_release_all
lines 17086-17170\r 0.12% 0.12% ksoftirqd/27 [kernel.vmlinux] [k] eth_type_trans
0.12% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] fib_validate_source
0.12% 0.07% ksoftirqd/23 [kernel.vmlinux] [k] __build_skb
0.12% 0.07% ksoftirqd/22 [kernel.vmlinux] [k] __build_skb
0.12% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] fib_validate_source
0.12% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] fib_validate_source
0.12% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] fib_validate_source
0.12% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] build_skb
0.12% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.12% 0.03% ksoftirqd/43 [kernel.vmlinux] [k] validate_xmit_skb
0.12% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] validate_xmit_skb
0.12% 0.03% ksoftirqd/14 [kernel.vmlinux] [k] validate_xmit_skb
0.12% 0.12% ksoftirqd/24 [kernel.vmlinux] [k] eth_type_trans
0.12% 0.07% ksoftirqd/21 [kernel.vmlinux] [k] __build_skb
0.12% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] skb_release_all
0.12% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.12% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] fib_validate_source
0.12% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] fib_validate_source
0.12% 0.12% ksoftirqd/17 [kernel.vmlinux] [k] eth_type_trans
0.12% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.12% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] skb_release_all
0.12% 0.02% ksoftirqd/19 [kernel.vmlinux] [k] fib_validate_source
0.12% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.12% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.12% 0.12% ksoftirqd/23 [kernel.vmlinux] [k] eth_type_trans
0.12% 0.12% ksoftirqd/19 [kernel.vmlinux] [k] eth_type_trans
0.12% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.12% 0.07% ksoftirqd/26 [kernel.vmlinux] [k] __build_skb
0.12% 0.00% swapper [kernel.vmlinux] [k] local_apic_timer_interrupt
0.12% 0.07% ksoftirqd/24 [kernel.vmlinux] [k] __build_skb
0.12% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.12% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] skb_release_all
0.12% 0.08% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.12% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.12% 0.12% ksoftirqd/18 [kernel.vmlinux] [k] eth_type_trans
0.12% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.11% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] fib_validate_source
0.11% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.11% 0.04% ksoftirqd/42 [kernel.vmlinux] [k] skb_release_data
0.11% 0.00% swapper [kernel.vmlinux] [k] hrtimer_interrupt
0.11% 0.03% ksoftirqd/14 [kernel.vmlinux] [k] nf_hook_slow
0.11% 0.03% ksoftirqd/43 [kernel.vmlinux] [k] nf_hook_slow
0.11% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] fib_validate_source
0.11% 0.11% ksoftirqd/22 [kernel.vmlinux] [k] eth_type_trans
0.11% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] fib_validate_source
0.11% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] nf_hook_slow
0.11% 0.06% ksoftirqd/16 [kernel.vmlinux] [k] __build_skb
0.11% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.11% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] napi_consume_skb
0.11% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] fib_validate_source
0.11% 0.02% ksoftirqd/23 [kernel.vmlinux] [k] fib_validate_source
0.11% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_xmit_frame
0.11% 0.11% ksoftirqd/25 [kernel.vmlinux] [k] eth_type_trans
0.11% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] napi_consume_skb
0.11% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.11% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] nf_hook_slow
0.11% 0.04% ksoftirqd/14 [kernel.vmlinux] [k] skb_release_data
0.11% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.11% 0.11% ksoftirqd/21 [kernel.vmlinux] [k] eth_type_trans
0.11% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] napi_consume_skb
0.11% 0.04% ksoftirqd/43 [kernel.vmlinux] [k] skb_release_data
0.11% 0.08% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.11% 0.08% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.11% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] napi_consume_skb
0.11% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.11% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] _kfree_skb_defer
0.11% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] napi_consume_skb
0.11% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] napi_consume_skb
0.11% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] napi_consume_skb
0.11% 0.04% ksoftirqd/15 [kernel.vmlinux] [k] skb_release_data
0.11% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] fib_validate_source
0.10% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] napi_consume_skb
0.10% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.08% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] validate_xmit_skb
0.10% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] fib_validate_source
0.10% 0.07% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] napi_consume_skb
0.10% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] fib_lookup.constprop.29
lines 17171-17255\r 0.10% 0.07% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.08% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.07% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.07% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.07% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] napi_consume_skb
0.10% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.10% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] napi_consume_skb
0.10% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] validate_xmit_skb
0.10% 0.07% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] validate_xmit_skb
0.10% 0.07% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_xmit_frame_ring
0.10% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] validate_xmit_skb
0.10% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.10% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] _kfree_skb_defer
0.10% 0.02% ksoftirqd/19 [kernel.vmlinux] [k] validate_xmit_skb
0.10% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] validate_xmit_skb
0.09% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] napi_consume_skb
0.09% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.09% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.09% 0.02% ksoftirqd/23 [kernel.vmlinux] [k] validate_xmit_skb
0.09% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] validate_xmit_skb
0.09% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] validate_xmit_skb
0.09% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] _kfree_skb_defer
0.09% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] iptable_filter_hook
0.09% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] _kfree_skb_defer
0.09% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.09% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] validate_xmit_skb
0.09% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] validate_xmit_skb
0.09% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.09% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] iptable_filter_hook
0.09% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] fib_lookup.constprop.29
0.09% 0.06% ksoftirqd/43 [kernel.vmlinux] [k] netif_skb_features
0.09% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] iptable_filter_hook
0.09% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] iptable_filter_hook
0.09% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] _kfree_skb_defer
0.09% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.09% 0.06% ksoftirqd/15 [kernel.vmlinux] [k] netif_skb_features
0.09% 0.06% ksoftirqd/14 [kernel.vmlinux] [k] netif_skb_features
0.08% 0.06% ksoftirqd/42 [kernel.vmlinux] [k] netif_skb_features
0.08% 0.00% swapper [kernel.vmlinux] [k] __hrtimer_run_queues
0.08% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] validate_xmit_skb
0.08% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] validate_xmit_skb_list
0.08% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] validate_xmit_skb_list
0.08% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.07% ksoftirqd/14 [kernel.vmlinux] [k] ipt_do_table
0.08% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.00% perf [kernel.vmlinux] [k] reiserfs_write_end
0.08% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] validate_xmit_skb_list
0.08% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.07% ksoftirqd/42 [kernel.vmlinux] [k] ipt_do_table
0.08% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] validate_xmit_skb_list
0.08% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] skb_release_all
0.08% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.07% ksoftirqd/43 [kernel.vmlinux] [k] ipt_do_table
0.08% 0.05% ksoftirqd/20 [kernel.vmlinux] [k] netif_skb_features
0.08% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] skb_release_all
0.08% 0.08% ksoftirqd/16 [kernel.vmlinux] [k] eth_type_trans
0.08% 0.07% ksoftirqd/15 [kernel.vmlinux] [k] ipt_do_table
0.08% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] skb_release_all
0.08% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] skb_release_all
0.08% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.08% 0.00% swapper [kernel.vmlinux] [k] irq_exit
0.08% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] skb_release_all
0.08% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.07% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] skb_release_all
0.07% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.07% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] skb_release_all
0.07% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] skb_release_all
0.07% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] skb_release_all
0.07% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] fib_lookup.constprop.49
0.07% 0.05% ksoftirqd/19 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.05% ksoftirqd/24 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] skb_release_all
lines 17256-17340\r 0.07% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] skb_release_all
0.07% 0.05% ksoftirqd/27 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.00% swapper [kernel.vmlinux] [k] tick_sched_timer
0.07% 0.05% ksoftirqd/23 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.00% swapper [kernel.vmlinux] [k] __softirqentry_text_start
0.07% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_free_head
0.07% 0.05% ksoftirqd/26 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.05% ksoftirqd/17 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.05% ksoftirqd/21 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.04% ksoftirqd/15 [kernel.vmlinux] [k] kmem_cache_alloc
0.07% 0.05% ksoftirqd/18 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.07% ksoftirqd/42 [kernel.vmlinux] [k] page_frag_free
0.07% 0.05% ksoftirqd/25 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.05% ksoftirqd/22 [kernel.vmlinux] [k] netif_skb_features
0.07% 0.03% ksoftirqd/27 [kernel.vmlinux] [k] skb_release_data
0.07% 0.03% ksoftirqd/17 [kernel.vmlinux] [k] skb_release_data
0.07% 0.03% ksoftirqd/19 [kernel.vmlinux] [k] skb_release_data
0.07% 0.03% ksoftirqd/24 [kernel.vmlinux] [k] skb_release_data
0.07% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] skb_release_all
0.07% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] kmem_cache_alloc
0.07% 0.03% ksoftirqd/23 [kernel.vmlinux] [k] skb_release_data
0.07% 0.00% perf [kernel.vmlinux] [k] reiserfs_write_begin
0.07% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_free_head
0.07% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_free_head
0.07% 0.06% ksoftirqd/43 [kernel.vmlinux] [k] page_frag_free
0.07% 0.04% ksoftirqd/14 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.03% ksoftirqd/26 [kernel.vmlinux] [k] skb_release_data
0.06% 0.04% ksoftirqd/43 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.03% ksoftirqd/20 [kernel.vmlinux] [k] skb_release_data
0.06% 0.06% ksoftirqd/14 [kernel.vmlinux] [k] page_frag_free
0.06% 0.05% ksoftirqd/16 [kernel.vmlinux] [k] netif_skb_features
0.06% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] skb_free_head
0.06% 0.03% ksoftirqd/21 [kernel.vmlinux] [k] skb_release_data
0.06% 0.03% ksoftirqd/18 [kernel.vmlinux] [k] skb_release_data
0.06% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.03% ksoftirqd/25 [kernel.vmlinux] [k] skb_release_data
0.06% 0.03% ksoftirqd/19 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.06% ksoftirqd/15 [kernel.vmlinux] [k] page_frag_free
0.06% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.01% swapper [kernel.vmlinux] [k] run_rebalance_domains
0.06% 0.03% ksoftirqd/22 [kernel.vmlinux] [k] skb_release_data
0.06% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.03% ksoftirqd/17 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.00% perf [kernel.vmlinux] [k] reiserfs_update_sd_size
0.06% 0.03% ksoftirqd/27 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.00% swapper [kernel.vmlinux] [k] update_process_times
0.06% 0.02% perf [kernel.vmlinux] [k] search_by_key
0.06% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.03% ksoftirqd/25 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.03% ksoftirqd/22 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.03% ksoftirqd/20 [kernel.vmlinux] [k] kmem_cache_alloc
0.06% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.03% ksoftirqd/16 [kernel.vmlinux] [k] skb_release_data
0.06% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] validate_xmit_skb_list
0.06% 0.00% perf [kernel.vmlinux] [k] __block_write_begin
0.06% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] nf_hook_slow
0.06% 0.00% perf [kernel.vmlinux] [k] __block_write_begin_int
0.06% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] validate_xmit_skb_list
0.05% 0.03% ksoftirqd/21 [kernel.vmlinux] [k] kmem_cache_alloc
0.05% 0.03% ksoftirqd/23 [kernel.vmlinux] [k] kmem_cache_alloc
0.05% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] validate_xmit_skb_list
0.05% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] nf_hook_slow
0.05% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] nf_hook_slow
0.05% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] nf_hook_slow
0.05% 0.03% ksoftirqd/24 [kernel.vmlinux] [k] kmem_cache_alloc
0.05% 0.03% ksoftirqd/18 [kernel.vmlinux] [k] kmem_cache_alloc
0.05% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] iptable_filter_hook
0.05% 0.03% ksoftirqd/26 [kernel.vmlinux] [k] kmem_cache_alloc
0.05% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] iptable_filter_hook
0.05% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] kmem_cache_alloc
0.05% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] iptable_filter_hook
0.05% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] nf_hook_slow
0.05% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] iptable_filter_hook
0.05% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] validate_xmit_skb_list
0.05% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] iptable_filter_hook
lines 17341-17425\r 0.05% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] iptable_filter_hook
0.05% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] iptable_filter_hook
0.05% 0.04% ksoftirqd/20 [kernel.vmlinux] [k] ipt_do_table
0.05% 0.00% swapper [kernel.vmlinux] [k] rest_init
0.05% 0.00% swapper [kernel.vmlinux].init.text [k] start_kernel
0.05% 0.00% swapper [kernel.vmlinux].init.text [k] x86_64_start_reservations
0.05% 0.00% swapper [kernel.vmlinux].init.text [k] x86_64_start_kernel
0.05% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] iptable_filter_hook
0.05% 0.04% ksoftirqd/27 [kernel.vmlinux] [k] ipt_do_table
0.05% 0.04% ksoftirqd/26 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] iptable_filter_hook
0.04% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] iptable_filter_hook
0.04% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] iptable_filter_hook
0.04% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] iptable_filter_hook
0.04% 0.04% ksoftirqd/24 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.04% ksoftirqd/23 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.00% perf [kernel.vmlinux] [k] reiserfs_get_block
0.04% 0.04% ksoftirqd/19 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.03% ksoftirqd/43 [kernel.vmlinux] [k] udp_v4_early_demux
0.04% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] udp_v4_early_demux
0.04% 0.03% ksoftirqd/14 [kernel.vmlinux] [k] udp_v4_early_demux
0.04% 0.04% ksoftirqd/22 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.03% ksoftirqd/25 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.03% ksoftirqd/15 [kernel.vmlinux] [k] udp_v4_early_demux
0.04% 0.03% ksoftirqd/18 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.03% ksoftirqd/16 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.03% ksoftirqd/21 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.03% ksoftirqd/18 [kernel.vmlinux] [k] udp_v4_early_demux
0.04% 0.03% ksoftirqd/19 [kernel.vmlinux] [k] udp_v4_early_demux
0.04% 0.03% ksoftirqd/17 [kernel.vmlinux] [k] ipt_do_table
0.04% 0.03% ksoftirqd/17 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.03% ksoftirqd/27 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.03% ksoftirqd/23 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.00% perf [kernel.vmlinux] [k] __mark_inode_dirty
0.03% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] udp_v4_early_demux
0.03% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.00% perf [kernel.vmlinux] [k] reiserfs_dirty_inode
0.03% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.01% swapper [kernel.vmlinux] [k] load_balance
0.03% 0.02% ksoftirqd/19 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.03% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.03% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_free_head
0.03% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.03% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] __local_bh_enable_ip
0.03% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.02% ksoftirqd/23 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.03% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.03% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_free_head
0.03% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.03% ksoftirqd/14 [kernel.vmlinux] [k] __local_bh_enable_ip
0.03% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_free_head
0.03% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_free_head
0.03% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_free_head
0.03% 0.00% perf [kernel.vmlinux] [k] reiserfs_update_sd
0.03% 0.03% ksoftirqd/19 [kernel.vmlinux] [k] page_frag_free
0.03% 0.02% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.03% ksoftirqd/23 [kernel.vmlinux] [k] page_frag_free
0.03% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.03% ksoftirqd/15 [kernel.vmlinux] [k] __local_bh_enable_ip
0.03% 0.03% ksoftirqd/42 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.03% 0.03% ksoftirqd/26 [kernel.vmlinux] [k] page_frag_free
0.03% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_free_head
lines 17426-17510\r 0.03% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.03% 0.03% ksoftirqd/43 [kernel.vmlinux] [k] __local_bh_enable_ip
0.03% 0.03% ksoftirqd/24 [kernel.vmlinux] [k] page_frag_free
0.03% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.03% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_free_head
0.03% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_free_head
0.03% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] apic_timer_interrupt
0.03% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_free_head
0.03% 0.03% ksoftirqd/26 [kernel.vmlinux] [k] __local_bh_enable_ip
0.03% 0.03% ksoftirqd/15 [kernel.vmlinux] [k] get_dma_ops
0.03% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.03% 0.03% ksoftirqd/20 [kernel.vmlinux] [k] __local_bh_enable_ip
0.03% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.03% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] page_frag_free
0.03% 0.00% perf [kernel.vmlinux] [k] iov_iter_copy_from_user_atomic
0.03% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.03% ksoftirqd/43 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.03% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.03% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] page_frag_free
0.03% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.03% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] netdev_pick_tx
0.03% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.03% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.03% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.03% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.03% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_free_head
0.03% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.03% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_free_head
0.03% 0.03% ksoftirqd/24 [kernel.vmlinux] [k] __local_bh_enable_ip
0.03% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] page_frag_free
0.03% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] netdev_pick_tx
0.02% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_free_head
0.02% 0.02% perf [kernel.vmlinux] [k] copy_user_enhanced_fast_string
0.02% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.02% ksoftirqd/23 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.02% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.02% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.02% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] page_frag_free
0.02% 0.02% ksoftirqd/19 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] page_frag_free
0.02% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.02% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] netdev_pick_tx
0.02% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.00% perf [kernel.vmlinux] [k] reiserfs_paste_into_item
0.02% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] page_frag_free
0.02% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] get_dma_ops
0.02% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] page_frag_free
0.02% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] page_frag_free
0.02% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.02% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.02% swapper [kernel.vmlinux] [k] find_busiest_group
0.02% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] kmem_cache_free_bulk
0.02% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.02% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] vlan_passthru_hard_header
lines 17511-17595\r 0.02% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.02% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] get_dma_ops
0.02% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] dev_hard_header.constprop.54
0.02% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] vlan_passthru_hard_header
0.02% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_alloc_rx_buffers
0.02% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] __local_bh_enable_ip
0.02% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.00% swapper [kernel.vmlinux] [k] scheduler_tick
0.02% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.02% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] get_dma_ops
0.02% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] eth_header
0.02% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __slab_alloc
0.02% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] smp_apic_timer_interrupt
0.02% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.02% ksoftirqd/43 [kernel.vmlinux] [k] eth_header
0.02% 0.00% kworker/u114:0 [kernel.vmlinux] [k] wb_workfn
0.02% 0.00% kworker/u114:0 [kernel.vmlinux] [k] process_one_work
0.02% 0.00% kworker/u114:0 [kernel.vmlinux] [k] worker_thread
0.02% 0.00% kworker/u114:0 [kernel.vmlinux] [k] kthread
0.02% 0.00% kworker/u114:0 [kernel.vmlinux] [k] ret_from_fork
0.02% 0.00% kworker/u114:0 [kernel.vmlinux] [k] wb_writeback
0.02% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.02% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] eth_header
0.02% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] eth_header
0.02% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] eth_header
0.02% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] eth_header
0.02% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] hrtimer_interrupt
0.02% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.02% ksoftirqd/19 [kernel.vmlinux] [k] get_dma_ops
0.02% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.02% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.02% ksoftirqd/18 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] swiotlb_map_page
0.02% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] get_dma_ops
0.02% 0.00% swapper [kernel.vmlinux] [k] scsi_softirq_done
0.02% 0.00% swapper [kernel.vmlinux] [k] scsi_finish_command
0.02% 0.02% ksoftirqd/19 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.00% swapper [kernel.vmlinux] [k] scsi_io_completion
0.02% 0.00% swapper [kernel.vmlinux] [k] scsi_end_request
0.02% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.02% ksoftirqd/26 [kernel.vmlinux] [k] eth_header
0.02% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.02% ksoftirqd/17 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] local_apic_timer_interrupt
0.02% 0.02% ksoftirqd/25 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.02% ksoftirqd/23 [kernel.vmlinux] [k] eth_header
0.02% 0.02% ksoftirqd/20 [kernel.vmlinux] [k] eth_header
lines 17596-17680\r 0.02% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.02% ksoftirqd/15 [kernel.vmlinux] [k] eth_header
0.02% 0.02% ksoftirqd/16 [kernel.vmlinux] [k] eth_header
0.02% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.00% swapper [kernel.vmlinux] [k] blk_update_request
0.02% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.01% swapper [kernel.vmlinux] [k] sched_clock_cpu
0.02% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.02% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.02% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.02% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.02% ksoftirqd/24 [kernel.vmlinux] [k] eth_header
0.02% 0.02% ksoftirqd/27 [kernel.vmlinux] [k] eth_header
0.02% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.02% ksoftirqd/22 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.02% ksoftirqd/23 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.02% 0.02% ksoftirqd/21 [kernel.vmlinux] [k] eth_header
0.02% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ___slab_alloc
0.02% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __kfree_skb_flush
0.02% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] skb_network_protocol
0.02% 0.02% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] dma_map_single_attrs.constprop.113
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% swapper [kernel.vmlinux] [k] _raw_spin_trylock
0.01% 0.00% swapper [kernel.vmlinux] [k] tick_program_event
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] deliver_ptype_list_skb
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.00% swapper [kernel.vmlinux] [k] bio_endio
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] eth_header
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ___slab_alloc
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% perf [kernel.vmlinux] [k] ll_rw_block
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] eth_header
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.00% swapper [kernel.vmlinux] [k] raid1_end_write_request
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.00% perf [kernel.vmlinux] [k] do_balance
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] generic_writepages
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] do_writepages
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __writeback_single_inode
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] writeback_sb_inodes
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __writeback_inodes_wb
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] write_cache_pages
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __slab_free
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% perf [kernel.vmlinux] [k] __getblk_gfp
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] swiotlb_sync_single
lines 17681-17765\r 0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% perf [kernel.vmlinux] [k] unlock_buffer
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __slab_free
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% perf [kernel.vmlinux] [k] search_for_position_by_key
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% swapper [kernel.vmlinux] [k] rcu_check_callbacks
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] skb_network_protocol
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] get_dma_ops
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __slab_free
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.01% perf [kernel.vmlinux] [k] __find_get_block
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __slab_free
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __slab_free
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __writepage
0.01% 0.00% perf perf [.] 0xffffffffffc15bea
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] reiserfs_writepage
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.00% swapper [kernel.vmlinux] [k] call_function_single_interrupt
0.01% 0.00% swapper [kernel.vmlinux] [k] flush_smp_call_function_queue
0.01% 0.00% swapper [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.01% 0.00% swapper [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% swapper [kernel.vmlinux] [k] r1_bio_write_done
lines 17766-17850\r 0.01% 0.00% swapper [kernel.vmlinux] [k] __blk_mq_complete_request_remote
0.01% 0.00% swapper [kernel.vmlinux] [k] perf_event_task_tick
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] generic_make_request
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% swapper [kernel.vmlinux] [k] hrtimer_update_base
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% perf libc-2.23.so [.] poll
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] local_bh_enable
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __kfree_skb_flush
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __slab_free
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] do_IRQ
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% perf [kernel.vmlinux] [k] sys_poll
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% perf [kernel.vmlinux] [k] do_sys_poll
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.00% swapper [kernel.vmlinux] [k] ktime_get
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] __netdev_pick_tx
lines 17851-17935\r 0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __hrtimer_run_queues
0.01% 0.00% swapper [kernel.vmlinux] [k] raid_end_bio_io
0.01% 0.00% perf [kernel.vmlinux] [k] grab_cache_page_write_begin
0.01% 0.01% perf [kernel.vmlinux] [k] reiserfs_commit_page
0.01% 0.00% perf perf [.] 0xffffffffffc17306
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] do_IRQ
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] handle_irq
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] handle_edge_irq
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% swapper [kernel.vmlinux] [k] ktime_get_update_offsets_now
0.01% 0.00% perf [kernel.vmlinux] [k] pagecache_get_page
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] swiotlb_map_page
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.00% perf [kernel.vmlinux] [k] reiserfs_prepare_for_journal
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] do_IRQ
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] get_partial_node.isra.78
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
lines 17936-18020\r 0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] handle_irq
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% swapper [kernel.vmlinux] [k] ret_from_intr
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] do_IRQ
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __slab_free
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] dma_sync_single_range_for_cpu.constprop.121
0.01% 0.00% perf [kernel.vmlinux] [k] fix_nodes
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] tick_sched_timer
0.01% 0.00% swapper [kernel.vmlinux] [k] call_bio_endio
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] handle_edge_irq
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% perf [kernel.vmlinux] [k] clean_bdev_aliases
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] update_process_times
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% perf [kernel.vmlinux] [k] pagevec_lookup_range
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] is_swiotlb_buffer
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] handle_irq
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.00% perf [kernel.vmlinux] [k] unfix_nodes
0.01% 0.00% perf [kernel.vmlinux] [k] find_get_pages_range
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] handle_edge_irq
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] handle_irq
0.01% 0.00% swapper [kernel.vmlinux] [k] timekeeping_get_ns
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] __jhash_nwords
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] dma_unmap_single_attrs.constprop.112
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] compound_head
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] handle_edge_irq
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] worker_thread
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] kthread
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] ret_from_fork
0.01% 0.00% swapper [kernel.vmlinux] [k] do_IRQ
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] do_journal_end
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] journal_end_sync
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] reiserfs_sync_fs
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] flush_old_commits
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] process_one_work
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] submit_bh_wbc.isra.43
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] submit_bh
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
lines 18021-18105\r 0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] skb_unref
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] flush_commit_list
0.01% 0.01% perf [kernel.vmlinux] [k] radix_tree_next_chunk
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] write_ordered_buffers.isra.19
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] skb_unref
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] dst_metric.constprop.59
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] swiotlb_sync_single
0.01% 0.00% swapper [kernel.vmlinux] [k] update_blocked_averages
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] eth_type_vlan
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] skb_unref
0.01% 0.00% perf [kernel.vmlinux] [k] iov_iter_fault_in_readable
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_select_queue
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] compound_head
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] handle_irq_event
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] __skb_put
0.01% 0.00% swapper [kernel.vmlinux] [k] handle_edge_irq
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.00% swapper [kernel.vmlinux] [k] handle_irq
0.01% 0.01% swapper [kernel.vmlinux] [k] __read_seqcount_begin.constprop.20
0.01% 0.00% swapper [kernel.vmlinux] [k] handle_irq_event
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] skb_zcopy
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] submit_bio
0.01% 0.00% swapper [kernel.vmlinux] [k] handle_irq_event_percpu
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.01% 0.00% swapper [kernel.vmlinux] [k] __handle_irq_event_percpu
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_features_check
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] skb_unref
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] dst_metric.constprop.59
0.01% 0.00% swapper [kernel.vmlinux] [k] ahci_single_level_irq_intr
0.01% 0.01% perf [kernel.vmlinux] [k] fault_in_pages_readable
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] dev_gro_receive
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] write_ordered_chunk
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] compound_head
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] dst_metric.constprop.59
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] skb_zcopy
0.01% 0.00% perf [kernel.vmlinux] [k] perf_poll
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.00% swapper [kernel.vmlinux] [k] clockevents_program_event
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] __skb_put
0.01% 0.00% swapper [kernel.vmlinux] [k] ahci_handle_port_intr
lines 18106-18190\r 0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] submit_bh_wbc.isra.43
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] submit_bh
0.01% 0.01% perf [kernel.vmlinux] [k] atomic64_cmpxchg
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] do_IRQ
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] find_exception
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_features_check
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] __skb_put
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] handle_irq_event
0.01% 0.00% perf [kernel.vmlinux] [k] wake_up_bit
0.01% 0.00% swapper [kernel.vmlinux] [k] ahci_handle_port_interrupt
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] __netdev_pick_tx
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] handle_irq_event
0.01% 0.01% perf [kernel.vmlinux] [k] __wake_up_bit
0.01% 0.01% swapper [kernel.vmlinux] [k] do_raw_spin_lock
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] apic_timer_interrupt
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] do_IRQ
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] dst_metric.constprop.59
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] handle_irq_event_percpu
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] skb_zcopy
0.01% 0.00% perf [kernel.vmlinux] [k] journal_end
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.01% ksoftirqd/19 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] skb_zcopy
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] handle_irq_event
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.01% 0.00% swapper [kernel.vmlinux] [k] ata_qc_complete_multiple
0.01% 0.00% swapper [kernel.vmlinux] [k] ata_qc_complete
0.01% 0.00% swapper [kernel.vmlinux] [k] cpumask_next_and
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] compound_head
0.01% 0.01% ksoftirqd/16 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] swiotlb_unmap_page
0.01% 0.00% swapper [kernel.vmlinux] [k] __ata_qc_complete
0.01% 0.00% swapper [kernel.vmlinux] [k] ata_scsi_qc_complete
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] do_IRQ
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] do_IRQ
0.01% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.00% swapper [kernel.vmlinux] [k] native_sched_clock
0.01% 0.00% swapper [kernel.vmlinux] [k] ata_qc_done
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.00% kworker/u114:0 [kernel.vmlinux] [k] md_make_request
0.01% 0.00% swapper [kernel.vmlinux] [k] blk_mq_complete_request
0.01% 0.00% swapper [kernel.vmlinux] [k] scsi_mq_done
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] generic_make_request
0.01% 0.00% swapper [kernel.vmlinux] [k] __blk_mq_complete_request
0.01% 0.01% ksoftirqd/26 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] _raw_spin_lock
lines 18191-18275\r 0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.01% ksoftirqd/42 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] submit_bio
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_features_check
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] compound_head
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.01% ksoftirqd/20 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] handle_edge_irq
0.01% 0.01% swapper [kernel.vmlinux] [k] lapic_next_deadline
0.01% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] do_IRQ
0.01% 0.00% swapper [kernel.vmlinux] [k] perf_pmu_disable
0.01% 0.00% swapper [kernel.vmlinux] [k] end_bio_bh_io_sync
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] swiotlb_unmap_page
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/25 [kernel.vmlinux] [k] _raw_spin_lock
0.01% 0.01% ksoftirqd/22 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] do_IRQ
0.01% 0.01% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_features_check
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] handle_edge_irq
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] do_IRQ
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_features_check
0.01% 0.00% swapper [kernel.vmlinux] [k] __remove_hrtimer
0.01% 0.00% kworker/1:0 [kernel.vmlinux] [k] md_make_request
0.01% 0.00% swapper [kernel.vmlinux] [k] irq_enter
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] do_IRQ
0.01% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.01% ksoftirqd/18 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] swiotlb_unmap_page
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_maybe_stop_tx
0.01% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] handle_irq
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] handle_irq_event_percpu
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ret_from_intr
0.01% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.01% ksoftirqd/24 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.01% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] dma_mapping_error
0.01% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] handle_irq
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] put_cpu_partial
0.01% 0.01% ksoftirqd/23 [kernel.vmlinux] [k] __skb_put
0.01% 0.01% ksoftirqd/43 [kernel.vmlinux] [k] irq_entries_start
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] __skb_put
0.01% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.01% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] handle_irq_event_percpu
0.01% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.01% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_features_check
0.01% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] swiotlb_unmap_page
0.01% 0.01% ksoftirqd/21 [kernel.vmlinux] [k] compound_head
0.01% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_zcopy_clear
0.01% 0.01% ksoftirqd/15 [kernel.vmlinux] [k] skb_gro_reset_offset
0.01% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] virt_to_head_page
0.01% 0.01% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] raid1_make_request
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] raid1_make_request
0.00% 0.00% swapper [kernel.vmlinux] [k] run_local_timers
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] do_IRQ
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_zcopy_clear
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] flush_bio_list
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] raid1_unplug
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_flush_plug_list
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_finish_plug
0.00% 0.00% perf [kernel.vmlinux] [k] mutex_lock
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] handle_edge_irq
lines 18276-18360\r 0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] dma_mapping_error
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_zcopy_clear
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] swiotlb_sync_single_for_cpu
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_zcopy_clear
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] irq_entries_start
0.00% 0.00% perf [kernel.vmlinux] [k] comp_short_keys
0.00% 0.00% swapper [kernel.vmlinux] [k] read_tsc
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] dma_mapping_error
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] _raw_spin_lock
0.00% 0.00% swapper [kernel.vmlinux] [k] x86_pmu_disable
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_zcopy_clear
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] dma_mapping_error
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ret_from_intr
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] dma_mapping_error
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% perf [kernel.vmlinux] [k] __brelse
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] do_IRQ
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ret_from_intr
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_zcopy_clear
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] hrtimer_update_base
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ktime_get_update_offsets_now
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] find_exception
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] raid1_write_request
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] do_IRQ
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] do_IRQ
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] put_cpu_partial
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% swapper [kernel.vmlinux] [k] timerqueue_del
0.00% 0.00% perf [kernel.vmlinux] [k] get_bh
0.00% 0.00% perf perf [.] 0xffffffffffc159ad
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ret_from_intr
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% perf [kernel.vmlinux] [k] balance_leaf
0.00% 0.00% perf [kernel.vmlinux] [k] do_journal_end
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] put_cpu_partial
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] dma_mapping_error
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] bstats_update
lines 18361-18445\r 0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ip_skb_dst_mtu
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] find_exception
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_eqs_enter_common
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] raid1_write_request
0.00% 0.00% swapper [kernel.vmlinux] [k] cpuidle_select
0.00% 0.00% swapper [kernel.vmlinux] [k] end_buffer_async_write
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_restore_prepared_buffer
0.00% 0.00% perf [kernel.vmlinux] [k] comp_keys
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_zcopy_clear
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] smp_apic_timer_interrupt
0.00% 0.00% swapper [kernel.vmlinux] [k] intel_pmu_disable_all
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] local_apic_timer_interrupt
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_gro_reset_offset
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% perf [kernel.vmlinux] [k] add_to_page_cache_lru
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] dma_mapping_error
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] compound_head
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% swapper [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] hrtimer_interrupt
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_tx_ctxtdesc
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] handle_irq
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% swapper [kernel.vmlinux] [k] tk_clock_read
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% swapper [kernel.vmlinux] [k] cpumask_next
0.00% 0.00% perf [kernel.vmlinux] [k] put_bh
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] raid1d
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] md_thread
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] kthread
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_release_head_state
lines 18446-18530\r 0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_gro_reset_offset
0.00% 0.00% swapper [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] swiotlb_unmap_page
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] skb_zcopy_clear
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% swapper [kernel.vmlinux] [k] run_posix_cpu_timers
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] dma_mapping_error
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_end_persistent_transaction
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] scheduler_tick
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% perf [kernel.vmlinux] [k] pathrelse
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_make_request
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] flush_pending_writes
0.00% 0.00% swapper [kernel.vmlinux] [k] mempool_free
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_push
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] flush_bio_list
0.00% 0.00% perf [kernel.vmlinux] [k] journal_begin
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_dynticks_eqs_exit
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_gro_reset_offset
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% swapper [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% perf [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% swapper [kernel.vmlinux] [k] update_rq_clock
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_mutex_lock_safe
0.00% 0.00% swapper [kernel.vmlinux] [k] find_next_bit
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_push
0.00% 0.00% perf [kernel.vmlinux] [k] lock_journal
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% perf [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% perf [kernel.vmlinux] [k] __page_cache_alloc
0.00% 0.00% perf [kernel.vmlinux] [k] journal_mark_dirty
lines 18531-18615\r 0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_features_check
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] generic_make_request
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_process_callbacks
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% perf [kernel.vmlinux] [k] radix_tree_descend
0.00% 0.00% swapper [kernel.vmlinux] [k] _find_next_bit
0.00% 0.00% swapper [kernel.vmlinux] [k] bio_put
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] find_exception
0.00% 0.00% swapper [kernel.vmlinux] [k] idle_cpu
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_dynticks_eqs_enter
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __hrtimer_run_queues
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_idle_exit
0.00% 0.00% perf [kernel.vmlinux] [k] trylock_buffer
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% perf [kernel.vmlinux] [k] mark_buffer_dirty
0.00% 0.00% perf [kernel.vmlinux] [k] mutex_unlock
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% swapper [kernel.vmlinux] [k] raise_softirq
0.00% 0.00% swapper [kernel.vmlinux] [k] bio_free
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] bstats_update
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] task_tick_fair
0.00% 0.00% perf [kernel.vmlinux] [k] leaf_paste_in_buffer
0.00% 0.00% perf [kernel.vmlinux] [k] _allocate_block
0.00% 0.00% swapper [kernel.vmlinux] [k] cycles_2_ns
0.00% 0.00% swapper [kernel.vmlinux] [k] calc_global_load_tick
0.00% 0.00% swapper [kernel.vmlinux] [k] __slab_free
0.00% 0.00% perf [kernel.vmlinux] [k] do_journal_begin_r
0.00% 0.00% swapper [kernel.vmlinux] [k] end_page_writeback
0.00% 0.00% swapper [kernel.vmlinux] [k] cpu_load_update_active
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __phys_addr_nodebug
lines 18616-18700\r 0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __read_seqcount_begin.constprop.20
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% swapper [kernel.vmlinux] [k] mempool_free_slab
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_irq_exit
0.00% 0.00% perf [kernel.vmlinux] [k] create_page_buffers
0.00% 0.00% swapper [kernel.vmlinux] [k] kmem_cache_free
0.00% 0.00% perf [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __skb_put
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] find_exception
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] skb_push
0.00% 0.00% swapper [kernel.vmlinux] [k] ladder_select_state
0.00% 0.00% swapper [kernel.vmlinux] [k] account_process_tick
0.00% 0.00% swapper [kernel.vmlinux] [k] test_clear_page_writeback
0.00% 0.00% swapper [kernel.vmlinux] [k] source_load
0.00% 0.00% swapper [kernel.vmlinux] [k] _allow_barrier
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] tick_sched_timer
0.00% 0.00% perf [kernel.vmlinux] [k] pathrelse_and_restore
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] xt_write_recseq_begin
0.00% 0.00% perf perf [.] 0xffffffffffc1591b
0.00% 0.00% perf perf [.] 0x000000000001591b
0.00% 0.00% swapper [kernel.vmlinux] [k] cpu_needs_another_gp
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_make_request
0.00% 0.00% perf [kernel.vmlinux] [k] inode2sd
0.00% 0.00% snmpd [kernel.vmlinux] [k] entry_SYSCALL_64_fastpath
0.00% 0.00% perf [kernel.vmlinux] [k] create_empty_buffers
0.00% 0.00% perf [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] update_process_times
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] swiotlb_dma_mapping_error
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] rcu_check_callbacks
0.00% 0.00% perf [kernel.vmlinux] [k] do_raw_spin_lock
0.00% 0.00% swapper [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% swapper [kernel.vmlinux] [k] __wake_up
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_allocate_blocknrs
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] dst_metric.constprop.59
lines 18701-18785\r 0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% swapper [kernel.vmlinux] [k] sched_clock_idle_wakeup_event
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% swapper [kernel.vmlinux] [k] pm_qos_request
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mempool_alloc
0.00% 0.00% swapper [kernel.vmlinux] [k] enqueue_hrtimer
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% swapper [kernel.vmlinux] [k] rb_next
0.00% 0.00% swapper [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% perf [kernel.vmlinux] [k] __add_to_page_cache_locked
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] skb_unref
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% perf [kernel.vmlinux] [k] lru_cache_add
0.00% 0.00% swapper [kernel.vmlinux] [k] rb_erase
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_clone_fast
0.00% 0.00% swapper [kernel.vmlinux] [k] trigger_load_balance
0.00% 0.00% perf [kernel.vmlinux] [k] __lru_cache_add
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __test_set_page_writeback
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_work_tick
0.00% 0.00% perf [kernel.vmlinux] [k] trylock_buffer
0.00% 0.00% perf [kernel.vmlinux] [k] scan_bitmap_block.constprop.12
0.00% 0.00% perf [kernel.vmlinux] [k] zero_user_segments
0.00% 0.00% perf [kernel.vmlinux] [k] __pollwait
0.00% 0.00% perf [kernel.vmlinux] [k] touch_buffer
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] skb_put_padto
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% swapper [kernel.vmlinux] [k] tick_check_oneshot_broadcast_this_cpu
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_persistent_transaction
0.00% 0.00% perf [kernel.vmlinux] [k] pagevec_lru_move_fn
0.00% 0.00% perf [kernel.vmlinux] [k] __pagevec_lru_add
lines 18786-18870\r 0.00% 0.00% swapper [kernel.vmlinux] [k] tsc_verify_tsc_adjust
0.00% 0.00% swapper [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% perf [kernel.vmlinux] [k] __set_page_dirty
0.00% 0.00% perf [kernel.vmlinux] [k] to_fake_used_blocks
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_attempt_plug_merge
0.00% 0.00% swapper [kernel.vmlinux] [k] _local_bh_enable
0.00% 0.00% snmpd [kernel.vmlinux] [k] seq_read
0.00% 0.00% snmpd [kernel.vmlinux] [k] __vfs_read
0.00% 0.00% snmpd [kernel.vmlinux] [k] vfs_read
0.00% 0.00% snmpd [kernel.vmlinux] [k] sys_read
0.00% 0.00% snmpd libc-2.23.so [.] read
0.00% 0.00% snmpd [kernel.vmlinux] [k] proc_reg_read
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_alloc_bioset
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] dst_metric.constprop.59
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] account_process_tick
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% perf [kernel.vmlinux] [k] alloc_page_buffers
0.00% 0.00% swapper [kernel.vmlinux] [k] __wake_up_common
0.00% 0.00% swapper [kernel.vmlinux] [k] cpuidle_reflect
0.00% 0.00% swapper [kernel.vmlinux] [k] set_ti_thread_flag
0.00% 0.00% perf [kernel.vmlinux] [k] add_wait_queue
0.00% 0.00% swapper [kernel.vmlinux] [k] task_tick_idle
0.00% 0.00% swapper [kernel.vmlinux] [k] update_group_capacity
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_idle_enter
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_write_lock_nested
0.00% 0.00% perf [kernel.vmlinux] [k] radix_tree_lookup_slot
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] skb_zcopy
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% perf [kernel.vmlinux] [k] find_get_entry
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] default_wake_function
0.00% 0.00% swapper [kernel.vmlinux] [k] autoremove_wake_function
0.00% 0.00% swapper [kernel.vmlinux] [k] arch_cpu_idle_enter
0.00% 0.00% perf [kernel.vmlinux] [k] poll_freewait
0.00% 0.00% perf [kernel.vmlinux] [k] do_balance_mark_leaf_dirty
0.00% 0.00% swapper [kernel.vmlinux] [k] current_clr_polling_and_test
0.00% 0.00% swapper [kernel.vmlinux] [k] cyc2ns_read_begin
0.00% 0.00% perf [kernel.vmlinux] [k] __memmove
0.00% 0.00% perf [kernel.vmlinux] [k] alloc_buffer_head
0.00% 0.00% perf [kernel.vmlinux] [k] page_fault
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] skb_push
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] account_system_time
0.00% 0.00% perf [kernel.vmlinux] [k] inode_get_bytes
0.00% 0.00% perf [kernel.vmlinux] [k] get_journal_hash_dev
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_attempt_back_merge
0.00% 0.00% swapper [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_sched_dispatch_requests
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __blk_mq_run_hw_queue
0.00% 0.00% swapper [kernel.vmlinux] [k] schedule_idle
0.00% 0.00% swapper [kernel.vmlinux] [k] close_write
0.00% 0.00% perf [kernel.vmlinux] [k] kmem_cache_alloc
0.00% 0.00% swapper [kernel.vmlinux] [k] clear_ti_thread_flag.constprop.8
lines 18871-18955\r 0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __slab_alloc
0.00% 0.00% perf [kernel.vmlinux] [k] le_key_version
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] mempool_alloc
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_rx_pg_order
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] xt_write_recseq_end
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] udp4_portaddr_hash.isra.30
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_dispatch_rq_list
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] update_load_avg
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% perf [kernel.vmlinux] [k] page_cache_tree_insert
0.00% 0.00% swapper [kernel.vmlinux] [k] __local_bh_enable
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_cpu_idle_rcuidle
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_run_work_fn
0.00% 0.00% swapper [kernel.vmlinux] [k] task_cputime_zero
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% swapper [kernel.vmlinux] [k] update_wall_time
0.00% 0.00% swapper [kernel.vmlinux] [k] _raw_spin_lock
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mempool_alloc_slab
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] journal_end_sync
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] reiserfs_sync_fs
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] flush_old_commits
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] do_journal_end
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] ___slab_alloc
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_in_journal
0.00% 0.00% swapper [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bio_alloc_bioset
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] flush_commit_list
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] prefetchw
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_segcblist_ready_cbs
0.00% 0.00% swapper [kernel.vmlinux] [k] profile_tick
0.00% 0.00% perf [kernel.vmlinux] [k] mark_page_accessed
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_queue_rq
0.00% 0.00% swapper [kernel.vmlinux] [k] ttwu_do_activate.isra.85
0.00% 0.00% swapper [kernel.vmlinux] [k] update_cfs_rq_load_avg
0.00% 0.00% swapper [kernel.vmlinux] [k] sched_idle_set_state
0.00% 0.00% swapper [kernel.vmlinux] [k] timerqueue_add
0.00% 0.00% perf [kernel.vmlinux] [k] get_parents
0.00% 0.00% swapper [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% swapper [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% perf [kernel.vmlinux] [k] poll_schedule_timeout
0.00% 0.00% perf [kernel.vmlinux] [k] schedule_hrtimeout_range
0.00% 0.00% perf [kernel.vmlinux] [k] schedule_hrtimeout_range_clock
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_write_unlock
0.00% 0.00% swapper [kernel.vmlinux] [k] account_idle_time
0.00% 0.00% perf [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] write_ordered_buffers.isra.19
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bio_clone_fast
0.00% 0.00% perf [kernel.vmlinux] [k] brelse
0.00% 0.00% perf [kernel.vmlinux] [k] remove_wait_queue
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] mempool_alloc_slab
0.00% 0.00% perf [kernel.vmlinux] [k] __radix_tree_lookup
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] add_interrupt_randomness
lines 18956-19040\r 0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] account_system_index_time
0.00% 0.00% perf [kernel.vmlinux] [k] kfree
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] scheduler_tick
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_add_ordered_list
0.00% 0.00% perf [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% perf [kernel.vmlinux] [k] __add_jh
0.00% 0.00% swapper [kernel.vmlinux] [k] bio_advance
0.00% 0.00% perf [kernel.vmlinux] [k] handle_mm_fault
0.00% 0.00% perf [kernel.vmlinux] [k] __do_page_fault
0.00% 0.00% perf [kernel.vmlinux] [k] do_page_fault
0.00% 0.00% perf [kernel.vmlinux] [k] __handle_mm_fault
0.00% 0.00% swapper [kernel.vmlinux] [k] raise_softirq_irqoff
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] new_slab
0.00% 0.00% swapper [kernel.vmlinux] [k] bitmap_endwrite
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] kmem_cache_alloc
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] bio_advance_iter
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_write_unlock_nested
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] ixgbe_service_task
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_write_lock
0.00% 0.00% swapper [kernel.vmlinux] [k] ladder_reflect
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] new_slab
0.00% 0.00% swapper [kernel.vmlinux] [k] hrtimer_forward
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] hrtimer_update_base
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ktime_get_update_offsets_now
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] task_tick_fair
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] trace_consume_skb
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __slab_alloc
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_attempt_plug_merge
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] kmem_cache_alloc
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_queue_split
0.00% 0.00% perf [kernel.vmlinux] [k] unlock_journal
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_queue_split
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] ixgbe_read_reg
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_hrtimer_cancel
0.00% 0.00% swapper [kernel.vmlinux] [k] sched_clock_tick
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] ixgbe_update_stats
0.00% 0.00% swapper [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% perf [kernel.vmlinux] [k] __fget_light
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_setup_cmnd
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_init_sgtable
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_init_io
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] sd_init_command
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] new_slab
0.00% 0.00% swapper [kernel.vmlinux] [k] activate_task
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] ___slab_alloc
0.00% 0.00% perf [kernel.vmlinux] [k] get_bh
0.00% 0.00% swapper [kernel.vmlinux] [k] __hrtimer_get_next_event
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] schedule
0.00% 0.00% swapper [kernel.vmlinux] [k] __blk_mq_end_request
0.00% 0.00% perf [kernel.vmlinux] [k] __fdget
lines 19041-19125\r 0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] new_slab
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] kzalloc
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] r1bio_pool_alloc
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] alloc_r1bio
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __kmalloc
0.00% 0.00% perf [kernel.vmlinux] [k] __pagevec_lru_add_fn
0.00% 0.00% perf [kernel.vmlinux] [k] __slab_alloc
0.00% 0.00% perf [kernel.vmlinux] [k] ___slab_alloc
0.00% 0.00% perf [kernel.vmlinux] [k] kmem_cache_alloc_trace
0.00% 0.00% swapper [kernel.vmlinux] [k] cpuidle_not_available
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_work_run_list
0.00% 0.00% swapper [kernel.vmlinux] [k] ktime_add_safe
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] generic_make_request_checks
0.00% 0.00% swapper [kernel.vmlinux] [k] kfree
0.00% 0.00% perf [kernel.vmlinux] [k] init_tb_struct
0.00% 0.00% swapper [kernel.vmlinux] [k] r1bio_pool_free
0.00% 0.00% swapper [kernel.vmlinux] [k] get_sd_balance_interval.isra.64
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% perf [kernel.vmlinux] [k] tp_item_head
0.00% 0.00% swapper [kernel.vmlinux] [k] clocksource_delta
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_rq_map_sg
0.00% 0.00% swapper [kernel.vmlinux] [k] enqueue_task
0.00% 0.00% swapper [kernel.vmlinux] [k] note_gp_changes
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% swapper [kernel.vmlinux] [k] hrtimer_run_queues
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] reiserfs_free_jh
0.00% 0.00% perf perf [.] 0xffffffffffc158d6
0.00% 0.00% perf [kernel.vmlinux] [k] __radix_tree_create
0.00% 0.00% swapper [kernel.vmlinux] [k] unlock_buffer
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_schedule_old_flush
0.00% 0.00% swapper [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] new_slab
0.00% 0.00% swapper [kernel.vmlinux] [k] rdev_dec_pending
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] clear_page_dirty_for_io
0.00% 0.00% perf [kernel.vmlinux] [k] compound_head
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] update_curr
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] run_local_timers
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bio_attempt_back_merge
0.00% 0.00% swapper [kernel.vmlinux] [k] target_load
0.00% 0.00% swapper [kernel.vmlinux] [k] __msecs_to_jiffies
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_sched_qs
0.00% 0.00% perf [kernel.vmlinux] [k] check_irqs_on
0.00% 0.00% perf [kernel.vmlinux] [k] copy_item_head
0.00% 0.00% perf [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] ll_back_merge_fn
0.00% 0.00% snmpd [kernel.vmlinux] [k] show_stat
0.00% 0.00% swapper [kernel.vmlinux] [k] pick_next_task_fair
lines 19126-19210\r 0.00% 0.00% perf perf [.] 0xffffffffffc1590b
0.00% 0.00% perf perf [.] 0x000000000001590b
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] kthread
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] rcu_gp_kthread
0.00% 0.00% perf [kernel.vmlinux] [k] new_slab
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% perf [kernel.vmlinux] [k] radix_tree_tag_set
0.00% 0.00% swapper [kernel.vmlinux] [k] native_save_fl
0.00% 0.00% perf [kernel.vmlinux] [k] inode_add_bytes
0.00% 0.00% perf [kernel.vmlinux] [k] __dquot_alloc_space
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] generic_make_request_checks
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] blk_mq_dispatch_rq_list
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] blk_mq_sched_dispatch_requests
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] __blk_mq_run_hw_queue
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] blk_mq_run_work_fn
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_rcu_utilization
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_init
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mark_buffer_async_write
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mark_buffer_async_write_endio
0.00% 0.00% perf [kernel.vmlinux] [k] lock_page
0.00% 0.00% perf [kernel.vmlinux] [k] node_dirty_ok
0.00% 0.00% swapper [kernel.vmlinux] [k] igb_poll
0.00% 0.00% swapper [kernel.vmlinux] [k] net_rx_action
0.00% 0.00% swapper [kernel.vmlinux] [k] sched_ttwu_pending
0.00% 0.00% perf [kernel.vmlinux] [k] entry_SYSCALL_64
0.00% 0.00% perf [kernel.vmlinux] [k] load_balance
0.00% 0.00% swapper [kernel.vmlinux] [k] update_load_avg
0.00% 0.00% swapper [kernel.vmlinux] [k] enqueue_task_fair
0.00% 0.00% snmpd [kernel.vmlinux] [k] kstat_irqs_usr
0.00% 0.00% swapper [kernel.vmlinux] [k] tick_check_broadcast_expired
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_list_merge
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] pfmemalloc_match
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __remove_hrtimer
lines 19211-19295\r 0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] timerqueue_del
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] timekeeping_get_ns
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] cpuacct_account_field
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] perf_event_task_tick
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] tick_program_event
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] scsi_queue_rq
0.00% 0.00% swapper [kernel.vmlinux] [k] local_touch_nmi
0.00% 0.00% perf [kernel.vmlinux] [k] bit_waitqueue
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% swapper [kernel.vmlinux] [k] rb_insert_color
0.00% 0.00% swapper [kernel.vmlinux] [k] timekeeping_update
0.00% 0.00% perf [kernel.vmlinux] [k] fdget_pos
0.00% 0.00% swapper [kernel.vmlinux] [k] find_bio_disk
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] reiserfs_free_jh
0.00% 0.00% perf [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% swapper [kernel.vmlinux] [k] wake_up_process
0.00% 0.00% swapper [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] account_process_tick
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] rcu_check_callbacks
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __read_seqcount_begin.constprop.20
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] alloc_r1bio
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] radix_tree_tag_clear
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_segcblist_future_gp_needed
0.00% 0.00% swapper [kernel.vmlinux] [k] set_next_entity
0.00% 0.00% perf [kernel.vmlinux] [k] trylock_page
0.00% 0.00% swapper [kernel.vmlinux] [k] radix_tree_descend
0.00% 0.00% snmpd [kernel.vmlinux] [k] diskstats_show
0.00% 0.00% perf [kernel.vmlinux] [k] map_bh
0.00% 0.00% swapper [kernel.vmlinux] [k] __intel_pmu_enable_all
0.00% 0.00% swapper [kernel.vmlinux] [k] radix_tree_tag_clear
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_account_io_start
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_free_request
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_flush_plug_list
0.00% 0.00% perf [kernel.vmlinux] [k] __fdget_pos
0.00% 0.00% swapper [kernel.vmlinux] [k] percpu_counter_add_batch
0.00% 0.00% swapper [kernel.vmlinux] [k] napi_gro_receive
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] scsi_init_sgtable
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] scsi_init_io
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] sd_init_command
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] scsi_setup_cmnd
0.00% 0.00% perf [kernel.vmlinux] [k] __rmqueue
0.00% 0.00% swapper [kernel.vmlinux] [k] __intel_pmu_disable_all
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] kfree
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] scsi_queue_rq
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] blk_mq_dispatch_rq_list
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] blk_mq_sched_dispatch_requests
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] __blk_mq_run_hw_queue
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] blk_mq_run_work_fn
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] quiet_vmstat
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_sched_insert_requests
0.00% 0.00% perf [kernel.vmlinux] [k] fput
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% swapper [kernel.vmlinux] [k] reiserfs_end_ordered_io
0.00% 0.00% perf [kernel.vmlinux] [k] unlock_page
0.00% 0.00% perf [kernel.vmlinux] [k] atomic64_dec_and_test
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] kthread_should_stop
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] rcu_irq_enter
lines 19296-19380\r 0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] inc_slabs_node
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] task_group_account_field
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] tk_clock_read
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] read_tsc
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% swapper [kernel.vmlinux] [k] restore_regs_and_iret
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] hrtimer_active
0.00% 0.00% perf [kernel.vmlinux] [k] put_page
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% syslog-ng [unknown] [k] 0000000000000000
0.00% 0.00% perf [kernel.vmlinux] [k] compound_head
0.00% 0.00% swapper [kernel.vmlinux] [k] sched_avg_update
0.00% 0.00% swapper [kernel.vmlinux] [k] __wake_up_bit
0.00% 0.00% swapper [kernel.vmlinux] [k] __netif_receive_skb
0.00% 0.00% perf [kernel.vmlinux] [k] __kmalloc
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] netif_receive_skb_internal
0.00% 0.00% swapper [kernel.vmlinux] [k] invoke_rcu_core
0.00% 0.00% perf [kernel.vmlinux] [k] __radix_tree_replace
0.00% 0.00% perf [kernel.vmlinux] [k] le_key_k_type
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] igb_rd32
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] igb_watchdog_task
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bio_init
0.00% 0.00% swapper [kernel.vmlinux] [k] __netif_receive_skb_core
0.00% 0.00% swapper [kernel.vmlinux] [k] __wb_writeout_inc
0.00% 0.00% perf perf [.] 0xffffffffffc6eb0a
0.00% 0.00% perf perf [.] 0x000000000006eb0a
0.00% 0.00% swapper [kernel.vmlinux] [k] discard_slab
lines 19381-19465\r 0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] ll_back_merge_fn
0.00% 0.00% perf [kernel.vmlinux] [k] copyin
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] get_bh
0.00% 0.00% snmpd [kernel.vmlinux] [k] atomic64_cmpxchg
0.00% 0.00% perf [kernel.vmlinux] [k] real_space_diff
0.00% 0.00% swapper [kernel.vmlinux] [k] tick_irq_enter
0.00% 0.00% perf [kernel.vmlinux] [k] PageHuge
0.00% 0.00% perf [kernel.vmlinux] [k] radix_tree_load_root
0.00% 0.00% swapper [kernel.vmlinux] [k] wake_up_bit
0.00% 0.00% perf [kernel.vmlinux] [k] release_pages
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_get_request
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_entries_start
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] account_system_time
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] update_load_avg
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_queue_tag_busy_iter
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_in_flight
0.00% 0.00% swapper [kernel.vmlinux] [k] part_in_flight
0.00% 0.00% swapper [kernel.vmlinux] [k] part_round_stats
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_account_io_done
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] radix_tree_tag_set
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] tag_get
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% perf [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] md_write_start
0.00% 0.00% perf [kernel.vmlinux] [k] attach_page_buffers
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] inode_to_bdi
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mod_node_page_state
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mod_node_state
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] find_get_pages_tag
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] pagevec_lookup_tag
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] page_cache_get_speculative
0.00% 0.00% perf [kernel.vmlinux] [k] kmap_atomic
0.00% 0.00% perf [kernel.vmlinux] [k] __inc_zone_state
0.00% 0.00% snmpd libc-2.23.so [.] vsscanf
0.00% 0.00% swapper [kernel.vmlinux] [k] sata_scr_read
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] blk_rq_map_sg
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] force_qs_rnp
0.00% 0.00% swapper [kernel.vmlinux] [k] arp_rcv
0.00% 0.00% swapper [kernel.vmlinux] [k] sata_async_notification
0.00% 0.00% swapper [kernel.vmlinux] [k] queue_delayed_work_on
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bio_check_eod.part.82
0.00% 0.00% perf [kernel.vmlinux] [k] get_bh
0.00% 0.00% swapper [kernel.vmlinux] [k] __queue_work
0.00% 0.00% perf [kernel.vmlinux] [k] file_update_time
0.00% 0.00% swapper [kernel.vmlinux] [k] ahci_scr_read
0.00% 0.00% swapper [kernel.vmlinux] [k] refresh_cpu_vm_stats
0.00% 0.00% swapper [kernel.vmlinux] [k] mod_node_state
0.00% 0.00% swapper [kernel.vmlinux] [k] decay_load
0.00% 0.00% perf [kernel.vmlinux] [k] __inc_node_state
0.00% 0.00% swapper [kernel.vmlinux] [k] __add_wb_stat
0.00% 0.00% swapper [kernel.vmlinux] [k] kblockd_schedule_delayed_work_on
0.00% 0.00% swapper [kernel.vmlinux] [k] __blk_mq_delay_run_hw_queue
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_run_hw_queue
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_end_request
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_io_completion
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_finish_command
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_softirq_done
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __blk_mq_complete_request_remote
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_update_request
0.00% 0.00% perf [kernel.vmlinux] [k] account_page_dirtied
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] scsi_init_sgtable
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] scsi_init_io
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] sd_init_command
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] scsi_setup_cmnd
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] scsi_queue_rq
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] blk_mq_dispatch_rq_list
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] blk_mq_sched_dispatch_requests
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] __blk_mq_run_hw_queue
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] blk_mq_run_work_fn
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] md_write_start
0.00% 0.00% perf [kernel.vmlinux] [k] __do_fault
lines 19466-19550\r 0.00% 0.00% perf [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.00% 0.00% swapper [kernel.vmlinux] [k] sbitmap_for_each_set
0.00% 0.00% swapper [kernel.vmlinux] [k] __queue_delayed_work
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_dispatch_rq_list
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_sched_dispatch_requests
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __blk_mq_run_hw_queue
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __blk_mq_delay_run_hw_queue
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_run_hw_queue
0.00% 0.00% snmpd [kernel.vmlinux] [k] path_openat
0.00% 0.00% snmpd [kernel.vmlinux] [k] do_filp_open
0.00% 0.00% snmpd [kernel.vmlinux] [k] do_sys_open
0.00% 0.00% snmpd [kernel.vmlinux] [k] sys_open
0.00% 0.00% snmpd libc-2.23.so [.] open64
0.00% 0.00% perf [kernel.vmlinux] [k] __fcheck_files
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] igb_update_stats
0.00% 0.00% perf [kernel.vmlinux] [k] inode_to_bdi
0.00% 0.00% perf [kernel.vmlinux] [k] wait_for_stable_page
0.00% 0.00% swapper [kernel.vmlinux] [k] __free_slab
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% perf [kernel.vmlinux] [k] __count_vm_events
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] trylock_buffer
0.00% 0.00% snmpd [kernel.vmlinux] [k] __netlink_dump_start
0.00% 0.00% snmpd [kernel.vmlinux] [k] rtnetlink_rcv_msg
0.00% 0.00% snmpd [kernel.vmlinux] [k] netlink_rcv_skb
0.00% 0.00% snmpd [kernel.vmlinux] [k] rtnetlink_rcv
0.00% 0.00% snmpd [kernel.vmlinux] [k] netlink_unicast
0.00% 0.00% snmpd [kernel.vmlinux] [k] netlink_sendmsg
0.00% 0.00% snmpd [kernel.vmlinux] [k] sock_sendmsg_nosec
0.00% 0.00% snmpd [kernel.vmlinux] [k] SYSC_sendto
0.00% 0.00% snmpd [kernel.vmlinux] [k] sys_sendto
0.00% 0.00% snmpd libc-2.23.so [.] send
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] do_raw_spin_lock
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] scsi_init_sgtable
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] scsi_init_io
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] sd_init_command
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] scsi_setup_cmnd
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] blk_rq_map_sg
0.00% 0.00% swapper [kernel.vmlinux] [k] insert_work
0.00% 0.00% swapper [kernel.vmlinux] [k] wake_up_worker
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_queue_bounce
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] put_bh
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_wait_on_write_block
0.00% 0.00% snmpd [kernel.vmlinux] [k] mutex_lock
0.00% 0.00% snmpd [kernel.vmlinux] [k] irq_lock_sparse
0.00% 0.00% swapper [kernel.vmlinux] [k] bt_iter
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] irq_may_run
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __zone_watermark_ok
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] free_one_page
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] free_one_page
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] trace_napi_poll
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_msix_clean_rings
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] _mix_pool_bytes
lines 19551-19635\r 0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] mempolicy_slab_node
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] napi_schedule_prep
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] load_balance
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] run_rebalance_domains
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] run_rebalance_domains
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] mempolicy_slab_node
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] decay_load
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] update_cfs_rq_load_avg
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] raise_softirq
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] perf_pmu_disable
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ktime_get
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_sched_restart
lines 19636-19720\r 0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] kzalloc
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] r1bio_pool_alloc
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __kmalloc
0.00% 0.00% perf [kernel.vmlinux] [k] file_start_write
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] kfree
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_sched_restart_hctx
0.00% 0.00% swapper [kernel.vmlinux] [k] arp_process
0.00% 0.00% perf [kernel.vmlinux] [k] __mod_node_page_state
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_transaction_running
0.00% 0.00% perf [kernel.vmlinux] [k] perf_mmap_fault
0.00% 0.00% swapper [kernel.vmlinux] [k] md_write_end
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] blk_rq_map_sg
0.00% 0.00% swapper [kernel.vmlinux] [k] update_fast_timekeeper
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] entry_SYSCALL_64_fastpath
0.00% 0.00% perf [kernel.vmlinux] [k] node_page_state
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bitmap_startwrite
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __bio_clone_fast
0.00% 0.00% swapper [kernel.vmlinux] [k] compound_head
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] rcu_implicit_dynticks_qs
0.00% 0.00% perf [kernel.vmlinux] [k] radix_tree_maybe_preload
0.00% 0.00% perf [kernel.vmlinux] [k] __radix_tree_preload
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_flush_plug_list
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_flush_plug_list
0.00% 0.00% perf [kernel.vmlinux] [k] _raw_spin_lock
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] rcu_irq_exit
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __remove_hrtimer
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] generic_make_request_checks
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] run_local_timers
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] task_group_account_field
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __pagevec_release
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] release_pages
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] update_cfs_rq_load_avg
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] update_curr
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] page_mapping
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] unlock_page
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] radix_tree_descend
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] percpu_ref_put_many.constprop.98
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_sched_dispatch_requests
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __blk_mq_run_hw_queue
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __blk_mq_delay_run_hw_queue
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_run_hw_queue
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_dispatch_rq_list
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_sched_insert_requests
0.00% 0.00% perf [kernel.vmlinux] [k] pick_next_task_idle
0.00% 0.00% perf [kernel.vmlinux] [k] PageUptodate
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_rq_get_max_sectors
0.00% 0.00% perf [kernel.vmlinux] [k] apic_timer_interrupt
0.00% 0.00% swapper [kernel.vmlinux] [k] __fprop_inc_percpu_max
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] r1_bio_write_done
0.00% 0.00% perf [kernel.vmlinux] [k] iov_iter_advance
0.00% 0.00% perf [kernel.vmlinux] [k] current_time
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_check_lock_depth
0.00% 0.00% perf [kernel.vmlinux] [k] tag_get
0.00% 0.00% perf [kernel.vmlinux] [k] compound_head
0.00% 0.00% perf [kernel.vmlinux] [k] page_cache_get_speculative
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] return_from_SYSCALL_64
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_check_inflight
0.00% 0.00% perf [kernel.vmlinux] [k] replace_slot
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_queue_rq
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] _wait_barrier
0.00% 0.00% swapper [kernel.vmlinux] [k] call_timer_fn
0.00% 0.00% swapper [kernel.vmlinux] [k] __free_pages
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_queue_bounce
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] ll_new_hw_segment
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_syscall_64
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] page_mkclean
0.00% 0.00% perf [kernel.vmlinux] [k] le_key_k_type
0.00% 0.00% perf [kernel.vmlinux] [k] timespec_trunc
0.00% 0.00% snmpd [unknown] [k] 0x3538322020202000
0.00% 0.00% perf [kernel.vmlinux] [k] __sb_start_write
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_clone
0.00% 0.00% syslog-ng libc-2.23.so [.] __clone
0.00% 0.00% swapper [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% snmpd libc-2.23.so [.] _IO_sputbackc
0.00% 0.00% perf [kernel.vmlinux] [k] first_zones_zonelist
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] md_wakeup_thread
lines 19721-19805\r 0.00% 0.00% rcu_sched [kernel.vmlinux] [k] rcu_dynticks_snap
0.00% 0.00% swapper [kernel.vmlinux] [k] ttwu_do_wakeup.isra.84
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] _do_fork
0.00% 0.00% snmpd [unknown] [.] 0x2520756c6c252075
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_endio
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] raid1_end_write_request
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_finish_plug
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __slab_free
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] sbitmap_get
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __sbitmap_queue_get
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __blk_mq_get_tag
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_mq_get_tag
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __sbitmap_get_word
0.00% 0.00% perf [kernel.vmlinux] [k] init_sync_kiocb
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% perf [kernel.vmlinux] [k] __rmqueue_smallest
0.00% 0.00% swapper [kernel.vmlinux] [k] update_vsyscall
0.00% 0.00% perf perf [.] 0xffffffffffc15908
0.00% 0.00% perf perf [.] 0x0000000000015908
0.00% 0.00% perf [kernel.vmlinux] [k] down_write
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] compound_head
0.00% 0.00% snmpd [kernel.vmlinux] [k] may_open
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] md_check_recovery
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bitmap_startwrite
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] load_balance
0.00% 0.00% perf [kernel.vmlinux] [k] get_empty_nodes
0.00% 0.00% swapper [kernel.vmlinux] [k] cpus_share_cache
0.00% 0.00% swapper [kernel.vmlinux] [k] wq_worker_waking_up
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_cond_resched
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] bio_advance_iter
0.00% 0.00% snmpd [kernel.vmlinux] [k] dev_get_stats
0.00% 0.00% snmpd [kernel.vmlinux] [k] dev_seq_printf_stats
0.00% 0.00% snmpd [kernel.vmlinux] [k] dev_seq_show
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] schedule_timeout
0.00% 0.00% snmpd [kernel.vmlinux] [k] inet6_dump_addr
0.00% 0.00% snmpd [kernel.vmlinux] [k] inet6_dump_ifaddr
0.00% 0.00% snmpd [kernel.vmlinux] [k] netlink_dump
0.00% 0.00% perf [kernel.vmlinux] [k] get_page
0.00% 0.00% swapper [kernel.vmlinux] [k] cpuidle_get_cpu_driver
0.00% 0.00% snmpd [kernel.vmlinux] [k] kstat_irqs
0.00% 0.00% perf [kernel.vmlinux] [k] __update_idle_core
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] dequeue_task
0.00% 0.00% perf [kernel.vmlinux] [k] page_mapping
0.00% 0.00% snmpd [kernel.vmlinux] [k] snmp6_dev_seq_show
0.00% 0.00% perf [kernel.vmlinux] [k] rw_verify_area
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] process_one_work
0.00% 0.00% swapper [kernel.vmlinux] [k] update_tg_load_avg.constprop.87
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ahci_handle_port_interrupt
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ahci_handle_port_intr
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ahci_single_level_irq_intr
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __handle_irq_event_percpu
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] handle_irq_event_percpu
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] handle_irq_event
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] handle_edge_irq
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] handle_irq
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] do_IRQ
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ret_from_intr
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] __slab_free
0.00% 0.00% snmpd [kernel.vmlinux] [k] vsnprintf
0.00% 0.00% perf [kernel.vmlinux] [k] alloc_set_pte
0.00% 0.00% perf [kernel.vmlinux] [k] finish_fault
0.00% 0.00% perf [kernel.vmlinux] [k] hpage_nr_pages
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% perf [kernel.vmlinux] [k] get_seconds
0.00% 0.00% swapper [kernel.vmlinux] [k] llist_add_batch
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bio_check_eod.part.82
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_account_io_start
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] idle_cpu
0.00% 0.00% perf [kernel.vmlinux] [k] copy_user_generic_unrolled
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] prepare_to_wait_event
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] dd_bio_merge
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __blk_mq_sched_bio_merge
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] _wait_barrier
0.00% 0.00% snmpd [kernel.vmlinux] [k] d_instantiate
0.00% 0.00% snmpd [kernel.vmlinux] [k] sock_alloc_file
0.00% 0.00% snmpd [kernel.vmlinux] [k] SYSC_socket
0.00% 0.00% snmpd [kernel.vmlinux] [k] sys_socket
0.00% 0.00% snmpd [kernel.vmlinux] [k] __d_instantiate
0.00% 0.00% snmpd libc-2.23.so [.] __socket
0.00% 0.00% perf [kernel.vmlinux] [k] is_indirect_le_ih
0.00% 0.00% swapper [kernel.vmlinux] [k] cpufreq_update_util
0.00% 0.00% perf [kernel.vmlinux] [k] page_is_file_cache
lines 19806-19890\r 0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bitmap_daemon_work
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_cblist_dequeue
0.00% 0.00% snmpd [unknown] [k] 0x000000000159b1a0
0.00% 0.00% swapper [kernel.vmlinux] [k] __free_one_page
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_rq_get_max_sectors
0.00% 0.00% perf [kernel.vmlinux] [k] get_lfree
0.00% 0.00% swapper [kernel.vmlinux] [k] hrtick_update
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bio_advance_iter
0.00% 0.00% swapper [kernel.vmlinux] [k] next_online_pgdat
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_check_eod.part.82
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] update_load_avg
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] update_cfs_rq_load_avg
0.00% 0.00% snmpd [unknown] [k] 0x00007fff3b472ac4
0.00% 0.00% perf [kernel.vmlinux] [k] fatal_signal_pending
0.00% 0.00% swapper [kernel.vmlinux] [k] free_one_page
0.00% 0.00% swapper [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% swapper [kernel.vmlinux] [k] mod_node_page_state
0.00% 0.00% swapper [kernel.vmlinux] [k] __task_rq_lock
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] _cond_resched
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] prep_compound_page
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] kthread_should_park
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] activate_task
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ttwu_do_activate.isra.85
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] wake_up_process
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] wake_up_worker
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] insert_work
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __queue_work
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] delayed_work_timer_fn
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] call_timer_fn
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] enqueue_task
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] inc_slabs_node
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] mod_node_state
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] _cond_resched
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ttwu_do_activate.isra.85
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] wake_up_process
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] wake_up_worker
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] insert_work
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __queue_work
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] delayed_work_timer_fn
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] call_timer_fn
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] enqueue_task
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] apic_chip_data
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] mempolicy_slab_node
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_write_reg
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] trace_napi_poll
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __rmqueue
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __rmqueue_smallest
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __mix_pool_bytes
lines 19891-19975\r 0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ksoftirqd_should_run
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] inc_slabs_node
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __zone_watermark_ok
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __free_one_page
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] first_zones_zonelist
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] trace_napi_poll
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __rmqueue
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __rmqueue_smallest
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] trace_napi_poll
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] page_expected_state
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] mod_node_state
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __radix_tree_lookup
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] radix_tree_lookup
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] get_work_pool
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __queue_work
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] delayed_work_timer_fn
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] call_timer_fn
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] radix_tree_descend
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ixgbe_write_reg
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] kthread_should_stop
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ixgbe_write_reg
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] trace_mm_page_alloc_zone_locked
lines 19976-20060\r 0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] irq_may_run
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] apic_chip_data
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] mod_node_page_state
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] mod_node_state
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __local_bh_enable
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] to_kthread
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] apic_chip_data
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __napi_schedule
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_read_reg
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __qdisc_run
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] trace_napi_poll
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] apic_chip_data
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] free_one_page
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] rcu_irq_exit
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ixgbe_write_reg
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_rx_bufsz
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] native_irq_return_iret
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] dql_completed
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] new_slab
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ixgbe_write_reg
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] apic_chip_data
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __napi_schedule_irqoff
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] irq_enter
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] native_irq_return_iret
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] run_timer_softirq
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] _raw_spin_lock_irqsave
lines 20061-20145\r 0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] source_load
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] load_balance
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __local_bh_enable
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __zone_watermark_ok
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] free_one_page
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __qdisc_run
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] trace_rcu_utilization
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] list_del_init
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] native_iret
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] kthread_should_stop
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] to_kthread
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] rcu_irq_exit
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __napi_schedule
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] cpumask_next_and
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] rb_erase
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] cpufreq_update_util
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] update_min_vruntime
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ixgbe_write_reg
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] neigh_event_send
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] native_apic_mem_write
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] finish_task_switch
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ksoftirqd_should_run
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] native_sched_clock
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] sched_clock_cpu
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] update_rq_clock
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] cycles_2_ns
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] rcu_bh_qs
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] rcu_irq_exit
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __local_bh_enable
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] restore_regs_and_iret
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] mempolicy_slab_node
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] intel_pmu_disable_all
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] x86_pmu_disable
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] load_balance
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] run_rebalance_domains
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __raise_softirq_irqoff
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] cpu_needs_another_gp
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] cpuacct_charge
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __hrtimer_get_next_event
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] calc_global_load_tick
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] lapic_next_deadline
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] get_pageblock_bitmap.isra.55
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __usecs_to_jiffies
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __accumulate_pelt_segments
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] rcu_irq_exit
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] trace_napi_poll
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] free_one_page
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] discard_slab
lines 20146-20230\r 0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __free_one_page
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] run_posix_cpu_timers
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] clockevents_program_event
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] credit_entropy_bits
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __wake_up
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% perf [kernel.vmlinux] [k] radix_tree_node_alloc.constprop.19
0.00% 0.00% swapper [kernel.vmlinux] [k] dec_zone_page_state
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] alloc_pages_current
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ixgbe_irq_enable_queues
0.00% 0.00% perf [kernel.vmlinux] [k] compound_head
0.00% 0.00% perf [kernel.vmlinux] [k] sd_is_left_mergeable
0.00% 0.00% snmpd libc-2.23.so [.] _IO_vfscanf
0.00% 0.00% perf [kernel.vmlinux] [k] le_key_version
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bitmap_get_counter
0.00% 0.00% swapper [kernel.vmlinux] [k] inc_node_page_state
0.00% 0.00% perf perf [.] 0xffffffffffc6eb07
0.00% 0.00% perf perf [.] 0x000000000006eb07
0.00% 0.00% perf [kernel.vmlinux] [k] pagefault_enable
0.00% 0.00% perf [kernel.vmlinux] [k] _cond_resched
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] scsi_queue_rq
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] put_bh
0.00% 0.00% snmpd [kernel.vmlinux] [k] seq_vprintf
0.00% 0.00% snmpd [kernel.vmlinux] [k] seq_printf
0.00% 0.00% swapper [kernel.vmlinux] [k] put_bh
0.00% 0.00% snmpd [kernel.vmlinux] [k] netstat_seq_show
0.00% 0.00% snmpd [kernel.vmlinux] [k] snmp_fold_field
0.00% 0.00% perf perf [.] 0xffffffffffc15932
0.00% 0.00% perf perf [.] 0x0000000000015932
0.00% 0.00% perf perf [.] 0xffffffffffc15943
0.00% 0.00% perf perf [.] 0x0000000000015943
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] create_io_context
0.00% 0.00% perf perf [.] 0xffffffffffcce928
0.00% 0.00% perf perf [.] 0x00000000000ce928
0.00% 0.00% perf [kernel.vmlinux] [k] compound_head
0.00% 0.00% perf [kernel.vmlinux] [k] tick_sched_timer
0.00% 0.00% perf [kernel.vmlinux] [k] __hrtimer_run_queues
0.00% 0.00% perf [kernel.vmlinux] [k] hrtimer_interrupt
0.00% 0.00% perf [kernel.vmlinux] [k] local_apic_timer_interrupt
0.00% 0.00% perf [kernel.vmlinux] [k] smp_apic_timer_interrupt
0.00% 0.00% perf [kernel.vmlinux] [k] profile_tick
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] free_one_page
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __free_pages_ok
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __free_pages
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __free_slab
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] discard_slab
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] timerqueue_del
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] inc_slabs_node
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] _raw_spin_trylock
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] rcu_irq_enter
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] prep_compound_page
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] apic_chip_data
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] arch_get_random_seed_long
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] node_match
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] raise_softirq_irqoff
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_get_driver_tag
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_get_tag
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] rb_erase
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] hrtimer_active
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] tk_clock_read
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] timekeeping_get_ns
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ktime_get
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] read_tsc
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ixgbe_write_reg
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] node_tag_clear
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] trigger_load_balance
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] compound_head
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __bio_clone_fast
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] tick_program_event
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] lapic_next_deadline
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] account_system_index_time
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __accumulate_pelt_segments
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] decay_load
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __add_wb_stat
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bitmap_get_counter
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bitmap_checkpage
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] hrtimer_forward
lines 20231-20315\r 0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mapping_cap_account_dirty.isra.33
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] update_min_vruntime
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] radix_tree_tagged
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] dec_zone_page_state
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mod_zone_state
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] trylock_page
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mempolicy_slab_node
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] vmstat_shepherd
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] perf_event_task_tick
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] compound_head
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] create_io_context
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] dd_insert_requests
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] elv_rb_add
0.00% 0.00% perf [kernel.vmlinux] [k] __mod_zone_page_state
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] page_index
0.00% 0.00% perf [kernel.vmlinux] [k] __add_wb_stat
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_dispatch_rq_list
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_sched_dispatch_requests
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] __blk_mq_run_hw_queue
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] __blk_mq_delay_run_hw_queue
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] scsi_queue_rq
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_run_hw_queue
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_run_hw_queues
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_requeue_work
0.00% 0.00% perf [kernel.vmlinux] [k] __bread_gfp
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] bio_advance_iter
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% snmpd [kernel.vmlinux] [k] igb_update_stats
0.00% 0.00% snmpd [kernel.vmlinux] [k] igb_get_stats64
0.00% 0.00% snmpd [kernel.vmlinux] [k] igb_rd32
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% swapper [kernel.vmlinux] [k] _raw_spin_lock_irq
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] mddev_is_clustered
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_route_input_noref
0.00% 0.00% snmpd libc-2.23.so [.] vfprintf
0.00% 0.00% snmpd libc-2.23.so [.] __vsnprintf_chk
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] radix_tree_load_root
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] req_gap_back_merge
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_start_gp
0.00% 0.00% perf [kernel.vmlinux] [k] generic_write_checks
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] __find_get_block
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] check_irqs_on
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __sg_alloc_table
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] sg_alloc_table_chained
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_init_sgtable
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_init_io
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] sd_init_command
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_setup_cmnd
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] sg_init_table
0.00% 0.00% swapper [kernel.vmlinux] [k] file_free_rcu
0.00% 0.00% perf [kernel.vmlinux] [k] inode_to_bdi
0.00% 0.00% swapper [kernel.vmlinux] [k] bit_spin_lock.constprop.55
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] ata_scsi_queuecmd
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] scsi_dispatch_cmd
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] raid_end_bio_io
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_end_request
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_io_completion
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_finish_command
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_softirq_done
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __blk_mq_complete_request
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_complete_request
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_mq_done
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_qc_done
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_scsi_qc_complete
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __ata_qc_complete
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_qc_complete
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_qc_complete_multiple
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] SYSC_epoll_wait
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_epoll_wait
0.00% 0.00% syslog-ng libc-2.23.so [.] epoll_wait
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_start_request
0.00% 0.00% perf perf [.] 0xffffffffffc6eb11
0.00% 0.00% perf perf [.] 0x000000000006eb11
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_dispatch_cmd
0.00% 0.00% snmpd [kernel.vmlinux] [k] radix_tree_lookup
0.00% 0.00% snmpd [kernel.vmlinux] [k] __nla_put
0.00% 0.00% snmpd [kernel.vmlinux] [k] nla_put
0.00% 0.00% snmpd [kernel.vmlinux] [k] put_cacheinfo
0.00% 0.00% snmpd [kernel.vmlinux] [k] inet6_fill_ifaddr
0.00% 0.00% snmpd [kernel.vmlinux] [k] __nla_reserve
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] bio_endio
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_update_request
0.00% 0.00% swapper [kernel.vmlinux] [k] pick_next_entity
0.00% 0.00% swapper [kernel.vmlinux] [k] page_zone
lines 20316-20400\r 0.00% 0.00% swapper [kernel.vmlinux] [k] ip_route_input_rcu
0.00% 0.00% swapper [kernel.vmlinux] [k] wake_bit_function
0.00% 0.00% perf [kernel.vmlinux] [k] __fsnotify_parent
0.00% 0.00% snmpd [kernel.vmlinux] [k] refcount_inc
0.00% 0.00% perf [kernel.vmlinux] [k] page_copy_sane
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] timekeeping_get_ns
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] ktime_get_ts64
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] read_tsc
0.00% 0.00% snmpd [unknown] [.] 0x756c2520756c2520
0.00% 0.00% swapper [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% swapper [kernel.vmlinux] [k] compound_head
0.00% 0.00% swapper [kernel.vmlinux] [k] arch_cpu_idle_exit
0.00% 0.00% swapper [kernel.vmlinux] [k] bitmap_get_counter
0.00% 0.00% swapper [kernel.vmlinux] [k] finish_task_switch
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bitmap_checkpage
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] del_timer_sync
0.00% 0.00% snmpd [kernel.vmlinux] [k] inode_permission
0.00% 0.00% snmpd [kernel.vmlinux] [k] link_path_walk
0.00% 0.00% snmpd [kernel.vmlinux] [k] __inode_permission
0.00% 0.00% swapper [kernel.vmlinux] [k] restore_c_regs_and_iret
0.00% 0.00% snmpd [kernel.vmlinux] [k] __dentry_kill
0.00% 0.00% snmpd [kernel.vmlinux] [k] dput
0.00% 0.00% snmpd [kernel.vmlinux] [k] __fput
0.00% 0.00% snmpd [kernel.vmlinux] [k] ____fput
0.00% 0.00% snmpd [kernel.vmlinux] [k] task_work_run
0.00% 0.00% snmpd [kernel.vmlinux] [k] prepare_exit_to_usermode
0.00% 0.00% snmpd [kernel.vmlinux] [k] syscall_return_slowpath
0.00% 0.00% snmpd libc-2.23.so [.] close
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] ata_scsi_queuecmd
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] scsi_dispatch_cmd
0.00% 0.00% swapper [kernel.vmlinux] [k] __pick_first_entity
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] lock_timer_base
0.00% 0.00% perf [kernel.vmlinux] [k] security_file_permission
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] _raw_spin_lock_irq
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] find_next_zero_bit
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] _find_next_bit
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] load_balance
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] blk_queue_split
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] schedule
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_gp_kthread_wake
0.00% 0.00% snmpd libc-2.23.so [.] __rawmemchr
0.00% 0.00% swapper [kernel.vmlinux] [k] check_preempt_curr
0.00% 0.00% perf perf [.] 0xffffffffffc1592f
0.00% 0.00% perf perf [.] 0x000000000001592f
0.00% 0.00% snmpd [kernel.vmlinux] [k] cpumask_next.constprop.34
0.00% 0.00% snmpd [kernel.vmlinux] [k] mutex_unlock
0.00% 0.00% snmpd [kernel.vmlinux] [k] irq_unlock_sparse
0.00% 0.00% perf [kernel.vmlinux] [k] balance_dirty_pages_ratelimited
0.00% 0.00% perf [kernel.vmlinux] [k] get_page
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] igb_ptp_rx_hang
0.00% 0.00% snmpd [kernel.vmlinux] [k] string
0.00% 0.00% swapper [kernel.vmlinux] [k] add_interrupt_randomness
0.00% 0.00% snmpd [kernel.vmlinux] [k] snmp_get_cpu_field
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __get_vm_area_node
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __vmalloc_node_range
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] copy_process.part.45
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] alloc_vmap_area
0.00% 0.00% perf perf [.] 0xffffffffffcce951
0.00% 0.00% perf perf [.] 0x00000000000ce951
0.00% 0.00% swapper [kernel.vmlinux] [k] process_timeout
0.00% 0.00% swapper [kernel.vmlinux] [k] fib_table_lookup
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __sbitmap_queue_get
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __blk_mq_get_tag
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __sbitmap_get_word
0.00% 0.00% swapper [kernel.vmlinux] [k] swake_up
0.00% 0.00% swapper [kernel.vmlinux] [k] percpu_ref_put_many.constprop.98
0.00% 0.00% snmpd [kernel.vmlinux] [k] seq_put_decimal_ull
0.00% 0.00% swapper [kernel.vmlinux] [k] mod_zone_state
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] io_schedule
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bitmap_get_counter
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] schedule
0.00% 0.00% swapper [kernel.vmlinux] [k] switch_mm_irqs_off
0.00% 0.00% perf perf [.] 0xffffffffffc6eb2c
0.00% 0.00% perf perf [.] 0x000000000006eb2c
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] sg_alloc_table_chained
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] __sg_alloc_table
0.00% 0.00% snmpd [kernel.vmlinux] [k] snmp6_seq_show_icmpv6msg
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_put_tag
0.00% 0.00% swapper [kernel.vmlinux] [k] sbitmap_queue_clear
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] prefetch_freepointer.isra.70
lines 20401-20485\r 0.00% 0.00% swapper [kernel.vmlinux] [k] next_zone
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_try_merge
0.00% 0.00% perf [kernel.vmlinux] [k] __get_free_pages
0.00% 0.00% swapper [kernel.vmlinux] [k] put_cpu_partial
0.00% 0.00% swapper [kernel.vmlinux] [k] unfreeze_partials.isra.77
0.00% 0.00% perf perf [.] 0xffffffffffc15a35
0.00% 0.00% perf perf [.] 0x0000000000015a35
0.00% 0.00% perf [kernel.vmlinux] [k] __sb_end_write
0.00% 0.00% snmpd [unknown] [k] 0000000000000000
0.00% 0.00% swapper [kernel.vmlinux] [k] __switch_to
0.00% 0.00% snmpd [kernel.vmlinux] [k] _raw_read_unlock_bh
0.00% 0.00% snmpd [kernel.vmlinux] [k] queued_read_unlock
0.00% 0.00% perf perf [.] 0xffffffffffc1595a
0.00% 0.00% perf perf [.] 0x000000000001595a
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] arch_dup_task_struct
0.00% 0.00% swapper [kernel.vmlinux] [k] update_head_pos
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __blk_mq_delay_run_hw_queue
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_run_hw_queue
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_run_hw_queues
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_requeue_work
0.00% 0.00% swapper [kernel.vmlinux] [k] __kfree_skb
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] get_device
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] kobject_get
0.00% 0.00% snmpd [kernel.vmlinux] [k] snmp6_seq_show_item
0.00% 0.00% snmpd [kernel.vmlinux] [k] snmp6_seq_show
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __blk_mq_sched_bio_merge
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] dd_bio_merge
0.00% 0.00% snmpd libc-2.23.so [.] 0x000000000007703e
0.00% 0.00% snmpd libc-2.23.so [.] 0xffff80c2d324e03e
0.00% 0.00% perf [kernel.vmlinux] [k] trace_kmalloc
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_rq_merge_ok
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] rq_mergeable
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_rq_is_passthrough
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% perf [kernel.vmlinux] [k] dequeue_task
0.00% 0.00% swapper [kernel.vmlinux] [k] consume_skb
0.00% 0.00% perf [kernel.vmlinux] [k] is_direntry_le_ih
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] _allow_barrier
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] call_bio_endio
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] raid_end_bio_io
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] r1_bio_write_done
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] raid1_end_write_request
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __wake_up
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_scsi_dispatch_cmd_done
0.00% 0.00% swapper [kernel.vmlinux] [k] resched_curr
0.00% 0.00% swapper [kernel.vmlinux] [k] percpu_ref_put_many.constprop.64
0.00% 0.00% snmpd libc-2.23.so [.] memchr
0.00% 0.00% snmpd [unknown] [.] 0x6164732031202020
0.00% 0.00% swapper [kernel.vmlinux] [k] page_index
0.00% 0.00% snmpd [kernel.vmlinux] [k] vlan_dev_get_stats64
0.00% 0.00% perf [kernel.vmlinux] [k] finish_task_switch
0.00% 0.00% swapper [kernel.vmlinux] [k] swake_up_locked
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] get_page_from_freelist
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __alloc_pages_nodemask
0.00% 0.00% perf perf [.] 0xffffffffffc15976
0.00% 0.00% perf perf [.] 0x0000000000015976
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] schedule
0.00% 0.00% snmpd [unknown] [.] 0x20756c2520756c25
0.00% 0.00% perf [kernel.vmlinux] [k] dirid_groups
0.00% 0.00% perf [kernel.vmlinux] [k] bmap_hash_id
0.00% 0.00% snmpd [unknown] [k] 0x6164732030202020
0.00% 0.00% swapper [kernel.vmlinux] [k] fib_validate_source
0.00% 0.00% swapper [kernel.vmlinux] [k] sk_free
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% perf perf [.] 0xffffffffffc15bc0
0.00% 0.00% perf perf [.] 0x0000000000015bc0
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] wq_worker_sleeping
0.00% 0.00% perf [kernel.vmlinux] [k] current_kernel_time64
0.00% 0.00% perf [kernel.vmlinux] [k] cpumask_next
0.00% 0.00% snmpd [unknown] [.] 0x00000000015ab3a0
0.00% 0.00% snmpd [unknown] [.] 0x0000000000020b41
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] scsi_host_queue_ready.isra.27
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] scsi_host_in_recovery
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __switch_to
0.00% 0.00% swapper [kernel.vmlinux] [k] bit_spin_unlock.constprop.54
0.00% 0.00% perf [kernel.vmlinux] [k] fix_tail_page_for_writing
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] load_balance
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] idle_cpu
0.00% 0.00% kworker/1:1H [kernel.vmlinux] [k] ret_from_fork
lines 20486-20570\r 0.00% 0.00% swapper [kernel.vmlinux] [k] irq_chip_ack_parent
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] ata_scsi_queuecmd
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] scsi_dispatch_cmd
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] ahci_pmp_qc_defer
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_scsi_queuecmd
0.00% 0.00% swapper [kernel.vmlinux] [k] fold_diff
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] __sbitmap_queue_get
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] __blk_mq_get_tag
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] blk_mq_get_tag
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] blk_mq_get_driver_tag
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] sbitmap_get
0.00% 0.00% perf [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] sg_pool_alloc
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] __sg_alloc_table
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] sg_alloc_table_chained
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] mempool_alloc_slab
0.00% 0.00% perf perf [.] 0xffffffffffc16f26
0.00% 0.00% perf perf [.] 0x0000000000016f26
0.00% 0.00% snmpd [unknown] [.] 0x2520732520642520
0.00% 0.00% perf [kernel.vmlinux] [k] page_add_file_rmap
0.00% 0.00% swapper [kernel.vmlinux] [k] __enqueue_entity
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] load_balance
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] source_load
0.00% 0.00% snmpd [kernel.vmlinux] [k] __fdget
0.00% 0.00% snmpd [kernel.vmlinux] [k] sys_ioctl
0.00% 0.00% snmpd libc-2.23.so [.] ioctl
0.00% 0.00% snmpd [kernel.vmlinux] [k] __fget_light
0.00% 0.00% snmpd [unknown] [k] 0x00000000015a4a80
0.00% 0.00% perf perf [.] 0xffffffffffc15be5
0.00% 0.00% perf perf [.] 0x0000000000015be5
0.00% 0.00% perf [kernel.vmlinux] [k] percpu_counter_add_batch
0.00% 0.00% swapper [kernel.vmlinux] [k] notifier_call_chain
0.00% 0.00% perf perf [.] 0xffffffffffc1731b
0.00% 0.00% perf perf [.] 0x000000000001731b
0.00% 0.00% snmpd libc-2.23.so [.] 0xffff80c2d324e82e
0.00% 0.00% snmpd libc-2.23.so [.] 0x000000000007782e
0.00% 0.00% perf perf [.] 0xffffffffffcd7c5a
0.00% 0.00% perf perf [.] 0x00000000000d7c5a
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_rq_is_passthrough
0.00% 0.00% snmpd libc-2.23.so [.] 0xffff80c2d324d12d
0.00% 0.00% snmpd libc-2.23.so [.] 0x000000000007612d
0.00% 0.00% perf [kernel.vmlinux] [k] reiserfs_read_bitmap_block
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_note_context_switch
0.00% 0.00% snmpd libc-2.23.so [.] __strdup
0.00% 0.00% snmpd [unknown] [k] 0x00007fff3b470033
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_queue_enter
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] percpu_ref_tryget_live
0.00% 0.00% perf [kernel.vmlinux] [k] B_IS_IN_TREE
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] sata_scr_read
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] sata_async_notification
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ahci_scr_read
0.00% 0.00% swapper [kernel.vmlinux] [k] build_skb
0.00% 0.00% swapper [kernel.vmlinux] [k] __build_skb
0.00% 0.00% perf [kernel.vmlinux] [k] perf_mmap_to_page
0.00% 0.00% swapper [kernel.vmlinux] [k] fib_lookup.constprop.29
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_dispatch_wake
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] prepare_to_swait
0.00% 0.00% perf [kernel.vmlinux] [k] fsnotify_modify
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] __ata_scsi_queuecmd
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] default_wake_function
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] autoremove_wake_function
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] __wake_up_common
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] md_wakeup_thread
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] native_smp_send_reschedule
0.00% 0.00% perf [kernel.vmlinux] [k] fsnotify
0.00% 0.00% perf [kernel.vmlinux] [k] le_ih_k_offset
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] get_device
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] kobject_get
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_write
0.00% 0.00% syslog-ng libpthread-2.23.so [.] 0xffff806b2da4b09d
0.00% 0.00% swapper [kernel.vmlinux] [k] __sk_free
0.00% 0.00% swapper [kernel.vmlinux] [k] deferred_put_nlk_sk
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] wake_up_bit
0.00% 0.00% swapper [kernel.vmlinux] [k] need_update
0.00% 0.00% swapper [kernel.vmlinux] [k] memchr_inv
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] get_partial_node.isra.78
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __cmpxchg_double_slab.isra.59
0.00% 0.00% snmpd [kernel.vmlinux] [k] __call_rcu.constprop.61
0.00% 0.00% snmpd [kernel.vmlinux] [k] kfree_call_rcu
0.00% 0.00% snmpd [kernel.vmlinux] [k] sock_destroy_inode
lines 20571-20655\r 0.00% 0.00% snmpd [kernel.vmlinux] [k] destroy_inode
0.00% 0.00% snmpd [kernel.vmlinux] [k] evict
0.00% 0.00% snmpd [kernel.vmlinux] [k] iput
0.00% 0.00% snmpd [kernel.vmlinux] [k] dentry_unlink_inode
0.00% 0.00% snmpd [kernel.vmlinux] [k] rcu_segcblist_enqueue
0.00% 0.00% perf perf [.] 0xffffffffffc17073
0.00% 0.00% perf perf [.] 0x0000000000017073
0.00% 0.00% perf perf [.] 0xffffffffffcce953
0.00% 0.00% perf perf [.] 0x00000000000ce953
0.00% 0.00% perf [kernel.vmlinux] [k] put_prev_task_fair
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_rq_map_sg
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] scsi_init_sgtable
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] scsi_init_io
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] sd_init_command
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] scsi_setup_cmnd
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] scsi_host_queue_ready.isra.27
0.00% 0.00% swapper [kernel.vmlinux] [k] sk_destruct
0.00% 0.00% swapper [kernel.vmlinux] [k] __sk_destruct
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_work_run
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_work_interrupt
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] need_update
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] add_wait_queue
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_flush_complete_seq
0.00% 0.00% swapper [kernel.vmlinux] [k] flush_end_io
0.00% 0.00% perf perf [.] 0xffffffffffc158be
0.00% 0.00% perf perf [.] 0x00000000000158be
0.00% 0.00% swapper [kernel.vmlinux] [k] node_tag_clear
0.00% 0.00% snmpd [kernel.vmlinux] [k] snmp_seq_show
0.00% 0.00% snmpd [kernel.vmlinux] [k] find_next_bit
0.00% 0.00% snmpd [kernel.vmlinux] [k] _find_next_bit
0.00% 0.00% swapper [kernel.vmlinux] [k] kthread_data
0.00% 0.00% swapper [kernel.vmlinux] [k] smp_irq_work_interrupt
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] sbitmap_get
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] find_next_zero_bit
0.00% 0.00% swapper [kernel.vmlinux] [k] skb_release_all
0.00% 0.00% perf [kernel.vmlinux] [k] page_expected_state
0.00% 0.00% swapper [kernel.vmlinux] [k] bitmap_checkpage
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] sg_init_table
0.00% 0.00% swapper [kernel.vmlinux] [k] clear_buddies
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] prepare_to_swait_event
0.00% 0.00% swapper [kernel.vmlinux] [k] free_hot_cold_page
0.00% 0.00% swapper [kernel.vmlinux] [k] swiotlb_unmap_sg_attrs
0.00% 0.00% perf perf [.] 0xffffffffffcd7c4b
0.00% 0.00% perf perf [.] 0x00000000000d7c4b
0.00% 0.00% perf [kernel.vmlinux] [k] _raw_spin_lock_irq
0.00% 0.00% perf [kernel.vmlinux] [k] zone_statistics
0.00% 0.00% swapper [kernel.vmlinux] [k] zone_page_state_add
0.00% 0.00% snmpd [kernel.vmlinux] [k] kmem_cache_free
0.00% 0.00% snmpd [kernel.vmlinux] [k] __d_free
0.00% 0.00% snmpd [kernel.vmlinux] [k] dentry_free
0.00% 0.00% snmpd [kernel.vmlinux] [k] compound_head
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bio_advance_iter
0.00% 0.00% swapper [kernel.vmlinux] [k] irqd_cfg
0.00% 0.00% perf [kernel.vmlinux] [k] do_syscall_64
0.00% 0.00% perf [kernel.vmlinux] [k] return_from_SYSCALL_64
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] trace_block_bio_remap
0.00% 0.00% swapper [kernel.vmlinux] [k] generic_exec_single
0.00% 0.00% swapper [kernel.vmlinux] [k] smp_call_function_single_async
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_init_command
0.00% 0.00% swapper [kernel.vmlinux] [k] fib_lookup.constprop.49
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __ata_scsi_queuecmd
0.00% 0.00% swapper [kernel.vmlinux] [k] raw_notifier_call_chain
0.00% 0.00% swapper [kernel.vmlinux] [k] account_entity_enqueue
0.00% 0.00% swapper [kernel.vmlinux] [k] pollwake
0.00% 0.00% swapper [kernel.vmlinux] [k] perf_event_wakeup
0.00% 0.00% swapper [kernel.vmlinux] [k] perf_pending_event
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] select_task_rq
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] memchr_inv
0.00% 0.00% swapper [kernel.vmlinux] [k] bit_waitqueue
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] kobject_get
0.00% 0.00% kworker/44:1H [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] sbitmap_get
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __sbitmap_queue_get
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __blk_mq_get_tag
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_get_tag
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_get_driver_tag
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __sbitmap_get_word
0.00% 0.00% snmpd [kernel.vmlinux] [k] snprintf
0.00% 0.00% snmpd [kernel.vmlinux] [k] disk_name
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] find_worker_executing_work
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_stall_kick_kthreads
0.00% 0.00% swapper [kernel.vmlinux] [k] to_kthread
0.00% 0.00% perf [kernel.vmlinux] [k] trace_mm_page_alloc_zone_locked
lines 20656-20740\r 0.00% 0.00% snmpd [kernel.vmlinux] [k] kvmalloc_node
0.00% 0.00% swapper [kernel.vmlinux] [k] sg_pool_free
0.00% 0.00% swapper [kernel.vmlinux] [k] __sg_free_table
0.00% 0.00% swapper [kernel.vmlinux] [k] sg_free_table_chained
0.00% 0.00% swapper [kernel.vmlinux] [k] scsi_mq_free_sgtables
0.00% 0.00% swapper [kernel.vmlinux] [k] scsi_mq_uninit_cmd
0.00% 0.00% swapper [kernel.vmlinux] [k] napi_complete_done
0.00% 0.00% swapper [kernel.vmlinux] [k] __accumulate_pelt_segments
0.00% 0.00% snmpd libnetsnmpmibs.so.30.0.3 [.] netsnmp_access_interface_container_init@plt
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __wake_up_locked
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] ep_poll_callback
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __wake_up_locked_key
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __wake_up_common
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] eventfd_write
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __vfs_write
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] vfs_write
0.00% 0.00% kworker/u112:0 [kernel.vmlinux] [k] ixgbe_ptp_rx_hang
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] load_balance
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] _find_next_bit
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] balance_callback
0.00% 0.00% swapper [kernel.vmlinux] [k] common_interrupt
0.00% 0.00% swapper [kernel.vmlinux] [k] node_page_state_add
0.00% 0.00% perf [kernel.vmlinux] [k] get_neighbors
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] select_task_rq_fair
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] wake_up_new_task
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] target_load
0.00% 0.00% perf [kernel.vmlinux] [k] search_for_position_by_key.part.16
0.00% 0.00% swapper [kernel.vmlinux] [k] __neigh_lookup.constprop.21
0.00% 0.00% swapper [kernel.vmlinux] [k] hrtimer_wakeup
0.00% 0.00% swapper [kernel.vmlinux] [k] do_timer
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] update_load_avg
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] enqueue_task_fair
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] enqueue_task
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] activate_task
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ttwu_do_activate.isra.85
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] default_wake_function
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] autoremove_wake_function
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __wake_up_common
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] update_cfs_rq_load_avg
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] list_del_init
0.00% 0.00% swapper [kernel.vmlinux] [k] skb_release_data
0.00% 0.00% swapper [kernel.vmlinux] [k] ixgbe_service_timer
0.00% 0.00% swapper [kernel.vmlinux] [k] mod_timer
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] __queue_delayed_work
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] queue_delayed_work_on
0.00% 0.00% swapper [kernel.vmlinux] [k] get_work_pool
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] dyntick_save_progress_counter
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] vmstat_update
0.00% 0.00% swapper [kernel.vmlinux] [k] delayed_work_timer_fn
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] vmstat_update
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] refresh_cpu_vm_stats
0.00% 0.00% swapper [kernel.vmlinux] [k] skb_queue_purge
0.00% 0.00% swapper [kernel.vmlinux] [k] netlink_sock_destruct
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __fdget_pos
0.00% 0.00% swapper [kernel.vmlinux] [k] calc_global_load
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] find_bio_disk
0.00% 0.00% swapper [kernel.vmlinux] [k] fprop_reflect_period_percpu.isra.5
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] schedule
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] cpumask_next
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] md_update_sb
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_end_request
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] write_page
0.00% 0.00% swapper [kernel.vmlinux] [k] radix_tree_lookup
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_timer_start
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_add_timer
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] irq_exit
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] do_IRQ
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] ret_from_intr
0.00% 0.00% swapper [kernel.vmlinux] [k] update_curr
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] io_schedule
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] bit_wait_io
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __wait_on_bit
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] out_of_line_wait_on_bit
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __wait_on_buffer
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] wait_on_buffer
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] con_write
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] n_tty_write
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] tty_write
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_loop_readv_writev
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_iter_write
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] vfs_writev
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_writev
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_writev
lines 20741-20825\r 0.00% 0.00% syslog-ng libc-2.23.so [.] writev
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_con_write
0.00% 0.00% init [unknown] [k] 0000000000000000
0.00% 0.00% bgpd [kernel.vmlinux] [k] do_select
0.00% 0.00% bgpd [kernel.vmlinux] [k] core_sys_select
0.00% 0.00% bgpd [kernel.vmlinux] [k] sys_pselect6
0.00% 0.00% bgpd [kernel.vmlinux] [k] entry_SYSCALL_64_fastpath
0.00% 0.00% bgpd libc-2.23.so [.] pselect
0.00% 0.00% swapper [kernel.vmlinux] [k] neigh_update
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] __cond_resched_lock
0.00% 0.00% swapper [kernel.vmlinux] [k] dev_watchdog
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] __sbitmap_get_word
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] __queue_work
0.00% 0.00% swapper [kernel.vmlinux] [k] radix_tree_load_root
0.00% 0.00% perf [kernel.vmlinux] [k] __free_pages
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] dequeue_task
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bitmap_update_sb
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] compound_head
0.00% 0.00% perf perf [.] 0xffffffffffc6eae0
0.00% 0.00% perf perf [.] 0x000000000006eae0
0.00% 0.00% swapper [kernel.vmlinux] [k] kfree_skbmem
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] reiserfs_bmap_count
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] md_super_write
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] ntp_tick_length
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_read
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_try_merge
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_rq_merge_ok
0.00% 0.00% swapper [kernel.vmlinux] [k] __mix_pool_bytes
0.00% 0.00% swapper [kernel.vmlinux] [k] _mix_pool_bytes
0.00% 0.00% kworker/46:1H [kernel.vmlinux] [k] finish_task_switch
0.00% 0.00% sshd [kernel.vmlinux] [k] schedule_hrtimeout_range
0.00% 0.00% sshd [kernel.vmlinux] [k] poll_schedule_timeout
0.00% 0.00% sshd [kernel.vmlinux] [k] do_select
0.00% 0.00% sshd [kernel.vmlinux] [k] core_sys_select
0.00% 0.00% sshd [kernel.vmlinux] [k] sys_select
0.00% 0.00% sshd [kernel.vmlinux] [k] entry_SYSCALL_64_fastpath
0.00% 0.00% sshd libc-2.23.so [.] __select
0.00% 0.00% sshd [kernel.vmlinux] [k] schedule
0.00% 0.00% sshd sshd [.] 0xffffffec25166ebb
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] put_pwq
0.00% 0.00% swapper [kernel.vmlinux] [k] select_task_rq
0.00% 0.00% swapper [kernel.vmlinux] [k] neigh_lookup
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] find_next_bit
0.00% 0.00% swapper [kernel.vmlinux] [k] kfree_skb
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_start_request
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] load_balance
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% snmpd [kernel.vmlinux] [k] do_syscall_64
0.00% 0.00% snmpd [kernel.vmlinux] [k] return_from_SYSCALL_64
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] wake_up_process
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] wake_up_worker
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] insert_work
0.00% 0.00% init [kernel.vmlinux] [k] do_select
0.00% 0.00% init [kernel.vmlinux] [k] core_sys_select
0.00% 0.00% init [kernel.vmlinux] [k] sys_select
0.00% 0.00% init [kernel.vmlinux] [k] entry_SYSCALL_64_fastpath
0.00% 0.00% init libc-2.23.so [.] __select
0.00% 0.00% swapper [kernel.vmlinux] [k] skb_free_head
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] update_rq_clock
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] percpu_ref_get_many.constprop.65
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] cpumask_next
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_qc_new_init
0.00% 0.00% syslog-ng libglib-2.0.so.0.5000.3 [.] g_string_insert_len
0.00% 0.00% swapper [kernel.vmlinux] [k] kblockd_schedule_delayed_work
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_kick_requeue_list
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_mq_add_to_requeue_list
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_flush_queue_rq
0.00% 0.00% swapper [kernel.vmlinux] [k] native_read_msr.constprop.2
0.00% 0.00% swapper [kernel.vmlinux] [k] igb_msix_ring
0.00% 0.00% swapper [kernel.vmlinux] [k] netif_schedule_queue
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] vmstat_update
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] refresh_cpu_vm_stats
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] mempool_alloc_slab
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] mempool_alloc
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bio_alloc_bioset
lines 20826-20910\r 0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] md_bio_alloc_sync
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] kmem_cache_alloc
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] load_balance
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] list_sort
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] merge
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] account_entity_dequeue
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_tf_init
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __fget
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __fget_light
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __fdget
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] fdget_pos
0.00% 0.00% syslog-ng libc-2.23.so [.] __read_chk
0.00% 0.00% syslog-ng libsyslog-ng-3.7.so.0.0.0 [.] 0xffff806b2d54e660
0.00% 0.00% syslog-ng libsyslog-ng-3.7.so.0.0.0 [.] 0xffff806b2d546990
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __fcheck_files
0.00% 0.00% syslog-ng [unknown] [k] 0x5b000001bc8389ff
0.00% 0.00% swapper [kernel.vmlinux] [k] neigh_release
0.00% 0.00% swapper [kernel.vmlinux] [k] dev_gro_receive
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] _find_next_bit
0.00% 0.00% init [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% init [kernel.vmlinux] [k] schedule
0.00% 0.00% init [kernel.vmlinux] [k] schedule_hrtimeout_range_clock
0.00% 0.00% init [kernel.vmlinux] [k] schedule_hrtimeout_range
0.00% 0.00% init [kernel.vmlinux] [k] poll_schedule_timeout
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] sg_alloc_table_chained
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] scsi_init_sgtable
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] scsi_init_io
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] sd_init_command
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] scsi_setup_cmnd
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] sg_init_table
0.00% 0.00% swapper [kernel.vmlinux] [k] kmem_cache_alloc
0.00% 0.00% bgpd [kernel.vmlinux] [k] apic_timer_interrupt
0.00% 0.00% bgpd libzebra.so.0.0.0 [.] mqueue_done_waiting
0.00% 0.00% swapper [kernel.vmlinux] [k] _raw_write_unlock_bh
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] first_online_pgdat
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_get_request
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] __ata_scsi_queuecmd
0.00% 0.00% kworker/45:1H [kernel.vmlinux] [k] ata_std_qc_defer
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] raid1_write_request
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] raid1_make_request
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] md_submit_flush_data
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] bitmap_startwrite
0.00% 0.00% perf [kernel.vmlinux] [k] kmalloc_slab
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] sched_clock_cpu
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] native_sched_clock
0.00% 0.00% swapper [kernel.vmlinux] [k] clocksource_watchdog
0.00% 0.00% swapper [kernel.vmlinux] [k] read_hpet
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] refresh_cpu_vm_stats
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] rcu_all_qs
0.00% 0.00% swapper [kernel.vmlinux] [k] accumulate_nsecs_to_secs.constprop.8
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] ktime_get
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] clockevents_program_event
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] tick_program_event
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] hrtimer_interrupt
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] local_apic_timer_interrupt
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] smp_apic_timer_interrupt
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] apic_timer_interrupt
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __read_seqcount_begin.constprop.20
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] blk_do_io_stat
0.00% 0.00% perf [kernel.vmlinux] [k] rcu_all_qs
0.00% 0.00% swapper [kernel.vmlinux] [k] queued_write_lock
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] unmap_single_vma
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] zap_page_range
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_madvise
0.00% 0.00% syslog-ng libc-2.23.so [.] __madvise
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] pmd_none_or_trans_huge_or_clear_bad
0.00% 0.00% swapper [kernel.vmlinux] [k] wakeup_softirqd
0.00% 0.00% swapper [kernel.vmlinux] [k] __dev_queue_xmit
0.00% 0.00% swapper [kernel.vmlinux] [k] dev_queue_xmit
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_finish_output2
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_finish_output
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_output
0.00% 0.00% swapper [kernel.vmlinux] [k] dst_output
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_local_out
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_send_skb
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_push_pending_frames
0.00% 0.00% swapper [kernel.vmlinux] [k] icmp_push_reply
0.00% 0.00% swapper [kernel.vmlinux] [k] icmp_send
0.00% 0.00% swapper [kernel.vmlinux] [k] __udp4_lib_rcv
0.00% 0.00% swapper [kernel.vmlinux] [k] udp_rcv
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_local_deliver_finish
lines 20911-20995\r 0.00% 0.00% swapper [kernel.vmlinux] [k] ip_local_deliver
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_rcv_finish
0.00% 0.00% swapper [kernel.vmlinux] [k] ip_rcv
0.00% 0.00% swapper [kernel.vmlinux] [k] netdev_pick_tx
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_accelerate_cbs
0.00% 0.00% swapper [kernel.vmlinux] [k] deliver_ptype_list_skb
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% swapper [kernel.vmlinux] [k] igb_write_itr
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] source_load
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_sched_wake_idle_without_ipi
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] calc_wheel_index
0.00% 0.00% swapper [kernel.vmlinux] [k] page_frag_free
0.00% 0.00% kworker/1:0 [kernel.vmlinux] [k] update_curr
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] sbitmap_for_each_set
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_queue_tag_busy_iter
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_timeout_work
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] blk_mq_check_expired
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_may_run
0.00% 0.00% swapper [kernel.vmlinux] [k] writeout_period
0.00% 0.00% sshd [unknown] [.] 0x00000013db547b40
0.00% 0.00% sshd libc-2.23.so [.] sigaddset
0.00% 0.00% sshd [unknown] [.] 0x0000000000000100
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_sched_try_merge
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] alloc_file
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] anon_inode_getfile
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] SYSC_epoll_create1
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_epoll_create1
0.00% 0.00% syslog-ng libc-2.23.so [.] epoll_create1
0.00% 0.00% syslog-ng [unknown] [.] 0x0000000100000002
0.00% 0.00% syslog-ng libsyslog-ng-3.7.so.0.0.0 [.] 0xffff806b2d579220
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] get_empty_filp
0.00% 0.00% syslog-ng [unknown] [k] 0x00002dbc3b031b01
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] load_balance
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] source_load
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] find_next_bit
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] cpumask_next
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] cpumask_next_and
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] do_raw_spin_lock
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] enqueue_task_fair
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] enqueue_task
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] activate_task
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] ttwu_do_activate.isra.85
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] wake_up_process
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] wake_up_worker
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] insert_work
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __queue_work
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] next_zone
0.00% 0.00% bgpd [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% bgpd [kernel.vmlinux] [k] dequeue_task
0.00% 0.00% bgpd [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% bgpd [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% bgpd [kernel.vmlinux] [k] schedule
0.00% 0.00% bgpd [kernel.vmlinux] [k] schedule_hrtimeout_range_clock
0.00% 0.00% bgpd [kernel.vmlinux] [k] schedule_hrtimeout_range
0.00% 0.00% bgpd [kernel.vmlinux] [k] poll_schedule_timeout
0.00% 0.00% bgpd [kernel.vmlinux] [k] update_curr
0.00% 0.00% bgpd [unknown] [k] 0x00007f576e605520
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] account_entity_dequeue
0.00% 0.00% swapper [kernel.vmlinux] [k] __napi_schedule
0.00% 0.00% swapper [kernel.vmlinux] [k] napi_schedule
0.00% 0.00% swapper [kernel.vmlinux] [k] arch_local_irq_save
0.00% 0.00% swapper [kernel.vmlinux] [k] rb_erase_cached
0.00% 0.00% swapper [kernel.vmlinux] [k] __radix_tree_lookup
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] local_clock
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] perf_event_task_output
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] perf_iterate_sb
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] perf_event_task
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] perf_event_exit_task
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_exit
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_group_exit
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sched_clock_cpu
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] get_nr_dirty_inodes
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] get_nr_dirty_pages
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] get_nr_inodes_unused
0.00% 0.00% swapper [kernel.vmlinux] [k] remote_function
0.00% 0.00% swapper [kernel.vmlinux] [k] __perf_event_disable
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] set_in_sync
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_do_io_stat
lines 20996-21080\r 0.00% 0.00% perf [kernel.vmlinux] [k] restore_regs_and_iret
0.00% 0.00% sshd libcrypto.so.1.0.0 [.] 0xffff80ffd4e6a672
0.00% 0.00% sshd libcrypto.so.1.0.0 [.] 0x000000000006c672
0.00% 0.00% sshd [unknown] [.] 0x656aae96cd78985b
0.00% 0.00% zebra libc-2.23.so [.] __clock_gettime
0.00% 0.00% zebra [vdso] [.] __vdso_clock_gettime
0.00% 0.00% zebra [unknown] [.] 0x0000000000000002
0.00% 0.00% swapper [kernel.vmlinux] [k] sg_next
0.00% 0.00% init [kernel.vmlinux] [k] load_balance
0.00% 0.00% init [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% init [kernel.vmlinux] [k] find_busiest_group
0.00% 0.00% swapper [kernel.vmlinux] [k] apic_ack_edge
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] try_to_wake_up
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] default_wake_function
0.00% 0.00% swapper [kernel.vmlinux] [k] skb_release_head_state
0.00% 0.00% swapper [kernel.vmlinux] [k] sock_rfree
0.00% 0.00% swapper [kernel.vmlinux] [k] free_pcppages_bulk
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] sg_assign_page.isra.29
0.00% 0.00% swapper [kernel.vmlinux] [k] netif_tx_unlock
0.00% 0.00% swapper [kernel.vmlinux] [k] __d_free
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] scsi_dev_queue_ready
0.00% 0.00% swapper [kernel.vmlinux] [k] fprop_new_period
0.00% 0.00% swapper [kernel.vmlinux] [k] __percpu_counter_sum
0.00% 0.00% swapper [kernel.vmlinux] [k] __end_buffer_read_notouch
0.00% 0.00% swapper [kernel.vmlinux] [k] end_buffer_read_sync
0.00% 0.00% swapper [kernel.vmlinux] [k] raid1_end_read_request
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_bio_to_request
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_move_irq
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] queue_work_on
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] kernfs_notify
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] sysfs_notify_dirent_safe
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] update_curr
0.00% 0.00% bgpd bgpd [.] qt_jitter@plt
0.00% 0.00% bgpd [unknown] [.] 0000000000000000
0.00% 0.00% zebra libc-2.23.so [.] __select
0.00% 0.00% zebra [kernel.vmlinux] [k] __switch_to
0.00% 0.00% zebra [unknown] [k] 0x00000000000100f9
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] round_jiffies_up
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_rq_timeout
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] round_jiffies_common
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] __ata_scsi_queuecmd
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] ata_scsi_queuecmd
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] scsi_dispatch_cmd
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] ata_qc_issue
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] ttwu_do_activate.isra.85
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] wq_worker_waking_up
0.00% 0.00% perf [kernel.vmlinux] [k] indirect_bytes_number
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] _raw_spin_lock_irq
0.00% 0.00% kworker/u114:0 [kernel.vmlinux] [k] queued_spin_lock_slowpath
0.00% 0.00% swapper [kernel.vmlinux] [k] end_buffer_write_sync
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] dequeue_task
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% kworker/31:1H [kernel.vmlinux] [k] update_load_avg
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] worker_clr_flags
0.00% 0.00% perf [kernel.vmlinux] [k] mnt_clone_write
0.00% 0.00% swapper [kernel.vmlinux] [k] scsi_device_unbusy
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] update_curr
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] md_super_wait
0.00% 0.00% syslog-ng libsyslog-ng-3.7.so.0.0.0 [.] iv_fd_poll_and_run
0.00% 0.00% syslog-ng [unknown] [k] 0x00007fff00ce1a00
0.00% 0.00% swapper [kernel.vmlinux] [k] virt_to_head_page
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] submit_bio
0.00% 0.00% swapper [kernel.vmlinux] [k] __note_gp_changes
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_advance_cbs
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] finish_swait
0.00% 0.00% swapper [kernel.vmlinux] [k] ___neigh_lookup_noref
0.00% 0.00% swapper [kernel.vmlinux] [k] __phys_addr_nodebug
0.00% 0.00% swapper [kernel.vmlinux] [k] igb_alloc_rx_buffers
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/5:1 [kernel.vmlinux] [k] finish_task_switch
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] fold_diff
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] mutex_unlock
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] devkmsg_read
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __vfs_read
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] vfs_read
0.00% 0.00% syslog-ng libpthread-2.23.so [.] __GI___libc_read
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] atomic64_cmpxchg
0.00% 0.00% syslog-ng [unknown] [k] 0x98000000000150ef
0.00% 0.00% swapper [kernel.vmlinux] [k] compound_order
lines 21081-21165\r 0.00% 0.00% swapper [kernel.vmlinux] [k] native_irq_return_iret
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] trace_timer_init
0.00% 0.00% snmpd libc-2.23.so [.] 0xffff80c2d324e229
0.00% 0.00% snmpd libc-2.23.so [.] 0x0000000000077229
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] update_curr
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] dequeue_task
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/4:2 [kernel.vmlinux] [k] update_min_vruntime
0.00% 0.00% swapper [kernel.vmlinux] [k] eth_type_trans
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] put_prev_task_fair
0.00% 0.00% swapper [kernel.vmlinux] [k] __netif_schedule
0.00% 0.00% syslog-ng libsyslog-ng-3.7.so.0.0.0 [.] 0xffff806b2d564910
0.00% 0.00% swapper [kernel.vmlinux] [k] arp_hash
0.00% 0.00% init [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% init [kernel.vmlinux] [k] dequeue_task
0.00% 0.00% init [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% init [kernel.vmlinux] [k] update_curr
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] hrtimer_cancel
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] schedule_hrtimeout_range_clock
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] schedule_hrtimeout_range
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] hrtimer_try_to_cancel
0.00% 0.00% syslog-ng [unknown] [k] 0x3800000000014e36
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] __mod_timer
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] mod_timer
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] add_timer
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] enqueue_timer
0.00% 0.00% snmpd [kernel.vmlinux] [k] vfs_open
0.00% 0.00% snmpd [kernel.vmlinux] [k] proc_reg_open
0.00% 0.00% kworker/33:2 [kernel.vmlinux] [k] vmstat_update
0.00% 0.00% kworker/33:2 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/33:2 [kernel.vmlinux] [k] worker_thread
0.00% 0.00% kworker/33:2 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/33:2 [kernel.vmlinux] [k] refresh_cpu_vm_stats
0.00% 0.00% kworker/33:2 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] update_cfs_shares
0.00% 0.00% perf libpthread-2.23.so [.] __GI___libc_write
0.00% 0.00% swapper [kernel.vmlinux] [k] find_first_bit
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] hlist_del_init
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] disk_map_sector_rcu
0.00% 0.00% swapper [kernel.vmlinux] [k] select_task_rq_fair
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] kblockd_schedule_delayed_work
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_kick_requeue_list
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_mq_add_to_requeue_list
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_flush_queue_rq
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] blk_flush_complete_seq
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] flush_end_io
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] __blk_mq_end_request
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] queue_delayed_work_on
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] radix_tree_lookup
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] get_work_pool
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] radix_tree_descend
0.00% 0.00% swapper [kernel.vmlinux] [k] queue_work_on
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] source_load
0.00% 0.00% swapper [kernel.vmlinux] [k] apic_eoi
0.00% 0.00% kworker/44:0 [kernel.vmlinux] [k] finish_task_switch
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] reiserfs_discard_all_prealloc
0.00% 0.00% zebra [kernel.vmlinux] [k] cputime_adjust
0.00% 0.00% zebra [kernel.vmlinux] [k] thread_group_cputime_adjusted
0.00% 0.00% zebra [kernel.vmlinux] [k] getrusage
0.00% 0.00% zebra [kernel.vmlinux] [k] SYSC_getrusage
0.00% 0.00% zebra [kernel.vmlinux] [k] sys_getrusage
0.00% 0.00% zebra [kernel.vmlinux] [k] entry_SYSCALL_64_fastpath
0.00% 0.00% zebra libc-2.23.so [.] getrusage
0.00% 0.00% zebra [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.00% 0.00% zebra [unknown] [k] 0x00000000000100be
0.00% 0.00% perf [kernel.vmlinux] [k] fsnotify_perm
0.00% 0.00% bgpd libzebra.so.0.0.0 [.] qps_pselect
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_scsi_rw_xlat
0.00% 0.00% kworker/0:1H [kernel.vmlinux] [k] ata_build_rw_tf
0.00% 0.00% swapper [kernel.vmlinux] [k] atomic_add_unless.constprop.56
0.00% 0.00% init init [.] 0x0000000000004f24
0.00% 0.00% init init [.] 0xffffffffffc04f24
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/3:1 [kernel.vmlinux] [k] neigh_periodic_work
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] vmstat_update
0.00% 0.00% kworker/0:2 [kernel.vmlinux] [k] refresh_cpu_vm_stats
0.00% 0.00% swapper [kernel.vmlinux] [k] super_written
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] get_work_pool
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] __queue_work
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] queue_work_on
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] tty_schedule_flip
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] tty_flip_buffer_push
lines 21166-21250\r 0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] pty_write
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] tty_put_char
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] __process_echoes
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] n_tty_receive_buf_common
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] n_tty_receive_buf2
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] tty_ldisc_receive_buf
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] tty_port_default_receive_buf
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] flush_to_ldisc
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] process_one_work
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] __radix_tree_lookup
0.00% 0.00% swapper [kernel.vmlinux] [k] cfs_rq_util_change
0.00% 0.00% snmpd [unknown] [.] 0x0000000000000001
0.00% 0.00% snmpd libnetsnmp.so.30.0.3 [.] netsnmp_container_get_binary_array
0.00% 0.00% snmpd [unknown] [.] 0x0072656e6961746e
0.00% 0.00% swapper [kernel.vmlinux] [k] ixgbe_service_event_schedule
0.00% 0.00% init libc-2.23.so [.] __xstat64
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] enqueue_task
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] activate_task
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] ttwu_do_activate.isra.85
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] enqueue_task_fair
0.00% 0.00% swapper [kernel.vmlinux] [k] memcmp
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] update_min_vruntime
0.00% 0.00% syslog-ng libc-2.23.so [.] vfprintf
0.00% 0.00% syslog-ng libc-2.23.so [.] __vsnprintf_chk
0.00% 0.00% swapper [kernel.vmlinux] [k] is_swiotlb_buffer
0.00% 0.00% swapper [kernel.vmlinux] [k] ahci_port_base
0.00% 0.00% swapper [kernel.vmlinux] [k] swiotlb_sync_single_for_device
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] cpuacct_charge
0.00% 0.00% swapper [kernel.vmlinux] [k] blk_account_io_completion
0.00% 0.00% kworker/u113:0 [kernel.vmlinux] [k] get_work_pwq
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] try_to_del_timer_sync
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] bitmap_unplug
0.00% 0.00% swapper [kernel.vmlinux] [k] note_interrupt
0.00% 0.00% swapper [kernel.vmlinux] [k] __mod_timer
0.00% 0.00% swapper [kernel.vmlinux] [k] enqueue_timer
0.00% 0.00% swapper [kernel.vmlinux] [k] scsi_handle_queue_ramp_up
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] schedule
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] futex_wait_queue_me
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] futex_wait
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] do_futex
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] sys_futex
0.00% 0.00% syslog-ng libpthread-2.23.so [.] pthread_join
0.00% 0.00% syslog-ng [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% syslog-ng libc-2.23.so [.] epoll_ctl
0.00% 0.00% syslog-ng [unknown] [k] 0x0000000000000008
0.00% 0.00% swapper [kernel.vmlinux] [k] irq_complete_move
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __accumulate_pelt_segments
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] update_load_avg
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __queue_delayed_work
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] queue_delayed_work_on
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] kblockd_schedule_delayed_work
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_kick_requeue_list
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_add_to_requeue_list
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_flush_queue_rq
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_flush_complete_seq
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_insert_flush
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] blk_mq_sched_insert_request
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] decay_load
0.00% 0.00% swapper [kernel.vmlinux] [k] __skb_put
0.00% 0.00% init init [.] 0xffffffffffc06153
0.00% 0.00% init init [.] 0x0000000000006153
0.00% 0.00% rcu_sched [kernel.vmlinux] [k] sched_clock_cpu
0.00% 0.00% swapper [kernel.vmlinux] [k] native_iret
0.00% 0.00% swapper [kernel.vmlinux] [k] __switch_to_asm
0.00% 0.00% bgpd [kernel.vmlinux] [k] sock_poll
0.00% 0.00% bgpd [kernel.vmlinux] [k] sock_rps_record_flow
0.00% 0.00% bgpd [unknown] [k] 0x4b2d32314344342d
0.00% 0.00% swapper [kernel.vmlinux] [k] sock_flag
0.00% 0.00% swapper [kernel.vmlinux] [k] __wake_up_locked
0.00% 0.00% swapper [kernel.vmlinux] [k] ep_poll_callback
0.00% 0.00% swapper [kernel.vmlinux] [k] wake_up_klogd_work_func
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] schedule
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] percpu_ref_switch_to_atomic_sync
0.00% 0.00% md2_raid1 [kernel.vmlinux] [k] pick_next_task_fair
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_timer_cancel
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/1:2 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_napi_poll
0.00% 0.00% kworker/45:1 [kernel.vmlinux] [k] deactivate_task
0.00% 0.00% kworker/45:1 [kernel.vmlinux] [k] __sched_text_start
0.00% 0.00% kworker/45:1 [kernel.vmlinux] [k] schedule
0.00% 0.00% kworker/45:1 [kernel.vmlinux] [k] worker_thread
lines 21251-21335\r 0.00% 0.00% kworker/45:1 [kernel.vmlinux] [k] kthread
0.00% 0.00% kworker/45:1 [kernel.vmlinux] [k] dequeue_task_fair
0.00% 0.00% kworker/45:1 [kernel.vmlinux] [k] ret_from_fork
0.00% 0.00% syslog-ng libc-2.23.so [.] 0x00000000000752d2
0.00% 0.00% syslog-ng libc-2.23.so [.] 0xffff806b2de492d2
0.00% 0.00% swapper [kernel.vmlinux] [k] hrtick_enabled
0.00% 0.00% swapper [kernel.vmlinux] [k] event_function
0.00% 0.00% swapper [kernel.vmlinux] [k] local_clock
0.00% 0.00% swapper [kernel.vmlinux] [k] update_context_time
0.00% 0.00% swapper [kernel.vmlinux] [k] default_send_IPI_single_phys
0.00% 0.00% swapper [kernel.vmlinux] [k] netlink_skb_destructor
0.00% 0.00% swapper [kernel.vmlinux] [k] ata_qc_free
0.00% 0.00% swapper [kernel.vmlinux] [k] rcu_report_qs_rnp
0.00% 0.00% swapper [kernel.vmlinux] [k] trace_sched_waking
0.00% 0.00% swapper [kernel.vmlinux] [k] apic_chip_data
0.00% 0.00% swapper [kernel.vmlinux] [k] igb_rx_bufsz
0.00% 0.00% swapper [kernel.vmlinux] [k] calc_load_fold_active
0.00% 0.00% swapper [kernel.vmlinux] [k] schedule_work
0.00% 0.00% swapper [kernel.vmlinux] [k] igb_watchdog
0.00% 0.00% swapper [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% swapper [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% perf [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/14 [kernel.vmlinux] [k] wrmsrl
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/15 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/16 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/17 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] ctx_resched
lines 21336-21420\r 0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/18 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/19 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/20 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/21 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/22 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/23 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/24 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/25 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] intel_pmu_enable_all
lines 21421-21505\r 0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/26 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/27 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/42 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] event_function
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] remote_function
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] flush_smp_call_function_queue
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] generic_smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] smp_call_function_single_interrupt
0.00% 0.00% ksoftirqd/43 [kernel.vmlinux] [k] call_function_single_interrupt
0.00% 0.00% perf [kernel.vmlinux] [k] intel_pmu_enable_all
0.00% 0.00% perf [kernel.vmlinux] [k] x86_pmu_enable
0.00% 0.00% perf [kernel.vmlinux] [k] perf_pmu_enable
0.00% 0.00% perf [kernel.vmlinux] [k] ctx_resched
0.00% 0.00% perf [kernel.vmlinux] [k] __perf_event_enable
0.00% 0.00% perf [kernel.vmlinux] [k] event_function
0.00% 0.00% perf [kernel.vmlinux] [k] remote_function
0.00% 0.00% perf [kernel.vmlinux] [k] generic_exec_single
0.00% 0.00% perf [kernel.vmlinux] [k] smp_call_function_single
0.00% 0.00% perf [kernel.vmlinux] [k] cpu_function_call
0.00% 0.00% perf [kernel.vmlinux] [k] event_function_call
0.00% 0.00% perf [kernel.vmlinux] [k] _perf_event_enable
0.00% 0.00% perf [kernel.vmlinux] [k] perf_event_for_each_child
0.00% 0.00% perf [kernel.vmlinux] [k] perf_ioctl
0.00% 0.00% perf [kernel.vmlinux] [k] vfs_ioctl
0.00% 0.00% perf [kernel.vmlinux] [k] do_vfs_ioctl
0.00% 0.00% perf [kernel.vmlinux] [k] sys_ioctl
0.00% 0.00% perf libc-2.23.so [.] ioctl
0.00% 0.00% perf [unknown] [k] 0000000000000000
0.00% 0.00% perf [unknown] [k] 0x00000000032185a0
0.00% 0.00% perf [unknown] [k] 0x000000000320b910
#
# (Cannot load tips.txt file, please install perf!)
#
lines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)\r\rlines 21490-21574/21574 (END)
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-12 17:27 ` Paweł Staszewski
@ 2017-08-13 16:58 ` Paweł Staszewski
2017-08-14 16:19 ` Jesper Dangaard Brouer
2017-08-14 0:07 ` Alexander Duyck
1 sibling, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-13 16:58 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Linux Kernel Network Developers
To show some difference below comparision vlan/no-vlan traffic
10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
(ixgbe in kernel driver kernel 4.13.0-rc4-next-20170811)
ethtool settings for both tests:
ethtool -K $ifc gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple off
ethtool -L $ifc combined 16
ethtool -C $ifc rx-usecs 2
ethtool -G $ifc rx 4096 tx 1024
16 CORES / 16 RSS QUEUES
Tx traffic on vlan:
RX Interface:
enp216s0f0
TX Interface
vlan1000 added to enp216s0f1 interface (with vlan 1000 ip address assigned)
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;6939008;416325120;6938696;402411192
1;16;64;6941952;416444160;6941745;402558918
2;16;64;6960576;417584640;6960707;403698718
3;16;64;6940736;416486400;6941820;402503876
4;16;64;6927680;415741440;6927420;401853870
5;16;64;6929792;415687680;6929917;401839196
6;16;64;6950400;416989440;6950661;403026166
7;16;64;6953664;417216000;6953454;403260544
8;16;64;6948480;416851200;6948800;403023266
9;16;64;6924160;415422720;6924092;401542468
100% load on all 16 Cores.
vs
RX interface from traffic generator:
enp216s0f0
TX interface to the sink:
enp216s0f1
No vlan used
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;10280176;793608540;10298496;596796568
1;16;64;10046928;600978780;10046022;582527002
2;16;64;10032956;601827420;10026097;581515656
3;16;64;10051503;602252460;10067880;582420804
4;16;64;10016204;602725140;10017358;582644800
5;16;64;10035575;602437620;10059504;582067294
6;16;64;10041667;603069780;10057865;582477412
7;16;64;10044448;600027420;10046526;581022018
8;16;64;10022436;601121100;10025946;581904314
9;16;64;10036231;602514960;10058724;582180684
So we have 10Mpps forwarded
- have problems with pktgen on my traffic generator to push more than
10M but this low budget hardware so.. :)
And there are still free cpu cycles so probabbly can forward at line 10G
rate 14Mpps
Average: CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %gnice %idle
Average: all 0.00 0.00 0.00 0.00 0.00 20.91
0.00 0.00 0.00 79.09
Average: 0 0.00 0.00 0.00 0.00 0.00 0.09 0.00
0.00 0.00 99.91
Average: 1 0.03 0.00 0.03 0.00 0.00 0.00 0.00
0.00 0.00 99.94
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 5 0.00 0.00 0.18 0.00 0.00 0.00 0.00
0.00 0.00 99.82
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 10 0.00 0.00 0.03 0.24 0.00 0.00 0.00
0.00 0.00 99.74
Average: 11 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 14 0.00 0.00 0.00 0.00 0.00 92.38
0.00 0.00 0.00 7.62
Average: 15 0.00 0.00 0.00 0.00 0.00 85.88
0.00 0.00 0.00 14.12
Average: 16 0.00 0.00 0.00 0.00 0.00 64.91
0.00 0.00 0.00 35.09
Average: 17 0.00 0.00 0.00 0.00 0.00 66.76
0.00 0.00 0.00 33.24
Average: 18 0.00 0.00 0.00 0.00 0.00 65.57
0.00 0.00 0.00 34.43
Average: 19 0.00 0.00 0.00 0.00 0.00 66.38
0.00 0.00 0.00 33.62
Average: 20 0.00 0.00 0.00 0.00 0.00 72.97
0.00 0.00 0.00 27.03
Average: 21 0.00 0.00 0.00 0.00 0.00 70.80
0.00 0.00 0.00 29.20
Average: 22 0.00 0.00 0.00 0.00 0.00 66.44
0.00 0.00 0.00 33.56
Average: 23 0.00 0.00 0.00 0.00 0.00 66.12
0.00 0.00 0.00 33.88
Average: 24 0.00 0.00 0.00 0.00 0.00 68.35
0.00 0.00 0.00 31.65
Average: 25 0.00 0.00 0.00 0.00 0.00 71.79
0.00 0.00 0.00 28.21
Average: 26 0.00 0.00 0.00 0.00 0.00 70.24
0.00 0.00 0.00 29.76
Average: 27 0.00 0.00 0.00 0.00 0.00 73.24
0.00 0.00 0.00 26.76
Average: 28 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 39 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 42 0.00 0.00 0.00 0.00 0.00 84.27
0.00 0.00 0.00 15.73
Average: 43 0.00 0.00 0.00 0.00 0.00 84.50
0.00 0.00 0.00 15.50
Average: 44 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 45 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 46 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 47 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 48 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 49 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 51 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 52 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 54 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: 55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00 100.00
Average: CPU intr/s
Average: all 3559661.68
Average: 0 628.53
Average: 1 537.62
Average: 2 525.00
Average: 3 558.29
Average: 4 546.79
Average: 5 522.85
Average: 6 508.06
Average: 7 568.88
Average: 8 529.56
Average: 9 535.29
Average: 10 530.09
Average: 11 539.53
Average: 12 520.82
Average: 13 531.32
Average: 14 73315.68
Average: 15 115983.15
Average: 16 254446.09
Average: 17 253067.79
Average: 18 254446.35
Average: 19 252457.29
Average: 20 213928.18
Average: 21 232770.32
Average: 22 263906.85
Average: 23 260065.09
Average: 24 243609.74
Average: 25 218122.53
Average: 26 237405.38
Average: 27 217582.76
Average: 28 548.29
Average: 29 569.12
Average: 30 540.74
Average: 31 517.50
Average: 32 521.59
Average: 33 544.85
Average: 34 520.91
Average: 35 553.29
Average: 36 545.32
Average: 37 518.44
Average: 38 557.26
Average: 39 541.71
Average: 40 515.21
Average: 41 520.82
Average: 42 137722.38
Average: 43 135737.59
Average: 44 524.97
Average: 45 538.24
Average: 46 580.38
Average: 47 567.62
Average: 48 555.53
Average: 49 561.50
Average: 50 537.65
Average: 51 565.09
Average: 52 536.12
Average: 53 570.44
Average: 54 535.38
Average: 55 567.88
Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
Average: 0 0.00 250.03 0.18 69.09 0.00
0.00 6.59 250.03 0.00 52.62
Average: 1 0.00 228.94 0.00 0.00 0.00
0.00 10.82 249.79 0.00 48.06
Average: 2 0.00 244.53 0.00 0.00 0.00
0.00 0.00 249.79 0.00 30.68
Average: 3 0.00 249.53 0.00 0.00 0.00
0.00 0.00 249.91 0.00 58.85
Average: 4 0.00 243.71 0.00 0.00 0.00
0.00 0.00 249.94 0.00 53.15
Average: 5 0.00 247.06 0.00 0.00 0.00
0.00 0.00 249.44 0.00 26.35
Average: 6 0.00 249.38 0.00 0.00 0.00
0.00 0.00 249.91 0.00 8.76
Average: 7 0.00 226.03 0.00 0.00 0.00
0.00 0.00 249.88 0.00 92.97
Average: 8 0.00 244.29 0.00 0.00 0.00
0.00 0.00 249.79 0.00 35.47
Average: 9 0.00 247.71 0.00 0.00 0.00
0.00 0.00 249.94 0.00 37.65
Average: 10 0.00 241.85 0.00 0.00 0.00
0.00 0.00 249.65 0.00 38.59
Average: 11 0.00 249.00 0.00 0.00 0.00
0.00 0.00 249.94 0.00 40.59
Average: 12 0.00 248.85 0.00 0.00 0.00
0.00 0.00 249.88 0.00 22.09
Average: 13 0.00 249.12 0.00 0.00 0.00
0.00 0.00 249.88 0.00 32.32
Average: 14 0.00 212.97 147.44 72760.47 0.00
0.00 0.00 189.35 0.00 5.44
Average: 15 0.00 233.94 139.21 115367.26 0.00
0.00 0.00 225.03 0.00 17.71
Average: 16 0.00 245.41 158.94 253784.06 0.00
0.00 0.00 244.65 0.00 13.03
Average: 17 0.00 245.74 164.09 252402.41 0.00
0.00 0.00 244.85 0.00 10.71
Average: 18 0.00 245.56 161.32 253778.00 0.00
0.00 0.00 244.97 0.00 16.50
Average: 19 0.00 245.38 161.35 251789.41 0.00
0.00 0.00 244.53 0.00 16.62
Average: 20 0.00 244.97 201.59 213226.74 0.00
0.00 0.00 244.24 0.00 10.65
Average: 21 0.00 245.59 174.82 232072.85 0.00
0.00 0.00 244.82 0.00 32.24
Average: 22 0.00 245.44 157.47 263244.59 0.00
0.00 0.00 244.74 0.00 14.62
Average: 23 0.00 245.53 165.09 259398.41 0.00
0.00 0.00 244.94 0.00 11.12
Average: 24 0.00 245.38 179.47 242922.85 0.00
0.00 0.00 244.85 0.00 17.18
Average: 25 0.00 245.47 195.15 217419.79 0.00
0.00 0.00 244.50 0.00 17.62
Average: 26 0.00 245.41 182.18 236714.38 0.00
0.00 0.00 244.62 0.00 18.79
Average: 27 0.00 244.94 196.65 216886.68 0.00
0.00 0.00 244.44 0.00 10.06
Average: 28 0.00 247.35 0.00 0.00 0.00
0.00 0.00 249.91 0.00 51.03
Average: 29 0.00 228.94 0.00 0.00 0.00
0.00 0.00 249.79 0.00 90.38
Average: 30 0.00 244.50 0.00 0.00 0.00
0.00 0.00 249.82 0.00 46.41
Average: 31 0.00 249.53 0.00 0.00 0.00
0.00 0.00 249.97 0.00 18.00
Average: 32 0.00 243.82 0.00 0.00 0.00
0.00 0.00 249.97 0.00 27.79
Average: 33 0.00 247.03 0.00 0.00 0.00
0.00 0.00 249.47 0.00 48.35
Average: 34 0.00 249.38 0.00 0.00 0.00
0.00 0.00 249.94 0.00 21.59
Average: 35 0.00 226.00 0.00 0.00 0.00
0.00 0.00 249.88 0.00 77.41
Average: 36 0.00 244.29 0.00 0.00 0.00
0.00 0.00 249.85 0.00 51.18
Average: 37 0.00 247.71 0.00 0.00 0.00
0.00 0.00 249.97 0.00 20.76
Average: 38 0.00 241.85 0.00 0.00 0.00
0.00 0.00 249.68 0.00 65.74
Average: 39 0.00 249.00 0.00 0.00 0.00
0.00 0.00 249.94 0.00 42.76
Average: 40 0.00 248.85 0.00 0.00 0.00
0.00 0.00 249.85 0.00 16.50
Average: 41 0.00 249.12 0.00 0.00 0.00
0.00 0.00 249.94 0.00 21.76
Average: 42 0.00 240.26 112.85 137160.06 0.00
0.00 0.00 203.44 0.00 5.76
Average: 43 0.00 237.91 122.91 135139.38 0.00
0.00 0.00 231.79 0.00 5.59
Average: 44 0.00 249.15 0.00 0.00 0.00
0.00 0.00 248.26 0.00 27.56
Average: 45 0.00 249.26 0.00 0.00 0.00
0.00 0.00 248.62 0.00 40.35
Average: 46 0.00 249.26 0.00 0.00 0.00
0.00 0.00 248.47 0.00 82.65
Average: 47 0.00 249.26 0.00 0.00 0.00
0.00 0.00 248.50 0.00 69.85
Average: 48 0.00 249.09 0.00 0.00 0.00
0.00 0.00 248.38 0.00 58.06
Average: 49 0.00 249.32 0.00 0.00 0.00
0.00 0.00 248.59 0.00 63.59
Average: 50 0.00 249.35 0.00 0.00 0.00
0.00 0.00 248.32 0.00 39.97
Average: 51 0.00 249.24 0.00 0.00 0.00
0.00 0.00 248.29 0.00 67.56
Average: 52 0.00 249.18 0.00 0.00 0.00
0.00 0.00 247.91 0.00 39.03
Average: 53 0.00 249.18 0.00 0.00 0.00
0.00 0.00 248.41 0.00 72.85
Average: 54 0.00 249.18 0.00 0.00 0.00
0.00 0.00 248.06 0.00 38.15
Average: 55 0.00 249.41 0.18 0.00 0.00
0.00 0.00 248.71 0.00 69.59
W dniu 2017-08-12 o 19:27, Paweł Staszewski pisze:
> Hi and thanks for reply
>
>
>
> W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
>> On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> Hi
>>>
>>> I made some tests for performance comparison.
>> Thanks for doing this. Feel free to Cc me, if you do more of these
>> tests (so I don't miss them on the mailing list).
>>
>> I don't understand stand if you are reporting a potential problem?
>>
>> It would be good if you can provide a short summary section (of the
>> issue) in the _start_ of the email, and then provide all this nice data
>> afterwards, to back your case.
>>
>> My understanding is, you report:
>>
>> 1. VLANs on ixgbe show a 30-40% slowdown
>> 2. System stopped scaling after 7+ CPUs
> This is not only problem/bug report - but some kind of comparision
> plus some toughts about possible problems :)
> And can help somebody when searching the net for possible expectations :)
> Also - dono better list where are the smartest people that know what
> is going in kernel with networking :)
>
> Next time i will place summary on top - sorry :)
>
>>
>>> Tested HW (FORWARDING HOST):
>>>
>>> Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>> Interesting, I've not heard about a Intel CPU called "Gold" before now,
>> but it does exist:
>> https://ark.intel.com/products/123541/Intel-Xeon-Gold-6132-Processor-19_25M-Cache-2_60-GHz
>>
>>
>>> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
>>> (rev 01)
>> This is one of my all time favorite NICs!
> Yes this is a good NIC - will have connectx-4 2x100G by monday so will
> also do some tests
>
>>> Test diagram:
>>>
>>>
>>> TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
>>> (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
>>>
>>> Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
>>> 172.16.0.1 to 172.16.0.255
>>>
>>> TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
>> What kind of traffic flow? E.g. distribution, many/few source IPs...
>
> Traffic generator is pktgen so udp flows - better paste parameters
> from pktgen:
> UDP_MIN=9
> UDP_MAX=19
>
> pg_set $dev "dst_min 172.16.0.1"
> pg_set $dev "dst_max 172.16.0.100"
>
> # Setup random UDP port src range
> #pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "udp_src_min $UDP_MIN"
> pg_set $dev "udp_src_max $UDP_MAX"
>
>
>>
>>> Settings used for FORWARDING HOST (changed param. was only number of
>>> RSS
>>> combined queues + set affinity assignment for them to fit with first
>>> numa node where 2x10G port card is installed)
>>>
>>> ixgbe driver used from kernel (in-kernel build - not a module)
>>>
>> Nice with a script showing you setup, thanks. I would be good if it had
>> comments, telling why you think this is a needed setup adjustment.
>>
>>> #!/bin/sh
>>> ifc='enp216s0f0 enp216s0f1'
>>> for i in $ifc
>>> do
>>> ip link set up dev $i
>>> ethtool -A $i autoneg off rx off tx off
>> Good:
>> Turning off Ethernet flow control, to avoid receiver being the
>> bottleneck via pause-frames.
> Yes - enabled flow controll is really bad :)
>>> ethtool -G $i rx 4096 tx 1024
>> You adjust the RX and TX ring queue sizes, this have effects that you
>> don't realize. Especially for the ixgbe driver, which have a page
>> recycle trick tied to the RX ring queue size.
> rx ring 4096 and tx ring 1024
> - this is because have best performance then with average packet size
> from 64 to 1500 bytes
>
> Can be a little better performance for smaller frames like 64 - with
> rx ring set to 1024
> below 1 core/1 RSS queue with rx ring set to 1024
>
> 0;1;64;1530112;91772160;1529919;88724208
> 1;1;64;1531584;91872000;1531520;88813196
> 2;1;64;1531392;91895040;1531262;88831930
> 3;1;64;1530880;91875840;1531201;88783558
> 4;1;64;1530688;91829760;1530688;88768826
> 5;1;64;1530432;91810560;1530624;88764940
> 6;1;64;1530880;91868160;1530878;88787328
> 7;1;64;1530496;91845120;1530560;88765114
> 8;1;64;1530496;91837440;1530687;88772538
> 9;1;64;1530176;91795200;1530496;88735360
>
> so from 1.47Mpps to 1.53Mpps
>
> But with bigger packets > 200 performance is better when rx is set to
> 4096
>
>
>>
>>> ip link set $i txqueuelen 1000
>> Setting tx queue len to the default 1000 seems redundant.
> Yes cause i'm changing this parameter also to see if any impact on
> performance we have
>>
>>> ethtool -C $i rx-usecs 10
>> Adjusting this also have effects you might not realize. This actually
>> also affect the page recycle scheme of ixgbe. And can sometimes be
>> used to solve stalling on DMA TX completions, which could be you issue
>> here.
> same here - rx-usecs - setting to 10 was kind of compromise to have
> good performance with big ans small packet sizes
>
> Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to
> 256 (1Core/1RSS queue):
> 0;1;64;1506304;90424320;1506626;87402868
> 1;1;64;1505536;90343680;1504830;87321088
> 2;1;64;1506880;90416640;1507522;87388120
> 3;1;64;1511040;90700800;1511682;87684864
> 4;1;64;1511040;90681600;1511102;87662476
> 5;1;64;1511488;90712320;1511614;87673728
> 6;1;64;1511296;90700800;1511038;87669900
> 7;1;64;1513344;90773760;1513280;87751680
> 8;1;64;1513536;90850560;1513470;87807360
> 9;1;64;1512128;90696960;1512000;87696000
>
> And rx-usecs set to 1
> 0;1;64;1533632;92037120;1533504;88954368
> 1;1;64;1533632;92006400;1533570;88943348
> 2;1;64;1533504;91994880;1533504;88931980
> 3;1;64;1532864;91979520;1532674;88902516
> 4;1;64;1533952;92044800;1534080;88961792
> 5;1;64;1533888;92048640;1534270;88969100
> 6;1;64;1533952;92037120;1534082;88969216
> 7;1;64;1533952;92021760;1534208;88969332
> 8;1;64;1533056;91983360;1532930;88883724
> 9;1;64;1533760;92021760;1533886;88946828
>
> rx-useck set to 2
> 0;1;64;1522432;91334400;1522304;88301056
> 1;1;64;1521920;91330560;1522496;88286208
> 2;1;64;1522496;91322880;1522432;88304768
> 3;1;64;1523456;91422720;1523649;88382762
> 4;1;64;1527680;91676160;1527424;88601728
> 5;1;64;1527104;91626240;1526912;88572032
> 6;1;64;1527424;91641600;1527424;88590592
> 7;1;64;1526336;91572480;1526912;88523776
> 8;1;64;1527040;91637760;1526912;88579456
> 9;1;64;1527040;91595520;1526784;88553472
>
> rx-usecs set to 3
> 0;1;64;1526272;91549440;1526592;88527488
> 1;1;64;1526528;91560960;1526272;88516352
> 2;1;64;1525952;91580160;1525888;88527488
> 3;1;64;1525504;91511040;1524864;88456960
> 4;1;64;1526272;91568640;1526208;88494080
> 5;1;64;1525568;91545600;1525312;88494080
> 6;1;64;1526144;91584000;1526080;88512640
> 7;1;64;1525376;91530240;1525376;88482944
> 8;1;64;1526784;91607040;1526592;88549760
> 9;1;64;1526208;91560960;1526528;88512640
>
>
>>
>>> ethtool -L $i combined 16
>>> ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
>>> tx-nocache-copy on ntuple on
>> Here are many setting above.
> Yes mostly NIC defaults besides the ntuple that is on (for testing
> some nfc drop filters - and trying to test also tc-offload )
>
>> GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
>> this result in approx 10% slowdown.
> Ok lets give a try :)
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on
> ntuple on
> rx-usecs 10
> 1 CPU / 1 RSS QUEUE
>
> 0;1;64;1609344;96537600;1609279;93327104
> 1;1;64;1608320;96514560;1608256;93293812
> 2;1;64;1608000;96487680;1608125;93267770
> 3;1;64;1608320;96522240;1608576;93297524
> 4;1;64;1605888;96387840;1606211;93148986
> 5;1;64;1601472;96072960;1601600;92870644
> 6;1;64;1602624;96180480;1602243;92959674
> 7;1;64;1601728;96107520;1602113;92907764
> 8;1;64;1602176;96122880;1602176;92933806
> 9;1;64;1603904;96253440;1603777;93045208
>
> A little better performance 1.6Mpps
> But wondering if disabling tso will have no performance impact for tcp
> traffic ...
> Will try to get some pktgen like pktgen-dpdk that can generate also
> tcp traffic - to compare this.
>
>
>>
>> AFAIK "tx-nocache-copy on" was also determined to be a bad option.
> I set this to on cause i have better performance (a little 10kpps for
> this test)
> below same test as above with tx-nocache-copy off
>
> 0;1;64;1591552;95496960;1591230;92313654
> 1;1;64;1596224;95738880;1595842;92555066
> 2;1;64;1595456;95700480;1595201;92521774
> 3;1;64;1595456;95723520;1595072;92528966
> 4;1;64;1595136;95692800;1595457;92503040
> 5;1;64;1594624;95631360;1594496;92473402
> 6;1;64;1596224;95761920;1595778;92551180
> 7;1;64;1595200;95700480;1595331;92521542
> 8;1;64;1595584;95692800;1595457;92521426
> 9;1;64;1594624;95662080;1594048;92469574
>
>
>
>
>>
>> The "ntuple on" AFAIK disables the flow-director in the NIC. I though
>> this would actually help VLAN traffic, but I guess not.
> yes I enabled this cause was thinking that can help with traffic on vlans
>
> below same test with ntuple off
> so all settings for ixgbe:
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off
> ntuple off
> rx-usecs 10
> rx-flow-hash udp4 sdfn
>
> 0;1;64;1611840;96691200;1611905;93460794
> 1;1;64;1610688;96645120;1610818;93427328
> 2;1;64;1610752;96668160;1610497;93442176
> 3;1;64;1610624;96664320;1610817;93427212
> 4;1;64;1610752;96652800;1610623;93412480
> 5;1;64;1610048;96614400;1610112;93404940
> 6;1;64;1611264;96641280;1611390;93427212
> 7;1;64;1611008;96691200;1610942;93468160
> 8;1;64;1610048;96652800;1609984;93408652
> 9;1;64;1611136;96641280;1610690;93434636
>
> Performance is a little better
> and now with tx-nocache-copy on
>
> 0;1;64;1597248;95834880;1597311;92644096
> 1;1;64;1597888;95865600;1597824;92677446
> 2;1;64;1597952;95834880;1597822;92644038
> 3;1;64;1597568;95877120;1597375;92685044
> 4;1;64;1597184;95827200;1597314;92629190
> 5;1;64;1597696;95842560;1597565;92625652
> 6;1;64;1597312;95834880;1597376;92644038
> 7;1;64;1597568;95873280;1597634;92647924
> 8;1;64;1598400;95919360;1598849;92699602
> 9;1;64;1597824;95873280;1598208;92684928
>
>
> That is weird - so enabling tx-nocache-copy with disabled ntuple have
> bad performance impact - but with enabled ntuple there is no
> performance impact
>
>
>
>>
>>
>>> ethtool -N $i rx-flow-hash udp4 sdfn
>> Why do you change the NICs flow-hash?
> whan used 16 cores / 16 rss queues - there was better load
> distribution over all cores when sdfn rx-flow-hash enabled
>
>>
>>> done
>>>
>>> ip link set up dev enp216s0f0
>>> ip link set up dev enp216s0f1
>>>
>>> ip a a 10.0.0.1/30 dev enp216s0f0
>>>
>>> ip link add link enp216s0f1 name vlan1000 type vlan id 1000
>>> ip link set up dev vlan1000
>>> ip a a 10.0.0.5/30 dev vlan1000
>>>
>>>
>>> ip route add 172.16.0.0/12 via 10.0.0.6
>>>
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
>>> #cat /sys/devices/system/node/node1/cpulist
>>> #14-27,42-55
>>> #cat /sys/devices/system/node/node0/cpulist
>>> #0-13,28-41
>> Is this a NUMA system?
> This is 2x CPU 6132 - so have two separate pcie access to the nic -
> need to check what cpu is assigned to pcie where network card is
> connected to have network card on local cpu where all irq's are binded
>
>>
>>> #################################################
>>>
>>>
>>> Looks like forwarding performance when using vlans on ixgbe is less
>>> that
>>> without vlans for about 30-40% (wondering if this is some vlan
>>> offloading problem and ixgbe)
>> I would see this as a problem/bug that enabling VLANs cost this much.
> Yes - was thinking that with tx/rx vlan offloading there will be not
> much performance impact when vlans used.
>
>>> settings below:
>>>
>>> ethtool -k enp216s0f0
>>> Features for enp216s0f0:
>>> Cannot get device udp-fragmentation-offload settings: Operation not
>>> supported
>>> rx-checksumming: on
>>> tx-checksumming: on
>>> tx-checksum-ipv4: off [fixed]
>>> tx-checksum-ip-generic: on
>>> tx-checksum-ipv6: off [fixed]
>>> tx-checksum-fcoe-crc: off [fixed]
>>> tx-checksum-sctp: on
>>> scatter-gather: on
>>> tx-scatter-gather: on
>>> tx-scatter-gather-fraglist: off [fixed]
>>> tcp-segmentation-offload: on
>>> tx-tcp-segmentation: on
>>> tx-tcp-ecn-segmentation: off [fixed]
>>> tx-tcp-mangleid-segmentation: on
>>> tx-tcp6-segmentation: on
>>> udp-fragmentation-offload: off
>>> generic-segmentation-offload: off
>>> generic-receive-offload: on
>>> large-receive-offload: off
>>> rx-vlan-offload: on
>>> tx-vlan-offload: on
>>> ntuple-filters: on
>>> receive-hashing: on
>>> highdma: on [fixed]
>>> rx-vlan-filter: on
>>> vlan-challenged: off [fixed]
>>> tx-lockless: off [fixed]
>>> netns-local: off [fixed]
>>> tx-gso-robust: off [fixed]
>>> tx-fcoe-segmentation: off [fixed]
>>> tx-gre-segmentation: on
>>> tx-gre-csum-segmentation: on
>>> tx-ipxip4-segmentation: on
>>> tx-ipxip6-segmentation: on
>>> tx-udp_tnl-segmentation: on
>>> tx-udp_tnl-csum-segmentation: on
>>> tx-gso-partial: on
>>> tx-sctp-segmentation: off [fixed]
>>> tx-esp-segmentation: off [fixed]
>>> fcoe-mtu: off [fixed]
>>> tx-nocache-copy: on
>>> loopback: off [fixed]
>>> rx-fcs: off [fixed]
>>> rx-all: off
>>> tx-vlan-stag-hw-insert: off [fixed]
>>> rx-vlan-stag-hw-parse: off [fixed]
>>> rx-vlan-stag-filter: off [fixed]
>>> l2-fwd-offload: off
>>> hw-tc-offload: off
>>> esp-hw-offload: off [fixed]
>>> esp-tx-csum-hw-offload: off [fixed]
>>> rx-udp_tunnel-port-offload: on
>>>
>>>
>>> Another thing is that forwarding performance does not scale with number
>>> of cores when 7+ cores are reached
>> I've seen problems with using Hyper-Threading CPUs. Could it be that
>> above 7 CPUs you are starting to use sibling-cores ?
> Turbostats can help here:
> Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI
> C1 C2 C1% C2% CPU%c1 CPU%c6 CoreTmp PkgTmp
> PkgWatt RAMWatt PKG_% RAM_%
> - - - 72 2.27 3188 2600 194844 0
> 64 69282 0.07 97.83 18.38 79.36 -4 54 123.49 16.08
> 0.00 0.00
> 0 0 0 8 0.74 1028 2600 1513 0
> 32 1462 1.50 97.99 10.92 88.34 47 51 58.34 5.34
> 0.00 0.00
> 0 0 28 7 0.67 1015 2600 1255 0
> 12 1249 0.96 98.61 10.99
> 0 1 1 7 0.68 1019 2600 1260 0
> 0 1260 0.00 99.54 8.44 90.88 49
> 0 1 29 9 0.71 1208 2600 1252 0
> 0 1253 0.00 99.48 8.41
> 0 2 2 7 0.67 1019 2600 1261 0
> 0 1260 0.00 99.54 8.44 90.89 48
> 0 2 30 7 0.67 1017 2600 1255 0
> 0 1255 0.00 99.55 8.44
> 0 3 3 7 0.68 1019 2600 1260 0
> 0 1259 0.00 99.53 8.46 90.86 -4
> 0 3 31 7 0.67 1017 2600 1256 0
> 0 1256 0.00 99.55 8.46
> 0 4 4 7 0.67 1027 2600 1260 0
> 0 1260 0.00 99.54 8.43 90.90 -4
> 0 4 32 7 0.66 1018 2600 1255 0
> 0 1255 0.00 99.55 8.44
> 0 5 5 7 0.68 1020 2600 1260 0
> 0 1257 0.00 99.54 8.44 90.89 50
> 0 5 33 7 0.68 1019 2600 1255 0
> 0 1255 0.00 99.55 8.43
> 0 6 6 7 0.70 1019 2600 1260 0
> 0 1259 0.00 99.53 8.43 90.87 -4
> 0 6 34 7 0.70 1019 2600 1255 0
> 0 1255 0.00 99.54 8.43
> 0 8 7 7 0.68 1019 2600 1262 0
> 0 1261 0.00 99.52 8.42 90.90 50
> 0 8 35 7 0.67 1019 2600 1255 0
> 0 1255 0.00 99.55 8.43
> 0 9 8 7 0.68 1019 2600 1260 0
> 0 1257 0.00 99.54 8.40 90.92 49
> 0 9 36 7 0.66 1017 2600 1255 0
> 0 1255 0.00 99.55 8.41
> 0 10 9 7 0.66 1018 2600 1257 0
> 0 1257 0.00 99.54 8.40 90.94 -4
> 0 10 37 7 0.66 1018 2600 1255 0
> 0 1255 0.00 99.55 8.41
> 0 11 10 7 0.66 1019 2600 1257 0
> 0 1259 0.00 99.54 8.56 90.77 -4
> 0 11 38 7 0.66 1018 2600 1255 0
> 3 1252 0.19 99.36 8.57
> 0 12 11 7 0.67 1019 2600 1260 0
> 0 1260 0.00 99.54 8.44 90.88 -4
> 0 12 39 7 0.67 1019 2600 1255 0
> 0 1256 0.00 99.55 8.44
> 0 13 12 7 0.68 1019 2600 1257 0
> 4 1254 0.32 99.22 8.67 90.65 -4
> 0 13 40 7 0.69 1019 2600 1256 0
> 4 1253 0.24 99.31 8.66
> 0 14 13 7 0.71 1020 2600 1260 0
> 0 1259 0.00 99.53 8.41 90.88 -4
> 0 14 41 7 0.72 1020 2600 1255 0
> 0 1255 0.00 99.54 8.40
> 1 0 14 3564 99.19 3594 2600 125472 0
> 0 0 0.00 0.00 0.81 0.00 54 54 65.15 10.74
> 0.00 0.00
> 1 0 42 3 0.07 3701 2600 1255 0
> 0 1255 0.00 99.95 99.93
> 1 1 15 11 0.32 3301 2600 1257 0
> 0 1257 0.00 99.81 26.37 73.31 42
> 1 1 43 10 0.31 3301 2600 1255 0
> 0 1255 0.00 99.82 26.38
> 1 2 16 10 0.31 3301 2600 1257 0
> 0 1257 0.00 99.81 26.37 73.32 39
> 1 2 44 10 0.32 3301 2600 1255 0
> 0 1255 0.00 99.82 26.36
> 1 3 17 10 0.32 3301 2600 1257 0
> 0 1257 0.00 99.81 26.40 73.28 39
> 1 3 45 11 0.32 3301 2600 1255 0
> 0 1255 0.00 99.81 26.40
> 1 4 18 10 0.32 3301 2600 1257 0
> 0 1257 0.00 99.82 26.40 73.28 40
> 1 4 46 11 0.32 3301 2600 1255 0
> 0 1255 0.00 99.82 26.40
> 1 5 19 11 0.33 3301 2600 1257 0
> 0 1257 0.00 99.81 26.40 73.27 39
> 1 5 47 11 0.33 3300 2600 1255 0
> 0 1255 0.00 99.82 26.40
> 1 6 20 12 0.35 3301 2600 1257 0
> 0 1257 0.00 99.81 26.38 73.27 42
> 1 6 48 12 0.36 3301 2600 1255 0
> 0 1255 0.00 99.81 26.37
> 1 8 21 11 0.33 3301 2600 1257 0
> 0 1257 0.00 99.82 26.37 73.29 42
> 1 8 49 11 0.33 3301 2600 1255 0
> 0 1255 0.00 99.82 26.38
> 1 9 22 10 0.32 3300 2600 1257 0
> 0 1257 0.00 99.82 26.35 73.34 41
> 1 9 50 10 0.30 3301 2600 1255 0
> 0 1255 0.00 99.82 26.36
> 1 10 23 10 0.31 3301 2600 1257 0
> 0 1257 0.00 99.82 26.37 73.33 41
> 1 10 51 10 0.31 3301 2600 1255 0
> 0 1255 0.00 99.82 26.36
> 1 11 24 10 0.32 3301 2600 1257 0
> 0 1257 0.00 99.81 26.62 73.06 41
> 1 11 52 10 0.32 3301 2600 1255 0
> 4 1251 0.32 99.50 26.62
> 1 12 25 11 0.33 3301 2600 1257 0
> 0 1257 0.00 99.81 26.39 73.28 41
> 1 12 53 11 0.33 3301 2600 1258 0
> 0 1254 0.00 99.82 26.38
> 1 13 26 12 0.36 3317 2600 1259 0
> 0 1258 0.00 99.79 26.41 73.23 39
> 1 13 54 11 0.34 3301 2600 1255 0
> 0 1254 0.00 99.82 26.42
> 1 14 27 12 0.36 3301 2600 1257 0
> 5 1251 0.24 99.58 26.54 73.10 41
> 1 14 55 12 0.36 3300 2600 1255 0
> 0 1254 0.00 99.82 26.54
>
>
> So it looks like in all tests i'm using core+sibling
> But side effect of this is that :
> 33 * 100.0 = 3300.0 MHz max turbo 28 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 24 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 20 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 14 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 12 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 8 active cores
> 35 * 100.0 = 3500.0 MHz max turbo 4 active cores
> 37 * 100.0 = 3700.0 MHz max turbo 2 active cores
>
> So more cores = less MHz per core/sibling
>
>>
>>> perf top:
>>>
>>> PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>>
>>> 16.32% [kernel] [k] skb_dst_force
>>> 16.30% [kernel] [k] dst_release
>>> 15.11% [kernel] [k] rt_cache_valid
>>> 12.62% [kernel] [k] ipv4_mtu
>> It seems a little strange that these 4 functions are on the top
> Yes dono why there is ipv4_mtu called and taking soo much cycles
>
>>
>>> 5.60% [kernel] [k] do_raw_spin_lock
>> Why is calling/taking this lock? (Use perf call-graph recording).
> can be hard to paste it here:)
> attached file
>
>>
>>> 3.03% [kernel] [k] fib_table_lookup
>>> 2.70% [kernel] [k] ip_finish_output2
>>> 2.10% [kernel] [k] dev_gro_receive
>>> 1.89% [kernel] [k] eth_type_trans
>>> 1.81% [kernel] [k] ixgbe_poll
>>> 1.15% [kernel] [k] ixgbe_xmit_frame_ring
>>> 1.06% [kernel] [k] __build_skb
>>> 1.04% [kernel] [k] __dev_queue_xmit
>>> 0.97% [kernel] [k] ip_rcv
>>> 0.78% [kernel] [k] netif_skb_features
>>> 0.74% [kernel] [k] ipt_do_table
>> Unloading netfilter modules, will give more performance, but it
>> semifake to do so.
> Compiled in kernel - only in filter mode - with ipv4+ipv6 - no other
> modules conntrack or other .
>>> 0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
>>> 0.64% [kernel] [k] ip_forward
>>> 0.59% [kernel] [k] __netif_receive_skb_core
>>> 0.55% [kernel] [k] dev_hard_start_xmit
>>> 0.53% [kernel] [k] ip_route_input_rcu
>>> 0.53% [kernel] [k] ip_rcv_finish
>>> 0.51% [kernel] [k] page_frag_free
>>> 0.50% [kernel] [k] kmem_cache_alloc
>>> 0.50% [kernel] [k] udp_v4_early_demux
>>> 0.44% [kernel] [k] skb_release_data
>>> 0.42% [kernel] [k] inet_gro_receive
>>> 0.40% [kernel] [k] sch_direct_xmit
>>> 0.39% [kernel] [k] __local_bh_enable_ip
>>> 0.33% [kernel] [k] netdev_pick_tx
>>> 0.33% [kernel] [k] validate_xmit_skb
>>> 0.28% [kernel] [k] fib_validate_source
>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>> 0.25% [kernel] [k] eth_header
>>> 0.23% [kernel] [k] get_dma_ops
>>> 0.22% [kernel] [k] skb_network_protocol
>>> 0.21% [kernel] [k] ip_output
>>> 0.21% [kernel] [k] vlan_dev_hard_start_xmit
>>> 0.20% [kernel] [k] ixgbe_alloc_rx_buffers
>>> 0.18% [kernel] [k] nf_hook_slow
>>> 0.18% [kernel] [k] apic_timer_interrupt
>>> 0.18% [kernel] [k] virt_to_head_page
>>> 0.18% [kernel] [k] build_skb
>>> 0.16% [kernel] [k] swiotlb_map_page
>>> 0.16% [kernel] [k] ip_finish_output
>>> 0.16% [kernel] [k] udp4_gro_receive
>>>
>>>
>>> RESULTS:
>>>
>>> CSV format - delimeter ";"
>>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;1;64;1470912;88247040;1470720;85305530
>>> 1;1;64;1470912;88285440;1470977;85335110
>>> 2;1;64;1470464;88247040;1470402;85290508
>>> 3;1;64;1471424;88262400;1471230;85353728
>>> 4;1;64;1468736;88166400;1468672;85201652
>>> 5;1;64;1470016;88181760;1469949;85234944
>>> 6;1;64;1470720;88247040;1470466;85290624
>>> 7;1;64;1471232;88277760;1471167;85346246
>>> 8;1;64;1469184;88170240;1469249;85216326
>>> 9;1;64;1470592;88227840;1470847;85294394
>> Single core 1.47Mpps seems a little low, I would expect 2Mpps.
>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;2;64;2413120;144802560;2413245;139975924
>>> 1;2;64;2415296;144913920;2415356;140098188
>>> 2;2;64;2416768;144898560;2416573;140105670
>>> 3;2;64;2418176;145056000;2418110;140261806
>>> 4;2;64;2416512;144990720;2416509;140172950
>>> 5;2;64;2415168;144860160;2414466;140064780
>>> 6;2;64;2416960;144983040;2416833;140190930
>>> 7;2;64;2413632;144768000;2413568;140001734
>>> 8;2;64;2415296;144898560;2414589;140087168
>>> 9;2;64;2416576;144963840;2416892;140190930
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;3;64;3419008;205155840;3418882;198239244
>>> 1;3;64;3428032;205585920;3427971;198744234
>>> 2;3;64;3425472;205536000;3425344;198677260
>>> 3;3;64;3425088;205470720;3425156;198603136
>>> 4;3;64;3427648;205693440;3426883;198773888
>>> 5;3;64;3426880;205670400;3427392;198796044
>>> 6;3;64;3429120;205678080;3430140;198848186
>>> 7;3;64;3422976;205355520;3423490;198458136
>>> 8;3;64;3423168;205336320;3423486;198495372
>>> 9;3;64;3424384;205493760;3425538;198617868
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;4;64;4406464;264364800;4405244;255560296
>>> 1;4;64;4404672;264349440;4405122;255541504
>>> 2;4;64;4402368;264049920;4403326;255188864
>>> 3;4;64;4401344;264076800;4400702;255207134
>>> 4;4;64;4385536;263074560;4386620;254312716
>>> 5;4;64;4386560;263189760;4385404;254379532
>>> 6;4;64;4398784;263857920;4399031;255025288
>>> 7;4;64;4407232;264445440;4407998;255637900
>>> 8;4;64;4413184;264698880;4413758;255875816
>>> 9;4;64;4411328;264526080;4411906;255712372
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;5;64;5094464;305871360;5094464;295657262
>>> 1;5;64;5090816;305514240;5091201;295274810
>>> 2;5;64;5088384;305387520;5089792;295175108
>>> 3;5;64;5079296;304869120;5079484;294680368
>>> 4;5;64;5092992;305544960;5094207;295349166
>>> 5;5;64;5092416;305502720;5093372;295334260
>>> 6;5;64;5080896;304896000;5081090;294677004
>>> 7;5;64;5085376;305114880;5086401;294933058
>>> 8;5;64;5092544;305575680;5092036;295356938
>>> 9;5;64;5093056;305652480;5093832;295449506
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;6;64;5705088;342351360;5705784;330965110
>>> 1;6;64;5710272;342743040;5707591;331373952
>>> 2;6;64;5703424;342182400;5701826;330776552
>>> 3;6;64;5708736;342604800;5707963;331147462
>>> 4;6;64;5710144;342654720;5712067;331202910
>>> 5;6;64;5712064;342777600;5711361;331292288
>>> 6;6;64;5710144;342585600;5708607;331144272
>>> 7;6;64;5699840;342021120;5697853;330609222
>>> 8;6;64;5701184;342124800;5702909;330653592
>>> 9;6;64;5711360;342735360;5713283;331247686
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;7;64;6244416;374603520;6243591;362180072
>>> 1;7;64;6230912;374016000;6231490;361534126
>>> 2;7;64;6244800;374776320;6244866;362224326
>>> 3;7;64;6238720;374376960;6238261;361838510
>>> 4;7;64;6218816;373079040;6220413;360683962
>>> 5;7;64;6224320;373566720;6225086;361017404
>>> 6;7;64;6224000;373570560;6221370;360936088
>>> 7;7;64;6210048;372741120;6210627;360212654
>>> 8;7;64;6231616;374035200;6231537;361445502
>>> 9;7;64;6227840;373724160;6228802;361162752
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;8;64;6251840;375144960;6251849;362609678
>>> 1;8;64;6250816;375014400;6250881;362547038
>>> 2;8;64;6257728;375432960;6257160;362911104
>>> 3;8;64;6255552;375325440;6255622;362822074
>>> 4;8;64;6243776;374576640;6243270;362120622
>>> 5;8;64;6237184;374296320;6237690;361790080
>>> 6;8;64;6240960;374415360;6240714;361927366
>>> 7;8;64;6222784;373317120;6223746;360854424
>>> 8;8;64;6225920;373593600;6227014;361154980
>>> 9;8;64;6238528;374304000;6237701;361845238
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;14;64;6486144;389184000;6486135;376236488
>>> 1;14;64;6454912;387390720;6454222;374466734
>>> 2;14;64;6441152;386480640;6440431;373572780
>>> 3;14;64;6450240;386972160;6450870;374070014
>>> 4;14;64;6465600;387997440;6467221;375089654
>>> 5;14;64;6448384;386860800;6448000;373980230
>>> 6;14;64;6452352;387095040;6452148;374168904
>>> 7;14;64;6441984;386507520;6443203;373665058
>>> 8;14;64;6456704;387340800;6455744;374429092
>>> 9;14;64;6464640;387901440;6465218;374949004
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;16;64;6939008;416325120;6938696;402411192
>>> 1;16;64;6941952;416444160;6941745;402558918
>>> 2;16;64;6960576;417584640;6960707;403698718
>>> 3;16;64;6940736;416486400;6941820;402503876
>>> 4;16;64;6927680;415741440;6927420;401853870
>>> 5;16;64;6929792;415687680;6929917;401839196
>>> 6;16;64;6950400;416989440;6950661;403026166
>>> 7;16;64;6953664;417216000;6953454;403260544
>>> 8;16;64;6948480;416851200;6948800;403023266
>>> 9;16;64;6924160;415422720;6924092;401542468
>> I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
>> issue/bug. You could be stalling on DMA TX completion being too slow,
>> but you already increased the interval and increased the TX ring queue
>> size. You could play with those setting and see if it changes this?
>>
>> Could you try my napi_monitor tool in:
>> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
>>
>> Also provide the output from:
>> mpstat -P ALL -u -I SCPU -I SUM 2
> with 16 cores / 16 RSS queues
> Average: CPU %usr %nice %sys %iowait %irq %soft
> %steal %guest %gnice %idle
> Average: all 0.00 0.00 0.01 0.00 0.00 28.57
> 0.00 0.00 0.00 71.42
> Average: 0 0.00 0.00 0.04 0.00 0.00 0.08
> 0.00 0.00 0.00 99.88
> Average: 1 0.00 0.00 0.12 0.00 0.00 0.00
> 0.00 0.00 0.00 99.88
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 7 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 11 0.08 0.00 0.04 0.00 0.00 0.00
> 0.00 0.00 0.00 99.88
> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 14 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 15 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 16 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 17 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 18 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 19 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 20 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 21 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 22 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 23 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 24 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 25 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 26 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 27 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 28 0.00 0.00 0.04 0.00 0.00 0.00
> 0.00 0.00 0.00 99.96
> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 33 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 36 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 37 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 39 0.04 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 99.96
> Average: 40 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 41 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 42 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 43 0.00 0.00 0.00 0.00 0.00 100.00
> 0.00 0.00 0.00 0.00
> Average: 44 0.00 0.00 0.04 0.17 0.00 0.00
> 0.00 0.00 0.00 99.79
> Average: 45 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 46 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 47 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 48 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 49 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 50 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 51 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 52 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 53 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 54 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
> Average: 55 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 0.00 100.00
>
> Average: CPU intr/s
> Average: all 123596.08
> Average: 0 646.38
> Average: 1 500.54
> Average: 2 511.67
> Average: 3 534.25
> Average: 4 542.21
> Average: 5 531.54
> Average: 6 554.58
> Average: 7 535.88
> Average: 8 544.58
> Average: 9 536.42
> Average: 10 575.46
> Average: 11 601.12
> Average: 12 502.08
> Average: 13 575.46
> Average: 14 5917.92
> Average: 15 5949.58
> Average: 16 7021.29
> Average: 17 7299.71
> Average: 18 7391.67
> Average: 19 7354.25
> Average: 20 7543.42
> Average: 21 7354.25
> Average: 22 7322.33
> Average: 23 7368.71
> Average: 24 7429.00
> Average: 25 7406.46
> Average: 26 7400.67
> Average: 27 7447.21
> Average: 28 517.00
> Average: 29 549.54
> Average: 30 529.33
> Average: 31 533.83
> Average: 32 541.25
> Average: 33 541.17
> Average: 34 532.50
> Average: 35 545.17
> Average: 36 528.96
> Average: 37 509.92
> Average: 38 520.12
> Average: 39 523.29
> Average: 40 530.75
> Average: 41 542.33
> Average: 42 5921.71
> Average: 43 5949.42
> Average: 44 503.04
> Average: 45 542.75
> Average: 46 582.50
> Average: 47 581.71
> Average: 48 495.29
> Average: 49 524.38
> Average: 50 527.92
> Average: 51 528.12
> Average: 52 456.38
> Average: 53 477.00
> Average: 54 440.92
> Average: 55 568.83
>
> Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
> IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
> Average: 0 0.00 250.00 0.17 87.00
> 0.00 0.00 45.46 250.00 0.00 13.75
> Average: 1 0.00 233.42 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 17.21
> Average: 2 0.00 249.04 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 12.67
> Average: 3 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 34.42
> Average: 4 0.00 248.67 0.17 0.00
> 0.00 0.00 0.00 249.96 0.00 43.42
> Average: 5 0.00 249.46 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 32.17
> Average: 6 0.00 249.79 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 54.92
> Average: 7 0.00 240.12 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 45.79
> Average: 8 0.00 247.42 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 47.25
> Average: 9 0.00 249.29 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 37.17
> Average: 10 0.00 248.75 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 76.79
> Average: 11 0.00 249.29 0.00 0.00
> 0.00 0.00 42.79 249.83 0.00 59.21
> Average: 12 0.00 249.83 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 2.29
> Average: 13 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 75.62
> Average: 14 0.00 148.21 0.17 5758.04
> 0.00 0.00 0.00 8.42 0.00 3.08
> Average: 15 0.00 148.42 0.46 5789.25
> 0.00 0.00 0.00 8.33 0.00 3.12
> Average: 16 0.00 142.62 0.79 6866.46
> 0.00 0.00 0.00 8.29 0.00 3.12
> Average: 17 0.00 143.17 0.42 7145.00
> 0.00 0.00 0.00 8.08 0.00 3.04
> Average: 18 0.00 153.62 0.42 7226.42
> 0.00 0.00 0.00 8.04 0.00 3.17
> Average: 19 0.00 150.46 0.46 7192.21
> 0.00 0.00 0.00 8.04 0.00 3.08
> Average: 20 0.00 145.21 0.17 7386.50
> 0.00 0.00 0.00 8.29 0.00 3.25
> Average: 21 0.00 150.96 0.46 7191.37
> 0.00 0.00 0.00 8.25 0.00 3.21
> Average: 22 0.00 146.67 0.54 7163.96
> 0.00 0.00 0.00 8.04 0.00 3.12
> Average: 23 0.00 151.38 0.42 7205.75
> 0.00 0.00 0.00 8.00 0.00 3.17
> Average: 24 0.00 153.33 0.17 7264.12
> 0.00 0.00 0.00 8.08 0.00 3.29
> Average: 25 0.00 153.21 0.17 7241.83
> 0.00 0.00 0.00 7.96 0.00 3.29
> Average: 26 0.00 153.96 0.17 7234.88
> 0.00 0.00 0.00 8.38 0.00 3.29
> Average: 27 0.00 151.71 0.79 7283.25
> 0.00 0.00 0.00 8.04 0.00 3.42
> Average: 28 0.00 245.71 0.00 0.00
> 0.00 0.00 0.00 249.50 0.00 21.79
> Average: 29 0.00 233.21 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 66.46
> Average: 30 0.00 248.92 0.00 0.00
> 0.00 0.00 0.00 250.00 0.00 30.42
> Average: 31 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 33.96
> Average: 32 0.00 248.67 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 42.62
> Average: 33 0.00 249.46 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 41.79
> Average: 34 0.00 249.79 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 32.83
> Average: 35 0.00 240.12 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 55.08
> Average: 36 0.00 247.42 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 31.58
> Average: 37 0.00 249.29 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 10.71
> Average: 38 0.00 248.75 0.00 0.00
> 0.00 0.00 0.00 249.87 0.00 21.50
> Average: 39 0.00 249.50 0.00 0.00
> 0.00 0.00 0.00 249.83 0.00 23.96
> Average: 40 0.00 249.83 0.00 0.00
> 0.00 0.00 0.00 249.96 0.00 30.96
> Average: 41 0.00 249.92 0.00 0.00
> 0.00 0.00 0.00 249.92 0.00 42.50
> Average: 42 0.00 148.38 0.71 5761.00
> 0.00 0.00 0.00 8.25 0.00 3.38
> Average: 43 0.00 147.21 0.50 5790.33
> 0.00 0.00 0.00 8.00 0.00 3.38
> Average: 44 0.00 248.96 0.00 0.00
> 0.00 0.00 0.00 248.13 0.00 5.96
> Average: 45 0.00 249.04 0.00 0.00
> 0.00 0.00 0.00 248.88 0.00 44.83
> Average: 46 0.00 248.96 0.00 0.00
> 0.00 0.00 0.00 248.58 0.00 84.96
> Average: 47 0.00 249.00 0.00 0.00
> 0.00 0.00 0.00 248.75 0.00 83.96
> Average: 48 0.00 249.12 0.00 0.00
> 0.00 0.00 0.00 132.83 0.00 113.33
> Average: 49 0.00 249.12 0.00 0.00
> 0.00 0.00 0.00 248.62 0.00 26.62
> Average: 50 0.00 248.92 0.00 0.00
> 0.00 0.00 0.00 248.58 0.00 30.42
> Average: 51 0.00 249.08 0.00 0.00
> 0.00 0.00 0.00 248.42 0.00 30.63
> Average: 52 0.00 249.21 0.00 0.00
> 0.00 0.00 0.00 131.96 0.00 75.21
> Average: 53 0.00 249.08 0.00 0.00
> 0.00 0.00 0.00 136.12 0.00 91.79
> Average: 54 0.00 249.00 0.00 0.00
> 0.00 0.00 0.00 136.79 0.00 55.12
> Average: 55 0.00 249.04 0.00 0.00
> 0.00 0.00 0.00 248.71 0.00 71.08
>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-12 17:27 ` Paweł Staszewski
2017-08-13 16:58 ` Paweł Staszewski
@ 2017-08-14 0:07 ` Alexander Duyck
2017-08-14 15:07 ` Paweł Staszewski
1 sibling, 1 reply; 37+ messages in thread
From: Alexander Duyck @ 2017-08-14 0:07 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Jesper Dangaard Brouer, Linux Kernel Network Developers
On Sat, Aug 12, 2017 at 10:27 AM, Paweł Staszewski
<pstaszewski@itcare.pl> wrote:
> Hi and thanks for reply
>
>
>
> W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
>>
>> On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> Hi
>>>
>>> I made some tests for performance comparison.
>>
>> Thanks for doing this. Feel free to Cc me, if you do more of these
>> tests (so I don't miss them on the mailing list).
>>
>> I don't understand stand if you are reporting a potential problem?
>>
>> It would be good if you can provide a short summary section (of the
>> issue) in the _start_ of the email, and then provide all this nice data
>> afterwards, to back your case.
>>
>> My understanding is, you report:
>>
>> 1. VLANs on ixgbe show a 30-40% slowdown
>> 2. System stopped scaling after 7+ CPUs
So I had read through most of this before I realized what it was you
were reporting. As far as the behavior there are a few things going
on. I have some additional comments below but they are mostly based on
what I had read up to that point.
As far as possible issues for item 1. The VLAN adds 4 bytes of data of
the payload, when it is stripped it can result in a packet that is 56
bytes. These missing 8 bytes can cause issues as it forces the CPU to
do a read/modify/write every time the device writes to the 64B cache
line instead of just doing it as a single write. This can be very
expensive and hurt performance. In addition it adds 4 bytes on the
wire, so if you are sending the same 64B packets over the VLAN
interface it is bumping them up to 68B to make room for the VLAN tag.
I am suspecting you are encountering one of these type of issues. You
might try tweaking the packet sizes in increments of 4 to see if there
is a sweet spot that you might be falling out of or into.
Item 2 is a known issue with the NICs supported by ixgbe, at least for
anything 82599 and later. The issue here is that there isn't really an
Rx descriptor cache so to try and optimize performance the hardware
will try to write back as many descriptors it has ready for the ring
requesting writeback. The problem is as you add more rings it means
the writes get smaller as they are triggering more often. So what you
end up seeing is that for each additional ring you add the performance
starts dropping as soon as the rings are no longer being fully
saturated. You can tell this has happened when the CPUs in use
suddenly all stop reporting 100% softirq use. So for example to
perform at line rate with 64B packets you would need something like
XDP and to keep the ring count small, like maybe 2 rings. Any more
than that and the performance will start to drop as you hit PCIe
bottlenecks.
> This is not only problem/bug report - but some kind of comparision plus
> some toughts about possible problems :)
> And can help somebody when searching the net for possible expectations :)
> Also - dono better list where are the smartest people that know what is
> going in kernel with networking :)
>
> Next time i will place summary on top - sorry :)
>
>>
>>> Tested HW (FORWARDING HOST):
>>>
>>> Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>
>> Interesting, I've not heard about a Intel CPU called "Gold" before now,
>> but it does exist:
>>
>> https://ark.intel.com/products/123541/Intel-Xeon-Gold-6132-Processor-19_25M-Cache-2_60-GHz
>>
>>
>>> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>>
>> This is one of my all time favorite NICs!
>
> Yes this is a good NIC - will have connectx-4 2x100G by monday so will also
> do some tests
>
>>
>>>
>>> Test diagram:
>>>
>>>
>>> TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
>>> (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
>>>
>>> Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
>>> 172.16.0.1 to 172.16.0.255
>>>
>>> TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
>>
>> What kind of traffic flow? E.g. distribution, many/few source IPs...
>
>
> Traffic generator is pktgen so udp flows - better paste parameters from
> pktgen:
> UDP_MIN=9
> UDP_MAX=19
>
> pg_set $dev "dst_min 172.16.0.1"
> pg_set $dev "dst_max 172.16.0.100"
>
> # Setup random UDP port src range
> #pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "flag UDPSRC_RND"
> pg_set $dev "udp_src_min $UDP_MIN"
> pg_set $dev "udp_src_max $UDP_MAX"
>
>
>>
>>
>>>
>>> Settings used for FORWARDING HOST (changed param. was only number of RSS
>>> combined queues + set affinity assignment for them to fit with first
>>> numa node where 2x10G port card is installed)
>>>
>>> ixgbe driver used from kernel (in-kernel build - not a module)
>>>
>> Nice with a script showing you setup, thanks. I would be good if it had
>> comments, telling why you think this is a needed setup adjustment.
>>
>>> #!/bin/sh
>>> ifc='enp216s0f0 enp216s0f1'
>>> for i in $ifc
>>> do
>>> ip link set up dev $i
>>> ethtool -A $i autoneg off rx off tx off
>>
>> Good:
>> Turning off Ethernet flow control, to avoid receiver being the
>> bottleneck via pause-frames.
>
> Yes - enabled flow controll is really bad :)
>>>
>>> ethtool -G $i rx 4096 tx 1024
>>
>> You adjust the RX and TX ring queue sizes, this have effects that you
>> don't realize. Especially for the ixgbe driver, which have a page
>> recycle trick tied to the RX ring queue size.
>
> rx ring 4096 and tx ring 1024
> - this is because have best performance then with average packet size from
> 64 to 1500 bytes
The problem is this has huge negative effects on the CPU caches.
Generally less is more. When I perform tests I will usually drop the
ring size for Tx to 128 and Rx to 256. That reduces the descriptor
caches per ring to 1 page each for the Tx and Rx. With an increased
interrupt rate you should be able to service this optimally without
too much issue.
Also for these type of tests the Tx ring never really gets over 64
packets anyway since a single Tx ring is always populated by a single
Rx ring so as long as there isn't any flow control in play the Tx
queue should always be empty when the Rx clean-up begins and it will
only be populated with up to NAPI poll weight worth of packets.
> Can be a little better performance for smaller frames like 64 - with rx ring
> set to 1024
> below 1 core/1 RSS queue with rx ring set to 1024
>
> 0;1;64;1530112;91772160;1529919;88724208
> 1;1;64;1531584;91872000;1531520;88813196
> 2;1;64;1531392;91895040;1531262;88831930
> 3;1;64;1530880;91875840;1531201;88783558
> 4;1;64;1530688;91829760;1530688;88768826
> 5;1;64;1530432;91810560;1530624;88764940
> 6;1;64;1530880;91868160;1530878;88787328
> 7;1;64;1530496;91845120;1530560;88765114
> 8;1;64;1530496;91837440;1530687;88772538
> 9;1;64;1530176;91795200;1530496;88735360
>
> so from 1.47Mpps to 1.53Mpps
>
> But with bigger packets > 200 performance is better when rx is set to 4096
This is likely due to the interrupt moderation on the adapter. Instead
of adjusting the ring size up you might try pushing the time between
interrupts down. I have generally found around 25 usecs is best. You
can change the rx-usecs value via ethtool -C to get the rate you want.
You should find that it will perform better that way since you put
less stress on the CPU caches.
>
>>
>>> ip link set $i txqueuelen 1000
>>
>> Setting tx queue len to the default 1000 seems redundant.
>
> Yes cause i'm changing this parameter also to see if any impact on
> performance we have
>>
>>
>>> ethtool -C $i rx-usecs 10
>>
>> Adjusting this also have effects you might not realize. This actually
>> also affect the page recycle scheme of ixgbe. And can sometimes be
>> used to solve stalling on DMA TX completions, which could be you issue
>> here.
>
> same here - rx-usecs - setting to 10 was kind of compromise to have good
> performance with big ans small packet sizes
>From my personal experience I can say that 10 is probably too
aggressive. The logic for trying to find an ideal interrupt rate for
these kind of tests is actually pretty simple. What you want to do is
have the updates coming fast enough that you never hit the point of
descriptor starvation, but at the same time you don't want them coming
too quickly otherwise you limit how many descriptors can be coalesced
into a single PCI DMA write since the descriptors have to be flushed
when an interrupt is triggered.
> Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256
> (1Core/1RSS queue):
> 0;1;64;1506304;90424320;1506626;87402868
> 1;1;64;1505536;90343680;1504830;87321088
> 2;1;64;1506880;90416640;1507522;87388120
> 3;1;64;1511040;90700800;1511682;87684864
> 4;1;64;1511040;90681600;1511102;87662476
> 5;1;64;1511488;90712320;1511614;87673728
> 6;1;64;1511296;90700800;1511038;87669900
> 7;1;64;1513344;90773760;1513280;87751680
> 8;1;64;1513536;90850560;1513470;87807360
> 9;1;64;1512128;90696960;1512000;87696000
>
> And rx-usecs set to 1
> 0;1;64;1533632;92037120;1533504;88954368
> 1;1;64;1533632;92006400;1533570;88943348
> 2;1;64;1533504;91994880;1533504;88931980
> 3;1;64;1532864;91979520;1532674;88902516
> 4;1;64;1533952;92044800;1534080;88961792
> 5;1;64;1533888;92048640;1534270;88969100
> 6;1;64;1533952;92037120;1534082;88969216
> 7;1;64;1533952;92021760;1534208;88969332
> 8;1;64;1533056;91983360;1532930;88883724
> 9;1;64;1533760;92021760;1533886;88946828
>
> rx-useck set to 2
> 0;1;64;1522432;91334400;1522304;88301056
> 1;1;64;1521920;91330560;1522496;88286208
> 2;1;64;1522496;91322880;1522432;88304768
> 3;1;64;1523456;91422720;1523649;88382762
> 4;1;64;1527680;91676160;1527424;88601728
> 5;1;64;1527104;91626240;1526912;88572032
> 6;1;64;1527424;91641600;1527424;88590592
> 7;1;64;1526336;91572480;1526912;88523776
> 8;1;64;1527040;91637760;1526912;88579456
> 9;1;64;1527040;91595520;1526784;88553472
>
> rx-usecs set to 3
> 0;1;64;1526272;91549440;1526592;88527488
> 1;1;64;1526528;91560960;1526272;88516352
> 2;1;64;1525952;91580160;1525888;88527488
> 3;1;64;1525504;91511040;1524864;88456960
> 4;1;64;1526272;91568640;1526208;88494080
> 5;1;64;1525568;91545600;1525312;88494080
> 6;1;64;1526144;91584000;1526080;88512640
> 7;1;64;1525376;91530240;1525376;88482944
> 8;1;64;1526784;91607040;1526592;88549760
> 9;1;64;1526208;91560960;1526528;88512640
>
>
>>
>>> ethtool -L $i combined 16
>>> ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
>>> tx-nocache-copy on ntuple on
>>
>> Here are many setting above.
>
> Yes mostly NIC defaults besides the ntuple that is on (for testing some nfc
> drop filters - and trying to test also tc-offload )
>
>> GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
>> this result in approx 10% slowdown.
>
> Ok lets give a try :)
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple
> on
> rx-usecs 10
> 1 CPU / 1 RSS QUEUE
>
> 0;1;64;1609344;96537600;1609279;93327104
> 1;1;64;1608320;96514560;1608256;93293812
> 2;1;64;1608000;96487680;1608125;93267770
> 3;1;64;1608320;96522240;1608576;93297524
> 4;1;64;1605888;96387840;1606211;93148986
> 5;1;64;1601472;96072960;1601600;92870644
> 6;1;64;1602624;96180480;1602243;92959674
> 7;1;64;1601728;96107520;1602113;92907764
> 8;1;64;1602176;96122880;1602176;92933806
> 9;1;64;1603904;96253440;1603777;93045208
>
> A little better performance 1.6Mpps
> But wondering if disabling tso will have no performance impact for tcp
> traffic ...
If you were passing TCP traffic through the router GRO/TSO would
impact things, but for UDP it just adds overhead.
> Will try to get some pktgen like pktgen-dpdk that can generate also tcp
> traffic - to compare this.
>
>
>>
>> AFAIK "tx-nocache-copy on" was also determined to be a bad option.
>
> I set this to on cause i have better performance (a little 10kpps for this
> test)
> below same test as above with tx-nocache-copy off
>
> 0;1;64;1591552;95496960;1591230;92313654
> 1;1;64;1596224;95738880;1595842;92555066
> 2;1;64;1595456;95700480;1595201;92521774
> 3;1;64;1595456;95723520;1595072;92528966
> 4;1;64;1595136;95692800;1595457;92503040
> 5;1;64;1594624;95631360;1594496;92473402
> 6;1;64;1596224;95761920;1595778;92551180
> 7;1;64;1595200;95700480;1595331;92521542
> 8;1;64;1595584;95692800;1595457;92521426
> 9;1;64;1594624;95662080;1594048;92469574
If I recall it should have no actual impact one way or the other. The
tx-nocache-copy option should only impact socket traffic, not routing
since if I recall correctly it only impacts copies from userspace.
>> The "ntuple on" AFAIK disables the flow-director in the NIC. I though
>> this would actually help VLAN traffic, but I guess not.
>
> yes I enabled this cause was thinking that can help with traffic on vlans
>
> below same test with ntuple off
> so all settings for ixgbe:
> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple
> off
> rx-usecs 10
> rx-flow-hash udp4 sdfn
>
> 0;1;64;1611840;96691200;1611905;93460794
> 1;1;64;1610688;96645120;1610818;93427328
> 2;1;64;1610752;96668160;1610497;93442176
> 3;1;64;1610624;96664320;1610817;93427212
> 4;1;64;1610752;96652800;1610623;93412480
> 5;1;64;1610048;96614400;1610112;93404940
> 6;1;64;1611264;96641280;1611390;93427212
> 7;1;64;1611008;96691200;1610942;93468160
> 8;1;64;1610048;96652800;1609984;93408652
> 9;1;64;1611136;96641280;1610690;93434636
>
> Performance is a little better
> and now with tx-nocache-copy on
>
> 0;1;64;1597248;95834880;1597311;92644096
> 1;1;64;1597888;95865600;1597824;92677446
> 2;1;64;1597952;95834880;1597822;92644038
> 3;1;64;1597568;95877120;1597375;92685044
> 4;1;64;1597184;95827200;1597314;92629190
> 5;1;64;1597696;95842560;1597565;92625652
> 6;1;64;1597312;95834880;1597376;92644038
> 7;1;64;1597568;95873280;1597634;92647924
> 8;1;64;1598400;95919360;1598849;92699602
> 9;1;64;1597824;95873280;1598208;92684928
>
>
> That is weird - so enabling tx-nocache-copy with disabled ntuple have bad
> performance impact - but with enabled ntuple there is no performance impact
I would leave the ntuple feature enabled if you are routing simply
because that disables the ixgbe feature ATR which can have a negative
impact on routing tests (causes reordering).
>>
>>
>>> ethtool -N $i rx-flow-hash udp4 sdfn
>>
>> Why do you change the NICs flow-hash?
>
> whan used 16 cores / 16 rss queues - there was better load distribution over
> all cores when sdfn rx-flow-hash enabled
That is to be expected. The default hash will only has on IPv4
addresses. Enabling the use of UDP ports would allow for more entropy.
If you want similar performance without resorting to hashing on ports
you would have to change the source/destination IP addresses.
>>
>>> done
>>>
>>> ip link set up dev enp216s0f0
>>> ip link set up dev enp216s0f1
>>>
>>> ip a a 10.0.0.1/30 dev enp216s0f0
>>>
>>> ip link add link enp216s0f1 name vlan1000 type vlan id 1000
>>> ip link set up dev vlan1000
>>> ip a a 10.0.0.5/30 dev vlan1000
>>>
>>>
>>> ip route add 172.16.0.0/12 via 10.0.0.6
>>>
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
>>> #cat /sys/devices/system/node/node1/cpulist
>>> #14-27,42-55
>>> #cat /sys/devices/system/node/node0/cpulist
>>> #0-13,28-41
>>
>> Is this a NUMA system?
>
> This is 2x CPU 6132 - so have two separate pcie access to the nic - need to
> check what cpu is assigned to pcie where network card is connected to have
> network card on local cpu where all irq's are binded
>
>>
>>
>>>
>>> #################################################
>>>
>>>
>>> Looks like forwarding performance when using vlans on ixgbe is less that
>>> without vlans for about 30-40% (wondering if this is some vlan
>>> offloading problem and ixgbe)
>>
>> I would see this as a problem/bug that enabling VLANs cost this much.
>
> Yes - was thinking that with tx/rx vlan offloading there will be not much
> performance impact when vlans used.
What is the rate difference? Also did you account for the header size
when noticing that there is a difference in rates? I just want to make
sure we aren't seen an issue where you are expecting a rate of
14.88Mpps when VLAN tags drop the rate due to header overhead down to
something like 14.2Mpps if I recall correctly.
>
>>
>>>
>>> settings below:
>>>
>>> ethtool -k enp216s0f0
>>> Features for enp216s0f0:
>>> Cannot get device udp-fragmentation-offload settings: Operation not
>>> supported
>>> rx-checksumming: on
>>> tx-checksumming: on
>>> tx-checksum-ipv4: off [fixed]
>>> tx-checksum-ip-generic: on
>>> tx-checksum-ipv6: off [fixed]
>>> tx-checksum-fcoe-crc: off [fixed]
>>> tx-checksum-sctp: on
>>> scatter-gather: on
>>> tx-scatter-gather: on
>>> tx-scatter-gather-fraglist: off [fixed]
>>> tcp-segmentation-offload: on
>>> tx-tcp-segmentation: on
>>> tx-tcp-ecn-segmentation: off [fixed]
>>> tx-tcp-mangleid-segmentation: on
>>> tx-tcp6-segmentation: on
>>> udp-fragmentation-offload: off
>>> generic-segmentation-offload: off
>>> generic-receive-offload: on
>>> large-receive-offload: off
>>> rx-vlan-offload: on
>>> tx-vlan-offload: on
>>> ntuple-filters: on
>>> receive-hashing: on
>>> highdma: on [fixed]
>>> rx-vlan-filter: on
>>> vlan-challenged: off [fixed]
>>> tx-lockless: off [fixed]
>>> netns-local: off [fixed]
>>> tx-gso-robust: off [fixed]
>>> tx-fcoe-segmentation: off [fixed]
>>> tx-gre-segmentation: on
>>> tx-gre-csum-segmentation: on
>>> tx-ipxip4-segmentation: on
>>> tx-ipxip6-segmentation: on
>>> tx-udp_tnl-segmentation: on
>>> tx-udp_tnl-csum-segmentation: on
>>> tx-gso-partial: on
>>> tx-sctp-segmentation: off [fixed]
>>> tx-esp-segmentation: off [fixed]
>>> fcoe-mtu: off [fixed]
>>> tx-nocache-copy: on
>>> loopback: off [fixed]
>>> rx-fcs: off [fixed]
>>> rx-all: off
>>> tx-vlan-stag-hw-insert: off [fixed]
>>> rx-vlan-stag-hw-parse: off [fixed]
>>> rx-vlan-stag-filter: off [fixed]
>>> l2-fwd-offload: off
>>> hw-tc-offload: off
>>> esp-hw-offload: off [fixed]
>>> esp-tx-csum-hw-offload: off [fixed]
>>> rx-udp_tunnel-port-offload: on
>>>
>>>
>>> Another thing is that forwarding performance does not scale with number
>>> of cores when 7+ cores are reached
>>
>> I've seen problems with using Hyper-Threading CPUs. Could it be that
>> above 7 CPUs you are starting to use sibling-cores ?
>>
I would suspect that it may be more than likely the case. One thing
you might look at doing is CPU pinning the interrupts for the NIC in a
1:1 fashion so that the queues are all bound to separate cores without
them sharing between Hyper-threads.
> Turbostats can help here:
> Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI C1
> C2 C1% C2% CPU%c1 CPU%c6 CoreTmp PkgTmp PkgWatt RAMWatt
> PKG_% RAM_%
> - - - 72 2.27 3188 2600 194844 0 64
> 69282 0.07 97.83 18.38 79.36 -4 54 123.49 16.08 0.00
> 0.00
> 0 0 0 8 0.74 1028 2600 1513 0 32
> 1462 1.50 97.99 10.92 88.34 47 51 58.34 5.34 0.00
> 0.00
> 0 0 28 7 0.67 1015 2600 1255 0 12
> 1249 0.96 98.61 10.99
> 0 1 1 7 0.68 1019 2600 1260 0 0
> 1260 0.00 99.54 8.44 90.88 49
> 0 1 29 9 0.71 1208 2600 1252 0 0
> 1253 0.00 99.48 8.41
> 0 2 2 7 0.67 1019 2600 1261 0 0
> 1260 0.00 99.54 8.44 90.89 48
> 0 2 30 7 0.67 1017 2600 1255 0 0
> 1255 0.00 99.55 8.44
> 0 3 3 7 0.68 1019 2600 1260 0 0
> 1259 0.00 99.53 8.46 90.86 -4
> 0 3 31 7 0.67 1017 2600 1256 0 0
> 1256 0.00 99.55 8.46
> 0 4 4 7 0.67 1027 2600 1260 0 0
> 1260 0.00 99.54 8.43 90.90 -4
> 0 4 32 7 0.66 1018 2600 1255 0 0
> 1255 0.00 99.55 8.44
> 0 5 5 7 0.68 1020 2600 1260 0 0
> 1257 0.00 99.54 8.44 90.89 50
> 0 5 33 7 0.68 1019 2600 1255 0 0
> 1255 0.00 99.55 8.43
> 0 6 6 7 0.70 1019 2600 1260 0 0
> 1259 0.00 99.53 8.43 90.87 -4
> 0 6 34 7 0.70 1019 2600 1255 0 0
> 1255 0.00 99.54 8.43
> 0 8 7 7 0.68 1019 2600 1262 0 0
> 1261 0.00 99.52 8.42 90.90 50
> 0 8 35 7 0.67 1019 2600 1255 0 0
> 1255 0.00 99.55 8.43
> 0 9 8 7 0.68 1019 2600 1260 0 0
> 1257 0.00 99.54 8.40 90.92 49
> 0 9 36 7 0.66 1017 2600 1255 0 0
> 1255 0.00 99.55 8.41
> 0 10 9 7 0.66 1018 2600 1257 0 0
> 1257 0.00 99.54 8.40 90.94 -4
> 0 10 37 7 0.66 1018 2600 1255 0 0
> 1255 0.00 99.55 8.41
> 0 11 10 7 0.66 1019 2600 1257 0 0
> 1259 0.00 99.54 8.56 90.77 -4
> 0 11 38 7 0.66 1018 2600 1255 0 3
> 1252 0.19 99.36 8.57
> 0 12 11 7 0.67 1019 2600 1260 0 0
> 1260 0.00 99.54 8.44 90.88 -4
> 0 12 39 7 0.67 1019 2600 1255 0 0
> 1256 0.00 99.55 8.44
> 0 13 12 7 0.68 1019 2600 1257 0 4
> 1254 0.32 99.22 8.67 90.65 -4
> 0 13 40 7 0.69 1019 2600 1256 0 4
> 1253 0.24 99.31 8.66
> 0 14 13 7 0.71 1020 2600 1260 0 0
> 1259 0.00 99.53 8.41 90.88 -4
> 0 14 41 7 0.72 1020 2600 1255 0 0
> 1255 0.00 99.54 8.40
> 1 0 14 3564 99.19 3594 2600 125472 0 0
> 0 0.00 0.00 0.81 0.00 54 54 65.15 10.74 0.00
> 0.00
> 1 0 42 3 0.07 3701 2600 1255 0 0
> 1255 0.00 99.95 99.93
> 1 1 15 11 0.32 3301 2600 1257 0 0
> 1257 0.00 99.81 26.37 73.31 42
> 1 1 43 10 0.31 3301 2600 1255 0 0
> 1255 0.00 99.82 26.38
> 1 2 16 10 0.31 3301 2600 1257 0 0
> 1257 0.00 99.81 26.37 73.32 39
> 1 2 44 10 0.32 3301 2600 1255 0 0
> 1255 0.00 99.82 26.36
> 1 3 17 10 0.32 3301 2600 1257 0 0
> 1257 0.00 99.81 26.40 73.28 39
> 1 3 45 11 0.32 3301 2600 1255 0 0
> 1255 0.00 99.81 26.40
> 1 4 18 10 0.32 3301 2600 1257 0 0
> 1257 0.00 99.82 26.40 73.28 40
> 1 4 46 11 0.32 3301 2600 1255 0 0
> 1255 0.00 99.82 26.40
> 1 5 19 11 0.33 3301 2600 1257 0 0
> 1257 0.00 99.81 26.40 73.27 39
> 1 5 47 11 0.33 3300 2600 1255 0 0
> 1255 0.00 99.82 26.40
> 1 6 20 12 0.35 3301 2600 1257 0 0
> 1257 0.00 99.81 26.38 73.27 42
> 1 6 48 12 0.36 3301 2600 1255 0 0
> 1255 0.00 99.81 26.37
> 1 8 21 11 0.33 3301 2600 1257 0 0
> 1257 0.00 99.82 26.37 73.29 42
> 1 8 49 11 0.33 3301 2600 1255 0 0
> 1255 0.00 99.82 26.38
> 1 9 22 10 0.32 3300 2600 1257 0 0
> 1257 0.00 99.82 26.35 73.34 41
> 1 9 50 10 0.30 3301 2600 1255 0 0
> 1255 0.00 99.82 26.36
> 1 10 23 10 0.31 3301 2600 1257 0 0
> 1257 0.00 99.82 26.37 73.33 41
> 1 10 51 10 0.31 3301 2600 1255 0 0
> 1255 0.00 99.82 26.36
> 1 11 24 10 0.32 3301 2600 1257 0 0
> 1257 0.00 99.81 26.62 73.06 41
> 1 11 52 10 0.32 3301 2600 1255 0 4
> 1251 0.32 99.50 26.62
> 1 12 25 11 0.33 3301 2600 1257 0 0
> 1257 0.00 99.81 26.39 73.28 41
> 1 12 53 11 0.33 3301 2600 1258 0 0
> 1254 0.00 99.82 26.38
> 1 13 26 12 0.36 3317 2600 1259 0 0
> 1258 0.00 99.79 26.41 73.23 39
> 1 13 54 11 0.34 3301 2600 1255 0 0
> 1254 0.00 99.82 26.42
> 1 14 27 12 0.36 3301 2600 1257 0 5
> 1251 0.24 99.58 26.54 73.10 41
> 1 14 55 12 0.36 3300 2600 1255 0 0
> 1254 0.00 99.82 26.54
>
>
> So it looks like in all tests i'm using core+sibling
> But side effect of this is that :
> 33 * 100.0 = 3300.0 MHz max turbo 28 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 24 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 20 active cores
> 33 * 100.0 = 3300.0 MHz max turbo 14 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 12 active cores
> 34 * 100.0 = 3400.0 MHz max turbo 8 active cores
> 35 * 100.0 = 3500.0 MHz max turbo 4 active cores
> 37 * 100.0 = 3700.0 MHz max turbo 2 active cores
>
> So more cores = less MHz per core/sibling
Yes that is always a trade off. Also the ixgbe is limited in terms of
PCIe bus bandwidth. The more queues you add the worse the descriptor
overhead will be. Generally I have found that about 6 queues is ideal.
As you start getting to more than 8 the performance for 64B packets
will start to drop off as each additional queue will hurt the
descriptor cache performance as it starts writing back fewer and fewer
descriptors per write which will increase the PCIe bus overhead for
the writes.
>>
>>> perf top:
>>>
>>> PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>> cycles], (all, 56 CPUs)
>>>
>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> 16.32% [kernel] [k] skb_dst_force
>>> 16.30% [kernel] [k] dst_release
>>> 15.11% [kernel] [k] rt_cache_valid
>>> 12.62% [kernel] [k] ipv4_mtu
>>
>> It seems a little strange that these 4 functions are on the top
>
> Yes dono why there is ipv4_mtu called and taking soo much cycles
>
>>
>>> 5.60% [kernel] [k] do_raw_spin_lock
>>
>> Why is calling/taking this lock? (Use perf call-graph recording).
>
> can be hard to paste it here:)
> attached file
>
>>
>>> 3.03% [kernel] [k] fib_table_lookup
>>> 2.70% [kernel] [k] ip_finish_output2
>>> 2.10% [kernel] [k] dev_gro_receive
>>> 1.89% [kernel] [k] eth_type_trans
>>> 1.81% [kernel] [k] ixgbe_poll
>>> 1.15% [kernel] [k] ixgbe_xmit_frame_ring
>>> 1.06% [kernel] [k] __build_skb
>>> 1.04% [kernel] [k] __dev_queue_xmit
>>> 0.97% [kernel] [k] ip_rcv
>>> 0.78% [kernel] [k] netif_skb_features
>>> 0.74% [kernel] [k] ipt_do_table
>>
>> Unloading netfilter modules, will give more performance, but it
>> semifake to do so.
>
> Compiled in kernel - only in filter mode - with ipv4+ipv6 - no other modules
> conntrack or other .
>
>>> 0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
>>> 0.64% [kernel] [k] ip_forward
>>> 0.59% [kernel] [k] __netif_receive_skb_core
>>> 0.55% [kernel] [k] dev_hard_start_xmit
>>> 0.53% [kernel] [k] ip_route_input_rcu
>>> 0.53% [kernel] [k] ip_rcv_finish
>>> 0.51% [kernel] [k] page_frag_free
>>> 0.50% [kernel] [k] kmem_cache_alloc
>>> 0.50% [kernel] [k] udp_v4_early_demux
>>> 0.44% [kernel] [k] skb_release_data
>>> 0.42% [kernel] [k] inet_gro_receive
>>> 0.40% [kernel] [k] sch_direct_xmit
>>> 0.39% [kernel] [k] __local_bh_enable_ip
>>> 0.33% [kernel] [k] netdev_pick_tx
>>> 0.33% [kernel] [k] validate_xmit_skb
>>> 0.28% [kernel] [k] fib_validate_source
>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>> 0.25% [kernel] [k] eth_header
>>> 0.23% [kernel] [k] get_dma_ops
>>> 0.22% [kernel] [k] skb_network_protocol
>>> 0.21% [kernel] [k] ip_output
>>> 0.21% [kernel] [k] vlan_dev_hard_start_xmit
>>> 0.20% [kernel] [k] ixgbe_alloc_rx_buffers
>>> 0.18% [kernel] [k] nf_hook_slow
>>> 0.18% [kernel] [k] apic_timer_interrupt
>>> 0.18% [kernel] [k] virt_to_head_page
>>> 0.18% [kernel] [k] build_skb
>>> 0.16% [kernel] [k] swiotlb_map_page
>>> 0.16% [kernel] [k] ip_finish_output
>>> 0.16% [kernel] [k] udp4_gro_receive
>>>
>>>
>>> RESULTS:
>>>
>>> CSV format - delimeter ";"
>>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;1;64;1470912;88247040;1470720;85305530
>>> 1;1;64;1470912;88285440;1470977;85335110
>>> 2;1;64;1470464;88247040;1470402;85290508
>>> 3;1;64;1471424;88262400;1471230;85353728
>>> 4;1;64;1468736;88166400;1468672;85201652
>>> 5;1;64;1470016;88181760;1469949;85234944
>>> 6;1;64;1470720;88247040;1470466;85290624
>>> 7;1;64;1471232;88277760;1471167;85346246
>>> 8;1;64;1469184;88170240;1469249;85216326
>>> 9;1;64;1470592;88227840;1470847;85294394
>>
>> Single core 1.47Mpps seems a little low, I would expect 2Mpps.
>>
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;2;64;2413120;144802560;2413245;139975924
>>> 1;2;64;2415296;144913920;2415356;140098188
>>> 2;2;64;2416768;144898560;2416573;140105670
>>> 3;2;64;2418176;145056000;2418110;140261806
>>> 4;2;64;2416512;144990720;2416509;140172950
>>> 5;2;64;2415168;144860160;2414466;140064780
>>> 6;2;64;2416960;144983040;2416833;140190930
>>> 7;2;64;2413632;144768000;2413568;140001734
>>> 8;2;64;2415296;144898560;2414589;140087168
>>> 9;2;64;2416576;144963840;2416892;140190930
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;3;64;3419008;205155840;3418882;198239244
>>> 1;3;64;3428032;205585920;3427971;198744234
>>> 2;3;64;3425472;205536000;3425344;198677260
>>> 3;3;64;3425088;205470720;3425156;198603136
>>> 4;3;64;3427648;205693440;3426883;198773888
>>> 5;3;64;3426880;205670400;3427392;198796044
>>> 6;3;64;3429120;205678080;3430140;198848186
>>> 7;3;64;3422976;205355520;3423490;198458136
>>> 8;3;64;3423168;205336320;3423486;198495372
>>> 9;3;64;3424384;205493760;3425538;198617868
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;4;64;4406464;264364800;4405244;255560296
>>> 1;4;64;4404672;264349440;4405122;255541504
>>> 2;4;64;4402368;264049920;4403326;255188864
>>> 3;4;64;4401344;264076800;4400702;255207134
>>> 4;4;64;4385536;263074560;4386620;254312716
>>> 5;4;64;4386560;263189760;4385404;254379532
>>> 6;4;64;4398784;263857920;4399031;255025288
>>> 7;4;64;4407232;264445440;4407998;255637900
>>> 8;4;64;4413184;264698880;4413758;255875816
>>> 9;4;64;4411328;264526080;4411906;255712372
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;5;64;5094464;305871360;5094464;295657262
>>> 1;5;64;5090816;305514240;5091201;295274810
>>> 2;5;64;5088384;305387520;5089792;295175108
>>> 3;5;64;5079296;304869120;5079484;294680368
>>> 4;5;64;5092992;305544960;5094207;295349166
>>> 5;5;64;5092416;305502720;5093372;295334260
>>> 6;5;64;5080896;304896000;5081090;294677004
>>> 7;5;64;5085376;305114880;5086401;294933058
>>> 8;5;64;5092544;305575680;5092036;295356938
>>> 9;5;64;5093056;305652480;5093832;295449506
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;6;64;5705088;342351360;5705784;330965110
>>> 1;6;64;5710272;342743040;5707591;331373952
>>> 2;6;64;5703424;342182400;5701826;330776552
>>> 3;6;64;5708736;342604800;5707963;331147462
>>> 4;6;64;5710144;342654720;5712067;331202910
>>> 5;6;64;5712064;342777600;5711361;331292288
>>> 6;6;64;5710144;342585600;5708607;331144272
>>> 7;6;64;5699840;342021120;5697853;330609222
>>> 8;6;64;5701184;342124800;5702909;330653592
>>> 9;6;64;5711360;342735360;5713283;331247686
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;7;64;6244416;374603520;6243591;362180072
>>> 1;7;64;6230912;374016000;6231490;361534126
>>> 2;7;64;6244800;374776320;6244866;362224326
>>> 3;7;64;6238720;374376960;6238261;361838510
>>> 4;7;64;6218816;373079040;6220413;360683962
>>> 5;7;64;6224320;373566720;6225086;361017404
>>> 6;7;64;6224000;373570560;6221370;360936088
>>> 7;7;64;6210048;372741120;6210627;360212654
>>> 8;7;64;6231616;374035200;6231537;361445502
>>> 9;7;64;6227840;373724160;6228802;361162752
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;8;64;6251840;375144960;6251849;362609678
>>> 1;8;64;6250816;375014400;6250881;362547038
>>> 2;8;64;6257728;375432960;6257160;362911104
>>> 3;8;64;6255552;375325440;6255622;362822074
>>> 4;8;64;6243776;374576640;6243270;362120622
>>> 5;8;64;6237184;374296320;6237690;361790080
>>> 6;8;64;6240960;374415360;6240714;361927366
>>> 7;8;64;6222784;373317120;6223746;360854424
>>> 8;8;64;6225920;373593600;6227014;361154980
>>> 9;8;64;6238528;374304000;6237701;361845238
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;14;64;6486144;389184000;6486135;376236488
>>> 1;14;64;6454912;387390720;6454222;374466734
>>> 2;14;64;6441152;386480640;6440431;373572780
>>> 3;14;64;6450240;386972160;6450870;374070014
>>> 4;14;64;6465600;387997440;6467221;375089654
>>> 5;14;64;6448384;386860800;6448000;373980230
>>> 6;14;64;6452352;387095040;6452148;374168904
>>> 7;14;64;6441984;386507520;6443203;373665058
>>> 8;14;64;6456704;387340800;6455744;374429092
>>> 9;14;64;6464640;387901440;6465218;374949004
>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>> 0;16;64;6939008;416325120;6938696;402411192
>>> 1;16;64;6941952;416444160;6941745;402558918
>>> 2;16;64;6960576;417584640;6960707;403698718
>>> 3;16;64;6940736;416486400;6941820;402503876
>>> 4;16;64;6927680;415741440;6927420;401853870
>>> 5;16;64;6929792;415687680;6929917;401839196
>>> 6;16;64;6950400;416989440;6950661;403026166
>>> 7;16;64;6953664;417216000;6953454;403260544
>>> 8;16;64;6948480;416851200;6948800;403023266
>>> 9;16;64;6924160;415422720;6924092;401542468
>>
>> I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
>> issue/bug. You could be stalling on DMA TX completion being too slow,
>> but you already increased the interval and increased the TX ring queue
>> size. You could play with those setting and see if it changes this?
>>
>> Could you try my napi_monitor tool in:
>>
>> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
>>
>> Also provide the output from:
>> mpstat -P ALL -u -I SCPU -I SUM 2
>
> with 16 cores / 16 RSS queues
> Average: CPU %usr %nice %sys %iowait %irq %soft %steal
> %guest %gnice %idle
> Average: all 0.00 0.00 0.01 0.00 0.00 28.57 0.00
> 0.00 0.00 71.42
> Average: 0 0.00 0.00 0.04 0.00 0.00 0.08 0.00
> 0.00 0.00 99.88
> Average: 1 0.00 0.00 0.12 0.00 0.00 0.00 0.00
> 0.00 0.00 99.88
> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 11 0.08 0.00 0.04 0.00 0.00 0.00 0.00
> 0.00 0.00 99.88
> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 14 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 15 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 16 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 17 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 18 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 19 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 20 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 21 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 22 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 23 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 24 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 25 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 26 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 27 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 28 0.00 0.00 0.04 0.00 0.00 0.00 0.00
> 0.00 0.00 99.96
> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 39 0.04 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 99.96
> Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 42 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 43 0.00 0.00 0.00 0.00 0.00 100.00 0.00
> 0.00 0.00 0.00
> Average: 44 0.00 0.00 0.04 0.17 0.00 0.00 0.00
> 0.00 0.00 99.79
> Average: 45 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 46 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 47 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 48 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 49 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 51 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 52 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 54 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
> Average: 55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> 0.00 0.00 100.00
>
> Average: CPU intr/s
> Average: all 123596.08
> Average: 0 646.38
> Average: 1 500.54
> Average: 2 511.67
> Average: 3 534.25
> Average: 4 542.21
> Average: 5 531.54
> Average: 6 554.58
> Average: 7 535.88
> Average: 8 544.58
> Average: 9 536.42
> Average: 10 575.46
> Average: 11 601.12
> Average: 12 502.08
> Average: 13 575.46
> Average: 14 5917.92
> Average: 15 5949.58
> Average: 16 7021.29
> Average: 17 7299.71
> Average: 18 7391.67
> Average: 19 7354.25
> Average: 20 7543.42
> Average: 21 7354.25
> Average: 22 7322.33
> Average: 23 7368.71
> Average: 24 7429.00
> Average: 25 7406.46
> Average: 26 7400.67
> Average: 27 7447.21
> Average: 28 517.00
> Average: 29 549.54
> Average: 30 529.33
> Average: 31 533.83
> Average: 32 541.25
> Average: 33 541.17
> Average: 34 532.50
> Average: 35 545.17
> Average: 36 528.96
> Average: 37 509.92
> Average: 38 520.12
> Average: 39 523.29
> Average: 40 530.75
> Average: 41 542.33
> Average: 42 5921.71
> Average: 43 5949.42
> Average: 44 503.04
> Average: 45 542.75
> Average: 46 582.50
> Average: 47 581.71
> Average: 48 495.29
> Average: 49 524.38
> Average: 50 527.92
> Average: 51 528.12
> Average: 52 456.38
> Average: 53 477.00
> Average: 54 440.92
> Average: 55 568.83
>
> Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
> IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
> Average: 0 0.00 250.00 0.17 87.00 0.00 0.00
> 45.46 250.00 0.00 13.75
> Average: 1 0.00 233.42 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 17.21
> Average: 2 0.00 249.04 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 12.67
> Average: 3 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 34.42
> Average: 4 0.00 248.67 0.17 0.00 0.00 0.00
> 0.00 249.96 0.00 43.42
> Average: 5 0.00 249.46 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 32.17
> Average: 6 0.00 249.79 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 54.92
> Average: 7 0.00 240.12 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 45.79
> Average: 8 0.00 247.42 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 47.25
> Average: 9 0.00 249.29 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 37.17
> Average: 10 0.00 248.75 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 76.79
> Average: 11 0.00 249.29 0.00 0.00 0.00 0.00
> 42.79 249.83 0.00 59.21
> Average: 12 0.00 249.83 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 2.29
> Average: 13 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 75.62
> Average: 14 0.00 148.21 0.17 5758.04 0.00 0.00
> 0.00 8.42 0.00 3.08
> Average: 15 0.00 148.42 0.46 5789.25 0.00 0.00
> 0.00 8.33 0.00 3.12
> Average: 16 0.00 142.62 0.79 6866.46 0.00 0.00
> 0.00 8.29 0.00 3.12
> Average: 17 0.00 143.17 0.42 7145.00 0.00 0.00
> 0.00 8.08 0.00 3.04
> Average: 18 0.00 153.62 0.42 7226.42 0.00 0.00
> 0.00 8.04 0.00 3.17
> Average: 19 0.00 150.46 0.46 7192.21 0.00 0.00
> 0.00 8.04 0.00 3.08
> Average: 20 0.00 145.21 0.17 7386.50 0.00 0.00
> 0.00 8.29 0.00 3.25
> Average: 21 0.00 150.96 0.46 7191.37 0.00 0.00
> 0.00 8.25 0.00 3.21
> Average: 22 0.00 146.67 0.54 7163.96 0.00 0.00
> 0.00 8.04 0.00 3.12
> Average: 23 0.00 151.38 0.42 7205.75 0.00 0.00
> 0.00 8.00 0.00 3.17
> Average: 24 0.00 153.33 0.17 7264.12 0.00 0.00
> 0.00 8.08 0.00 3.29
> Average: 25 0.00 153.21 0.17 7241.83 0.00 0.00
> 0.00 7.96 0.00 3.29
> Average: 26 0.00 153.96 0.17 7234.88 0.00 0.00
> 0.00 8.38 0.00 3.29
> Average: 27 0.00 151.71 0.79 7283.25 0.00 0.00
> 0.00 8.04 0.00 3.42
> Average: 28 0.00 245.71 0.00 0.00 0.00 0.00
> 0.00 249.50 0.00 21.79
> Average: 29 0.00 233.21 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 66.46
> Average: 30 0.00 248.92 0.00 0.00 0.00 0.00
> 0.00 250.00 0.00 30.42
> Average: 31 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 33.96
> Average: 32 0.00 248.67 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 42.62
> Average: 33 0.00 249.46 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 41.79
> Average: 34 0.00 249.79 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 32.83
> Average: 35 0.00 240.12 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 55.08
> Average: 36 0.00 247.42 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 31.58
> Average: 37 0.00 249.29 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 10.71
> Average: 38 0.00 248.75 0.00 0.00 0.00 0.00
> 0.00 249.87 0.00 21.50
> Average: 39 0.00 249.50 0.00 0.00 0.00 0.00
> 0.00 249.83 0.00 23.96
> Average: 40 0.00 249.83 0.00 0.00 0.00 0.00
> 0.00 249.96 0.00 30.96
> Average: 41 0.00 249.92 0.00 0.00 0.00 0.00
> 0.00 249.92 0.00 42.50
> Average: 42 0.00 148.38 0.71 5761.00 0.00 0.00
> 0.00 8.25 0.00 3.38
> Average: 43 0.00 147.21 0.50 5790.33 0.00 0.00
> 0.00 8.00 0.00 3.38
> Average: 44 0.00 248.96 0.00 0.00 0.00 0.00
> 0.00 248.13 0.00 5.96
> Average: 45 0.00 249.04 0.00 0.00 0.00 0.00
> 0.00 248.88 0.00 44.83
> Average: 46 0.00 248.96 0.00 0.00 0.00 0.00
> 0.00 248.58 0.00 84.96
> Average: 47 0.00 249.00 0.00 0.00 0.00 0.00
> 0.00 248.75 0.00 83.96
> Average: 48 0.00 249.12 0.00 0.00 0.00 0.00
> 0.00 132.83 0.00 113.33
> Average: 49 0.00 249.12 0.00 0.00 0.00 0.00
> 0.00 248.62 0.00 26.62
> Average: 50 0.00 248.92 0.00 0.00 0.00 0.00
> 0.00 248.58 0.00 30.42
> Average: 51 0.00 249.08 0.00 0.00 0.00 0.00
> 0.00 248.42 0.00 30.63
> Average: 52 0.00 249.21 0.00 0.00 0.00 0.00
> 0.00 131.96 0.00 75.21
> Average: 53 0.00 249.08 0.00 0.00 0.00 0.00
> 0.00 136.12 0.00 91.79
> Average: 54 0.00 249.00 0.00 0.00 0.00 0.00
> 0.00 136.79 0.00 55.12
> Average: 55 0.00 249.04 0.00 0.00 0.00 0.00
> 0.00 248.71 0.00 71.08
>
>
>>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-14 0:07 ` Alexander Duyck
@ 2017-08-14 15:07 ` Paweł Staszewski
0 siblings, 0 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-14 15:07 UTC (permalink / raw)
To: Alexander Duyck; +Cc: Jesper Dangaard Brouer, Linux Kernel Network Developers
W dniu 2017-08-14 o 02:07, Alexander Duyck pisze:
> On Sat, Aug 12, 2017 at 10:27 AM, Paweł Staszewski
> <pstaszewski@itcare.pl> wrote:
>> Hi and thanks for reply
>>
>>
>>
>> W dniu 2017-08-12 o 14:23, Jesper Dangaard Brouer pisze:
>>> On Fri, 11 Aug 2017 19:51:10 +0200 Paweł Staszewski
>>> <pstaszewski@itcare.pl> wrote:
>>>
>>>> Hi
>>>>
>>>> I made some tests for performance comparison.
>>> Thanks for doing this. Feel free to Cc me, if you do more of these
>>> tests (so I don't miss them on the mailing list).
>>>
>>> I don't understand stand if you are reporting a potential problem?
>>>
>>> It would be good if you can provide a short summary section (of the
>>> issue) in the _start_ of the email, and then provide all this nice data
>>> afterwards, to back your case.
>>>
>>> My understanding is, you report:
>>>
>>> 1. VLANs on ixgbe show a 30-40% slowdown
>>> 2. System stopped scaling after 7+ CPUs
> So I had read through most of this before I realized what it was you
> were reporting. As far as the behavior there are a few things going
> on. I have some additional comments below but they are mostly based on
> what I had read up to that point.
>
> As far as possible issues for item 1. The VLAN adds 4 bytes of data of
> the payload, when it is stripped it can result in a packet that is 56
> bytes. These missing 8 bytes can cause issues as it forces the CPU to
> do a read/modify/write every time the device writes to the 64B cache
> line instead of just doing it as a single write. This can be very
> expensive and hurt performance. In addition it adds 4 bytes on the
> wire, so if you are sending the same 64B packets over the VLAN
> interface it is bumping them up to 68B to make room for the VLAN tag.
> I am suspecting you are encountering one of these type of issues. You
> might try tweaking the packet sizes in increments of 4 to see if there
> is a sweet spot that you might be falling out of or into.
No this is not a problem with 4byte header or soo
Cause topology is like this
TX generator (pktgen) physical interface no vlan -> RX physical
interface (no vlan) [ FORWARDING HOST ] TX vlan interface binded to
physical interface -> SINK
below data for packet size 70 (pktgen PKT_SIZE: 70)
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;70;7246720;434749440;7245917;420269856
1;16;70;7249152;434872320;7248885;420434344
2;16;70;7249024;434926080;7249225;420401400
3;16;70;7249984;434952960;7249448;420435736
4;16;70;7251200;435064320;7250990;420495244
5;16;70;7241408;434592000;7241781;420068074
6;16;70;7229696;433689600;7229750;419268196
7;16;70;7236032;434127360;7236133;419669092
8;16;70;7236608;434161920;7236274;419695830
9;16;70;7226496;433578240;7227501;419107826
100% cpu load on all 16 cores
the difference vlan/no vlan currently on this host varries from 40 to
even 50% (but cant check if can reach 50% performance degradation cause
pktgen can give me only 10Mpps with 70% of cpu load for forwarding host
(soo still place to forward maybee at line rate 14Mpps)
> Item 2 is a known issue with the NICs supported by ixgbe, at least for
> anything 82599 and later. The issue here is that there isn't really an
> Rx descriptor cache so to try and optimize performance the hardware
> will try to write back as many descriptors it has ready for the ring
> requesting writeback. The problem is as you add more rings it means
> the writes get smaller as they are triggering more often. So what you
> end up seeing is that for each additional ring you add the performance
> starts dropping as soon as the rings are no longer being fully
> saturated. You can tell this has happened when the CPUs in use
> suddenly all stop reporting 100% softirq use. So for example to
> perform at line rate with 64B packets you would need something like
> XDP and to keep the ring count small, like maybe 2 rings. Any more
> than that and the performance will start to drop as you hit PCIe
> bottlenecks.
>
>> This is not only problem/bug report - but some kind of comparision plus
>> some toughts about possible problems :)
>> And can help somebody when searching the net for possible expectations :)
>> Also - dono better list where are the smartest people that know what is
>> going in kernel with networking :)
>>
>> Next time i will place summary on top - sorry :)
>>
>>>> Tested HW (FORWARDING HOST):
>>>>
>>>> Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz
>>> Interesting, I've not heard about a Intel CPU called "Gold" before now,
>>> but it does exist:
>>>
>>> https://ark.intel.com/products/123541/Intel-Xeon-Gold-6132-Processor-19_25M-Cache-2_60-GHz
>>>
>>>
>>>> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
>>> This is one of my all time favorite NICs!
>> Yes this is a good NIC - will have connectx-4 2x100G by monday so will also
>> do some tests
>>
>>>> Test diagram:
>>>>
>>>>
>>>> TRAFFIC GENERATOR (ethX) -> (enp216s0f0 - RX Traffic) FORWARDING HOST
>>>> (enp216s0f1(vlan1000) - TX Traffic) -> (ethY) SINK
>>>>
>>>> Forwarder traffic: UDP random ports from 9 to 19 with random hosts from
>>>> 172.16.0.1 to 172.16.0.255
>>>>
>>>> TRAFFIC GENERATOR TX is stable 9.9Mpps (in kernel pktgen)
>>> What kind of traffic flow? E.g. distribution, many/few source IPs...
>>
>> Traffic generator is pktgen so udp flows - better paste parameters from
>> pktgen:
>> UDP_MIN=9
>> UDP_MAX=19
>>
>> pg_set $dev "dst_min 172.16.0.1"
>> pg_set $dev "dst_max 172.16.0.100"
>>
>> # Setup random UDP port src range
>> #pg_set $dev "flag UDPSRC_RND"
>> pg_set $dev "flag UDPSRC_RND"
>> pg_set $dev "udp_src_min $UDP_MIN"
>> pg_set $dev "udp_src_max $UDP_MAX"
>>
>>
>>>
>>>> Settings used for FORWARDING HOST (changed param. was only number of RSS
>>>> combined queues + set affinity assignment for them to fit with first
>>>> numa node where 2x10G port card is installed)
>>>>
>>>> ixgbe driver used from kernel (in-kernel build - not a module)
>>>>
>>> Nice with a script showing you setup, thanks. I would be good if it had
>>> comments, telling why you think this is a needed setup adjustment.
>>>
>>>> #!/bin/sh
>>>> ifc='enp216s0f0 enp216s0f1'
>>>> for i in $ifc
>>>> do
>>>> ip link set up dev $i
>>>> ethtool -A $i autoneg off rx off tx off
>>> Good:
>>> Turning off Ethernet flow control, to avoid receiver being the
>>> bottleneck via pause-frames.
>> Yes - enabled flow controll is really bad :)
>>>> ethtool -G $i rx 4096 tx 1024
>>> You adjust the RX and TX ring queue sizes, this have effects that you
>>> don't realize. Especially for the ixgbe driver, which have a page
>>> recycle trick tied to the RX ring queue size.
>> rx ring 4096 and tx ring 1024
>> - this is because have best performance then with average packet size from
>> 64 to 1500 bytes
> The problem is this has huge negative effects on the CPU caches.
> Generally less is more. When I perform tests I will usually drop the
> ring size for Tx to 128 and Rx to 256. That reduces the descriptor
> caches per ring to 1 page each for the Tx and Rx. With an increased
> interrupt rate you should be able to service this optimally without
> too much issue.
>
> Also for these type of tests the Tx ring never really gets over 64
> packets anyway since a single Tx ring is always populated by a single
> Rx ring so as long as there isn't any flow control in play the Tx
> queue should always be empty when the Rx clean-up begins and it will
> only be populated with up to NAPI poll weight worth of packets.
Will check different ring sizes also to compare this.
>
>> Can be a little better performance for smaller frames like 64 - with rx ring
>> set to 1024
>> below 1 core/1 RSS queue with rx ring set to 1024
>>
>> 0;1;64;1530112;91772160;1529919;88724208
>> 1;1;64;1531584;91872000;1531520;88813196
>> 2;1;64;1531392;91895040;1531262;88831930
>> 3;1;64;1530880;91875840;1531201;88783558
>> 4;1;64;1530688;91829760;1530688;88768826
>> 5;1;64;1530432;91810560;1530624;88764940
>> 6;1;64;1530880;91868160;1530878;88787328
>> 7;1;64;1530496;91845120;1530560;88765114
>> 8;1;64;1530496;91837440;1530687;88772538
>> 9;1;64;1530176;91795200;1530496;88735360
>>
>> so from 1.47Mpps to 1.53Mpps
>>
>> But with bigger packets > 200 performance is better when rx is set to 4096
> This is likely due to the interrupt moderation on the adapter. Instead
> of adjusting the ring size up you might try pushing the time between
> interrupts down. I have generally found around 25 usecs is best. You
> can change the rx-usecs value via ethtool -C to get the rate you want.
> You should find that it will perform better that way since you put
> less stress on the CPU caches.
>
>>>> ip link set $i txqueuelen 1000
>>> Setting tx queue len to the default 1000 seems redundant.
>> Yes cause i'm changing this parameter also to see if any impact on
>> performance we have
>>>
>>>> ethtool -C $i rx-usecs 10
>>> Adjusting this also have effects you might not realize. This actually
>>> also affect the page recycle scheme of ixgbe. And can sometimes be
>>> used to solve stalling on DMA TX completions, which could be you issue
>>> here.
>> same here - rx-usecs - setting to 10 was kind of compromise to have good
>> performance with big ans small packet sizes
> >From my personal experience I can say that 10 is probably too
> aggressive. The logic for trying to find an ideal interrupt rate for
> these kind of tests is actually pretty simple. What you want to do is
> have the updates coming fast enough that you never hit the point of
> descriptor starvation, but at the same time you don't want them coming
> too quickly otherwise you limit how many descriptors can be coalesced
> into a single PCI DMA write since the descriptors have to be flushed
> when an interrupt is triggered.
>
>> Same test as above with rx ring 1024 tx ring 1024 and rxusecs set to 256
>> (1Core/1RSS queue):
>> 0;1;64;1506304;90424320;1506626;87402868
>> 1;1;64;1505536;90343680;1504830;87321088
>> 2;1;64;1506880;90416640;1507522;87388120
>> 3;1;64;1511040;90700800;1511682;87684864
>> 4;1;64;1511040;90681600;1511102;87662476
>> 5;1;64;1511488;90712320;1511614;87673728
>> 6;1;64;1511296;90700800;1511038;87669900
>> 7;1;64;1513344;90773760;1513280;87751680
>> 8;1;64;1513536;90850560;1513470;87807360
>> 9;1;64;1512128;90696960;1512000;87696000
>>
>> And rx-usecs set to 1
>> 0;1;64;1533632;92037120;1533504;88954368
>> 1;1;64;1533632;92006400;1533570;88943348
>> 2;1;64;1533504;91994880;1533504;88931980
>> 3;1;64;1532864;91979520;1532674;88902516
>> 4;1;64;1533952;92044800;1534080;88961792
>> 5;1;64;1533888;92048640;1534270;88969100
>> 6;1;64;1533952;92037120;1534082;88969216
>> 7;1;64;1533952;92021760;1534208;88969332
>> 8;1;64;1533056;91983360;1532930;88883724
>> 9;1;64;1533760;92021760;1533886;88946828
>>
>> rx-useck set to 2
>> 0;1;64;1522432;91334400;1522304;88301056
>> 1;1;64;1521920;91330560;1522496;88286208
>> 2;1;64;1522496;91322880;1522432;88304768
>> 3;1;64;1523456;91422720;1523649;88382762
>> 4;1;64;1527680;91676160;1527424;88601728
>> 5;1;64;1527104;91626240;1526912;88572032
>> 6;1;64;1527424;91641600;1527424;88590592
>> 7;1;64;1526336;91572480;1526912;88523776
>> 8;1;64;1527040;91637760;1526912;88579456
>> 9;1;64;1527040;91595520;1526784;88553472
>>
>> rx-usecs set to 3
>> 0;1;64;1526272;91549440;1526592;88527488
>> 1;1;64;1526528;91560960;1526272;88516352
>> 2;1;64;1525952;91580160;1525888;88527488
>> 3;1;64;1525504;91511040;1524864;88456960
>> 4;1;64;1526272;91568640;1526208;88494080
>> 5;1;64;1525568;91545600;1525312;88494080
>> 6;1;64;1526144;91584000;1526080;88512640
>> 7;1;64;1525376;91530240;1525376;88482944
>> 8;1;64;1526784;91607040;1526592;88549760
>> 9;1;64;1526208;91560960;1526528;88512640
>>
>>
>>>> ethtool -L $i combined 16
>>>> ethtool -K $i gro on tso on gso off sg on l2-fwd-offload off
>>>> tx-nocache-copy on ntuple on
>>> Here are many setting above.
>> Yes mostly NIC defaults besides the ntuple that is on (for testing some nfc
>> drop filters - and trying to test also tc-offload )
>>
>>> GRO/GSO/TSO for _forwarding_ is actually bad... in my tests, enabling
>>> this result in approx 10% slowdown.
>> Ok lets give a try :)
>> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy on ntuple
>> on
>> rx-usecs 10
>> 1 CPU / 1 RSS QUEUE
>>
>> 0;1;64;1609344;96537600;1609279;93327104
>> 1;1;64;1608320;96514560;1608256;93293812
>> 2;1;64;1608000;96487680;1608125;93267770
>> 3;1;64;1608320;96522240;1608576;93297524
>> 4;1;64;1605888;96387840;1606211;93148986
>> 5;1;64;1601472;96072960;1601600;92870644
>> 6;1;64;1602624;96180480;1602243;92959674
>> 7;1;64;1601728;96107520;1602113;92907764
>> 8;1;64;1602176;96122880;1602176;92933806
>> 9;1;64;1603904;96253440;1603777;93045208
>>
>> A little better performance 1.6Mpps
>> But wondering if disabling tso will have no performance impact for tcp
>> traffic ...
> If you were passing TCP traffic through the router GRO/TSO would
> impact things, but for UDP it just adds overhead.
>
>> Will try to get some pktgen like pktgen-dpdk that can generate also tcp
>> traffic - to compare this.
>>
>>
>>> AFAIK "tx-nocache-copy on" was also determined to be a bad option.
>> I set this to on cause i have better performance (a little 10kpps for this
>> test)
>> below same test as above with tx-nocache-copy off
>>
>> 0;1;64;1591552;95496960;1591230;92313654
>> 1;1;64;1596224;95738880;1595842;92555066
>> 2;1;64;1595456;95700480;1595201;92521774
>> 3;1;64;1595456;95723520;1595072;92528966
>> 4;1;64;1595136;95692800;1595457;92503040
>> 5;1;64;1594624;95631360;1594496;92473402
>> 6;1;64;1596224;95761920;1595778;92551180
>> 7;1;64;1595200;95700480;1595331;92521542
>> 8;1;64;1595584;95692800;1595457;92521426
>> 9;1;64;1594624;95662080;1594048;92469574
> If I recall it should have no actual impact one way or the other. The
> tx-nocache-copy option should only impact socket traffic, not routing
> since if I recall correctly it only impacts copies from userspace.
>
>>> The "ntuple on" AFAIK disables the flow-director in the NIC. I though
>>> this would actually help VLAN traffic, but I guess not.
>> yes I enabled this cause was thinking that can help with traffic on vlans
>>
>> below same test with ntuple off
>> so all settings for ixgbe:
>> gro off tso off gso off sg on l2-fwd-offload off tx-nocache-copy off ntuple
>> off
>> rx-usecs 10
>> rx-flow-hash udp4 sdfn
>>
>> 0;1;64;1611840;96691200;1611905;93460794
>> 1;1;64;1610688;96645120;1610818;93427328
>> 2;1;64;1610752;96668160;1610497;93442176
>> 3;1;64;1610624;96664320;1610817;93427212
>> 4;1;64;1610752;96652800;1610623;93412480
>> 5;1;64;1610048;96614400;1610112;93404940
>> 6;1;64;1611264;96641280;1611390;93427212
>> 7;1;64;1611008;96691200;1610942;93468160
>> 8;1;64;1610048;96652800;1609984;93408652
>> 9;1;64;1611136;96641280;1610690;93434636
>>
>> Performance is a little better
>> and now with tx-nocache-copy on
>>
>> 0;1;64;1597248;95834880;1597311;92644096
>> 1;1;64;1597888;95865600;1597824;92677446
>> 2;1;64;1597952;95834880;1597822;92644038
>> 3;1;64;1597568;95877120;1597375;92685044
>> 4;1;64;1597184;95827200;1597314;92629190
>> 5;1;64;1597696;95842560;1597565;92625652
>> 6;1;64;1597312;95834880;1597376;92644038
>> 7;1;64;1597568;95873280;1597634;92647924
>> 8;1;64;1598400;95919360;1598849;92699602
>> 9;1;64;1597824;95873280;1598208;92684928
>>
>>
>> That is weird - so enabling tx-nocache-copy with disabled ntuple have bad
>> performance impact - but with enabled ntuple there is no performance impact
> I would leave the ntuple feature enabled if you are routing simply
> because that disables the ixgbe feature ATR which can have a negative
> impact on routing tests (causes reordering).
>
>>>
>>>> ethtool -N $i rx-flow-hash udp4 sdfn
>>> Why do you change the NICs flow-hash?
>> whan used 16 cores / 16 rss queues - there was better load distribution over
>> all cores when sdfn rx-flow-hash enabled
> That is to be expected. The default hash will only has on IPv4
> addresses. Enabling the use of UDP ports would allow for more entropy.
> If you want similar performance without resorting to hashing on ports
> you would have to change the source/destination IP addresses.
>
>>>> done
>>>>
>>>> ip link set up dev enp216s0f0
>>>> ip link set up dev enp216s0f1
>>>>
>>>> ip a a 10.0.0.1/30 dev enp216s0f0
>>>>
>>>> ip link add link enp216s0f1 name vlan1000 type vlan id 1000
>>>> ip link set up dev vlan1000
>>>> ip a a 10.0.0.5/30 dev vlan1000
>>>>
>>>>
>>>> ip route add 172.16.0.0/12 via 10.0.0.6
>>>>
>>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f0
>>>> ./set_irq_affinity.sh -x 14-27,42-43 enp216s0f1
>>>> #cat /sys/devices/system/node/node1/cpulist
>>>> #14-27,42-55
>>>> #cat /sys/devices/system/node/node0/cpulist
>>>> #0-13,28-41
>>> Is this a NUMA system?
>> This is 2x CPU 6132 - so have two separate pcie access to the nic - need to
>> check what cpu is assigned to pcie where network card is connected to have
>> network card on local cpu where all irq's are binded
>>
>>>
>>>> #################################################
>>>>
>>>>
>>>> Looks like forwarding performance when using vlans on ixgbe is less that
>>>> without vlans for about 30-40% (wondering if this is some vlan
>>>> offloading problem and ixgbe)
>>> I would see this as a problem/bug that enabling VLANs cost this much.
>> Yes - was thinking that with tx/rx vlan offloading there will be not much
>> performance impact when vlans used.
> What is the rate difference? Also did you account for the header size
> when noticing that there is a difference in rates? I just want to make
> sure we aren't seen an issue where you are expecting a rate of
> 14.88Mpps when VLAN tags drop the rate due to header overhead down to
> something like 14.2Mpps if I recall correctly.
As reply above - the difference is
With vlan: 7Mpps (100%cpu on 16Cores with 16RSS queues)
Without vlan: 10Mpps (70% cpu load on 16Cores with 16RSS queues)
So this is really big difference.
>
>>>> settings below:
>>>>
>>>> ethtool -k enp216s0f0
>>>> Features for enp216s0f0:
>>>> Cannot get device udp-fragmentation-offload settings: Operation not
>>>> supported
>>>> rx-checksumming: on
>>>> tx-checksumming: on
>>>> tx-checksum-ipv4: off [fixed]
>>>> tx-checksum-ip-generic: on
>>>> tx-checksum-ipv6: off [fixed]
>>>> tx-checksum-fcoe-crc: off [fixed]
>>>> tx-checksum-sctp: on
>>>> scatter-gather: on
>>>> tx-scatter-gather: on
>>>> tx-scatter-gather-fraglist: off [fixed]
>>>> tcp-segmentation-offload: on
>>>> tx-tcp-segmentation: on
>>>> tx-tcp-ecn-segmentation: off [fixed]
>>>> tx-tcp-mangleid-segmentation: on
>>>> tx-tcp6-segmentation: on
>>>> udp-fragmentation-offload: off
>>>> generic-segmentation-offload: off
>>>> generic-receive-offload: on
>>>> large-receive-offload: off
>>>> rx-vlan-offload: on
>>>> tx-vlan-offload: on
>>>> ntuple-filters: on
>>>> receive-hashing: on
>>>> highdma: on [fixed]
>>>> rx-vlan-filter: on
>>>> vlan-challenged: off [fixed]
>>>> tx-lockless: off [fixed]
>>>> netns-local: off [fixed]
>>>> tx-gso-robust: off [fixed]
>>>> tx-fcoe-segmentation: off [fixed]
>>>> tx-gre-segmentation: on
>>>> tx-gre-csum-segmentation: on
>>>> tx-ipxip4-segmentation: on
>>>> tx-ipxip6-segmentation: on
>>>> tx-udp_tnl-segmentation: on
>>>> tx-udp_tnl-csum-segmentation: on
>>>> tx-gso-partial: on
>>>> tx-sctp-segmentation: off [fixed]
>>>> tx-esp-segmentation: off [fixed]
>>>> fcoe-mtu: off [fixed]
>>>> tx-nocache-copy: on
>>>> loopback: off [fixed]
>>>> rx-fcs: off [fixed]
>>>> rx-all: off
>>>> tx-vlan-stag-hw-insert: off [fixed]
>>>> rx-vlan-stag-hw-parse: off [fixed]
>>>> rx-vlan-stag-filter: off [fixed]
>>>> l2-fwd-offload: off
>>>> hw-tc-offload: off
>>>> esp-hw-offload: off [fixed]
>>>> esp-tx-csum-hw-offload: off [fixed]
>>>> rx-udp_tunnel-port-offload: on
>>>>
>>>>
>>>> Another thing is that forwarding performance does not scale with number
>>>> of cores when 7+ cores are reached
>>> I've seen problems with using Hyper-Threading CPUs. Could it be that
>>> above 7 CPUs you are starting to use sibling-cores ?
>>>
> I would suspect that it may be more than likely the case. One thing
> you might look at doing is CPU pinning the interrupts for the NIC in a
> 1:1 fashion so that the queues are all bound to separate cores without
> them sharing between Hyper-threads.
>
>> Turbostats can help here:
>> Package Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IRQ SMI C1
>> C2 C1% C2% CPU%c1 CPU%c6 CoreTmp PkgTmp PkgWatt RAMWatt
>> PKG_% RAM_%
>> - - - 72 2.27 3188 2600 194844 0 64
>> 69282 0.07 97.83 18.38 79.36 -4 54 123.49 16.08 0.00
>> 0.00
>> 0 0 0 8 0.74 1028 2600 1513 0 32
>> 1462 1.50 97.99 10.92 88.34 47 51 58.34 5.34 0.00
>> 0.00
>> 0 0 28 7 0.67 1015 2600 1255 0 12
>> 1249 0.96 98.61 10.99
>> 0 1 1 7 0.68 1019 2600 1260 0 0
>> 1260 0.00 99.54 8.44 90.88 49
>> 0 1 29 9 0.71 1208 2600 1252 0 0
>> 1253 0.00 99.48 8.41
>> 0 2 2 7 0.67 1019 2600 1261 0 0
>> 1260 0.00 99.54 8.44 90.89 48
>> 0 2 30 7 0.67 1017 2600 1255 0 0
>> 1255 0.00 99.55 8.44
>> 0 3 3 7 0.68 1019 2600 1260 0 0
>> 1259 0.00 99.53 8.46 90.86 -4
>> 0 3 31 7 0.67 1017 2600 1256 0 0
>> 1256 0.00 99.55 8.46
>> 0 4 4 7 0.67 1027 2600 1260 0 0
>> 1260 0.00 99.54 8.43 90.90 -4
>> 0 4 32 7 0.66 1018 2600 1255 0 0
>> 1255 0.00 99.55 8.44
>> 0 5 5 7 0.68 1020 2600 1260 0 0
>> 1257 0.00 99.54 8.44 90.89 50
>> 0 5 33 7 0.68 1019 2600 1255 0 0
>> 1255 0.00 99.55 8.43
>> 0 6 6 7 0.70 1019 2600 1260 0 0
>> 1259 0.00 99.53 8.43 90.87 -4
>> 0 6 34 7 0.70 1019 2600 1255 0 0
>> 1255 0.00 99.54 8.43
>> 0 8 7 7 0.68 1019 2600 1262 0 0
>> 1261 0.00 99.52 8.42 90.90 50
>> 0 8 35 7 0.67 1019 2600 1255 0 0
>> 1255 0.00 99.55 8.43
>> 0 9 8 7 0.68 1019 2600 1260 0 0
>> 1257 0.00 99.54 8.40 90.92 49
>> 0 9 36 7 0.66 1017 2600 1255 0 0
>> 1255 0.00 99.55 8.41
>> 0 10 9 7 0.66 1018 2600 1257 0 0
>> 1257 0.00 99.54 8.40 90.94 -4
>> 0 10 37 7 0.66 1018 2600 1255 0 0
>> 1255 0.00 99.55 8.41
>> 0 11 10 7 0.66 1019 2600 1257 0 0
>> 1259 0.00 99.54 8.56 90.77 -4
>> 0 11 38 7 0.66 1018 2600 1255 0 3
>> 1252 0.19 99.36 8.57
>> 0 12 11 7 0.67 1019 2600 1260 0 0
>> 1260 0.00 99.54 8.44 90.88 -4
>> 0 12 39 7 0.67 1019 2600 1255 0 0
>> 1256 0.00 99.55 8.44
>> 0 13 12 7 0.68 1019 2600 1257 0 4
>> 1254 0.32 99.22 8.67 90.65 -4
>> 0 13 40 7 0.69 1019 2600 1256 0 4
>> 1253 0.24 99.31 8.66
>> 0 14 13 7 0.71 1020 2600 1260 0 0
>> 1259 0.00 99.53 8.41 90.88 -4
>> 0 14 41 7 0.72 1020 2600 1255 0 0
>> 1255 0.00 99.54 8.40
>> 1 0 14 3564 99.19 3594 2600 125472 0 0
>> 0 0.00 0.00 0.81 0.00 54 54 65.15 10.74 0.00
>> 0.00
>> 1 0 42 3 0.07 3701 2600 1255 0 0
>> 1255 0.00 99.95 99.93
>> 1 1 15 11 0.32 3301 2600 1257 0 0
>> 1257 0.00 99.81 26.37 73.31 42
>> 1 1 43 10 0.31 3301 2600 1255 0 0
>> 1255 0.00 99.82 26.38
>> 1 2 16 10 0.31 3301 2600 1257 0 0
>> 1257 0.00 99.81 26.37 73.32 39
>> 1 2 44 10 0.32 3301 2600 1255 0 0
>> 1255 0.00 99.82 26.36
>> 1 3 17 10 0.32 3301 2600 1257 0 0
>> 1257 0.00 99.81 26.40 73.28 39
>> 1 3 45 11 0.32 3301 2600 1255 0 0
>> 1255 0.00 99.81 26.40
>> 1 4 18 10 0.32 3301 2600 1257 0 0
>> 1257 0.00 99.82 26.40 73.28 40
>> 1 4 46 11 0.32 3301 2600 1255 0 0
>> 1255 0.00 99.82 26.40
>> 1 5 19 11 0.33 3301 2600 1257 0 0
>> 1257 0.00 99.81 26.40 73.27 39
>> 1 5 47 11 0.33 3300 2600 1255 0 0
>> 1255 0.00 99.82 26.40
>> 1 6 20 12 0.35 3301 2600 1257 0 0
>> 1257 0.00 99.81 26.38 73.27 42
>> 1 6 48 12 0.36 3301 2600 1255 0 0
>> 1255 0.00 99.81 26.37
>> 1 8 21 11 0.33 3301 2600 1257 0 0
>> 1257 0.00 99.82 26.37 73.29 42
>> 1 8 49 11 0.33 3301 2600 1255 0 0
>> 1255 0.00 99.82 26.38
>> 1 9 22 10 0.32 3300 2600 1257 0 0
>> 1257 0.00 99.82 26.35 73.34 41
>> 1 9 50 10 0.30 3301 2600 1255 0 0
>> 1255 0.00 99.82 26.36
>> 1 10 23 10 0.31 3301 2600 1257 0 0
>> 1257 0.00 99.82 26.37 73.33 41
>> 1 10 51 10 0.31 3301 2600 1255 0 0
>> 1255 0.00 99.82 26.36
>> 1 11 24 10 0.32 3301 2600 1257 0 0
>> 1257 0.00 99.81 26.62 73.06 41
>> 1 11 52 10 0.32 3301 2600 1255 0 4
>> 1251 0.32 99.50 26.62
>> 1 12 25 11 0.33 3301 2600 1257 0 0
>> 1257 0.00 99.81 26.39 73.28 41
>> 1 12 53 11 0.33 3301 2600 1258 0 0
>> 1254 0.00 99.82 26.38
>> 1 13 26 12 0.36 3317 2600 1259 0 0
>> 1258 0.00 99.79 26.41 73.23 39
>> 1 13 54 11 0.34 3301 2600 1255 0 0
>> 1254 0.00 99.82 26.42
>> 1 14 27 12 0.36 3301 2600 1257 0 5
>> 1251 0.24 99.58 26.54 73.10 41
>> 1 14 55 12 0.36 3300 2600 1255 0 0
>> 1254 0.00 99.82 26.54
>>
>>
>> So it looks like in all tests i'm using core+sibling
>> But side effect of this is that :
>> 33 * 100.0 = 3300.0 MHz max turbo 28 active cores
>> 33 * 100.0 = 3300.0 MHz max turbo 24 active cores
>> 33 * 100.0 = 3300.0 MHz max turbo 20 active cores
>> 33 * 100.0 = 3300.0 MHz max turbo 14 active cores
>> 34 * 100.0 = 3400.0 MHz max turbo 12 active cores
>> 34 * 100.0 = 3400.0 MHz max turbo 8 active cores
>> 35 * 100.0 = 3500.0 MHz max turbo 4 active cores
>> 37 * 100.0 = 3700.0 MHz max turbo 2 active cores
>>
>> So more cores = less MHz per core/sibling
> Yes that is always a trade off. Also the ixgbe is limited in terms of
> PCIe bus bandwidth. The more queues you add the worse the descriptor
> overhead will be. Generally I have found that about 6 queues is ideal.
> As you start getting to more than 8 the performance for 64B packets
> will start to drop off as each additional queue will hurt the
> descriptor cache performance as it starts writing back fewer and fewer
> descriptors per write which will increase the PCIe bus overhead for
> the writes.
>
>>>> perf top:
>>>>
>>>> PerfTop: 77835 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>>> cycles], (all, 56 CPUs)
>>>>
>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> 16.32% [kernel] [k] skb_dst_force
>>>> 16.30% [kernel] [k] dst_release
>>>> 15.11% [kernel] [k] rt_cache_valid
>>>> 12.62% [kernel] [k] ipv4_mtu
>>> It seems a little strange that these 4 functions are on the top
>> Yes dono why there is ipv4_mtu called and taking soo much cycles
>>
>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>> Why is calling/taking this lock? (Use perf call-graph recording).
>> can be hard to paste it here:)
>> attached file
>>
>>>> 3.03% [kernel] [k] fib_table_lookup
>>>> 2.70% [kernel] [k] ip_finish_output2
>>>> 2.10% [kernel] [k] dev_gro_receive
>>>> 1.89% [kernel] [k] eth_type_trans
>>>> 1.81% [kernel] [k] ixgbe_poll
>>>> 1.15% [kernel] [k] ixgbe_xmit_frame_ring
>>>> 1.06% [kernel] [k] __build_skb
>>>> 1.04% [kernel] [k] __dev_queue_xmit
>>>> 0.97% [kernel] [k] ip_rcv
>>>> 0.78% [kernel] [k] netif_skb_features
>>>> 0.74% [kernel] [k] ipt_do_table
>>> Unloading netfilter modules, will give more performance, but it
>>> semifake to do so.
>> Compiled in kernel - only in filter mode - with ipv4+ipv6 - no other modules
>> conntrack or other .
>>
>>>> 0.70% [kernel] [k] acpi_processor_ffh_cstate_enter
>>>> 0.64% [kernel] [k] ip_forward
>>>> 0.59% [kernel] [k] __netif_receive_skb_core
>>>> 0.55% [kernel] [k] dev_hard_start_xmit
>>>> 0.53% [kernel] [k] ip_route_input_rcu
>>>> 0.53% [kernel] [k] ip_rcv_finish
>>>> 0.51% [kernel] [k] page_frag_free
>>>> 0.50% [kernel] [k] kmem_cache_alloc
>>>> 0.50% [kernel] [k] udp_v4_early_demux
>>>> 0.44% [kernel] [k] skb_release_data
>>>> 0.42% [kernel] [k] inet_gro_receive
>>>> 0.40% [kernel] [k] sch_direct_xmit
>>>> 0.39% [kernel] [k] __local_bh_enable_ip
>>>> 0.33% [kernel] [k] netdev_pick_tx
>>>> 0.33% [kernel] [k] validate_xmit_skb
>>>> 0.28% [kernel] [k] fib_validate_source
>>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>>> 0.25% [kernel] [k] eth_header
>>>> 0.23% [kernel] [k] get_dma_ops
>>>> 0.22% [kernel] [k] skb_network_protocol
>>>> 0.21% [kernel] [k] ip_output
>>>> 0.21% [kernel] [k] vlan_dev_hard_start_xmit
>>>> 0.20% [kernel] [k] ixgbe_alloc_rx_buffers
>>>> 0.18% [kernel] [k] nf_hook_slow
>>>> 0.18% [kernel] [k] apic_timer_interrupt
>>>> 0.18% [kernel] [k] virt_to_head_page
>>>> 0.18% [kernel] [k] build_skb
>>>> 0.16% [kernel] [k] swiotlb_map_page
>>>> 0.16% [kernel] [k] ip_finish_output
>>>> 0.16% [kernel] [k] udp4_gro_receive
>>>>
>>>>
>>>> RESULTS:
>>>>
>>>> CSV format - delimeter ";"
>>>>
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;1;64;1470912;88247040;1470720;85305530
>>>> 1;1;64;1470912;88285440;1470977;85335110
>>>> 2;1;64;1470464;88247040;1470402;85290508
>>>> 3;1;64;1471424;88262400;1471230;85353728
>>>> 4;1;64;1468736;88166400;1468672;85201652
>>>> 5;1;64;1470016;88181760;1469949;85234944
>>>> 6;1;64;1470720;88247040;1470466;85290624
>>>> 7;1;64;1471232;88277760;1471167;85346246
>>>> 8;1;64;1469184;88170240;1469249;85216326
>>>> 9;1;64;1470592;88227840;1470847;85294394
>>> Single core 1.47Mpps seems a little low, I would expect 2Mpps.
>>>
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;2;64;2413120;144802560;2413245;139975924
>>>> 1;2;64;2415296;144913920;2415356;140098188
>>>> 2;2;64;2416768;144898560;2416573;140105670
>>>> 3;2;64;2418176;145056000;2418110;140261806
>>>> 4;2;64;2416512;144990720;2416509;140172950
>>>> 5;2;64;2415168;144860160;2414466;140064780
>>>> 6;2;64;2416960;144983040;2416833;140190930
>>>> 7;2;64;2413632;144768000;2413568;140001734
>>>> 8;2;64;2415296;144898560;2414589;140087168
>>>> 9;2;64;2416576;144963840;2416892;140190930
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;3;64;3419008;205155840;3418882;198239244
>>>> 1;3;64;3428032;205585920;3427971;198744234
>>>> 2;3;64;3425472;205536000;3425344;198677260
>>>> 3;3;64;3425088;205470720;3425156;198603136
>>>> 4;3;64;3427648;205693440;3426883;198773888
>>>> 5;3;64;3426880;205670400;3427392;198796044
>>>> 6;3;64;3429120;205678080;3430140;198848186
>>>> 7;3;64;3422976;205355520;3423490;198458136
>>>> 8;3;64;3423168;205336320;3423486;198495372
>>>> 9;3;64;3424384;205493760;3425538;198617868
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;4;64;4406464;264364800;4405244;255560296
>>>> 1;4;64;4404672;264349440;4405122;255541504
>>>> 2;4;64;4402368;264049920;4403326;255188864
>>>> 3;4;64;4401344;264076800;4400702;255207134
>>>> 4;4;64;4385536;263074560;4386620;254312716
>>>> 5;4;64;4386560;263189760;4385404;254379532
>>>> 6;4;64;4398784;263857920;4399031;255025288
>>>> 7;4;64;4407232;264445440;4407998;255637900
>>>> 8;4;64;4413184;264698880;4413758;255875816
>>>> 9;4;64;4411328;264526080;4411906;255712372
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;5;64;5094464;305871360;5094464;295657262
>>>> 1;5;64;5090816;305514240;5091201;295274810
>>>> 2;5;64;5088384;305387520;5089792;295175108
>>>> 3;5;64;5079296;304869120;5079484;294680368
>>>> 4;5;64;5092992;305544960;5094207;295349166
>>>> 5;5;64;5092416;305502720;5093372;295334260
>>>> 6;5;64;5080896;304896000;5081090;294677004
>>>> 7;5;64;5085376;305114880;5086401;294933058
>>>> 8;5;64;5092544;305575680;5092036;295356938
>>>> 9;5;64;5093056;305652480;5093832;295449506
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;6;64;5705088;342351360;5705784;330965110
>>>> 1;6;64;5710272;342743040;5707591;331373952
>>>> 2;6;64;5703424;342182400;5701826;330776552
>>>> 3;6;64;5708736;342604800;5707963;331147462
>>>> 4;6;64;5710144;342654720;5712067;331202910
>>>> 5;6;64;5712064;342777600;5711361;331292288
>>>> 6;6;64;5710144;342585600;5708607;331144272
>>>> 7;6;64;5699840;342021120;5697853;330609222
>>>> 8;6;64;5701184;342124800;5702909;330653592
>>>> 9;6;64;5711360;342735360;5713283;331247686
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;7;64;6244416;374603520;6243591;362180072
>>>> 1;7;64;6230912;374016000;6231490;361534126
>>>> 2;7;64;6244800;374776320;6244866;362224326
>>>> 3;7;64;6238720;374376960;6238261;361838510
>>>> 4;7;64;6218816;373079040;6220413;360683962
>>>> 5;7;64;6224320;373566720;6225086;361017404
>>>> 6;7;64;6224000;373570560;6221370;360936088
>>>> 7;7;64;6210048;372741120;6210627;360212654
>>>> 8;7;64;6231616;374035200;6231537;361445502
>>>> 9;7;64;6227840;373724160;6228802;361162752
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;8;64;6251840;375144960;6251849;362609678
>>>> 1;8;64;6250816;375014400;6250881;362547038
>>>> 2;8;64;6257728;375432960;6257160;362911104
>>>> 3;8;64;6255552;375325440;6255622;362822074
>>>> 4;8;64;6243776;374576640;6243270;362120622
>>>> 5;8;64;6237184;374296320;6237690;361790080
>>>> 6;8;64;6240960;374415360;6240714;361927366
>>>> 7;8;64;6222784;373317120;6223746;360854424
>>>> 8;8;64;6225920;373593600;6227014;361154980
>>>> 9;8;64;6238528;374304000;6237701;361845238
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;14;64;6486144;389184000;6486135;376236488
>>>> 1;14;64;6454912;387390720;6454222;374466734
>>>> 2;14;64;6441152;386480640;6440431;373572780
>>>> 3;14;64;6450240;386972160;6450870;374070014
>>>> 4;14;64;6465600;387997440;6467221;375089654
>>>> 5;14;64;6448384;386860800;6448000;373980230
>>>> 6;14;64;6452352;387095040;6452148;374168904
>>>> 7;14;64;6441984;386507520;6443203;373665058
>>>> 8;14;64;6456704;387340800;6455744;374429092
>>>> 9;14;64;6464640;387901440;6465218;374949004
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;16;64;6939008;416325120;6938696;402411192
>>>> 1;16;64;6941952;416444160;6941745;402558918
>>>> 2;16;64;6960576;417584640;6960707;403698718
>>>> 3;16;64;6940736;416486400;6941820;402503876
>>>> 4;16;64;6927680;415741440;6927420;401853870
>>>> 5;16;64;6929792;415687680;6929917;401839196
>>>> 6;16;64;6950400;416989440;6950661;403026166
>>>> 7;16;64;6953664;417216000;6953454;403260544
>>>> 8;16;64;6948480;416851200;6948800;403023266
>>>> 9;16;64;6924160;415422720;6924092;401542468
>>> I've seen Linux scale beyond 6.9Mpps, thus I also see this as an
>>> issue/bug. You could be stalling on DMA TX completion being too slow,
>>> but you already increased the interval and increased the TX ring queue
>>> size. You could play with those setting and see if it changes this?
>>>
>>> Could you try my napi_monitor tool in:
>>>
>>> https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf
>>>
>>> Also provide the output from:
>>> mpstat -P ALL -u -I SCPU -I SUM 2
>> with 16 cores / 16 RSS queues
>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal
>> %guest %gnice %idle
>> Average: all 0.00 0.00 0.01 0.00 0.00 28.57 0.00
>> 0.00 0.00 71.42
>> Average: 0 0.00 0.00 0.04 0.00 0.00 0.08 0.00
>> 0.00 0.00 99.88
>> Average: 1 0.00 0.00 0.12 0.00 0.00 0.00 0.00
>> 0.00 0.00 99.88
>> Average: 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 4 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 8 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 9 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 10 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 11 0.08 0.00 0.04 0.00 0.00 0.00 0.00
>> 0.00 0.00 99.88
>> Average: 12 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 14 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 15 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 16 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 17 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 18 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 19 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 20 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 21 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 22 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 23 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 24 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 25 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 26 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 27 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 28 0.00 0.00 0.04 0.00 0.00 0.00 0.00
>> 0.00 0.00 99.96
>> Average: 29 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 30 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 31 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 32 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 34 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 35 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 36 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 38 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 39 0.04 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 99.96
>> Average: 40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 41 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 42 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 43 0.00 0.00 0.00 0.00 0.00 100.00 0.00
>> 0.00 0.00 0.00
>> Average: 44 0.00 0.00 0.04 0.17 0.00 0.00 0.00
>> 0.00 0.00 99.79
>> Average: 45 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 46 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 47 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 48 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 49 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 50 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 51 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 52 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 54 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>> Average: 55 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>> 0.00 0.00 100.00
>>
>> Average: CPU intr/s
>> Average: all 123596.08
>> Average: 0 646.38
>> Average: 1 500.54
>> Average: 2 511.67
>> Average: 3 534.25
>> Average: 4 542.21
>> Average: 5 531.54
>> Average: 6 554.58
>> Average: 7 535.88
>> Average: 8 544.58
>> Average: 9 536.42
>> Average: 10 575.46
>> Average: 11 601.12
>> Average: 12 502.08
>> Average: 13 575.46
>> Average: 14 5917.92
>> Average: 15 5949.58
>> Average: 16 7021.29
>> Average: 17 7299.71
>> Average: 18 7391.67
>> Average: 19 7354.25
>> Average: 20 7543.42
>> Average: 21 7354.25
>> Average: 22 7322.33
>> Average: 23 7368.71
>> Average: 24 7429.00
>> Average: 25 7406.46
>> Average: 26 7400.67
>> Average: 27 7447.21
>> Average: 28 517.00
>> Average: 29 549.54
>> Average: 30 529.33
>> Average: 31 533.83
>> Average: 32 541.25
>> Average: 33 541.17
>> Average: 34 532.50
>> Average: 35 545.17
>> Average: 36 528.96
>> Average: 37 509.92
>> Average: 38 520.12
>> Average: 39 523.29
>> Average: 40 530.75
>> Average: 41 542.33
>> Average: 42 5921.71
>> Average: 43 5949.42
>> Average: 44 503.04
>> Average: 45 542.75
>> Average: 46 582.50
>> Average: 47 581.71
>> Average: 48 495.29
>> Average: 49 524.38
>> Average: 50 527.92
>> Average: 51 528.12
>> Average: 52 456.38
>> Average: 53 477.00
>> Average: 54 440.92
>> Average: 55 568.83
>>
>> Average: CPU HI/s TIMER/s NET_TX/s NET_RX/s BLOCK/s
>> IRQ_POLL/s TASKLET/s SCHED/s HRTIMER/s RCU/s
>> Average: 0 0.00 250.00 0.17 87.00 0.00 0.00
>> 45.46 250.00 0.00 13.75
>> Average: 1 0.00 233.42 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 17.21
>> Average: 2 0.00 249.04 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 12.67
>> Average: 3 0.00 249.92 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 34.42
>> Average: 4 0.00 248.67 0.17 0.00 0.00 0.00
>> 0.00 249.96 0.00 43.42
>> Average: 5 0.00 249.46 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 32.17
>> Average: 6 0.00 249.79 0.00 0.00 0.00 0.00
>> 0.00 249.87 0.00 54.92
>> Average: 7 0.00 240.12 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 45.79
>> Average: 8 0.00 247.42 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 47.25
>> Average: 9 0.00 249.29 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 37.17
>> Average: 10 0.00 248.75 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 76.79
>> Average: 11 0.00 249.29 0.00 0.00 0.00 0.00
>> 42.79 249.83 0.00 59.21
>> Average: 12 0.00 249.83 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 2.29
>> Average: 13 0.00 249.92 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 75.62
>> Average: 14 0.00 148.21 0.17 5758.04 0.00 0.00
>> 0.00 8.42 0.00 3.08
>> Average: 15 0.00 148.42 0.46 5789.25 0.00 0.00
>> 0.00 8.33 0.00 3.12
>> Average: 16 0.00 142.62 0.79 6866.46 0.00 0.00
>> 0.00 8.29 0.00 3.12
>> Average: 17 0.00 143.17 0.42 7145.00 0.00 0.00
>> 0.00 8.08 0.00 3.04
>> Average: 18 0.00 153.62 0.42 7226.42 0.00 0.00
>> 0.00 8.04 0.00 3.17
>> Average: 19 0.00 150.46 0.46 7192.21 0.00 0.00
>> 0.00 8.04 0.00 3.08
>> Average: 20 0.00 145.21 0.17 7386.50 0.00 0.00
>> 0.00 8.29 0.00 3.25
>> Average: 21 0.00 150.96 0.46 7191.37 0.00 0.00
>> 0.00 8.25 0.00 3.21
>> Average: 22 0.00 146.67 0.54 7163.96 0.00 0.00
>> 0.00 8.04 0.00 3.12
>> Average: 23 0.00 151.38 0.42 7205.75 0.00 0.00
>> 0.00 8.00 0.00 3.17
>> Average: 24 0.00 153.33 0.17 7264.12 0.00 0.00
>> 0.00 8.08 0.00 3.29
>> Average: 25 0.00 153.21 0.17 7241.83 0.00 0.00
>> 0.00 7.96 0.00 3.29
>> Average: 26 0.00 153.96 0.17 7234.88 0.00 0.00
>> 0.00 8.38 0.00 3.29
>> Average: 27 0.00 151.71 0.79 7283.25 0.00 0.00
>> 0.00 8.04 0.00 3.42
>> Average: 28 0.00 245.71 0.00 0.00 0.00 0.00
>> 0.00 249.50 0.00 21.79
>> Average: 29 0.00 233.21 0.00 0.00 0.00 0.00
>> 0.00 249.87 0.00 66.46
>> Average: 30 0.00 248.92 0.00 0.00 0.00 0.00
>> 0.00 250.00 0.00 30.42
>> Average: 31 0.00 249.92 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 33.96
>> Average: 32 0.00 248.67 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 42.62
>> Average: 33 0.00 249.46 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 41.79
>> Average: 34 0.00 249.79 0.00 0.00 0.00 0.00
>> 0.00 249.87 0.00 32.83
>> Average: 35 0.00 240.12 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 55.08
>> Average: 36 0.00 247.42 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 31.58
>> Average: 37 0.00 249.29 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 10.71
>> Average: 38 0.00 248.75 0.00 0.00 0.00 0.00
>> 0.00 249.87 0.00 21.50
>> Average: 39 0.00 249.50 0.00 0.00 0.00 0.00
>> 0.00 249.83 0.00 23.96
>> Average: 40 0.00 249.83 0.00 0.00 0.00 0.00
>> 0.00 249.96 0.00 30.96
>> Average: 41 0.00 249.92 0.00 0.00 0.00 0.00
>> 0.00 249.92 0.00 42.50
>> Average: 42 0.00 148.38 0.71 5761.00 0.00 0.00
>> 0.00 8.25 0.00 3.38
>> Average: 43 0.00 147.21 0.50 5790.33 0.00 0.00
>> 0.00 8.00 0.00 3.38
>> Average: 44 0.00 248.96 0.00 0.00 0.00 0.00
>> 0.00 248.13 0.00 5.96
>> Average: 45 0.00 249.04 0.00 0.00 0.00 0.00
>> 0.00 248.88 0.00 44.83
>> Average: 46 0.00 248.96 0.00 0.00 0.00 0.00
>> 0.00 248.58 0.00 84.96
>> Average: 47 0.00 249.00 0.00 0.00 0.00 0.00
>> 0.00 248.75 0.00 83.96
>> Average: 48 0.00 249.12 0.00 0.00 0.00 0.00
>> 0.00 132.83 0.00 113.33
>> Average: 49 0.00 249.12 0.00 0.00 0.00 0.00
>> 0.00 248.62 0.00 26.62
>> Average: 50 0.00 248.92 0.00 0.00 0.00 0.00
>> 0.00 248.58 0.00 30.42
>> Average: 51 0.00 249.08 0.00 0.00 0.00 0.00
>> 0.00 248.42 0.00 30.63
>> Average: 52 0.00 249.21 0.00 0.00 0.00 0.00
>> 0.00 131.96 0.00 75.21
>> Average: 53 0.00 249.08 0.00 0.00 0.00 0.00
>> 0.00 136.12 0.00 91.79
>> Average: 54 0.00 249.00 0.00 0.00 0.00 0.00
>> 0.00 136.79 0.00 55.12
>> Average: 55 0.00 249.04 0.00 0.00 0.00 0.00
>> 0.00 248.71 0.00 71.08
>>
>>
>>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-13 16:58 ` Paweł Staszewski
@ 2017-08-14 16:19 ` Jesper Dangaard Brouer
2017-08-14 16:33 ` Eric Dumazet
` (2 more replies)
0 siblings, 3 replies; 37+ messages in thread
From: Jesper Dangaard Brouer @ 2017-08-14 16:19 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Linux Kernel Network Developers, brouer, Alexander Duyck
On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> To show some difference below comparision vlan/no-vlan traffic
>
> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
I'm trying to reproduce in my testlab (with ixgbe). I do see, a
performance reduction of about 10-19% when I forward out a VLAN
interface. This is larger than I expected, but still lower than what
you reported 30-40% slowdown.
[...]
> >>> perf top:
> >>>
> >>> PerfTop: 77835 irqs/sec kernel:99.7%
> >>> ---------------------------------------------
> >>>
> >>> 16.32% [kernel] [k] skb_dst_force
> >>> 16.30% [kernel] [k] dst_release
> >>> 15.11% [kernel] [k] rt_cache_valid
> >>> 12.62% [kernel] [k] ipv4_mtu
> >> It seems a little strange that these 4 functions are on the top
I don't see these in my test.
> >>
> >>> 5.60% [kernel] [k] do_raw_spin_lock
> >> Why is calling/taking this lock? (Use perf call-graph recording).
> > can be hard to paste it here:)
> > attached file
The attached was very big. Please don't attach so big file on mailing
lists. Next time plase share them via e.g. pastebin. The output was a
capture from your terminal, which made the output more difficult to
read. Hint: You can/could use perf --stdio and place it in a file
instead.
The output (extracted below) didn't show who called 'do_raw_spin_lock',
BUT it showed another interesting thing. The kernel code
__dev_queue_xmit() in might create route dst-cache problem for itself(?),
as it will first call skb_dst_force() and then skb_dst_drop() when the
packet is transmitted on a VLAN.
static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
{
[...]
/* If device/qdisc don't need skb->dst, release it right now while
* its hot in this cpu cache.
*/
if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
skb_dst_drop(skb);
else
skb_dst_force(skb);
- -
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
Extracted part of attached perf output:
--5.37%--ip_rcv_finish
|
|--4.02%--ip_forward
| |
| --3.92%--ip_forward_finish
| |
| --3.91%--ip_output
| |
| --3.90%--ip_finish_output
| |
| --3.88%--ip_finish_output2
| |
| --2.77%--neigh_connected_output
| |
| --2.74%--dev_queue_xmit
| |
| --2.73%--__dev_queue_xmit
| |
| |--1.66%--dev_hard_start_xmit
| | |
| | --1.64%--vlan_dev_hard_start_xmit
| | |
| | --1.63%--dev_queue_xmit
| | |
| | --1.62%--__dev_queue_xmit
| | |
| | |--0.99%--skb_dst_drop.isra.77
| | | |
| | | --0.99%--dst_release
| | |
| | --0.55%--sch_direct_xmit
| |
| --0.99%--skb_dst_force
|
--1.29%--ip_route_input_noref
|
--1.29%--ip_route_input_rcu
|
--1.05%--rt_cache_valid
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-14 16:19 ` Jesper Dangaard Brouer
@ 2017-08-14 16:33 ` Eric Dumazet
2017-08-14 16:57 ` Paolo Abeni
2017-08-15 0:38 ` Paweł Staszewski
2 siblings, 0 replies; 37+ messages in thread
From: Eric Dumazet @ 2017-08-14 16:33 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Paweł Staszewski, Linux Kernel Network Developers, Alexander Duyck
On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
> The output (extracted below) didn't show who called 'do_raw_spin_lock',
> BUT it showed another interesting thing. The kernel code
> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> as it will first call skb_dst_force() and then skb_dst_drop() when the
> packet is transmitted on a VLAN.
>
> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> {
> [...]
> /* If device/qdisc don't need skb->dst, release it right now while
> * its hot in this cpu cache.
> */
> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> skb_dst_drop(skb);
> else
> skb_dst_force(skb);
This is explained in this commit changelog.
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/?id=93f154b594fe47e4a7e5358b309add449a046cd3
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-14 16:19 ` Jesper Dangaard Brouer
2017-08-14 16:33 ` Eric Dumazet
@ 2017-08-14 16:57 ` Paolo Abeni
2017-08-15 0:45 ` Paweł Staszewski
2017-08-15 9:35 ` Jesper Dangaard Brouer
2017-08-15 0:38 ` Paweł Staszewski
2 siblings, 2 replies; 37+ messages in thread
From: Paolo Abeni @ 2017-08-14 16:57 UTC (permalink / raw)
To: Jesper Dangaard Brouer, Paweł Staszewski
Cc: Linux Kernel Network Developers, Alexander Duyck, Eric Dumazet
On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
> The output (extracted below) didn't show who called 'do_raw_spin_lock',
> BUT it showed another interesting thing. The kernel code
> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> as it will first call skb_dst_force() and then skb_dst_drop() when the
> packet is transmitted on a VLAN.
>
> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> {
> [...]
> /* If device/qdisc don't need skb->dst, release it right now while
> * its hot in this cpu cache.
> */
> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> skb_dst_drop(skb);
> else
> skb_dst_force(skb);
I think that the high impact of the above code in this specific test is
mostly due to the following:
- ingress packets with different RSS rx hash lands on different CPUs
- but they use the same dst entry, since the destination IPs belong to
the same subnet
- the dst refcnt cacheline is contented between all the CPUs
Perhaps we can inprove the situation setting the IFF_XMIT_DST_RELEASE
flag for vlan if the underlaying device does not have (relevant)
classifier attached? (and clearing it as needed)
Paolo
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-14 16:19 ` Jesper Dangaard Brouer
2017-08-14 16:33 ` Eric Dumazet
2017-08-14 16:57 ` Paolo Abeni
@ 2017-08-15 0:38 ` Paweł Staszewski
2017-08-15 9:23 ` Jesper Dangaard Brouer
2 siblings, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-15 0:38 UTC (permalink / raw)
To: Jesper Dangaard Brouer; +Cc: Linux Kernel Network Developers, Alexander Duyck
W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> To show some difference below comparision vlan/no-vlan traffic
>>
>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
> performance reduction of about 10-19% when I forward out a VLAN
> interface. This is larger than I expected, but still lower than what
> you reported 30-40% slowdown.
>
> [...]
Ok mellanox afrrived (MT27700 - mlnx5 driver)
And to compare melannox with vlans and without: 33% performance
degradation (less than with ixgbe where i reach ~40% with same settings)
Mellanox without TX traffix on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;11089305;709715520;8871553;567779392
1;16;64;11096292;710162688;11095566;710116224
2;16;64;11095770;710129280;11096799;710195136
3;16;64;11097199;710220736;11097702;710252928
4;16;64;11080984;567081856;11079662;709098368
5;16;64;11077696;708972544;11077039;708930496
6;16;64;11082991;709311424;8864802;567347328
7;16;64;11089596;709734144;8870927;709789184
8;16;64;11094043;710018752;11095391;710105024
Mellanox with TX traffic on vlan:
ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
0;16;64;7369914;471674496;7370281;471697980
1;16;64;7368896;471609408;7368043;471554752
2;16;64;7367577;471524864;7367759;471536576
3;16;64;7368744;377305344;7369391;471641024
4;16;64;7366824;471476736;7364330;471237120
5;16;64;7368352;471574528;7367239;471503296
6;16;64;7367459;471517376;7367806;471539584
7;16;64;7367190;471500160;7367988;471551232
8;16;64;7368023;471553472;7368076;471556864
ethtool settings for both tests:
ifc='enp175s0f0 enp175s0f1'
for i in $ifc
do
ip link set up dev $i
ethtool -A $i autoneg off rx off tx off
ethtool -G $i rx 128 tx 256
ip link set $i txqueuelen 1000
ethtool -C $i rx-usecs 25
ethtool -L $i combined 16
ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
tx-nocache-copy off ntuple on
ethtool -N $i rx-flow-hash udp4 sdfn
done
and perf top:
PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
cycles], (all, 56 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
14.25% [kernel] [k] dst_release
14.17% [kernel] [k] skb_dst_force
13.41% [kernel] [k] rt_cache_valid
11.47% [kernel] [k] ip_finish_output2
7.01% [kernel] [k] do_raw_spin_lock
5.07% [kernel] [k] page_frag_free
3.47% [mlx5_core] [k] mlx5e_xmit
2.88% [kernel] [k] fib_table_lookup
2.43% [mlx5_core] [k] skb_from_cqe.isra.32
1.97% [kernel] [k] virt_to_head_page
1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
0.93% [kernel] [k] __dev_queue_xmit
0.87% [kernel] [k] __build_skb
0.84% [kernel] [k] ipt_do_table
0.79% [kernel] [k] ip_rcv
0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
0.78% [kernel] [k] netif_skb_features
0.73% [kernel] [k] __netif_receive_skb_core
0.52% [kernel] [k] dev_hard_start_xmit
0.52% [kernel] [k] build_skb
0.51% [kernel] [k] ip_route_input_rcu
0.50% [kernel] [k] skb_unref
0.49% [kernel] [k] ip_forward
0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
0.44% [kernel] [k] udp_v4_early_demux
0.41% [kernel] [k] napi_consume_skb
0.40% [kernel] [k] __local_bh_enable_ip
0.39% [kernel] [k] ip_rcv_finish
0.39% [kernel] [k] kmem_cache_alloc
0.38% [kernel] [k] sch_direct_xmit
0.33% [kernel] [k] validate_xmit_skb
0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
0.29% [kernel] [k] netdev_pick_tx
0.28% [mlx5_core] [k] mlx5e_build_rx_skb
0.27% [kernel] [k] deliver_ptype_list_skb
0.26% [kernel] [k] fib_validate_source
0.26% [mlx5_core] [k] mlx5e_napi_poll
0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
0.26% [mlx5_core] [k] mlx5e_rx_cache_get
0.25% [kernel] [k] eth_header
0.23% [kernel] [k] skb_network_protocol
0.20% [kernel] [k] nf_hook_slow
0.20% [kernel] [k] vlan_passthru_hard_header
0.20% [kernel] [k] vlan_dev_hard_start_xmit
0.19% [kernel] [k] swiotlb_map_page
0.18% [kernel] [k] compound_head
0.18% [kernel] [k] neigh_connected_output
0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
0.18% [kernel] [k] ip_output
0.17% [kernel] [k] prefetch_freepointer.isra.70
0.17% [kernel] [k] __slab_free
0.16% [kernel] [k] eth_type_vlan
0.16% [kernel] [k] ip_finish_output
0.15% [kernel] [k] kmem_cache_free_bulk
0.14% [kernel] [k] netif_receive_skb_internal
wondering why this:
1.97% [kernel] [k] virt_to_head_page
is in top...
>>>>> perf top:
>>>>>
>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
>>>>> ---------------------------------------------
>>>>>
>>>>> 16.32% [kernel] [k] skb_dst_force
>>>>> 16.30% [kernel] [k] dst_release
>>>>> 15.11% [kernel] [k] rt_cache_valid
>>>>> 12.62% [kernel] [k] ipv4_mtu
>>>> It seems a little strange that these 4 functions are on the top
> I don't see these in my test.
>
>>>>
>>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>>> Why is calling/taking this lock? (Use perf call-graph recording).
>>> can be hard to paste it here:)
>>> attached file
> The attached was very big. Please don't attach so big file on mailing
> lists. Next time plase share them via e.g. pastebin. The output was a
> capture from your terminal, which made the output more difficult to
> read. Hint: You can/could use perf --stdio and place it in a file
> instead.
>
> The output (extracted below) didn't show who called 'do_raw_spin_lock',
> BUT it showed another interesting thing. The kernel code
> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> as it will first call skb_dst_force() and then skb_dst_drop() when the
> packet is transmitted on a VLAN.
>
> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> {
> [...]
> /* If device/qdisc don't need skb->dst, release it right now while
> * its hot in this cpu cache.
> */
> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> skb_dst_drop(skb);
> else
> skb_dst_force(skb);
>
>
> - -
> Best regards,
> Jesper Dangaard Brouer
> MSc.CS, Principal Kernel Engineer at Red Hat
> LinkedIn: http://www.linkedin.com/in/brouer
>
> Extracted part of attached perf output:
>
> --5.37%--ip_rcv_finish
> |
> |--4.02%--ip_forward
> | |
> | --3.92%--ip_forward_finish
> | |
> | --3.91%--ip_output
> | |
> | --3.90%--ip_finish_output
> | |
> | --3.88%--ip_finish_output2
> | |
> | --2.77%--neigh_connected_output
> | |
> | --2.74%--dev_queue_xmit
> | |
> | --2.73%--__dev_queue_xmit
> | |
> | |--1.66%--dev_hard_start_xmit
> | | |
> | | --1.64%--vlan_dev_hard_start_xmit
> | | |
> | | --1.63%--dev_queue_xmit
> | | |
> | | --1.62%--__dev_queue_xmit
> | | |
> | | |--0.99%--skb_dst_drop.isra.77
> | | | |
> | | | --0.99%--dst_release
> | | |
> | | --0.55%--sch_direct_xmit
> | |
> | --0.99%--skb_dst_force
> |
> --1.29%--ip_route_input_noref
> |
> --1.29%--ip_route_input_rcu
> |
> --1.05%--rt_cache_valid
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-14 16:57 ` Paolo Abeni
@ 2017-08-15 0:45 ` Paweł Staszewski
2017-08-15 1:07 ` Eric Dumazet
2017-08-15 9:35 ` Jesper Dangaard Brouer
1 sibling, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-15 0:45 UTC (permalink / raw)
To: Paolo Abeni, Jesper Dangaard Brouer
Cc: Linux Kernel Network Developers, Alexander Duyck, Eric Dumazet
W dniu 2017-08-14 o 18:57, Paolo Abeni pisze:
> On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
>> The output (extracted below) didn't show who called 'do_raw_spin_lock',
>> BUT it showed another interesting thing. The kernel code
>> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
>> as it will first call skb_dst_force() and then skb_dst_drop() when the
>> packet is transmitted on a VLAN.
>>
>> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
>> {
>> [...]
>> /* If device/qdisc don't need skb->dst, release it right now while
>> * its hot in this cpu cache.
>> */
>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
>> skb_dst_drop(skb);
>> else
>> skb_dst_force(skb);
> I think that the high impact of the above code in this specific test is
> mostly due to the following:
>
> - ingress packets with different RSS rx hash lands on different CPUs
yes but isn't this normal ?
everybody that want to ballance load over cores will try tu use as many
as possible :)
With some limitations ... best are 6 to 7 RSS queues - so need to use 6
to 7 cpu cores
> - but they use the same dst entry, since the destination IPs belong to
> the same subnet
typical for ddos - many sources one destination
> - the dst refcnt cacheline is contented between all the CPUs
>
> Perhaps we can inprove the situation setting the IFF_XMIT_DST_RELEASE
> flag for vlan if the underlaying device does not have (relevant)
> classifier attached? (and clearing it as needed)
>
> Paolo
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 0:45 ` Paweł Staszewski
@ 2017-08-15 1:07 ` Eric Dumazet
2017-08-15 1:17 ` Eric Dumazet
0 siblings, 1 reply; 37+ messages in thread
From: Eric Dumazet @ 2017-08-15 1:07 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On Tue, 2017-08-15 at 02:45 +0200, Paweł Staszewski wrote:
>
> W dniu 2017-08-14 o 18:57, Paolo Abeni pisze:
> > On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
> >> The output (extracted below) didn't show who called 'do_raw_spin_lock',
> >> BUT it showed another interesting thing. The kernel code
> >> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> >> as it will first call skb_dst_force() and then skb_dst_drop() when the
> >> packet is transmitted on a VLAN.
> >>
> >> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> >> {
> >> [...]
> >> /* If device/qdisc don't need skb->dst, release it right now while
> >> * its hot in this cpu cache.
> >> */
> >> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> >> skb_dst_drop(skb);
> >> else
> >> skb_dst_force(skb);
> > I think that the high impact of the above code in this specific test is
> > mostly due to the following:
> >
> > - ingress packets with different RSS rx hash lands on different CPUs
> yes but isn't this normal ?
> everybody that want to ballance load over cores will try tu use as many
> as possible :)
> With some limitations ... best are 6 to 7 RSS queues - so need to use 6
> to 7 cpu cores
>
> > - but they use the same dst entry, since the destination IPs belong to
> > the same subnet
> typical for ddos - many sources one destination
Nobody hit this issue yet.
We usually change the kernel, given typical workloads.
In this case, we might need per cpu nh_rth_input
Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 1:07 ` Eric Dumazet
@ 2017-08-15 1:17 ` Eric Dumazet
2017-08-15 9:11 ` Paweł Staszewski
2017-09-09 9:03 ` Paweł Staszewski
0 siblings, 2 replies; 37+ messages in thread
From: Eric Dumazet @ 2017-08-15 1:17 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
Something like :
diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
index 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7 100644
--- a/net/8021q/vlan_netlink.c
+++ b/net/8021q/vlan_netlink.c
@@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net, struct net_device *dev,
vlan->vlan_proto = proto;
vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
vlan->real_dev = real_dev;
+ dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
vlan->flags = VLAN_FLAG_REORDER_HDR;
err = vlan_check_real_dev(real_dev, vlan->vlan_proto, vlan->vlan_id);
^ permalink raw reply related [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 1:17 ` Eric Dumazet
@ 2017-08-15 9:11 ` Paweł Staszewski
2017-08-15 9:19 ` Paweł Staszewski
` (2 more replies)
2017-09-09 9:03 ` Paweł Staszewski
1 sibling, 3 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-15 9:11 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
Hi
Yes it helped - now there is almost no difference when using vlans or not:
10.5Mpps - with vlan
11Mpps - without vlan
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
> On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
>
>> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
> Something like :
>
> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
> index 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7 100644
> --- a/net/8021q/vlan_netlink.c
> +++ b/net/8021q/vlan_netlink.c
> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net, struct net_device *dev,
> vlan->vlan_proto = proto;
> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
> vlan->real_dev = real_dev;
> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
> vlan->flags = VLAN_FLAG_REORDER_HDR;
>
> err = vlan_check_real_dev(real_dev, vlan->vlan_proto, vlan->vlan_id);
>
>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 9:11 ` Paweł Staszewski
@ 2017-08-15 9:19 ` Paweł Staszewski
2017-08-15 10:05 ` Jesper Dangaard Brouer
2017-09-21 21:26 ` Paweł Staszewski
2 siblings, 0 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-15 9:19 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
With hack:
14.44% [kernel] [k] do_raw_spin_lock
8.30% [kernel] [k] page_frag_free
7.06% [mlx5_core] [k] mlx5e_xmit
5.97% [kernel] [k] acpi_processor_ffh_cstate_enter
5.73% [kernel] [k] fib_table_lookup
4.81% [mlx5_core] [k] mlx5e_poll_tx_cq
4.51% [mlx5_core] [k] skb_from_cqe.isra.32
3.81% [kernel] [k] virt_to_head_page
2.45% [kernel] [k] __dev_queue_xmit
1.84% [kernel] [k] ipt_do_table
1.77% [kernel] [k] napi_consume_skb
1.62% [kernel] [k] __build_skb
1.46% [kernel] [k] netif_skb_features
1.43% [kernel] [k] __netif_receive_skb_core
1.41% [kernel] [k] ip_rcv
1.08% [kernel] [k] dev_hard_start_xmit
1.02% [kernel] [k] build_skb
1.00% [mlx5_core] [k] mlx5_cqwq_get_cqe
0.96% [kernel] [k] ip_route_input_rcu
0.95% [kernel] [k] ip_forward
0.89% [kernel] [k] ip_finish_output2
0.89% [kernel] [k] kmem_cache_alloc
0.78% [kernel] [k] __local_bh_enable_ip
0.76% [kernel] [k] udp_v4_early_demux
0.75% [kernel] [k] compound_head
0.75% [kernel] [k] __netdev_pick_tx
0.73% [kernel] [k] sch_direct_xmit
0.65% [kernel] [k] irq_entries_start
0.63% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
0.61% [kernel] [k] netdev_pick_tx
0.61% [kernel] [k] validate_xmit_skb
0.55% [kernel] [k] skb_network_protocol
0.53% [mlx5_core] [k] mlx5e_rx_cache_get
0.53% [mlx5_core] [k] mlx5e_build_rx_skb
0.51% [kernel] [k] ip_rcv_finish
0.50% [kernel] [k] eth_header
0.50% [kernel] [k] fib_validate_source
0.50% [mlx5_core] [k] mlx5e_handle_rx_cqe
0.48% [mlx5_core] [k] eq_update_ci
0.47% [kernel] [k] kmem_cache_free_bulk
0.44% [kernel] [k] deliver_ptype_list_skb
0.43% [kernel] [k] skb_release_data
0.42% [kernel] [k] cpuidle_enter_state
0.40% [kernel] [k] virt_to_head_page
0.39% [kernel] [k] vlan_dev_hard_start_xmit
0.39% [kernel] [k] neigh_connected_output
0.38% [kernel] [k] eth_type_vlan
0.35% [mlx5_core] [k] mlx5e_alloc_rx_wqe
0.32% [kernel] [k] nf_hook_slow
0.32% [kernel] [k] swiotlb_map_page
0.31% [kernel] [k] ip_finish_output
0.29% [kernel] [k] ip_output
0.28% [kernel] [k] skb_free_head
0.25% [kernel] [k] netif_receive_skb_internal
0.25% [kernel] [k] __jhash_nwords
Without hack:
14.25% [kernel] [k] dst_release
14.17% [kernel] [k] skb_dst_force
13.41% [kernel] [k] rt_cache_valid
11.47% [kernel] [k] ip_finish_output2
7.01% [kernel] [k] do_raw_spin_lock
5.07% [kernel] [k] page_frag_free
3.47% [mlx5_core] [k] mlx5e_xmit
2.88% [kernel] [k] fib_table_lookup
2.43% [mlx5_core] [k] skb_from_cqe.isra.32
1.97% [kernel] [k] virt_to_head_page
1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
0.93% [kernel] [k] __dev_queue_xmit
0.87% [kernel] [k] __build_skb
0.84% [kernel] [k] ipt_do_table
0.79% [kernel] [k] ip_rcv
0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
0.78% [kernel] [k] netif_skb_features
0.73% [kernel] [k] __netif_receive_skb_core
0.52% [kernel] [k] dev_hard_start_xmit
0.52% [kernel] [k] build_skb
0.51% [kernel] [k] ip_route_input_rcu
0.50% [kernel] [k] skb_unref
0.49% [kernel] [k] ip_forward
0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
0.44% [kernel] [k] udp_v4_early_demux
0.41% [kernel] [k] napi_consume_skb
0.40% [kernel] [k] __local_bh_enable_ip
0.39% [kernel] [k] ip_rcv_finish
0.39% [kernel] [k] kmem_cache_alloc
0.38% [kernel] [k] sch_direct_xmit
0.33% [kernel] [k] validate_xmit_skb
0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
0.29% [kernel] [k] netdev_pick_tx
0.28% [mlx5_core] [k] mlx5e_build_rx_skb
0.27% [kernel] [k] deliver_ptype_list_skb
0.26% [kernel] [k] fib_validate_source
0.26% [mlx5_core] [k] mlx5e_napi_poll
0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
0.26% [mlx5_core] [k] mlx5e_rx_cache_get
0.25% [kernel] [k] eth_header
0.23% [kernel] [k] skb_network_protocol
0.20% [kernel] [k] nf_hook_slow
0.20% [kernel] [k] vlan_passthru_hard_header
0.20% [kernel] [k] vlan_dev_hard_start_xmit
0.19% [kernel] [k] swiotlb_map_page
0.18% [kernel] [k] compound_head
0.18% [kernel] [k] neigh_connected_output
0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
0.18% [kernel] [k] ip_output
0.17% [kernel] [k] prefetch_freepointer.isra.70
0.17% [kernel] [k] __slab_free
0.16% [kernel] [k] eth_type_vlan
0.16% [kernel] [k] ip_finish_output
0.15% [kernel] [k] kmem_cache_free_bulk
0.14% [kernel] [k] netif_receive_skb_internal
W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
> Hi
>
>
> Yes it helped - now there is almost no difference when using vlans or
> not:
>
> 10.5Mpps - with vlan
>
> 11Mpps - without vlan
>
>
>
>
> W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
>> On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
>>
>>> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
>> Something like :
>>
>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>> index
>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>> 100644
>> --- a/net/8021q/vlan_netlink.c
>> +++ b/net/8021q/vlan_netlink.c
>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>> struct net_device *dev,
>> vlan->vlan_proto = proto;
>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>> vlan->real_dev = real_dev;
>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>> vlan->vlan_id);
>>
>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 0:38 ` Paweł Staszewski
@ 2017-08-15 9:23 ` Jesper Dangaard Brouer
2017-08-15 9:30 ` Paweł Staszewski
0 siblings, 1 reply; 37+ messages in thread
From: Jesper Dangaard Brouer @ 2017-08-15 9:23 UTC (permalink / raw)
To: Paweł Staszewski
Cc: brouer, Linux Kernel Network Developers, Alexander Duyck,
Saeed Mahameed, Tariq Toukan
On Tue, 15 Aug 2017 02:38:56 +0200
Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
> > On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >
> >> To show some difference below comparision vlan/no-vlan traffic
> >>
> >> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
> > I'm trying to reproduce in my testlab (with ixgbe). I do see, a
> > performance reduction of about 10-19% when I forward out a VLAN
> > interface. This is larger than I expected, but still lower than what
> > you reported 30-40% slowdown.
> >
> > [...]
> Ok mellanox afrrived (MT27700 - mlnx5 driver)
> And to compare melannox with vlans and without: 33% performance
> degradation (less than with ixgbe where i reach ~40% with same settings)
>
> Mellanox without TX traffix on vlan:
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;16;64;11089305;709715520;8871553;567779392
> 1;16;64;11096292;710162688;11095566;710116224
> 2;16;64;11095770;710129280;11096799;710195136
> 3;16;64;11097199;710220736;11097702;710252928
> 4;16;64;11080984;567081856;11079662;709098368
> 5;16;64;11077696;708972544;11077039;708930496
> 6;16;64;11082991;709311424;8864802;567347328
> 7;16;64;11089596;709734144;8870927;709789184
> 8;16;64;11094043;710018752;11095391;710105024
>
> Mellanox with TX traffic on vlan:
> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> 0;16;64;7369914;471674496;7370281;471697980
> 1;16;64;7368896;471609408;7368043;471554752
> 2;16;64;7367577;471524864;7367759;471536576
> 3;16;64;7368744;377305344;7369391;471641024
> 4;16;64;7366824;471476736;7364330;471237120
> 5;16;64;7368352;471574528;7367239;471503296
> 6;16;64;7367459;471517376;7367806;471539584
> 7;16;64;7367190;471500160;7367988;471551232
> 8;16;64;7368023;471553472;7368076;471556864
I wonder if the drivers page recycler is active/working or not, and if
the situation is different between VLAN vs no-vlan (given
page_frag_free is so high in you perf top). The Mellanox drivers
fortunately have a stats counter to tell us this explicitly (which the
ixgbe driver doesn't).
You can use my ethtool_stats.pl script watch these stats:
https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
(Hint perl dependency: dnf install perl-Time-HiRes)
> ethtool settings for both tests:
> ifc='enp175s0f0 enp175s0f1'
> for i in $ifc
> do
> ip link set up dev $i
> ethtool -A $i autoneg off rx off tx off
> ethtool -G $i rx 128 tx 256
The ring queue size recommendations, might be different for the mlx5
driver (Cc'ing Mellanox maintainers).
> ip link set $i txqueuelen 1000
> ethtool -C $i rx-usecs 25
> ethtool -L $i combined 16
> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
> tx-nocache-copy off ntuple on
> ethtool -N $i rx-flow-hash udp4 sdfn
> done
Thanks for being explicit about what you setup is :-)
> and perf top:
> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
> cycles], (all, 56 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> 14.25% [kernel] [k] dst_release
> 14.17% [kernel] [k] skb_dst_force
> 13.41% [kernel] [k] rt_cache_valid
> 11.47% [kernel] [k] ip_finish_output2
> 7.01% [kernel] [k] do_raw_spin_lock
> 5.07% [kernel] [k] page_frag_free
> 3.47% [mlx5_core] [k] mlx5e_xmit
> 2.88% [kernel] [k] fib_table_lookup
> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
> 1.97% [kernel] [k] virt_to_head_page
> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
> 0.93% [kernel] [k] __dev_queue_xmit
> 0.87% [kernel] [k] __build_skb
> 0.84% [kernel] [k] ipt_do_table
> 0.79% [kernel] [k] ip_rcv
> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
> 0.78% [kernel] [k] netif_skb_features
> 0.73% [kernel] [k] __netif_receive_skb_core
> 0.52% [kernel] [k] dev_hard_start_xmit
> 0.52% [kernel] [k] build_skb
> 0.51% [kernel] [k] ip_route_input_rcu
> 0.50% [kernel] [k] skb_unref
> 0.49% [kernel] [k] ip_forward
> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
> 0.44% [kernel] [k] udp_v4_early_demux
> 0.41% [kernel] [k] napi_consume_skb
> 0.40% [kernel] [k] __local_bh_enable_ip
> 0.39% [kernel] [k] ip_rcv_finish
> 0.39% [kernel] [k] kmem_cache_alloc
> 0.38% [kernel] [k] sch_direct_xmit
> 0.33% [kernel] [k] validate_xmit_skb
> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
> 0.29% [kernel] [k] netdev_pick_tx
> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
> 0.27% [kernel] [k] deliver_ptype_list_skb
> 0.26% [kernel] [k] fib_validate_source
> 0.26% [mlx5_core] [k] mlx5e_napi_poll
> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
> 0.25% [kernel] [k] eth_header
> 0.23% [kernel] [k] skb_network_protocol
> 0.20% [kernel] [k] nf_hook_slow
> 0.20% [kernel] [k] vlan_passthru_hard_header
> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
> 0.19% [kernel] [k] swiotlb_map_page
> 0.18% [kernel] [k] compound_head
> 0.18% [kernel] [k] neigh_connected_output
> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
> 0.18% [kernel] [k] ip_output
> 0.17% [kernel] [k] prefetch_freepointer.isra.70
> 0.17% [kernel] [k] __slab_free
> 0.16% [kernel] [k] eth_type_vlan
> 0.16% [kernel] [k] ip_finish_output
> 0.15% [kernel] [k] kmem_cache_free_bulk
> 0.14% [kernel] [k] netif_receive_skb_internal
>
>
>
>
> wondering why this:
> 1.97% [kernel] [k] virt_to_head_page
> is in top...
This is related to the page_frag_free() call, but it is weird that it
shows up because it is suppose to be inlined (it is explicitly marked
inline in include/linux/mm.h).
> >>>>> perf top:
> >>>>>
> >>>>> PerfTop: 77835 irqs/sec kernel:99.7%
> >>>>> ---------------------------------------------
> >>>>>
> >>>>> 16.32% [kernel] [k] skb_dst_force
> >>>>> 16.30% [kernel] [k] dst_release
> >>>>> 15.11% [kernel] [k] rt_cache_valid
> >>>>> 12.62% [kernel] [k] ipv4_mtu
> >>>> It seems a little strange that these 4 functions are on the top
> > I don't see these in my test.
> >
> >>>>
> >>>>> 5.60% [kernel] [k] do_raw_spin_lock
> >>>> Why is calling/taking this lock? (Use perf call-graph recording).
> >>> can be hard to paste it here:)
> >>> attached file
> > The attached was very big. Please don't attach so big file on mailing
> > lists. Next time plase share them via e.g. pastebin. The output was a
> > capture from your terminal, which made the output more difficult to
> > read. Hint: You can/could use perf --stdio and place it in a file
> > instead.
> >
> > The output (extracted below) didn't show who called 'do_raw_spin_lock',
> > BUT it showed another interesting thing. The kernel code
> > __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> > as it will first call skb_dst_force() and then skb_dst_drop() when the
> > packet is transmitted on a VLAN.
> >
> > static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> > {
> > [...]
> > /* If device/qdisc don't need skb->dst, release it right now while
> > * its hot in this cpu cache.
> > */
> > if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> > skb_dst_drop(skb);
> > else
> > skb_dst_force(skb);
> >
> >
> >
> > Extracted part of attached perf output:
> >
> > --5.37%--ip_rcv_finish
> > |
> > |--4.02%--ip_forward
> > | |
> > | --3.92%--ip_forward_finish
> > | |
> > | --3.91%--ip_output
> > | |
> > | --3.90%--ip_finish_output
> > | |
> > | --3.88%--ip_finish_output2
> > | |
> > | --2.77%--neigh_connected_output
> > | |
> > | --2.74%--dev_queue_xmit
> > | |
> > | --2.73%--__dev_queue_xmit
> > | |
> > | |--1.66%--dev_hard_start_xmit
> > | | |
> > | | --1.64%--vlan_dev_hard_start_xmit
> > | | |
> > | | --1.63%--dev_queue_xmit
> > | | |
> > | | --1.62%--__dev_queue_xmit
> > | | |
> > | | |--0.99%--skb_dst_drop.isra.77
> > | | | |
> > | | | --0.99%--dst_release
> > | | |
> > | | --0.55%--sch_direct_xmit
> > | |
> > | --0.99%--skb_dst_force
> > |
> > --1.29%--ip_route_input_noref
> > |
> > --1.29%--ip_route_input_rcu
> > |
> > --1.05%--rt_cache_valid
> >
>
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 9:23 ` Jesper Dangaard Brouer
@ 2017-08-15 9:30 ` Paweł Staszewski
2017-08-15 9:57 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-15 9:30 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan
W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
> On Tue, 15 Aug 2017 02:38:56 +0200
> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>
>>>> To show some difference below comparision vlan/no-vlan traffic
>>>>
>>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
>>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
>>> performance reduction of about 10-19% when I forward out a VLAN
>>> interface. This is larger than I expected, but still lower than what
>>> you reported 30-40% slowdown.
>>>
>>> [...]
>> Ok mellanox afrrived (MT27700 - mlnx5 driver)
>> And to compare melannox with vlans and without: 33% performance
>> degradation (less than with ixgbe where i reach ~40% with same settings)
>>
>> Mellanox without TX traffix on vlan:
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;16;64;11089305;709715520;8871553;567779392
>> 1;16;64;11096292;710162688;11095566;710116224
>> 2;16;64;11095770;710129280;11096799;710195136
>> 3;16;64;11097199;710220736;11097702;710252928
>> 4;16;64;11080984;567081856;11079662;709098368
>> 5;16;64;11077696;708972544;11077039;708930496
>> 6;16;64;11082991;709311424;8864802;567347328
>> 7;16;64;11089596;709734144;8870927;709789184
>> 8;16;64;11094043;710018752;11095391;710105024
>>
>> Mellanox with TX traffic on vlan:
>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>> 0;16;64;7369914;471674496;7370281;471697980
>> 1;16;64;7368896;471609408;7368043;471554752
>> 2;16;64;7367577;471524864;7367759;471536576
>> 3;16;64;7368744;377305344;7369391;471641024
>> 4;16;64;7366824;471476736;7364330;471237120
>> 5;16;64;7368352;471574528;7367239;471503296
>> 6;16;64;7367459;471517376;7367806;471539584
>> 7;16;64;7367190;471500160;7367988;471551232
>> 8;16;64;7368023;471553472;7368076;471556864
> I wonder if the drivers page recycler is active/working or not, and if
> the situation is different between VLAN vs no-vlan (given
> page_frag_free is so high in you perf top). The Mellanox drivers
> fortunately have a stats counter to tell us this explicitly (which the
> ixgbe driver doesn't).
>
> You can use my ethtool_stats.pl script watch these stats:
> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> (Hint perl dependency: dnf install perl-Time-HiRes)
For RX NIC:
Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <= rx0_bytes /sec
Ethtool(enp175s0f0) stat: 230978 ( 230,978) <=
rx0_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <=
rx0_csum_complete /sec
Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_packets /sec
Ethtool(enp175s0f0) stat: 921614 ( 921,614) <=
rx0_page_reuse /sec
Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <= rx1_bytes /sec
Ethtool(enp175s0f0) stat: 233343 ( 233,343) <=
rx1_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <=
rx1_csum_complete /sec
Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_packets /sec
Ethtool(enp175s0f0) stat: 927793 ( 927,793) <=
rx1_page_reuse /sec
Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <= rx2_bytes /sec
Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
rx2_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <=
rx2_csum_complete /sec
Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_packets /sec
Ethtool(enp175s0f0) stat: 937989 ( 937,989) <=
rx2_page_reuse /sec
Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <= rx3_bytes /sec
Ethtool(enp175s0f0) stat: 230311 ( 230,311) <=
rx3_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <=
rx3_csum_complete /sec
Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_packets /sec
Ethtool(enp175s0f0) stat: 922513 ( 922,513) <=
rx3_page_reuse /sec
Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <= rx4_bytes /sec
Ethtool(enp175s0f0) stat: 191969 ( 191,969) <=
rx4_cache_reuse /sec
Ethtool(enp175s0f0) stat: 958317 ( 958,317) <=
rx4_csum_complete /sec
Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_packets /sec
Ethtool(enp175s0f0) stat: 766332 ( 766,332) <=
rx4_page_reuse /sec
Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <= rx5_bytes /sec
Ethtool(enp175s0f0) stat: 197150 ( 197,150) <=
rx5_cache_reuse /sec
Ethtool(enp175s0f0) stat: 984128 ( 984,128) <=
rx5_csum_complete /sec
Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_packets /sec
Ethtool(enp175s0f0) stat: 786978 ( 786,978) <=
rx5_page_reuse /sec
Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <= rx6_bytes /sec
Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
rx6_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <=
rx6_csum_complete /sec
Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_packets /sec
Ethtool(enp175s0f0) stat: 929163 ( 929,163) <=
rx6_page_reuse /sec
Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <= rx7_bytes /sec
Ethtool(enp175s0f0) stat: 230413 ( 230,413) <=
rx7_cache_reuse /sec
Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <=
rx7_csum_complete /sec
Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_packets /sec
Ethtool(enp175s0f0) stat: 926376 ( 926,376) <=
rx7_page_reuse /sec
Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes /sec
Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <= rx_bytes_phy
/sec
Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <=
rx_cache_reuse /sec
Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <=
rx_csum_complete /sec
Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <=
rx_out_of_buffer /sec
Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_packets /sec
Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <=
rx_packets_phy /sec
Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <=
rx_page_reuse /sec
Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <=
rx_prio0_bytes /sec
Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <=
rx_prio0_packets /sec
Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <=
rx_vport_unicast_bytes /sec
Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <=
rx_vport_unicast_packets /sec
For TX nic with vlan:
Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_65_to_127_bytes_phy /sec
Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_bytes_phy
/sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_multicast_phy /sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_packets_phy /sec
Ethtool(enp175s0f1) stat: 71 ( 71) <=
rx_prio0_bytes /sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_prio0_packets /sec
Ethtool(enp175s0f1) stat: 67 ( 67) <=
rx_vport_multicast_bytes /sec
Ethtool(enp175s0f1) stat: 1 ( 1) <=
rx_vport_multicast_packets /sec
Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <= tx0_bytes /sec
Ethtool(enp175s0f1) stat: 955222 ( 955,222) <=
tx0_csum_none /sec
Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop /sec
Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_packets /sec
Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <= tx1_bytes /sec
Ethtool(enp175s0f1) stat: 982341 ( 982,341) <=
tx1_csum_none /sec
Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop /sec
Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_packets /sec
Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <= tx2_bytes /sec
Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <=
tx2_csum_none /sec
Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop /sec
Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_packets /sec
Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <= tx3_bytes /sec
Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <=
tx3_csum_none /sec
Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop /sec
Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_packets /sec
Ethtool(enp175s0f1) stat: 234 ( 234) <=
tx3_xmit_more /sec
Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <= tx4_bytes /sec
Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <=
tx4_csum_none /sec
Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop /sec
Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_packets /sec
Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <= tx5_bytes /sec
Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <=
tx5_csum_none /sec
Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop /sec
Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_packets /sec
Ethtool(enp175s0f1) stat: 264 ( 264) <=
tx5_xmit_more /sec
Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <= tx6_bytes /sec
Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <=
tx6_csum_none /sec
Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop /sec
Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_packets /sec
Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <= tx7_bytes /sec
Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <=
tx7_csum_none /sec
Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop /sec
Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_packets /sec
Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes /sec
Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <= tx_bytes_phy
/sec
Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <= tx_packets /sec
Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <=
tx_packets_phy /sec
Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <=
tx_prio0_bytes /sec
Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <=
tx_prio0_packets /sec
Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <=
tx_vport_unicast_bytes /sec
Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <=
tx_vport_unicast_packets /sec
Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more
/sec
>
>> ethtool settings for both tests:
>> ifc='enp175s0f0 enp175s0f1'
>> for i in $ifc
>> do
>> ip link set up dev $i
>> ethtool -A $i autoneg off rx off tx off
>> ethtool -G $i rx 128 tx 256
> The ring queue size recommendations, might be different for the mlx5
> driver (Cc'ing Mellanox maintainers).
>
>
>> ip link set $i txqueuelen 1000
>> ethtool -C $i rx-usecs 25
>> ethtool -L $i combined 16
>> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
>> tx-nocache-copy off ntuple on
>> ethtool -N $i rx-flow-hash udp4 sdfn
>> done
> Thanks for being explicit about what you setup is :-)
>
>> and perf top:
>> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>> cycles], (all, 56 CPUs)
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> 14.25% [kernel] [k] dst_release
>> 14.17% [kernel] [k] skb_dst_force
>> 13.41% [kernel] [k] rt_cache_valid
>> 11.47% [kernel] [k] ip_finish_output2
>> 7.01% [kernel] [k] do_raw_spin_lock
>> 5.07% [kernel] [k] page_frag_free
>> 3.47% [mlx5_core] [k] mlx5e_xmit
>> 2.88% [kernel] [k] fib_table_lookup
>> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
>> 1.97% [kernel] [k] virt_to_head_page
>> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
>> 0.93% [kernel] [k] __dev_queue_xmit
>> 0.87% [kernel] [k] __build_skb
>> 0.84% [kernel] [k] ipt_do_table
>> 0.79% [kernel] [k] ip_rcv
>> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
>> 0.78% [kernel] [k] netif_skb_features
>> 0.73% [kernel] [k] __netif_receive_skb_core
>> 0.52% [kernel] [k] dev_hard_start_xmit
>> 0.52% [kernel] [k] build_skb
>> 0.51% [kernel] [k] ip_route_input_rcu
>> 0.50% [kernel] [k] skb_unref
>> 0.49% [kernel] [k] ip_forward
>> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
>> 0.44% [kernel] [k] udp_v4_early_demux
>> 0.41% [kernel] [k] napi_consume_skb
>> 0.40% [kernel] [k] __local_bh_enable_ip
>> 0.39% [kernel] [k] ip_rcv_finish
>> 0.39% [kernel] [k] kmem_cache_alloc
>> 0.38% [kernel] [k] sch_direct_xmit
>> 0.33% [kernel] [k] validate_xmit_skb
>> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
>> 0.29% [kernel] [k] netdev_pick_tx
>> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
>> 0.27% [kernel] [k] deliver_ptype_list_skb
>> 0.26% [kernel] [k] fib_validate_source
>> 0.26% [mlx5_core] [k] mlx5e_napi_poll
>> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
>> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
>> 0.25% [kernel] [k] eth_header
>> 0.23% [kernel] [k] skb_network_protocol
>> 0.20% [kernel] [k] nf_hook_slow
>> 0.20% [kernel] [k] vlan_passthru_hard_header
>> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
>> 0.19% [kernel] [k] swiotlb_map_page
>> 0.18% [kernel] [k] compound_head
>> 0.18% [kernel] [k] neigh_connected_output
>> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
>> 0.18% [kernel] [k] ip_output
>> 0.17% [kernel] [k] prefetch_freepointer.isra.70
>> 0.17% [kernel] [k] __slab_free
>> 0.16% [kernel] [k] eth_type_vlan
>> 0.16% [kernel] [k] ip_finish_output
>> 0.15% [kernel] [k] kmem_cache_free_bulk
>> 0.14% [kernel] [k] netif_receive_skb_internal
>>
>>
>>
>>
>> wondering why this:
>> 1.97% [kernel] [k] virt_to_head_page
>> is in top...
> This is related to the page_frag_free() call, but it is weird that it
> shows up because it is suppose to be inlined (it is explicitly marked
> inline in include/linux/mm.h).
>
>
>>>>>>> perf top:
>>>>>>>
>>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
>>>>>>> ---------------------------------------------
>>>>>>>
>>>>>>> 16.32% [kernel] [k] skb_dst_force
>>>>>>> 16.30% [kernel] [k] dst_release
>>>>>>> 15.11% [kernel] [k] rt_cache_valid
>>>>>>> 12.62% [kernel] [k] ipv4_mtu
>>>>>> It seems a little strange that these 4 functions are on the top
>>> I don't see these in my test.
>>>
>>>>>>
>>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
>>>>> can be hard to paste it here:)
>>>>> attached file
>>> The attached was very big. Please don't attach so big file on mailing
>>> lists. Next time plase share them via e.g. pastebin. The output was a
>>> capture from your terminal, which made the output more difficult to
>>> read. Hint: You can/could use perf --stdio and place it in a file
>>> instead.
>>>
>>> The output (extracted below) didn't show who called 'do_raw_spin_lock',
>>> BUT it showed another interesting thing. The kernel code
>>> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
>>> as it will first call skb_dst_force() and then skb_dst_drop() when the
>>> packet is transmitted on a VLAN.
>>>
>>> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
>>> {
>>> [...]
>>> /* If device/qdisc don't need skb->dst, release it right now while
>>> * its hot in this cpu cache.
>>> */
>>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
>>> skb_dst_drop(skb);
>>> else
>>> skb_dst_force(skb);
>>>
>>>
>>>
>>> Extracted part of attached perf output:
>>>
>>> --5.37%--ip_rcv_finish
>>> |
>>> |--4.02%--ip_forward
>>> | |
>>> | --3.92%--ip_forward_finish
>>> | |
>>> | --3.91%--ip_output
>>> | |
>>> | --3.90%--ip_finish_output
>>> | |
>>> | --3.88%--ip_finish_output2
>>> | |
>>> | --2.77%--neigh_connected_output
>>> | |
>>> | --2.74%--dev_queue_xmit
>>> | |
>>> | --2.73%--__dev_queue_xmit
>>> | |
>>> | |--1.66%--dev_hard_start_xmit
>>> | | |
>>> | | --1.64%--vlan_dev_hard_start_xmit
>>> | | |
>>> | | --1.63%--dev_queue_xmit
>>> | | |
>>> | | --1.62%--__dev_queue_xmit
>>> | | |
>>> | | |--0.99%--skb_dst_drop.isra.77
>>> | | | |
>>> | | | --0.99%--dst_release
>>> | | |
>>> | | --0.55%--sch_direct_xmit
>>> | |
>>> | --0.99%--skb_dst_force
>>> |
>>> --1.29%--ip_route_input_noref
>>> |
>>> --1.29%--ip_route_input_rcu
>>> |
>>> --1.05%--rt_cache_valid
>>>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-14 16:57 ` Paolo Abeni
2017-08-15 0:45 ` Paweł Staszewski
@ 2017-08-15 9:35 ` Jesper Dangaard Brouer
1 sibling, 0 replies; 37+ messages in thread
From: Jesper Dangaard Brouer @ 2017-08-15 9:35 UTC (permalink / raw)
To: Paolo Abeni
Cc: Paweł Staszewski, Linux Kernel Network Developers,
Alexander Duyck, Eric Dumazet, brouer
On Mon, 14 Aug 2017 18:57:50 +0200
Paolo Abeni <pabeni@redhat.com> wrote:
> On Mon, 2017-08-14 at 18:19 +0200, Jesper Dangaard Brouer wrote:
> > The output (extracted below) didn't show who called 'do_raw_spin_lock',
> > BUT it showed another interesting thing. The kernel code
> > __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> > as it will first call skb_dst_force() and then skb_dst_drop() when the
> > packet is transmitted on a VLAN.
> >
> > static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> > {
> > [...]
> > /* If device/qdisc don't need skb->dst, release it right now while
> > * its hot in this cpu cache.
> > */
> > if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> > skb_dst_drop(skb);
> > else
> > skb_dst_force(skb);
>
> I think that the high impact of the above code in this specific test is
> mostly due to the following:
>
> - ingress packets with different RSS rx hash lands on different CPUs
> - but they use the same dst entry, since the destination IPs belong to
> the same subnet
> - the dst refcnt cacheline is contented between all the CPUs
Good point and explanation Paolo :-)
I changed my pktgen setup to be closer to Pawel's to provoke this
situation some more, and I get closer to provoke this although not as
clearly as Pawel.
A perf diff does show, that the overhead in the VLAN cause originates
from the routing "dst_release" code. Diff Baseline==non-vlan case.
[jbrouer@canyon ~]$ sudo ~/perf diff
# Event 'cycles'
#
# Baseline Delta Abs Shared Object Symbol
# ........ ......... ................ .........................................
#
3.23% +4.32% [kernel.vmlinux] [k] __dev_queue_xmit
+3.43% [kernel.vmlinux] [k] dst_release
13.54% -3.17% [kernel.vmlinux] [k] fib_table_lookup
9.33% -2.73% [kernel.vmlinux] [k] _raw_spin_lock
7.91% -1.75% [ixgbe] [k] ixgbe_poll
+1.64% [8021q] [k] vlan_dev_hard_start_xmit
7.23% -1.26% [ixgbe] [k] ixgbe_xmit_frame_ring
3.34% -1.10% [kernel.vmlinux] [k] eth_type_trans
5.20% +0.97% [kernel.vmlinux] [k] ip_route_input_rcu
1.13% +0.95% [kernel.vmlinux] [k] ip_rcv_finish
2.49% -0.82% [kernel.vmlinux] [k] ip_forward
3.05% -0.80% [kernel.vmlinux] [k] __build_skb
0.44% +0.74% [kernel.vmlinux] [k] __netif_receive_skb
+0.71% [kernel.vmlinux] [k] neigh_connected_output
1.70% +0.68% [kernel.vmlinux] [k] validate_xmit_skb
1.42% +0.67% [kernel.vmlinux] [k] dev_hard_start_xmit
0.49% +0.66% [kernel.vmlinux] [k] netif_receive_skb_internal
+0.62% [kernel.vmlinux] [k] eth_header
+0.57% [ixgbe] [k] ixgbe_tx_ctxtdesc
1.19% -0.55% [kernel.vmlinux] [k] __netdev_pick_tx
2.54% -0.48% [kernel.vmlinux] [k] fib_validate_source
2.83% +0.46% [kernel.vmlinux] [k] ip_finish_output2
1.45% +0.45% [kernel.vmlinux] [k] netif_skb_features
1.66% -0.45% [kernel.vmlinux] [k] napi_gro_receive
0.90% -0.40% [kernel.vmlinux] [k] validate_xmit_skb_list
1.45% -0.39% [kernel.vmlinux] [k] ip_finish_output
+0.36% [8021q] [k] vlan_passthru_hard_header
1.28% -0.33% [kernel.vmlinux] [k] netdev_pick_tx
> Perhaps we can inprove the situation setting the IFF_XMIT_DST_RELEASE
> flag for vlan if the underlaying device does not have (relevant)
> classifier attached? (and clearing it as needed)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 9:30 ` Paweł Staszewski
@ 2017-08-15 9:57 ` Jesper Dangaard Brouer
2017-08-15 10:02 ` Paweł Staszewski
0 siblings, 1 reply; 37+ messages in thread
From: Jesper Dangaard Brouer @ 2017-08-15 9:57 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan, brouer
On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
> > On Tue, 15 Aug 2017 02:38:56 +0200
> > Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >
> >> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
> >>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >>>
> >>>> To show some difference below comparision vlan/no-vlan traffic
> >>>>
> >>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
> >>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
> >>> performance reduction of about 10-19% when I forward out a VLAN
> >>> interface. This is larger than I expected, but still lower than what
> >>> you reported 30-40% slowdown.
> >>>
> >>> [...]
> >> Ok mellanox afrrived (MT27700 - mlnx5 driver)
> >> And to compare melannox with vlans and without: 33% performance
> >> degradation (less than with ixgbe where i reach ~40% with same settings)
> >>
> >> Mellanox without TX traffix on vlan:
> >> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> >> 0;16;64;11089305;709715520;8871553;567779392
> >> 1;16;64;11096292;710162688;11095566;710116224
> >> 2;16;64;11095770;710129280;11096799;710195136
> >> 3;16;64;11097199;710220736;11097702;710252928
> >> 4;16;64;11080984;567081856;11079662;709098368
> >> 5;16;64;11077696;708972544;11077039;708930496
> >> 6;16;64;11082991;709311424;8864802;567347328
> >> 7;16;64;11089596;709734144;8870927;709789184
> >> 8;16;64;11094043;710018752;11095391;710105024
> >>
> >> Mellanox with TX traffic on vlan:
> >> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
> >> 0;16;64;7369914;471674496;7370281;471697980
> >> 1;16;64;7368896;471609408;7368043;471554752
> >> 2;16;64;7367577;471524864;7367759;471536576
> >> 3;16;64;7368744;377305344;7369391;471641024
> >> 4;16;64;7366824;471476736;7364330;471237120
> >> 5;16;64;7368352;471574528;7367239;471503296
> >> 6;16;64;7367459;471517376;7367806;471539584
> >> 7;16;64;7367190;471500160;7367988;471551232
> >> 8;16;64;7368023;471553472;7368076;471556864
> > I wonder if the drivers page recycler is active/working or not, and if
> > the situation is different between VLAN vs no-vlan (given
> > page_frag_free is so high in you perf top). The Mellanox drivers
> > fortunately have a stats counter to tell us this explicitly (which the
> > ixgbe driver doesn't).
> >
> > You can use my ethtool_stats.pl script watch these stats:
> > https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
> > (Hint perl dependency: dnf install perl-Time-HiRes)
> For RX NIC:
> Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
> Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <= rx0_bytes /sec
> Ethtool(enp175s0f0) stat: 230978 ( 230,978) <= rx0_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_packets /sec
> Ethtool(enp175s0f0) stat: 921614 ( 921,614) <= rx0_page_reuse /sec
> Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <= rx1_bytes /sec
> Ethtool(enp175s0f0) stat: 233343 ( 233,343) <= rx1_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_packets /sec
> Ethtool(enp175s0f0) stat: 927793 ( 927,793) <= rx1_page_reuse /sec
> Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <= rx2_bytes /sec
> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx2_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_packets /sec
> Ethtool(enp175s0f0) stat: 937989 ( 937,989) <= rx2_page_reuse /sec
> Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <= rx3_bytes /sec
> Ethtool(enp175s0f0) stat: 230311 ( 230,311) <= rx3_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_packets /sec
> Ethtool(enp175s0f0) stat: 922513 ( 922,513) <= rx3_page_reuse /sec
> Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <= rx4_bytes /sec
> Ethtool(enp175s0f0) stat: 191969 ( 191,969) <= rx4_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_csum_complete /sec
> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_packets /sec
> Ethtool(enp175s0f0) stat: 766332 ( 766,332) <= rx4_page_reuse /sec
> Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <= rx5_bytes /sec
> Ethtool(enp175s0f0) stat: 197150 ( 197,150) <= rx5_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_csum_complete /sec
> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_packets /sec
> Ethtool(enp175s0f0) stat: 786978 ( 786,978) <= rx5_page_reuse /sec
> Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <= rx6_bytes /sec
> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx6_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_packets /sec
> Ethtool(enp175s0f0) stat: 929163 ( 929,163) <= rx6_page_reuse /sec
> Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <= rx7_bytes /sec
> Ethtool(enp175s0f0) stat: 230413 ( 230,413) <= rx7_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_packets /sec
> Ethtool(enp175s0f0) stat: 926376 ( 926,376) <= rx7_page_reuse /sec
> Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <= rx_65_to_127_bytes_phy /sec
> Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes /sec
> Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <= rx_bytes_phy /sec
> Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <= rx_cache_reuse /sec
> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_csum_complete /sec
> Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <= rx_out_of_buffer /sec
> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_packets /sec
> Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <= rx_packets_phy /sec
> Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <= rx_page_reuse /sec
> Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <= rx_prio0_bytes /sec
> Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <= rx_prio0_packets /sec
> Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <= rx_vport_unicast_bytes /sec
> Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <= rx_vport_unicast_packets /sec
It looks like the mlx5 page recycle mechanism works:
230413 ( 230,413) <= rx7_cache_reuse /sec
+ 926376 ( 926,376) <= rx7_page_reuse /sec
=1156789 (230413+926376)
-1156775 ( 1,156,775) <= rx7_packets /sec
= 14
You can also determine this as there are no counters for:
rx_cache_full or
rx_cache_empty or
rx1_cache_empty
rx1_cache_busy
> For TX nic with vlan:
> Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_65_to_127_bytes_phy /sec
> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_bytes_phy /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_multicast_phy /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_packets_phy /sec
> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_prio0_bytes /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_prio0_packets /sec
> Ethtool(enp175s0f1) stat: 67 ( 67) <= rx_vport_multicast_bytes /sec
> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_vport_multicast_packets /sec
> Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <= tx0_bytes /sec
> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_csum_none /sec
> Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop /sec
> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_packets /sec
> Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <= tx1_bytes /sec
> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_csum_none /sec
> Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop /sec
> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_packets /sec
> Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <= tx2_bytes /sec
> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_csum_none /sec
> Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop /sec
> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_packets /sec
> Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <= tx3_bytes /sec
> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_csum_none /sec
> Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop /sec
> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_packets /sec
> Ethtool(enp175s0f1) stat: 234 ( 234) <= tx3_xmit_more /sec
> Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <= tx4_bytes /sec
> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_csum_none /sec
> Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop /sec
> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_packets /sec
> Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <= tx5_bytes /sec
> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_csum_none /sec
> Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop /sec
> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_packets /sec
> Ethtool(enp175s0f1) stat: 264 ( 264) <= tx5_xmit_more /sec
> Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <= tx6_bytes /sec
> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_csum_none /sec
> Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop /sec
> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_packets /sec
> Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <= tx7_bytes /sec
> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_csum_none /sec
> Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop /sec
> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_packets /sec
> Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes /sec
> Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <= tx_bytes_phy /sec
> Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <= tx_packets /sec
> Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <= tx_packets_phy /sec
> Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <= tx_prio0_bytes /sec
> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <= tx_prio0_packets /sec
> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <= tx_vport_unicast_bytes /sec
> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <= tx_vport_unicast_packets /sec
> Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more /sec
We are seeing some xmit_more, this is interesting. Have you noticed,
if (in the VLAN case) there is a queue in the qdisc layer?
Simply inspect with: tc -s qdisc show dev ixgbe2
> >
> >> ethtool settings for both tests:
> >> ifc='enp175s0f0 enp175s0f1'
> >> for i in $ifc
> >> do
> >> ip link set up dev $i
> >> ethtool -A $i autoneg off rx off tx off
> >> ethtool -G $i rx 128 tx 256
> > The ring queue size recommendations, might be different for the mlx5
> > driver (Cc'ing Mellanox maintainers).
> >
> >
> >> ip link set $i txqueuelen 1000
> >> ethtool -C $i rx-usecs 25
> >> ethtool -L $i combined 16
> >> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
> >> tx-nocache-copy off ntuple on
> >> ethtool -N $i rx-flow-hash udp4 sdfn
> >> done
> > Thanks for being explicit about what you setup is :-)
> >
> >> and perf top:
> >> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
> >> cycles], (all, 56 CPUs)
> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >>
> >> 14.25% [kernel] [k] dst_release
> >> 14.17% [kernel] [k] skb_dst_force
> >> 13.41% [kernel] [k] rt_cache_valid
> >> 11.47% [kernel] [k] ip_finish_output2
> >> 7.01% [kernel] [k] do_raw_spin_lock
> >> 5.07% [kernel] [k] page_frag_free
> >> 3.47% [mlx5_core] [k] mlx5e_xmit
> >> 2.88% [kernel] [k] fib_table_lookup
> >> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
> >> 1.97% [kernel] [k] virt_to_head_page
> >> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
> >> 0.93% [kernel] [k] __dev_queue_xmit
> >> 0.87% [kernel] [k] __build_skb
> >> 0.84% [kernel] [k] ipt_do_table
> >> 0.79% [kernel] [k] ip_rcv
> >> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
> >> 0.78% [kernel] [k] netif_skb_features
> >> 0.73% [kernel] [k] __netif_receive_skb_core
> >> 0.52% [kernel] [k] dev_hard_start_xmit
> >> 0.52% [kernel] [k] build_skb
> >> 0.51% [kernel] [k] ip_route_input_rcu
> >> 0.50% [kernel] [k] skb_unref
> >> 0.49% [kernel] [k] ip_forward
> >> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
> >> 0.44% [kernel] [k] udp_v4_early_demux
> >> 0.41% [kernel] [k] napi_consume_skb
> >> 0.40% [kernel] [k] __local_bh_enable_ip
> >> 0.39% [kernel] [k] ip_rcv_finish
> >> 0.39% [kernel] [k] kmem_cache_alloc
> >> 0.38% [kernel] [k] sch_direct_xmit
> >> 0.33% [kernel] [k] validate_xmit_skb
> >> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
> >> 0.29% [kernel] [k] netdev_pick_tx
> >> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
> >> 0.27% [kernel] [k] deliver_ptype_list_skb
> >> 0.26% [kernel] [k] fib_validate_source
> >> 0.26% [mlx5_core] [k] mlx5e_napi_poll
> >> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
> >> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
> >> 0.25% [kernel] [k] eth_header
> >> 0.23% [kernel] [k] skb_network_protocol
> >> 0.20% [kernel] [k] nf_hook_slow
> >> 0.20% [kernel] [k] vlan_passthru_hard_header
> >> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
> >> 0.19% [kernel] [k] swiotlb_map_page
> >> 0.18% [kernel] [k] compound_head
> >> 0.18% [kernel] [k] neigh_connected_output
> >> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
> >> 0.18% [kernel] [k] ip_output
> >> 0.17% [kernel] [k] prefetch_freepointer.isra.70
> >> 0.17% [kernel] [k] __slab_free
> >> 0.16% [kernel] [k] eth_type_vlan
> >> 0.16% [kernel] [k] ip_finish_output
> >> 0.15% [kernel] [k] kmem_cache_free_bulk
> >> 0.14% [kernel] [k] netif_receive_skb_internal
> >>
> >>
> >>
> >>
> >> wondering why this:
> >> 1.97% [kernel] [k] virt_to_head_page
> >> is in top...
> > This is related to the page_frag_free() call, but it is weird that it
> > shows up because it is suppose to be inlined (it is explicitly marked
> > inline in include/linux/mm.h).
> >
> >
> >>>>>>> perf top:
> >>>>>>>
> >>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
> >>>>>>> ---------------------------------------------
> >>>>>>>
> >>>>>>> 16.32% [kernel] [k] skb_dst_force
> >>>>>>> 16.30% [kernel] [k] dst_release
> >>>>>>> 15.11% [kernel] [k] rt_cache_valid
> >>>>>>> 12.62% [kernel] [k] ipv4_mtu
> >>>>>> It seems a little strange that these 4 functions are on the top
> >>> I don't see these in my test.
> >>>
> >>>>>>
> >>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
> >>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
> >>>>> can be hard to paste it here:)
> >>>>> attached file
> >>> The attached was very big. Please don't attach so big file on mailing
> >>> lists. Next time plase share them via e.g. pastebin. The output was a
> >>> capture from your terminal, which made the output more difficult to
> >>> read. Hint: You can/could use perf --stdio and place it in a file
> >>> instead.
> >>>
> >>> The output (extracted below) didn't show who called 'do_raw_spin_lock',
> >>> BUT it showed another interesting thing. The kernel code
> >>> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
> >>> as it will first call skb_dst_force() and then skb_dst_drop() when the
> >>> packet is transmitted on a VLAN.
> >>>
> >>> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
> >>> {
> >>> [...]
> >>> /* If device/qdisc don't need skb->dst, release it right now while
> >>> * its hot in this cpu cache.
> >>> */
> >>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
> >>> skb_dst_drop(skb);
> >>> else
> >>> skb_dst_force(skb);
> >>>
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 9:57 ` Jesper Dangaard Brouer
@ 2017-08-15 10:02 ` Paweł Staszewski
2017-08-15 10:05 ` Paweł Staszewski
0 siblings, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-15 10:02 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan
W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
> On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>
>> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
>>> On Tue, 15 Aug 2017 02:38:56 +0200
>>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>
>>>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
>>>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>>>
>>>>>> To show some difference below comparision vlan/no-vlan traffic
>>>>>>
>>>>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
>>>>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
>>>>> performance reduction of about 10-19% when I forward out a VLAN
>>>>> interface. This is larger than I expected, but still lower than what
>>>>> you reported 30-40% slowdown.
>>>>>
>>>>> [...]
>>>> Ok mellanox afrrived (MT27700 - mlnx5 driver)
>>>> And to compare melannox with vlans and without: 33% performance
>>>> degradation (less than with ixgbe where i reach ~40% with same settings)
>>>>
>>>> Mellanox without TX traffix on vlan:
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;16;64;11089305;709715520;8871553;567779392
>>>> 1;16;64;11096292;710162688;11095566;710116224
>>>> 2;16;64;11095770;710129280;11096799;710195136
>>>> 3;16;64;11097199;710220736;11097702;710252928
>>>> 4;16;64;11080984;567081856;11079662;709098368
>>>> 5;16;64;11077696;708972544;11077039;708930496
>>>> 6;16;64;11082991;709311424;8864802;567347328
>>>> 7;16;64;11089596;709734144;8870927;709789184
>>>> 8;16;64;11094043;710018752;11095391;710105024
>>>>
>>>> Mellanox with TX traffic on vlan:
>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>> 0;16;64;7369914;471674496;7370281;471697980
>>>> 1;16;64;7368896;471609408;7368043;471554752
>>>> 2;16;64;7367577;471524864;7367759;471536576
>>>> 3;16;64;7368744;377305344;7369391;471641024
>>>> 4;16;64;7366824;471476736;7364330;471237120
>>>> 5;16;64;7368352;471574528;7367239;471503296
>>>> 6;16;64;7367459;471517376;7367806;471539584
>>>> 7;16;64;7367190;471500160;7367988;471551232
>>>> 8;16;64;7368023;471553472;7368076;471556864
>>> I wonder if the drivers page recycler is active/working or not, and if
>>> the situation is different between VLAN vs no-vlan (given
>>> page_frag_free is so high in you perf top). The Mellanox drivers
>>> fortunately have a stats counter to tell us this explicitly (which the
>>> ixgbe driver doesn't).
>>>
>>> You can use my ethtool_stats.pl script watch these stats:
>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>> (Hint perl dependency: dnf install perl-Time-HiRes)
>> For RX NIC:
>> Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
>> Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <= rx0_bytes /sec
>> Ethtool(enp175s0f0) stat: 230978 ( 230,978) <= rx0_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <= rx0_packets /sec
>> Ethtool(enp175s0f0) stat: 921614 ( 921,614) <= rx0_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <= rx1_bytes /sec
>> Ethtool(enp175s0f0) stat: 233343 ( 233,343) <= rx1_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <= rx1_packets /sec
>> Ethtool(enp175s0f0) stat: 927793 ( 927,793) <= rx1_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <= rx2_bytes /sec
>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx2_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <= rx2_packets /sec
>> Ethtool(enp175s0f0) stat: 937989 ( 937,989) <= rx2_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <= rx3_bytes /sec
>> Ethtool(enp175s0f0) stat: 230311 ( 230,311) <= rx3_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <= rx3_packets /sec
>> Ethtool(enp175s0f0) stat: 922513 ( 922,513) <= rx3_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <= rx4_bytes /sec
>> Ethtool(enp175s0f0) stat: 191969 ( 191,969) <= rx4_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <= rx4_packets /sec
>> Ethtool(enp175s0f0) stat: 766332 ( 766,332) <= rx4_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <= rx5_bytes /sec
>> Ethtool(enp175s0f0) stat: 197150 ( 197,150) <= rx5_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <= rx5_packets /sec
>> Ethtool(enp175s0f0) stat: 786978 ( 786,978) <= rx5_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <= rx6_bytes /sec
>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <= rx6_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <= rx6_packets /sec
>> Ethtool(enp175s0f0) stat: 929163 ( 929,163) <= rx6_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <= rx7_bytes /sec
>> Ethtool(enp175s0f0) stat: 230413 ( 230,413) <= rx7_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <= rx7_packets /sec
>> Ethtool(enp175s0f0) stat: 926376 ( 926,376) <= rx7_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <= rx_65_to_127_bytes_phy /sec
>> Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes /sec
>> Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <= rx_bytes_phy /sec
>> Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <= rx_cache_reuse /sec
>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_csum_complete /sec
>> Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <= rx_out_of_buffer /sec
>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <= rx_packets /sec
>> Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <= rx_packets_phy /sec
>> Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <= rx_page_reuse /sec
>> Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <= rx_prio0_bytes /sec
>> Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <= rx_prio0_packets /sec
>> Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <= rx_vport_unicast_bytes /sec
>> Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <= rx_vport_unicast_packets /sec
>
> It looks like the mlx5 page recycle mechanism works:
>
> 230413 ( 230,413) <= rx7_cache_reuse /sec
> + 926376 ( 926,376) <= rx7_page_reuse /sec
> =1156789 (230413+926376)
> -1156775 ( 1,156,775) <= rx7_packets /sec
> = 14
>
> You can also determine this as there are no counters for:
> rx_cache_full or
> rx_cache_empty or
> rx1_cache_empty
> rx1_cache_busy
>
>
>> For TX nic with vlan:
>> Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_65_to_127_bytes_phy /sec
>> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_bytes_phy /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_multicast_phy /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_packets_phy /sec
>> Ethtool(enp175s0f1) stat: 71 ( 71) <= rx_prio0_bytes /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_prio0_packets /sec
>> Ethtool(enp175s0f1) stat: 67 ( 67) <= rx_vport_multicast_bytes /sec
>> Ethtool(enp175s0f1) stat: 1 ( 1) <= rx_vport_multicast_packets /sec
>> Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <= tx0_bytes /sec
>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_csum_none /sec
>> Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop /sec
>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <= tx0_packets /sec
>> Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <= tx1_bytes /sec
>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_csum_none /sec
>> Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop /sec
>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <= tx1_packets /sec
>> Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <= tx2_bytes /sec
>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop /sec
>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <= tx2_packets /sec
>> Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <= tx3_bytes /sec
>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_csum_none /sec
>> Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop /sec
>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <= tx3_packets /sec
>> Ethtool(enp175s0f1) stat: 234 ( 234) <= tx3_xmit_more /sec
>> Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <= tx4_bytes /sec
>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_csum_none /sec
>> Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop /sec
>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <= tx4_packets /sec
>> Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <= tx5_bytes /sec
>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop /sec
>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <= tx5_packets /sec
>> Ethtool(enp175s0f1) stat: 264 ( 264) <= tx5_xmit_more /sec
>> Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <= tx6_bytes /sec
>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop /sec
>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <= tx6_packets /sec
>> Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <= tx7_bytes /sec
>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_csum_none /sec
>> Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop /sec
>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <= tx7_packets /sec
>> Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes /sec
>> Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <= tx_bytes_phy /sec
>> Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <= tx_packets /sec
>> Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <= tx_packets_phy /sec
>> Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <= tx_prio0_bytes /sec
>> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <= tx_prio0_packets /sec
>> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <= tx_vport_unicast_bytes /sec
>> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <= tx_vport_unicast_packets /sec
>> Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more /sec
> We are seeing some xmit_more, this is interesting. Have you noticed,
> if (in the VLAN case) there is a queue in the qdisc layer?
>
> Simply inspect with: tc -s qdisc show dev ixgbe2
for vlan no qdisc:
tc -s -d qdisc show dev vlan1000
qdisc noqueue 0: root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
physical interface mq attached with pfifo_fast:
tc -s -d qdisc show dev enp175s0f1
qdisc mq 0: root
Sent 1397200697212 bytes 3965888669 pkt (dropped 78065663, overlimits
0 requeues 629868)
backlog 0b 0p requeues 629868
qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :36 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :35 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :34 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :33 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :32 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :31 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :30 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :2a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :29 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :28 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :27 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :26 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :25 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :24 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :23 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :22 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :21 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :20 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :1a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :19 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :18 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 176 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :17 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :16 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :15 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :14 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :13 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :12 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :11 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc pfifo_fast 0: parent :10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 18534143578 bytes 289595993 pkt (dropped 0, overlimits 0 requeues
1191)
backlog 0b 0p requeues 1191
qdisc pfifo_fast 0: parent :f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 17022213952 bytes 265972093 pkt (dropped 0, overlimits 0 requeues
1126)
backlog 0b 0p requeues 1126
qdisc pfifo_fast 0: parent :e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 19610732552 bytes 306417696 pkt (dropped 0, overlimits 0 requeues
147499)
backlog 0b 0p requeues 147499
qdisc pfifo_fast 0: parent :d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 18153006206 bytes 283640721 pkt (dropped 0, overlimits 0 requeues
171576)
backlog 0b 0p requeues 171576
qdisc pfifo_fast 0: parent :c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 20642461846 bytes 322538466 pkt (dropped 0, overlimits 0 requeues
67124)
backlog 0b 0p requeues 67124
qdisc pfifo_fast 0: parent :b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 19343910144 bytes 302248596 pkt (dropped 0, overlimits 0 requeues
224627)
backlog 0b 0p requeues 224627
qdisc pfifo_fast 0: parent :a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 20735716968 bytes 323995576 pkt (dropped 0, overlimits 0 requeues
1451)
backlog 0b 0p requeues 1451
qdisc pfifo_fast 0: parent :9 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 19371526272 bytes 302680098 pkt (dropped 0, overlimits 0 requeues
2244)
backlog 0b 0p requeues 2244
qdisc pfifo_fast 0: parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 162202525454 bytes 2444195441 pkt (dropped 31716535, overlimits 0
requeues 2456)
backlog 0b 0p requeues 2456
qdisc pfifo_fast 0: parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 163308408548 bytes 2461561759 pkt (dropped 0, overlimits 0
requeues 1454)
backlog 0b 0p requeues 1454
qdisc pfifo_fast 0: parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 164214830346 bytes 2475734074 pkt (dropped 19272, overlimits 0
requeues 2216)
backlog 0b 0p requeues 2216
qdisc pfifo_fast 0: parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 160398101830 bytes 2417635038 pkt (dropped 0, overlimits 0
requeues 1509)
backlog 0b 0p requeues 1509
qdisc pfifo_fast 0: parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 161760263056 bytes 2437965677 pkt (dropped 0, overlimits 0
requeues 1442)
backlog 0b 0p requeues 1442
qdisc pfifo_fast 0: parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 160300203040 bytes 2415206290 pkt (dropped 0, overlimits 0
requeues 1615)
backlog 0b 0p requeues 1615
qdisc pfifo_fast 0: parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 140144336184 bytes 2114089739 pkt (dropped 0, overlimits 0
requeues 1177)
backlog 0b 0p requeues 1177
qdisc pfifo_fast 0: parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1
1 1 1
Sent 131458317060 bytes 1982280594 pkt (dropped 46329856, overlimits 0
requeues 1161)
backlog 0b 0p requeues 1161
>
>
>>>
>>>> ethtool settings for both tests:
>>>> ifc='enp175s0f0 enp175s0f1'
>>>> for i in $ifc
>>>> do
>>>> ip link set up dev $i
>>>> ethtool -A $i autoneg off rx off tx off
>>>> ethtool -G $i rx 128 tx 256
>>> The ring queue size recommendations, might be different for the mlx5
>>> driver (Cc'ing Mellanox maintainers).
>>>
>>>
>>>> ip link set $i txqueuelen 1000
>>>> ethtool -C $i rx-usecs 25
>>>> ethtool -L $i combined 16
>>>> ethtool -K $i gro off tso off gso off sg on l2-fwd-offload off
>>>> tx-nocache-copy off ntuple on
>>>> ethtool -N $i rx-flow-hash udp4 sdfn
>>>> done
>>> Thanks for being explicit about what you setup is :-)
>>>
>>>> and perf top:
>>>> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>>> cycles], (all, 56 CPUs)
>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>
>>>> 14.25% [kernel] [k] dst_release
>>>> 14.17% [kernel] [k] skb_dst_force
>>>> 13.41% [kernel] [k] rt_cache_valid
>>>> 11.47% [kernel] [k] ip_finish_output2
>>>> 7.01% [kernel] [k] do_raw_spin_lock
>>>> 5.07% [kernel] [k] page_frag_free
>>>> 3.47% [mlx5_core] [k] mlx5e_xmit
>>>> 2.88% [kernel] [k] fib_table_lookup
>>>> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
>>>> 1.97% [kernel] [k] virt_to_head_page
>>>> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
>>>> 0.93% [kernel] [k] __dev_queue_xmit
>>>> 0.87% [kernel] [k] __build_skb
>>>> 0.84% [kernel] [k] ipt_do_table
>>>> 0.79% [kernel] [k] ip_rcv
>>>> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
>>>> 0.78% [kernel] [k] netif_skb_features
>>>> 0.73% [kernel] [k] __netif_receive_skb_core
>>>> 0.52% [kernel] [k] dev_hard_start_xmit
>>>> 0.52% [kernel] [k] build_skb
>>>> 0.51% [kernel] [k] ip_route_input_rcu
>>>> 0.50% [kernel] [k] skb_unref
>>>> 0.49% [kernel] [k] ip_forward
>>>> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
>>>> 0.44% [kernel] [k] udp_v4_early_demux
>>>> 0.41% [kernel] [k] napi_consume_skb
>>>> 0.40% [kernel] [k] __local_bh_enable_ip
>>>> 0.39% [kernel] [k] ip_rcv_finish
>>>> 0.39% [kernel] [k] kmem_cache_alloc
>>>> 0.38% [kernel] [k] sch_direct_xmit
>>>> 0.33% [kernel] [k] validate_xmit_skb
>>>> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
>>>> 0.29% [kernel] [k] netdev_pick_tx
>>>> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
>>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>>> 0.26% [kernel] [k] fib_validate_source
>>>> 0.26% [mlx5_core] [k] mlx5e_napi_poll
>>>> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
>>>> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
>>>> 0.25% [kernel] [k] eth_header
>>>> 0.23% [kernel] [k] skb_network_protocol
>>>> 0.20% [kernel] [k] nf_hook_slow
>>>> 0.20% [kernel] [k] vlan_passthru_hard_header
>>>> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
>>>> 0.19% [kernel] [k] swiotlb_map_page
>>>> 0.18% [kernel] [k] compound_head
>>>> 0.18% [kernel] [k] neigh_connected_output
>>>> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
>>>> 0.18% [kernel] [k] ip_output
>>>> 0.17% [kernel] [k] prefetch_freepointer.isra.70
>>>> 0.17% [kernel] [k] __slab_free
>>>> 0.16% [kernel] [k] eth_type_vlan
>>>> 0.16% [kernel] [k] ip_finish_output
>>>> 0.15% [kernel] [k] kmem_cache_free_bulk
>>>> 0.14% [kernel] [k] netif_receive_skb_internal
>>>>
>>>>
>>>>
>>>>
>>>> wondering why this:
>>>> 1.97% [kernel] [k] virt_to_head_page
>>>> is in top...
>>> This is related to the page_frag_free() call, but it is weird that it
>>> shows up because it is suppose to be inlined (it is explicitly marked
>>> inline in include/linux/mm.h).
>>>
>>>
>>>>>>>>> perf top:
>>>>>>>>>
>>>>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
>>>>>>>>> ---------------------------------------------
>>>>>>>>>
>>>>>>>>> 16.32% [kernel] [k] skb_dst_force
>>>>>>>>> 16.30% [kernel] [k] dst_release
>>>>>>>>> 15.11% [kernel] [k] rt_cache_valid
>>>>>>>>> 12.62% [kernel] [k] ipv4_mtu
>>>>>>>> It seems a little strange that these 4 functions are on the top
>>>>> I don't see these in my test.
>>>>>
>>>>>>>>
>>>>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>>>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
>>>>>>> can be hard to paste it here:)
>>>>>>> attached file
>>>>> The attached was very big. Please don't attach so big file on mailing
>>>>> lists. Next time plase share them via e.g. pastebin. The output was a
>>>>> capture from your terminal, which made the output more difficult to
>>>>> read. Hint: You can/could use perf --stdio and place it in a file
>>>>> instead.
>>>>>
>>>>> The output (extracted below) didn't show who called 'do_raw_spin_lock',
>>>>> BUT it showed another interesting thing. The kernel code
>>>>> __dev_queue_xmit() in might create route dst-cache problem for itself(?),
>>>>> as it will first call skb_dst_force() and then skb_dst_drop() when the
>>>>> packet is transmitted on a VLAN.
>>>>>
>>>>> static int __dev_queue_xmit(struct sk_buff *skb, void *accel_priv)
>>>>> {
>>>>> [...]
>>>>> /* If device/qdisc don't need skb->dst, release it right now while
>>>>> * its hot in this cpu cache.
>>>>> */
>>>>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
>>>>> skb_dst_drop(skb);
>>>>> else
>>>>> skb_dst_force(skb);
>>>>>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 10:02 ` Paweł Staszewski
@ 2017-08-15 10:05 ` Paweł Staszewski
2017-08-15 10:28 ` Jesper Dangaard Brouer
0 siblings, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-08-15 10:05 UTC (permalink / raw)
To: Jesper Dangaard Brouer
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan
W dniu 2017-08-15 o 12:02, Paweł Staszewski pisze:
>
>
> W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
>> On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski
>> <pstaszewski@itcare.pl> wrote:
>>
>>> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
>>>> On Tue, 15 Aug 2017 02:38:56 +0200
>>>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
>>>>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
>>>>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski
>>>>>> <pstaszewski@itcare.pl> wrote:
>>>>>>> To show some difference below comparision vlan/no-vlan traffic
>>>>>>>
>>>>>>> 10Mpps forwarded traffic vith no-vlan vs 6.9Mpps with vlan
>>>>>> I'm trying to reproduce in my testlab (with ixgbe). I do see, a
>>>>>> performance reduction of about 10-19% when I forward out a VLAN
>>>>>> interface. This is larger than I expected, but still lower than
>>>>>> what
>>>>>> you reported 30-40% slowdown.
>>>>>>
>>>>>> [...]
>>>>> Ok mellanox afrrived (MT27700 - mlnx5 driver)
>>>>> And to compare melannox with vlans and without: 33% performance
>>>>> degradation (less than with ixgbe where i reach ~40% with same
>>>>> settings)
>>>>>
>>>>> Mellanox without TX traffix on vlan:
>>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>>> 0;16;64;11089305;709715520;8871553;567779392
>>>>> 1;16;64;11096292;710162688;11095566;710116224
>>>>> 2;16;64;11095770;710129280;11096799;710195136
>>>>> 3;16;64;11097199;710220736;11097702;710252928
>>>>> 4;16;64;11080984;567081856;11079662;709098368
>>>>> 5;16;64;11077696;708972544;11077039;708930496
>>>>> 6;16;64;11082991;709311424;8864802;567347328
>>>>> 7;16;64;11089596;709734144;8870927;709789184
>>>>> 8;16;64;11094043;710018752;11095391;710105024
>>>>>
>>>>> Mellanox with TX traffic on vlan:
>>>>> ID;CPU_CORES / RSS QUEUES;PKT_SIZE;PPS_RX;BPS_RX;PPS_TX;BPS_TX
>>>>> 0;16;64;7369914;471674496;7370281;471697980
>>>>> 1;16;64;7368896;471609408;7368043;471554752
>>>>> 2;16;64;7367577;471524864;7367759;471536576
>>>>> 3;16;64;7368744;377305344;7369391;471641024
>>>>> 4;16;64;7366824;471476736;7364330;471237120
>>>>> 5;16;64;7368352;471574528;7367239;471503296
>>>>> 6;16;64;7367459;471517376;7367806;471539584
>>>>> 7;16;64;7367190;471500160;7367988;471551232
>>>>> 8;16;64;7368023;471553472;7368076;471556864
>>>> I wonder if the drivers page recycler is active/working or not, and if
>>>> the situation is different between VLAN vs no-vlan (given
>>>> page_frag_free is so high in you perf top). The Mellanox drivers
>>>> fortunately have a stats counter to tell us this explicitly (which the
>>>> ixgbe driver doesn't).
>>>>
>>>> You can use my ethtool_stats.pl script watch these stats:
>>>> https://github.com/netoptimizer/network-testing/blob/master/bin/ethtool_stats.pl
>>>> (Hint perl dependency: dnf install perl-Time-HiRes)
>>> For RX NIC:
>>> Show adapter(s) (enp175s0f0) statistics (ONLY that changed!)
>>> Ethtool(enp175s0f0) stat: 78380071 ( 78,380,071) <=
>>> rx0_bytes /sec
>>> Ethtool(enp175s0f0) stat: 230978 ( 230,978) <=
>>> rx0_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <=
>>> rx0_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1152648 ( 1,152,648) <=
>>> rx0_packets /sec
>>> Ethtool(enp175s0f0) stat: 921614 ( 921,614) <=
>>> rx0_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 78956591 ( 78,956,591) <=
>>> rx1_bytes /sec
>>> Ethtool(enp175s0f0) stat: 233343 ( 233,343) <=
>>> rx1_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <=
>>> rx1_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1161126 ( 1,161,126) <=
>>> rx1_packets /sec
>>> Ethtool(enp175s0f0) stat: 927793 ( 927,793) <=
>>> rx1_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 79677124 ( 79,677,124) <=
>>> rx2_bytes /sec
>>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
>>> rx2_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <=
>>> rx2_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1171722 ( 1,171,722) <=
>>> rx2_packets /sec
>>> Ethtool(enp175s0f0) stat: 937989 ( 937,989) <=
>>> rx2_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 78392893 ( 78,392,893) <=
>>> rx3_bytes /sec
>>> Ethtool(enp175s0f0) stat: 230311 ( 230,311) <=
>>> rx3_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <=
>>> rx3_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1152837 ( 1,152,837) <=
>>> rx3_packets /sec
>>> Ethtool(enp175s0f0) stat: 922513 ( 922,513) <=
>>> rx3_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 65165583 ( 65,165,583) <=
>>> rx4_bytes /sec
>>> Ethtool(enp175s0f0) stat: 191969 ( 191,969) <=
>>> rx4_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <=
>>> rx4_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 958317 ( 958,317) <=
>>> rx4_packets /sec
>>> Ethtool(enp175s0f0) stat: 766332 ( 766,332) <=
>>> rx4_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 66920721 ( 66,920,721) <=
>>> rx5_bytes /sec
>>> Ethtool(enp175s0f0) stat: 197150 ( 197,150) <=
>>> rx5_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <=
>>> rx5_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 984128 ( 984,128) <=
>>> rx5_packets /sec
>>> Ethtool(enp175s0f0) stat: 786978 ( 786,978) <=
>>> rx5_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 79076984 ( 79,076,984) <=
>>> rx6_bytes /sec
>>> Ethtool(enp175s0f0) stat: 233735 ( 233,735) <=
>>> rx6_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <=
>>> rx6_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1162897 ( 1,162,897) <=
>>> rx6_packets /sec
>>> Ethtool(enp175s0f0) stat: 929163 ( 929,163) <=
>>> rx6_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 78660672 ( 78,660,672) <=
>>> rx7_bytes /sec
>>> Ethtool(enp175s0f0) stat: 230413 ( 230,413) <=
>>> rx7_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <=
>>> rx7_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1156775 ( 1,156,775) <=
>>> rx7_packets /sec
>>> Ethtool(enp175s0f0) stat: 926376 ( 926,376) <=
>>> rx7_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 10674565 ( 10,674,565) <=
>>> rx_65_to_127_bytes_phy /sec
>>> Ethtool(enp175s0f0) stat: 605241031 ( 605,241,031) <= rx_bytes
>>> /sec
>>> Ethtool(enp175s0f0) stat: 768585608 ( 768,585,608) <=
>>> rx_bytes_phy /sec
>>> Ethtool(enp175s0f0) stat: 1781569 ( 1,781,569) <=
>>> rx_cache_reuse /sec
>>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <=
>>> rx_csum_complete /sec
>>> Ethtool(enp175s0f0) stat: 1773785 ( 1,773,785) <=
>>> rx_out_of_buffer /sec
>>> Ethtool(enp175s0f0) stat: 8900603 ( 8,900,603) <=
>>> rx_packets /sec
>>> Ethtool(enp175s0f0) stat: 10674799 ( 10,674,799) <=
>>> rx_packets_phy /sec
>>> Ethtool(enp175s0f0) stat: 7118993 ( 7,118,993) <=
>>> rx_page_reuse /sec
>>> Ethtool(enp175s0f0) stat: 768565744 ( 768,565,744) <=
>>> rx_prio0_bytes /sec
>>> Ethtool(enp175s0f0) stat: 10674522 ( 10,674,522) <=
>>> rx_prio0_packets /sec
>>> Ethtool(enp175s0f0) stat: 725871089 ( 725,871,089) <=
>>> rx_vport_unicast_bytes /sec
>>> Ethtool(enp175s0f0) stat: 10674575 ( 10,674,575) <=
>>> rx_vport_unicast_packets /sec
>>
>> It looks like the mlx5 page recycle mechanism works:
>>
>> 230413 ( 230,413) <= rx7_cache_reuse /sec
>> + 926376 ( 926,376) <= rx7_page_reuse /sec
>> =1156789 (230413+926376)
>> -1156775 ( 1,156,775) <= rx7_packets /sec
>> = 14
>>
>> You can also determine this as there are no counters for:
>> rx_cache_full or
>> rx_cache_empty or
>> rx1_cache_empty
>> rx1_cache_busy
>>
>>> For TX nic with vlan:
>>> Show adapter(s) (enp175s0f1) statistics (ONLY that changed!)
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_65_to_127_bytes_phy /sec
>>> Ethtool(enp175s0f1) stat: 71 ( 71) <=
>>> rx_bytes_phy /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_multicast_phy /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_packets_phy /sec
>>> Ethtool(enp175s0f1) stat: 71 ( 71) <=
>>> rx_prio0_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_prio0_packets /sec
>>> Ethtool(enp175s0f1) stat: 67 ( 67) <=
>>> rx_vport_multicast_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1 ( 1) <=
>>> rx_vport_multicast_packets /sec
>>> Ethtool(enp175s0f1) stat: 64955114 ( 64,955,114) <=
>>> tx0_bytes /sec
>>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <=
>>> tx0_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 26489 ( 26,489) <= tx0_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 955222 ( 955,222) <=
>>> tx0_packets /sec
>>> Ethtool(enp175s0f1) stat: 66799214 ( 66,799,214) <=
>>> tx1_bytes /sec
>>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <=
>>> tx1_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 27225 ( 27,225) <= tx1_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 982341 ( 982,341) <=
>>> tx1_packets /sec
>>> Ethtool(enp175s0f1) stat: 78650421 ( 78,650,421) <=
>>> tx2_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <=
>>> tx2_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32059 ( 32,059) <= tx2_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1156624 ( 1,156,624) <=
>>> tx2_packets /sec
>>> Ethtool(enp175s0f1) stat: 78186849 ( 78,186,849) <=
>>> tx3_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <=
>>> tx3_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 31879 ( 31,879) <= tx3_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1149807 ( 1,149,807) <=
>>> tx3_packets /sec
>>> Ethtool(enp175s0f1) stat: 234 ( 234) <=
>>> tx3_xmit_more /sec
>>> Ethtool(enp175s0f1) stat: 78466099 ( 78,466,099) <=
>>> tx4_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <=
>>> tx4_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 31990 ( 31,990) <= tx4_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1153913 ( 1,153,913) <=
>>> tx4_packets /sec
>>> Ethtool(enp175s0f1) stat: 78765724 ( 78,765,724) <=
>>> tx5_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <=
>>> tx5_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32115 ( 32,115) <= tx5_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1158319 ( 1,158,319) <=
>>> tx5_packets /sec
>>> Ethtool(enp175s0f1) stat: 264 ( 264) <=
>>> tx5_xmit_more /sec
>>> Ethtool(enp175s0f1) stat: 79669524 ( 79,669,524) <=
>>> tx6_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <=
>>> tx6_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32490 ( 32,490) <= tx6_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1171611 ( 1,171,611) <=
>>> tx6_packets /sec
>>> Ethtool(enp175s0f1) stat: 79389329 ( 79,389,329) <=
>>> tx7_bytes /sec
>>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <=
>>> tx7_csum_none /sec
>>> Ethtool(enp175s0f1) stat: 32365 ( 32,365) <= tx7_nop
>>> /sec
>>> Ethtool(enp175s0f1) stat: 1167490 ( 1,167,490) <=
>>> tx7_packets /sec
>>> Ethtool(enp175s0f1) stat: 604885175 ( 604,885,175) <= tx_bytes
>>> /sec
>>> Ethtool(enp175s0f1) stat: 676059749 ( 676,059,749) <=
>>> tx_bytes_phy /sec
>>> Ethtool(enp175s0f1) stat: 8895370 ( 8,895,370) <=
>>> tx_packets /sec
>>> Ethtool(enp175s0f1) stat: 8895522 ( 8,895,522) <=
>>> tx_packets_phy /sec
>>> Ethtool(enp175s0f1) stat: 676063067 ( 676,063,067) <=
>>> tx_prio0_bytes /sec
>>> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <=
>>> tx_prio0_packets /sec
>>> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <=
>>> tx_vport_unicast_bytes /sec
>>> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <=
>>> tx_vport_unicast_packets /sec
>>> Ethtool(enp175s0f1) stat: 498 ( 498) <=
>>> tx_xmit_more /sec
>> We are seeing some xmit_more, this is interesting. Have you noticed,
>> if (in the VLAN case) there is a queue in the qdisc layer?
>>
>> Simply inspect with: tc -s qdisc show dev ixgbe2
> for vlan no qdisc:
> tc -s -d qdisc show dev vlan1000
>
> qdisc noqueue 0: root refcnt 2
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
>
> physical interface mq attached with pfifo_fast:
>
> tc -s -d qdisc show dev enp175s0f1
> qdisc mq 0: root
> Sent 1397200697212 bytes 3965888669 pkt (dropped 78065663, overlimits
> 0 requeues 629868)
> backlog 0b 0p requeues 629868
> qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :36 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :35 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :34 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :33 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :32 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :31 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :30 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :2a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :29 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :28 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :27 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :26 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :25 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :24 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :23 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :22 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :21 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :20 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :1a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :19 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :18 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 176 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :17 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :16 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :15 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :14 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :13 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :12 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :11 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> backlog 0b 0p requeues 0
> qdisc pfifo_fast 0: parent :10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1
> 1 1 1 1 1
> Sent 18534143578 bytes 289595993 pkt (dropped 0, overlimits 0
> requeues 1191)
> backlog 0b 0p requeues 1191
> qdisc pfifo_fast 0: parent :f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 17022213952 bytes 265972093 pkt (dropped 0, overlimits 0
> requeues 1126)
> backlog 0b 0p requeues 1126
> qdisc pfifo_fast 0: parent :e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 19610732552 bytes 306417696 pkt (dropped 0, overlimits 0
> requeues 147499)
> backlog 0b 0p requeues 147499
> qdisc pfifo_fast 0: parent :d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 18153006206 bytes 283640721 pkt (dropped 0, overlimits 0
> requeues 171576)
> backlog 0b 0p requeues 171576
> qdisc pfifo_fast 0: parent :c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 20642461846 bytes 322538466 pkt (dropped 0, overlimits 0
> requeues 67124)
> backlog 0b 0p requeues 67124
> qdisc pfifo_fast 0: parent :b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 19343910144 bytes 302248596 pkt (dropped 0, overlimits 0
> requeues 224627)
> backlog 0b 0p requeues 224627
> qdisc pfifo_fast 0: parent :a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 20735716968 bytes 323995576 pkt (dropped 0, overlimits 0
> requeues 1451)
> backlog 0b 0p requeues 1451
> qdisc pfifo_fast 0: parent :9 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 19371526272 bytes 302680098 pkt (dropped 0, overlimits 0
> requeues 2244)
> backlog 0b 0p requeues 2244
> qdisc pfifo_fast 0: parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 162202525454 bytes 2444195441 pkt (dropped 31716535, overlimits
> 0 requeues 2456)
> backlog 0b 0p requeues 2456
> qdisc pfifo_fast 0: parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 163308408548 bytes 2461561759 pkt (dropped 0, overlimits 0
> requeues 1454)
> backlog 0b 0p requeues 1454
> qdisc pfifo_fast 0: parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 164214830346 bytes 2475734074 pkt (dropped 19272, overlimits 0
> requeues 2216)
> backlog 0b 0p requeues 2216
> qdisc pfifo_fast 0: parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 160398101830 bytes 2417635038 pkt (dropped 0, overlimits 0
> requeues 1509)
> backlog 0b 0p requeues 1509
> qdisc pfifo_fast 0: parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 161760263056 bytes 2437965677 pkt (dropped 0, overlimits 0
> requeues 1442)
> backlog 0b 0p requeues 1442
> qdisc pfifo_fast 0: parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 160300203040 bytes 2415206290 pkt (dropped 0, overlimits 0
> requeues 1615)
> backlog 0b 0p requeues 1615
> qdisc pfifo_fast 0: parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 140144336184 bytes 2114089739 pkt (dropped 0, overlimits 0
> requeues 1177)
> backlog 0b 0p requeues 1177
> qdisc pfifo_fast 0: parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
> 1 1 1 1
> Sent 131458317060 bytes 1982280594 pkt (dropped 46329856, overlimits
> 0 requeues 1161)
> backlog 0b 0p requeues 1161
>
>
just see that after changing RSS on nics did't deleted qdisc and added
again:
Here situation with qdisc del / add
tc -s -d qdisc show dev enp175s0f1
qdisc mq 1: root
Sent 43738523966 bytes 683414438 pkt (dropped 0, overlimits 0 requeues
1886)
backlog 0b 0p requeues 1886
qdisc pfifo_fast 0: parent 1:10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2585011904 bytes 40390811 pkt (dropped 0, overlimits 0 requeues 110)
backlog 0b 0p requeues 110
qdisc pfifo_fast 0: parent 1:f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2602068416 bytes 40657319 pkt (dropped 0, overlimits 0 requeues 121)
backlog 0b 0p requeues 121
qdisc pfifo_fast 0: parent 1:e bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2793524544 bytes 43648821 pkt (dropped 0, overlimits 0 requeues 12)
backlog 0b 0p requeues 12
qdisc pfifo_fast 0: parent 1:d bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2792512768 bytes 43633012 pkt (dropped 0, overlimits 0 requeues 8)
backlog 0b 0p requeues 8
qdisc pfifo_fast 0: parent 1:c bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789822080 bytes 43590970 pkt (dropped 0, overlimits 0 requeues 8)
backlog 0b 0p requeues 8
qdisc pfifo_fast 0: parent 1:b bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789904640 bytes 43592260 pkt (dropped 0, overlimits 0 requeues 9)
backlog 0b 0p requeues 9
qdisc pfifo_fast 0: parent 1:a bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2793279146 bytes 43644987 pkt (dropped 0, overlimits 0 requeues 192)
backlog 0b 0p requeues 192
qdisc pfifo_fast 0: parent 1:9 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789506688 bytes 43586042 pkt (dropped 0, overlimits 0 requeues 171)
backlog 0b 0p requeues 171
qdisc pfifo_fast 0: parent 1:8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789948928 bytes 43592952 pkt (dropped 0, overlimits 0 requeues 179)
backlog 0b 0p requeues 179
qdisc pfifo_fast 0: parent 1:7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2790191552 bytes 43596743 pkt (dropped 0, overlimits 0 requeues 177)
backlog 0b 0p requeues 177
qdisc pfifo_fast 0: parent 1:6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789649280 bytes 43588270 pkt (dropped 0, overlimits 0 requeues 182)
backlog 0b 0p requeues 182
qdisc pfifo_fast 0: parent 1:5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789654612 bytes 43588354 pkt (dropped 0, overlimits 0 requeues 151)
backlog 0b 0p requeues 151
qdisc pfifo_fast 0: parent 1:4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2789409280 bytes 43584520 pkt (dropped 0, overlimits 0 requeues 168)
backlog 0b 0p requeues 168
qdisc pfifo_fast 0: parent 1:3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2792827648 bytes 43637932 pkt (dropped 0, overlimits 0 requeues 165)
backlog 0b 0p requeues 165
qdisc pfifo_fast 0: parent 1:2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2572823936 bytes 40200374 pkt (dropped 0, overlimits 0 requeues 119)
backlog 0b 0p requeues 119
qdisc pfifo_fast 0: parent 1:1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1
1 1 1 1
Sent 2488427264 bytes 38881676 pkt (dropped 0, overlimits 0 requeues 114)
backlog 0b 0p requeues 114
>
>>
>>
>>>>> ethtool settings for both tests:
>>>>> ifc='enp175s0f0 enp175s0f1'
>>>>> for i in $ifc
>>>>> do
>>>>> ip link set up dev $i
>>>>> ethtool -A $i autoneg off rx off tx off
>>>>> ethtool -G $i rx 128 tx 256
>>>> The ring queue size recommendations, might be different for the mlx5
>>>> driver (Cc'ing Mellanox maintainers).
>>>>
>>>>> ip link set $i txqueuelen 1000
>>>>> ethtool -C $i rx-usecs 25
>>>>> ethtool -L $i combined 16
>>>>> ethtool -K $i gro off tso off gso off sg on
>>>>> l2-fwd-offload off
>>>>> tx-nocache-copy off ntuple on
>>>>> ethtool -N $i rx-flow-hash udp4 sdfn
>>>>> done
>>>> Thanks for being explicit about what you setup is :-)
>>>>> and perf top:
>>>>> PerfTop: 83650 irqs/sec kernel:99.7% exact: 0.0% [4000Hz
>>>>> cycles], (all, 56 CPUs)
>>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> 14.25% [kernel] [k] dst_release
>>>>> 14.17% [kernel] [k] skb_dst_force
>>>>> 13.41% [kernel] [k] rt_cache_valid
>>>>> 11.47% [kernel] [k] ip_finish_output2
>>>>> 7.01% [kernel] [k] do_raw_spin_lock
>>>>> 5.07% [kernel] [k] page_frag_free
>>>>> 3.47% [mlx5_core] [k] mlx5e_xmit
>>>>> 2.88% [kernel] [k] fib_table_lookup
>>>>> 2.43% [mlx5_core] [k] skb_from_cqe.isra.32
>>>>> 1.97% [kernel] [k] virt_to_head_page
>>>>> 1.81% [mlx5_core] [k] mlx5e_poll_tx_cq
>>>>> 0.93% [kernel] [k] __dev_queue_xmit
>>>>> 0.87% [kernel] [k] __build_skb
>>>>> 0.84% [kernel] [k] ipt_do_table
>>>>> 0.79% [kernel] [k] ip_rcv
>>>>> 0.79% [kernel] [k] acpi_processor_ffh_cstate_enter
>>>>> 0.78% [kernel] [k] netif_skb_features
>>>>> 0.73% [kernel] [k] __netif_receive_skb_core
>>>>> 0.52% [kernel] [k] dev_hard_start_xmit
>>>>> 0.52% [kernel] [k] build_skb
>>>>> 0.51% [kernel] [k] ip_route_input_rcu
>>>>> 0.50% [kernel] [k] skb_unref
>>>>> 0.49% [kernel] [k] ip_forward
>>>>> 0.48% [mlx5_core] [k] mlx5_cqwq_get_cqe
>>>>> 0.44% [kernel] [k] udp_v4_early_demux
>>>>> 0.41% [kernel] [k] napi_consume_skb
>>>>> 0.40% [kernel] [k] __local_bh_enable_ip
>>>>> 0.39% [kernel] [k] ip_rcv_finish
>>>>> 0.39% [kernel] [k] kmem_cache_alloc
>>>>> 0.38% [kernel] [k] sch_direct_xmit
>>>>> 0.33% [kernel] [k] validate_xmit_skb
>>>>> 0.32% [mlx5_core] [k] mlx5e_free_rx_wqe_reuse
>>>>> 0.29% [kernel] [k] netdev_pick_tx
>>>>> 0.28% [mlx5_core] [k] mlx5e_build_rx_skb
>>>>> 0.27% [kernel] [k] deliver_ptype_list_skb
>>>>> 0.26% [kernel] [k] fib_validate_source
>>>>> 0.26% [mlx5_core] [k] mlx5e_napi_poll
>>>>> 0.26% [mlx5_core] [k] mlx5e_handle_rx_cqe
>>>>> 0.26% [mlx5_core] [k] mlx5e_rx_cache_get
>>>>> 0.25% [kernel] [k] eth_header
>>>>> 0.23% [kernel] [k] skb_network_protocol
>>>>> 0.20% [kernel] [k] nf_hook_slow
>>>>> 0.20% [kernel] [k] vlan_passthru_hard_header
>>>>> 0.20% [kernel] [k] vlan_dev_hard_start_xmit
>>>>> 0.19% [kernel] [k] swiotlb_map_page
>>>>> 0.18% [kernel] [k] compound_head
>>>>> 0.18% [kernel] [k] neigh_connected_output
>>>>> 0.18% [mlx5_core] [k] mlx5e_alloc_rx_wqe
>>>>> 0.18% [kernel] [k] ip_output
>>>>> 0.17% [kernel] [k] prefetch_freepointer.isra.70
>>>>> 0.17% [kernel] [k] __slab_free
>>>>> 0.16% [kernel] [k] eth_type_vlan
>>>>> 0.16% [kernel] [k] ip_finish_output
>>>>> 0.15% [kernel] [k] kmem_cache_free_bulk
>>>>> 0.14% [kernel] [k] netif_receive_skb_internal
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> wondering why this:
>>>>> 1.97% [kernel] [k] virt_to_head_page
>>>>> is in top...
>>>> This is related to the page_frag_free() call, but it is weird that it
>>>> shows up because it is suppose to be inlined (it is explicitly marked
>>>> inline in include/linux/mm.h).
>>>>
>>>>>>>>>> perf top:
>>>>>>>>>>
>>>>>>>>>> PerfTop: 77835 irqs/sec kernel:99.7%
>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>
>>>>>>>>>> 16.32% [kernel] [k] skb_dst_force
>>>>>>>>>> 16.30% [kernel] [k] dst_release
>>>>>>>>>> 15.11% [kernel] [k] rt_cache_valid
>>>>>>>>>> 12.62% [kernel] [k] ipv4_mtu
>>>>>>>>> It seems a little strange that these 4 functions are on the top
>>>>>> I don't see these in my test.
>>>>>>>>>> 5.60% [kernel] [k] do_raw_spin_lock
>>>>>>>>> Why is calling/taking this lock? (Use perf call-graph recording).
>>>>>>>> can be hard to paste it here:)
>>>>>>>> attached file
>>>>>> The attached was very big. Please don't attach so big file on
>>>>>> mailing
>>>>>> lists. Next time plase share them via e.g. pastebin. The output
>>>>>> was a
>>>>>> capture from your terminal, which made the output more difficult to
>>>>>> read. Hint: You can/could use perf --stdio and place it in a file
>>>>>> instead.
>>>>>>
>>>>>> The output (extracted below) didn't show who called
>>>>>> 'do_raw_spin_lock',
>>>>>> BUT it showed another interesting thing. The kernel code
>>>>>> __dev_queue_xmit() in might create route dst-cache problem for
>>>>>> itself(?),
>>>>>> as it will first call skb_dst_force() and then skb_dst_drop()
>>>>>> when the
>>>>>> packet is transmitted on a VLAN.
>>>>>>
>>>>>> static int __dev_queue_xmit(struct sk_buff *skb, void
>>>>>> *accel_priv)
>>>>>> {
>>>>>> [...]
>>>>>> /* If device/qdisc don't need skb->dst, release it right now
>>>>>> while
>>>>>> * its hot in this cpu cache.
>>>>>> */
>>>>>> if (dev->priv_flags & IFF_XMIT_DST_RELEASE)
>>>>>> skb_dst_drop(skb);
>>>>>> else
>>>>>> skb_dst_force(skb);
>>>>>>
>>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 9:11 ` Paweł Staszewski
2017-08-15 9:19 ` Paweł Staszewski
@ 2017-08-15 10:05 ` Jesper Dangaard Brouer
2017-09-21 21:26 ` Paweł Staszewski
2 siblings, 0 replies; 37+ messages in thread
From: Jesper Dangaard Brouer @ 2017-08-15 10:05 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Eric Dumazet, Paolo Abeni, Linux Kernel Network Developers,
Alexander Duyck, brouer
On Tue, 15 Aug 2017 11:11:57 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> Yes it helped - now there is almost no difference when using vlans or not:
>
> 10.5Mpps - with vlan
>
> 11Mpps - without vlan
Great! - it seems like we have pinpointed the root-cause. It also
demonstrate how big the benefit is of Eric commit (thanks!):
https://git.kernel.org/torvalds/c/93f154b594fe
> W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
> > On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
> >
> >> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
> > Something like :
> >
> > diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
> > index 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7 100644
> > --- a/net/8021q/vlan_netlink.c
> > +++ b/net/8021q/vlan_netlink.c
> > @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net, struct net_device *dev,
> > vlan->vlan_proto = proto;
> > vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
> > vlan->real_dev = real_dev;
> > + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
> > vlan->flags = VLAN_FLAG_REORDER_HDR;
> >
> > err = vlan_check_real_dev(real_dev, vlan->vlan_proto, vlan->vlan_id);
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 10:05 ` Paweł Staszewski
@ 2017-08-15 10:28 ` Jesper Dangaard Brouer
0 siblings, 0 replies; 37+ messages in thread
From: Jesper Dangaard Brouer @ 2017-08-15 10:28 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Linux Kernel Network Developers, Alexander Duyck, Saeed Mahameed,
Tariq Toukan, brouer
On Tue, 15 Aug 2017 12:05:37 +0200 Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> W dniu 2017-08-15 o 12:02, Paweł Staszewski pisze:
> > W dniu 2017-08-15 o 11:57, Jesper Dangaard Brouer pisze:
> >> On Tue, 15 Aug 2017 11:30:43 +0200 Paweł Staszewski
> >> <pstaszewski@itcare.pl> wrote:
> >>> W dniu 2017-08-15 o 11:23, Jesper Dangaard Brouer pisze:
> >>>> On Tue, 15 Aug 2017 02:38:56 +0200
> >>>> Paweł Staszewski <pstaszewski@itcare.pl> wrote:
> >>>>> W dniu 2017-08-14 o 18:19, Jesper Dangaard Brouer pisze:
> >>>>>> On Sun, 13 Aug 2017 18:58:58 +0200 Paweł Staszewski
> >>>>>> <pstaszewski@itcare.pl> wrote:
[... cut ...]
> >>> Ethtool(enp175s0f1) stat: 8895566 ( 8,895,566) <= tx_prio0_packets /sec
> >>> Ethtool(enp175s0f1) stat: 640470657 ( 640,470,657) <= tx_vport_unicast_bytes /sec
> >>> Ethtool(enp175s0f1) stat: 8895427 ( 8,895,427) <= tx_vport_unicast_packets /sec
> >>> Ethtool(enp175s0f1) stat: 498 ( 498) <= tx_xmit_more /sec
> >>
> >> We are seeing some xmit_more, this is interesting. Have you noticed,
> >> if (in the VLAN case) there is a queue in the qdisc layer?
> >>
> >> Simply inspect with: tc -s qdisc show dev ixgbe2
[...]
> > physical interface mq attached with pfifo_fast:
> >
> > tc -s -d qdisc show dev enp175s0f1
> > qdisc mq 0: root
> > Sent 1397200697212 bytes 3965888669 pkt (dropped 78065663, overlimits 0 requeues 629868)
> > backlog 0b 0p requeues 629868
> > qdisc pfifo_fast 0: parent :38 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> > backlog 0b 0p requeues 0
> > qdisc pfifo_fast 0: parent :37 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> > Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
> > backlog 0b 0p requeues 0
[...]
So, it doesn't look like there is any backlog queue. Although, this
can be difficult to measure/see this way (as the kernel empty the queue
quickly via bulk deq), also given the small amount of xmit_more which
indicate that the queue was likely very small.
There is a "dropped" counter, which indicate that you likely had a
setup (earlier) where you managed to overflow the qdisc queues.
> just see that after changing RSS on nics did't deleted qdisc and added
> again:
> Here situation with qdisc del / add
> tc -s -d qdisc show dev enp175s0f1
> qdisc mq 1: root
> Sent 43738523966 bytes 683414438 pkt (dropped 0, overlimits 0 requeues 1886)
> backlog 0b 0p requeues 1886
> qdisc pfifo_fast 0: parent 1:10 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> Sent 2585011904 bytes 40390811 pkt (dropped 0, overlimits 0 requeues 110)
> backlog 0b 0p requeues 110
> qdisc pfifo_fast 0: parent 1:f bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
> Sent 2602068416 bytes 40657319 pkt (dropped 0, overlimits 0 requeues 121)
> backlog 0b 0p requeues 121
[...]
Exactly as you indicated above, these "dropped" stats came from another
(earlier) test case. (Great that you caught this yourself)
While trying to reproduce you case, I also managed to cause a situation
with qdisc overload. This caused some weird behavior, where I saw
RX=8Mpps and TX only 4Mpps. (I didn't figure out the exact tuning that
caused this, and cannot reproduce it now).
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 1:17 ` Eric Dumazet
2017-08-15 9:11 ` Paweł Staszewski
@ 2017-09-09 9:03 ` Paweł Staszewski
2017-09-11 16:57 ` Paweł Staszewski
1 sibling, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-09-09 9:03 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
Hi
Are there any plans to have this fix normally in kernel ?
Or it is mostly only hack - not longterm fix and need to be different ?
All tests that was done shows that without this patch there is about
20-30% network forwarding performance degradation when using vlan
interfaces
Thanks
Paweł
W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
> On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
>
>> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
> Something like :
>
> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
> index 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7 100644
> --- a/net/8021q/vlan_netlink.c
> +++ b/net/8021q/vlan_netlink.c
> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net, struct net_device *dev,
> vlan->vlan_proto = proto;
> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
> vlan->real_dev = real_dev;
> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
> vlan->flags = VLAN_FLAG_REORDER_HDR;
>
> err = vlan_check_real_dev(real_dev, vlan->vlan_proto, vlan->vlan_id);
>
>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-09 9:03 ` Paweł Staszewski
@ 2017-09-11 16:57 ` Paweł Staszewski
2017-09-11 22:11 ` Paweł Staszewski
0 siblings, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-09-11 16:57 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
Tested with connectx-5
Without patch
10Mpps - > 16 cores used
PerfTop: 66258 irqs/sec kernel:99.3% exact: 0.0% [4000Hz
cycles], (all, 32 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.12% [kernel] [k] do_raw_spin_lock
6.31% [kernel] [k] fib_table_lookup
6.12% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
4.90% [kernel] [k] rt_cache_valid
3.99% [kernel] [k] mlx5e_xmit
3.03% [kernel] [k] ip_rcv
2.68% [kernel] [k] __netif_receive_skb_core
2.54% [kernel] [k] skb_dst_force
2.41% [kernel] [k] ip_finish_output2
2.21% [kernel] [k] __build_skb
2.03% [kernel] [k] __dev_queue_xmit
1.96% [kernel] [k] mlx5e_txwqe_complete
1.79% [kernel] [k] ipt_do_table
1.78% [kernel] [k] inet_gro_receive
1.69% [kernel] [k] ip_forward
1.66% [kernel] [k] udp_v4_early_demux
1.65% [kernel] [k] dst_release
1.56% [kernel] [k] ip_rcv_finish
1.45% [kernel] [k] dev_gro_receive
1.45% [kernel] [k] netif_skb_features
1.39% [kernel] [k] mlx5e_poll_tx_cq
1.35% [kernel] [k] mlx5e_txwqe_build_dsegs
1.35% [kernel] [k] ip_route_input_rcu
1.15% [kernel] [k] dev_hard_start_xmit
1.12% [kernel] [k] napi_gro_receive
1.07% [kernel] [k] netif_receive_skb_internal
0.98% [kernel] [k] sch_direct_xmit
0.95% [kernel] [k] kmem_cache_alloc
0.89% [kernel] [k] read_tsc
0.88% [kernel] [k] mlx5e_build_rx_skb
0.86% [kernel] [k] mlx5_cqwq_get_cqe
0.82% [kernel] [k] page_frag_free
0.78% [kernel] [k] __local_bh_enable_ip
0.69% [kernel] [k] skb_network_protocol
0.68% [kernel] [k] __netif_receive_skb
0.67% [kernel] [k] vlan_dev_hard_start_xmit
0.65% [kernel] [k] mlx5e_poll_rx_cq
0.65% [kernel] [k] validate_xmit_skb
0.60% [kernel] [k] eth_type_trans
0.60% [kernel] [k] deliver_ptype_list_skb
0.60% [kernel] [k] fib_validate_source
0.55% [kernel] [k] eth_header
0.53% [kernel] [k] netdev_pick_tx
0.53% [kernel] [k] __napi_alloc_skb
0.51% [kernel] [k] __udp4_lib_lookup
0.50% [kernel] [k] eth_type_vlan
0.49% [kernel] [k] ip_output
0.49% [kernel] [k] page_frag_alloc
0.49% [kernel] [k] ip_finish_output
0.48% [kernel] [k] neigh_connected_output
0.45% [kernel] [k] nf_hook_slow
0.44% [kernel] [k] udp4_gro_receive
0.39% [kernel] [k] mlx5e_features_check
0.39% [kernel] [k] mlx5e_napi_poll
0.37% [kernel] [k] __jhash_nwords
0.37% [kernel] [k] udp_gro_receive
0.36% [kernel] [k] swiotlb_map_page
0.33% [kernel] [k] mlx5_cqwq_get_wqe
0.33% [kernel] [k] __netdev_pick_tx
0.29% [kernel] [k] ktime_get_with_offset
0.29% [kernel] [k] get_dma_ops
0.29% [kernel] [k] validate_xmit_skb_list
0.26% [kernel] [k] vlan_passthru_hard_header
0.26% [kernel] [k] __udp4_lib_lookup_skb
0.24% [kernel] [k] get_dma_ops
0.24% [kernel] [k] skb_release_data
0.23% [kernel] [k] ip_forward_finish
0.23% [kernel] [k] kmem_cache_free_bulk
0.23% [kernel] [k] timekeeping_get_ns
0.22% [kernel] [k] ip_skb_dst_mtu
0.21% [kernel] [k] compound_head
0.20% [kernel] [k] skb_gro_reset_offset
0.20% [kernel] [k] is_swiotlb_buffer
0.19% [kernel] [k] __net_timestamp.isra.90
0.19% [kernel] [k] dst_metric.constprop.61
0.18% [kernel] [k] skb_orphan_frags.constprop.126
0.18% [kernel] [k] _kfree_skb_defer
0.18% [kernel] [k] irq_entries_start
0.17% [kernel] [k] dev_hard_header.constprop.54
0.17% [kernel] [k] dma_mapping_error
0.17% [kernel] [k] neigh_resolve_output
With patch
12Mpps -> 16 cores
PerfTop: 66209 irqs/sec kernel:99.3% exact: 0.0% [4000Hz
cycles], (all, 32 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.67% [kernel] [k] do_raw_spin_lock
6.96% [kernel] [k] fib_table_lookup
6.53% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
4.17% [kernel] [k] mlx5e_xmit
3.22% [kernel] [k] ip_rcv
3.07% [kernel] [k] __netif_receive_skb_core
2.86% [kernel] [k] __dev_queue_xmit
2.36% [kernel] [k] __build_skb
2.33% [kernel] [k] ip_forward
2.05% [kernel] [k] mlx5e_txwqe_complete
2.02% [kernel] [k] ip_finish_output2
2.00% [kernel] [k] ipt_do_table
1.84% [kernel] [k] ip_rcv_finish
1.83% [kernel] [k] inet_gro_receive
1.80% [kernel] [k] udp_v4_early_demux
1.61% [kernel] [k] dev_gro_receive
1.55% [kernel] [k] netif_skb_features
1.52% [kernel] [k] mlx5e_txwqe_build_dsegs
1.47% [kernel] [k] mlx5e_poll_tx_cq
1.39% [kernel] [k] ip_route_input_rcu
1.38% [kernel] [k] dev_hard_start_xmit
1.17% [kernel] [k] netif_receive_skb_internal
1.16% [kernel] [k] napi_gro_receive
1.03% [kernel] [k] kmem_cache_alloc
1.02% [kernel] [k] sch_direct_xmit
0.97% [kernel] [k] read_tsc
0.94% [kernel] [k] page_frag_free
0.91% [kernel] [k] mlx5_cqwq_get_cqe
0.90% [kernel] [k] mlx5e_build_rx_skb
0.89% [kernel] [k] skb_network_protocol
0.83% [kernel] [k] __local_bh_enable_ip
0.79% [kernel] [k] validate_xmit_skb
0.77% [kernel] [k] vlan_dev_hard_start_xmit
0.74% [kernel] [k] __netif_receive_skb
0.72% [kernel] [k] mlx5e_poll_rx_cq
0.70% [kernel] [k] netdev_pick_tx
0.69% [kernel] [k] eth_type_vlan
0.68% [kernel] [k] __netdev_pick_tx
0.66% [kernel] [k] nf_hook_slow
0.65% [kernel] [k] deliver_ptype_list_skb
0.62% [kernel] [k] fib_validate_source
0.61% [kernel] [k] eth_header
0.60% [kernel] [k] eth_type_trans
0.59% [kernel] [k] __udp4_lib_lookup
0.58% [kernel] [k] __napi_alloc_skb
0.53% [kernel] [k] ip_finish_output
0.51% [kernel] [k] neigh_connected_output
0.50% [kernel] [k] ip_output
0.50% [kernel] [k] rt_cache_valid
0.44% [kernel] [k] udp4_gro_receive
0.43% [kernel] [k] mlx5e_napi_poll
0.40% [kernel] [k] udp_gro_receive
0.40% [kernel] [k] page_frag_alloc
0.40% [kernel] [k] __jhash_nwords
0.39% [kernel] [k] swiotlb_map_page
0.38% [kernel] [k] mlx5_cqwq_get_wqe
0.36% [kernel] [k] mlx5e_features_check
0.32% [kernel] [k] get_dma_ops
0.31% [kernel] [k] ktime_get_with_offset
0.31% [kernel] [k] validate_xmit_skb_list
0.28% [kernel] [k] vlan_passthru_hard_header
0.28% [kernel] [k] get_dma_ops
0.27% [kernel] [k] __udp4_lib_lookup_skb
0.26% [kernel] [k] skb_gro_reset_offset
0.25% [kernel] [k] skb_release_data
0.25% [kernel] [k] timekeeping_get_ns
0.24% [kernel] [k] kmem_cache_free_bulk
0.24% [kernel] [k] ip_forward_finish
0.23% [kernel] [k] compound_head
0.23% [kernel] [k] ip_skb_dst_mtu
0.22% [kernel] [k] __net_timestamp.isra.90
0.22% [kernel] [k] is_swiotlb_buffer
0.21% [kernel] [k] neigh_resolve_output
0.21% [kernel] [k] dst_metric.constprop.61
0.20% [kernel] [k] skb_orphan_frags.constprop.126
0.20% [kernel] [k] irq_entries_start
0.19% [kernel] [k] mlx5e_calc_min_inline
0.19% [kernel] [k] dev_hard_header.constprop.54
0.19% [kernel] [k] _kfree_skb_defer
0.18% [kernel] [k] _raw_spin_lock
0.18% [kernel] [k] ip_route_input_noref
W dniu 2017-09-09 o 11:03, Paweł Staszewski pisze:
> Hi
>
>
> Are there any plans to have this fix normally in kernel ?
>
> Or it is mostly only hack - not longterm fix and need to be different ?
>
>
> All tests that was done shows that without this patch there is about
> 20-30% network forwarding performance degradation when using vlan
> interfaces
>
>
> Thanks
> Paweł
>
>
>
> W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
>> On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
>>
>>> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
>> Something like :
>>
>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>> index
>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>> 100644
>> --- a/net/8021q/vlan_netlink.c
>> +++ b/net/8021q/vlan_netlink.c
>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>> struct net_device *dev,
>> vlan->vlan_proto = proto;
>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>> vlan->real_dev = real_dev;
>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>> vlan->vlan_id);
>>
>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-11 16:57 ` Paweł Staszewski
@ 2017-09-11 22:11 ` Paweł Staszewski
0 siblings, 0 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-09-11 22:11 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
Another test for this patch with linux-next tree
with patch:
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
- iface Rx Tx Total
==============================================================================
vlan1004: 1.00 P/s 606842.31 P/s 606843.31 P/s
lo: 0.00 P/s 0.00 P/s
0.00 P/s
vlan1016: 0.00 P/s 607730.56 P/s 607730.56 P/s
vlan1020: 0.00 P/s 606891.25 P/s 606891.25 P/s
vlan1018: 0.00 P/s 607580.88 P/s 607580.88 P/s
vlan1014: 0.00 P/s 607606.81 P/s 607606.81 P/s
vlan1005: 0.00 P/s 606788.44 P/s 606788.44 P/s
enp2s0f0: 2.00 P/s 2.00 P/s
3.99 P/s
vlan1017: 0.00 P/s 607643.75 P/s 607643.75 P/s
enp132s0: 13079658.00 P/s 0.00 P/s 13079658.00 P/s
vlan1000: 0.00 P/s 604409.19 P/s 604409.19 P/s
vlan1010: 0.00 P/s 606984.06 P/s 606984.06 P/s
vlan1019: 0.00 P/s 607452.12 P/s 607452.12 P/s
vlan1008: 0.00 P/s 606803.44 P/s 606803.44 P/s
vlan1011: 0.00 P/s 607048.94 P/s 607048.94 P/s
vlan1001: 0.00 P/s 606773.50 P/s 606773.50 P/s
vlan1006: 0.00 P/s 606811.38 P/s 606811.38 P/s
vlan1012: 0.00 P/s 607051.94 P/s 607051.94 P/s
vlan1013: 0.00 P/s 607067.88 P/s 607067.88 P/s
enp4s0: 2.00 P/s 13020803.00 P/s 13020805.00 P/s
vlan1007: 0.00 P/s 606798.44 P/s 606798.44 P/s
vlan1002: 0.00 P/s 606840.31 P/s 606840.31 P/s
vlan1009: 0.00 P/s 606809.38 P/s 606809.38 P/s
enp2s0f1: 100.80 P/s 0.00 P/s
100.80 P/s
vlan1015: 0.00 P/s 607089.81 P/s 607089.81 P/s
vlan1003: 1.00 P/s 606928.19 P/s 606929.19 P/s
------------------------------------------------------------------------------
total: 13079765.00 P/s 25766758.00 P/s 38846524.00 P/s
13Mpps forwarded (32cores active for two mlx5 nics)
80% cpu load (20%idle all cores)
PerfTop: 126552 irqs/sec kernel:99.3% exact: 0.0% [4000Hz
cycles], (all, 32 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
8.25% [kernel] [k] fib_table_lookup
7.98% [kernel] [k] do_raw_spin_lock
6.20% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
4.21% [kernel] [k] mlx5e_xmit
3.37% [kernel] [k] __dev_queue_xmit
2.95% [kernel] [k] ip_rcv
2.72% [kernel] [k] ipt_do_table
2.24% [kernel] [k] ip_finish_output2
2.22% [kernel] [k] __netif_receive_skb_core
2.17% [kernel] [k] ip_forward
2.15% [kernel] [k] __build_skb
1.99% [kernel] [k] ip_route_input_rcu
1.70% [kernel] [k] mlx5e_txwqe_complete
1.54% [kernel] [k] dev_gro_receive
1.45% [kernel] [k] mlx5_cqwq_get_cqe
1.38% [kernel] [k] udp_v4_early_demux
1.35% [kernel] [k] netif_skb_features
1.33% [kernel] [k] inet_gro_receive
1.29% [kernel] [k] dev_hard_start_xmit
1.27% [kernel] [k] ip_rcv_finish
1.19% [kernel] [k] mlx5e_build_rx_skb
1.15% [kernel] [k] __netdev_pick_tx
1.11% [kernel] [k] kmem_cache_alloc
1.09% [kernel] [k] mlx5e_poll_tx_cq
1.07% [kernel] [k] mlx5e_txwqe_build_dsegs
1.00% [kernel] [k] vlan_dev_hard_start_xmit
0.90% [kernel] [k] __napi_alloc_skb
0.87% [kernel] [k] validate_xmit_skb
0.87% [kernel] [k] read_tsc
0.83% [kernel] [k] napi_gro_receive
0.79% [kernel] [k] skb_network_protocol
0.79% [kernel] [k] sch_direct_xmit
0.78% [kernel] [k] __local_bh_enable_ip
0.78% [kernel] [k] netdev_pick_tx
0.75% [kernel] [k] __udp4_lib_lookup
0.72% [kernel] [k] netif_receive_skb_internal
0.71% [kernel] [k] page_frag_free
0.71% [kernel] [k] deliver_ptype_list_skb
0.70% [kernel] [k] fib_validate_source
0.69% [kernel] [k] mlx5_cqwq_get_cqe
0.69% [kernel] [k] __netif_receive_skb
0.68% [kernel] [k] vlan_passthru_hard_header
0.61% [kernel] [k] rt_cache_valid
0.59% [kernel] [k] iptable_filter_hook
Without patch:
12,7Mpps forwarded (32cores active for two mlx5 nics)
100% cpu load ( rx drops on receiving side )
TX is about 13.05Mpps from pktgen
bwm-ng v0.6.1 (probing every 1.000s), press 'h' for help
input: /proc/net/dev type: rate
/ iface Rx Tx Total
==============================================================================
vlan1004: 0.00 P/s 589709.31 P/s 589709.31 P/s
lo: 0.00 P/s 0.00 P/s
0.00 P/s
vlan1016: 0.00 P/s 589495.50 P/s 589495.50 P/s
vlan1020: 0.00 P/s 589968.06 P/s 589968.06 P/s
vlan1018: 0.00 P/s 589896.12 P/s 589896.12 P/s
vlan1014: 0.00 P/s 589496.50 P/s 589496.50 P/s
vlan1005: 0.00 P/s 589502.50 P/s 589502.50 P/s
enp2s0f0: 42.96 P/s 2.00 P/s
44.96 P/s
vlan1017: 0.00 P/s 589508.50 P/s 589508.50 P/s
enp132s0: 12700689.00 P/s 0.00 P/s 12700689.00 P/s
vlan1000: 0.00 P/s 587671.38 P/s 587671.38 P/s
vlan1010: 0.00 P/s 589330.69 P/s 589330.69 P/s
vlan1019: 0.00 P/s 589808.19 P/s 589808.19 P/s
vlan1008: 0.00 P/s 589283.75 P/s 589283.75 P/s
vlan1011: 0.00 P/s 589482.56 P/s 589482.56 P/s
vlan1001: 0.00 P/s 589971.06 P/s 589971.06 P/s
vlan1006: 0.00 P/s 589785.25 P/s 589785.25 P/s
vlan1012: 0.00 P/s 589494.50 P/s 589494.50 P/s
vlan1013: 0.00 P/s 589495.50 P/s 589495.50 P/s
enp4s0: 0.00 P/s 12601841.00 P/s 12601841.00 P/s
vlan1007: 0.00 P/s 589537.50 P/s 589537.50 P/s
vlan1002: 0.00 P/s 589943.06 P/s 589943.06 P/s
vlan1009: 0.00 P/s 589306.69 P/s 589306.69 P/s
enp2s0f1: 99.90 P/s 0.00 P/s
99.90 P/s
vlan1015: 0.00 P/s 589553.44 P/s 589553.44 P/s
vlan1003: 0.00 P/s 589823.19 P/s 589823.19 P/s
------------------------------------------------------------------------------
total: 12700832.00 P/s 24981906.00 P/s 37682740.00 P/s
PerfTop: 124056 irqs/sec kernel:99.3% exact: 0.0% [4000Hz
cycles], (all, 32 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
7.77% [kernel] [k] fib_table_lookup
7.37% [kernel] [k] do_raw_spin_lock
5.43% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
3.84% [kernel] [k] mlx5e_xmit
3.52% [kernel] [k] rt_cache_valid
3.46% [kernel] [k] ip_finish_output2
3.45% [kernel] [k] skb_dst_force
2.96% [kernel] [k] dst_release
2.62% [kernel] [k] __dev_queue_xmit
2.54% [kernel] [k] ip_rcv
2.26% [kernel] [k] ipt_do_table
1.97% [kernel] [k] __build_skb
1.94% [kernel] [k] __netif_receive_skb_core
1.94% [kernel] [k] ip_route_input_rcu
1.83% [kernel] [k] ip_forward
1.52% [kernel] [k] mlx5e_txwqe_complete
1.42% [kernel] [k] dev_gro_receive
1.31% [kernel] [k] mlx5_cqwq_get_cqe
1.28% [kernel] [k] netif_skb_features
1.21% [kernel] [k] ip_rcv_finish
1.18% [kernel] [k] inet_gro_receive
1.17% [kernel] [k] udp_v4_early_demux
1.14% [kernel] [k] dev_hard_start_xmit
1.03% [kernel] [k] mlx5e_txwqe_build_dsegs
1.02% [kernel] [k] mlx5e_poll_tx_cq
1.02% [kernel] [k] mlx5e_build_rx_skb
1.00% [kernel] [k] kmem_cache_alloc
0.95% [kernel] [k] __netdev_pick_tx
0.93% [kernel] [k] vlan_dev_hard_start_xmit
0.82% [kernel] [k] __napi_alloc_skb
0.79% [kernel] [k] read_tsc
0.76% [kernel] [k] validate_xmit_skb
0.75% [kernel] [k] napi_gro_receive
0.71% [kernel] [k] __local_bh_enable_ip
0.70% [kernel] [k] sch_direct_xmit
0.67% [kernel] [k] page_frag_free
0.66% [kernel] [k] skb_network_protocol
0.66% [kernel] [k] nf_hook_slow
0.63% [kernel] [k] netif_receive_skb_internal
0.62% [kernel] [k] vlan_passthru_hard_header
0.62% [kernel] [k] deliver_ptype_list_skb
0.62% [kernel] [k] __netif_receive_skb
0.61% [kernel] [k] fib_validate_source
0.61% [kernel] [k] mlx5_cqwq_get_cqe
0.55% [kernel] [k] eth_type_trans
0.55% [kernel] [k] iptable_filter_hook
0.53% [kernel] [k] __udp4_lib_lookup
0.53% [kernel] [k] udp_gro_receive
0.51% [kernel] [k] eth_header
0.49% [kernel] [k] fib_lookup.constprop.49
0.48% [kernel] [k] ip_output
0.44% [kernel] [k] netdev_pick_tx
0.43% [kernel] [k] validate_xmit_skb_list
0.42% [kernel] [k] swiotlb_map_page
0.41% [kernel] [k] dma_sync_single_for_cpu.constprop.36
0.41% [kernel] [k] mlx5_cqwq_get_wqe
0.39% [kernel] [k] eth_type_vlan
0.38% [kernel] [k] udp4_gro_receive
0.37% [kernel] [k] __jhash_nwords
0.37% [kernel] [k] ip_forward_finish
0.33% [kernel] [k] ip_finish_output
0.33% [kernel] [k] neigh_connected_output
0.31% [kernel] [k] mlx5e_features_check
0.29% [kernel] [k] ktime_get_with_offset
0.27% [kernel] [k] __udp4_lib_lookup_skb
0.27% [kernel] [k] napi_consume_skb
0.26% [kernel] [k] page_frag_alloc
0.26% [kernel] [k] get_dma_ops
0.25% [kernel] [k] udp4_portaddr_hash.isra.29
0.24% [kernel] [k] skb_dst_drop.isra.76
0.24% [kernel] [k] neigh_resolve_output
0.24% [kernel] [k] skb_release_data
0.22% [kernel] [k] kmem_cache_free_bulk
0.22% [kernel] [k] get_dma_ops
0.22% [kernel] [k] _kfree_skb_defer
0.21% [kernel] [k] ip_skb_dst_mtu
0.21% [kernel] [k] is_swiotlb_buffer
0.21% [kernel] [k] find_exception
0.21% [kernel] [k] compound_head
0.21% [kernel] [k] __net_timestamp.isra.89
W dniu 2017-09-11 o 18:57, Paweł Staszewski pisze:
> Tested with connectx-5
>
> Without patch
>
> 10Mpps - > 16 cores used
>
> PerfTop: 66258 irqs/sec kernel:99.3% exact: 0.0% [4000Hz
> cycles], (all, 32 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> 10.12% [kernel] [k] do_raw_spin_lock
> 6.31% [kernel] [k] fib_table_lookup
> 6.12% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
> 4.90% [kernel] [k] rt_cache_valid
> 3.99% [kernel] [k] mlx5e_xmit
> 3.03% [kernel] [k] ip_rcv
> 2.68% [kernel] [k] __netif_receive_skb_core
> 2.54% [kernel] [k] skb_dst_force
> 2.41% [kernel] [k] ip_finish_output2
> 2.21% [kernel] [k] __build_skb
> 2.03% [kernel] [k] __dev_queue_xmit
> 1.96% [kernel] [k] mlx5e_txwqe_complete
> 1.79% [kernel] [k] ipt_do_table
> 1.78% [kernel] [k] inet_gro_receive
> 1.69% [kernel] [k] ip_forward
> 1.66% [kernel] [k] udp_v4_early_demux
> 1.65% [kernel] [k] dst_release
> 1.56% [kernel] [k] ip_rcv_finish
> 1.45% [kernel] [k] dev_gro_receive
> 1.45% [kernel] [k] netif_skb_features
> 1.39% [kernel] [k] mlx5e_poll_tx_cq
> 1.35% [kernel] [k] mlx5e_txwqe_build_dsegs
> 1.35% [kernel] [k] ip_route_input_rcu
> 1.15% [kernel] [k] dev_hard_start_xmit
> 1.12% [kernel] [k] napi_gro_receive
> 1.07% [kernel] [k] netif_receive_skb_internal
> 0.98% [kernel] [k] sch_direct_xmit
> 0.95% [kernel] [k] kmem_cache_alloc
> 0.89% [kernel] [k] read_tsc
> 0.88% [kernel] [k] mlx5e_build_rx_skb
> 0.86% [kernel] [k] mlx5_cqwq_get_cqe
> 0.82% [kernel] [k] page_frag_free
> 0.78% [kernel] [k] __local_bh_enable_ip
> 0.69% [kernel] [k] skb_network_protocol
> 0.68% [kernel] [k] __netif_receive_skb
> 0.67% [kernel] [k] vlan_dev_hard_start_xmit
> 0.65% [kernel] [k] mlx5e_poll_rx_cq
> 0.65% [kernel] [k] validate_xmit_skb
> 0.60% [kernel] [k] eth_type_trans
> 0.60% [kernel] [k] deliver_ptype_list_skb
> 0.60% [kernel] [k] fib_validate_source
> 0.55% [kernel] [k] eth_header
> 0.53% [kernel] [k] netdev_pick_tx
> 0.53% [kernel] [k] __napi_alloc_skb
> 0.51% [kernel] [k] __udp4_lib_lookup
> 0.50% [kernel] [k] eth_type_vlan
> 0.49% [kernel] [k] ip_output
> 0.49% [kernel] [k] page_frag_alloc
> 0.49% [kernel] [k] ip_finish_output
> 0.48% [kernel] [k] neigh_connected_output
> 0.45% [kernel] [k] nf_hook_slow
> 0.44% [kernel] [k] udp4_gro_receive
> 0.39% [kernel] [k] mlx5e_features_check
> 0.39% [kernel] [k] mlx5e_napi_poll
> 0.37% [kernel] [k] __jhash_nwords
> 0.37% [kernel] [k] udp_gro_receive
> 0.36% [kernel] [k] swiotlb_map_page
> 0.33% [kernel] [k] mlx5_cqwq_get_wqe
> 0.33% [kernel] [k] __netdev_pick_tx
> 0.29% [kernel] [k] ktime_get_with_offset
> 0.29% [kernel] [k] get_dma_ops
> 0.29% [kernel] [k] validate_xmit_skb_list
> 0.26% [kernel] [k] vlan_passthru_hard_header
> 0.26% [kernel] [k] __udp4_lib_lookup_skb
> 0.24% [kernel] [k] get_dma_ops
> 0.24% [kernel] [k] skb_release_data
> 0.23% [kernel] [k] ip_forward_finish
> 0.23% [kernel] [k] kmem_cache_free_bulk
> 0.23% [kernel] [k] timekeeping_get_ns
> 0.22% [kernel] [k] ip_skb_dst_mtu
> 0.21% [kernel] [k] compound_head
> 0.20% [kernel] [k] skb_gro_reset_offset
> 0.20% [kernel] [k] is_swiotlb_buffer
> 0.19% [kernel] [k] __net_timestamp.isra.90
> 0.19% [kernel] [k] dst_metric.constprop.61
> 0.18% [kernel] [k] skb_orphan_frags.constprop.126
> 0.18% [kernel] [k] _kfree_skb_defer
> 0.18% [kernel] [k] irq_entries_start
> 0.17% [kernel] [k] dev_hard_header.constprop.54
> 0.17% [kernel] [k] dma_mapping_error
> 0.17% [kernel] [k] neigh_resolve_output
>
>
>
>
> With patch
>
> 12Mpps -> 16 cores
>
> PerfTop: 66209 irqs/sec kernel:99.3% exact: 0.0% [4000Hz
> cycles], (all, 32 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>
> 10.67% [kernel] [k] do_raw_spin_lock
> 6.96% [kernel] [k] fib_table_lookup
> 6.53% [kernel] [k] mlx5e_handle_rx_cqe_mpwrq
> 4.17% [kernel] [k] mlx5e_xmit
> 3.22% [kernel] [k] ip_rcv
> 3.07% [kernel] [k] __netif_receive_skb_core
> 2.86% [kernel] [k] __dev_queue_xmit
> 2.36% [kernel] [k] __build_skb
> 2.33% [kernel] [k] ip_forward
> 2.05% [kernel] [k] mlx5e_txwqe_complete
> 2.02% [kernel] [k] ip_finish_output2
> 2.00% [kernel] [k] ipt_do_table
> 1.84% [kernel] [k] ip_rcv_finish
> 1.83% [kernel] [k] inet_gro_receive
> 1.80% [kernel] [k] udp_v4_early_demux
> 1.61% [kernel] [k] dev_gro_receive
> 1.55% [kernel] [k] netif_skb_features
> 1.52% [kernel] [k] mlx5e_txwqe_build_dsegs
> 1.47% [kernel] [k] mlx5e_poll_tx_cq
> 1.39% [kernel] [k] ip_route_input_rcu
> 1.38% [kernel] [k] dev_hard_start_xmit
> 1.17% [kernel] [k] netif_receive_skb_internal
> 1.16% [kernel] [k] napi_gro_receive
> 1.03% [kernel] [k] kmem_cache_alloc
> 1.02% [kernel] [k] sch_direct_xmit
> 0.97% [kernel] [k] read_tsc
> 0.94% [kernel] [k] page_frag_free
> 0.91% [kernel] [k] mlx5_cqwq_get_cqe
> 0.90% [kernel] [k] mlx5e_build_rx_skb
> 0.89% [kernel] [k] skb_network_protocol
> 0.83% [kernel] [k] __local_bh_enable_ip
> 0.79% [kernel] [k] validate_xmit_skb
> 0.77% [kernel] [k] vlan_dev_hard_start_xmit
> 0.74% [kernel] [k] __netif_receive_skb
> 0.72% [kernel] [k] mlx5e_poll_rx_cq
> 0.70% [kernel] [k] netdev_pick_tx
> 0.69% [kernel] [k] eth_type_vlan
> 0.68% [kernel] [k] __netdev_pick_tx
> 0.66% [kernel] [k] nf_hook_slow
> 0.65% [kernel] [k] deliver_ptype_list_skb
> 0.62% [kernel] [k] fib_validate_source
> 0.61% [kernel] [k] eth_header
> 0.60% [kernel] [k] eth_type_trans
> 0.59% [kernel] [k] __udp4_lib_lookup
> 0.58% [kernel] [k] __napi_alloc_skb
> 0.53% [kernel] [k] ip_finish_output
> 0.51% [kernel] [k] neigh_connected_output
> 0.50% [kernel] [k] ip_output
> 0.50% [kernel] [k] rt_cache_valid
> 0.44% [kernel] [k] udp4_gro_receive
> 0.43% [kernel] [k] mlx5e_napi_poll
> 0.40% [kernel] [k] udp_gro_receive
> 0.40% [kernel] [k] page_frag_alloc
> 0.40% [kernel] [k] __jhash_nwords
> 0.39% [kernel] [k] swiotlb_map_page
> 0.38% [kernel] [k] mlx5_cqwq_get_wqe
> 0.36% [kernel] [k] mlx5e_features_check
> 0.32% [kernel] [k] get_dma_ops
> 0.31% [kernel] [k] ktime_get_with_offset
> 0.31% [kernel] [k] validate_xmit_skb_list
> 0.28% [kernel] [k] vlan_passthru_hard_header
> 0.28% [kernel] [k] get_dma_ops
> 0.27% [kernel] [k] __udp4_lib_lookup_skb
> 0.26% [kernel] [k] skb_gro_reset_offset
> 0.25% [kernel] [k] skb_release_data
> 0.25% [kernel] [k] timekeeping_get_ns
> 0.24% [kernel] [k] kmem_cache_free_bulk
> 0.24% [kernel] [k] ip_forward_finish
> 0.23% [kernel] [k] compound_head
> 0.23% [kernel] [k] ip_skb_dst_mtu
> 0.22% [kernel] [k] __net_timestamp.isra.90
> 0.22% [kernel] [k] is_swiotlb_buffer
> 0.21% [kernel] [k] neigh_resolve_output
> 0.21% [kernel] [k] dst_metric.constprop.61
> 0.20% [kernel] [k] skb_orphan_frags.constprop.126
> 0.20% [kernel] [k] irq_entries_start
> 0.19% [kernel] [k] mlx5e_calc_min_inline
> 0.19% [kernel] [k] dev_hard_header.constprop.54
> 0.19% [kernel] [k] _kfree_skb_defer
> 0.18% [kernel] [k] _raw_spin_lock
> 0.18% [kernel] [k] ip_route_input_noref
>
>
>
>
>
> W dniu 2017-09-09 o 11:03, Paweł Staszewski pisze:
>> Hi
>>
>>
>> Are there any plans to have this fix normally in kernel ?
>>
>> Or it is mostly only hack - not longterm fix and need to be different ?
>>
>>
>> All tests that was done shows that without this patch there is about
>> 20-30% network forwarding performance degradation when using vlan
>> interfaces
>>
>>
>> Thanks
>> Paweł
>>
>>
>>
>> W dniu 2017-08-15 o 03:17, Eric Dumazet pisze:
>>> On Mon, 2017-08-14 at 18:07 -0700, Eric Dumazet wrote:
>>>
>>>> Or try to hack the IFF_XMIT_DST_RELEASE flag on the vlan netdev.
>>> Something like :
>>>
>>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>>> index
>>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>>> 100644
>>> --- a/net/8021q/vlan_netlink.c
>>> +++ b/net/8021q/vlan_netlink.c
>>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>>> struct net_device *dev,
>>> vlan->vlan_proto = proto;
>>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>>> vlan->real_dev = real_dev;
>>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>>> vlan->vlan_id);
>>>
>>>
>>>
>>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-08-15 9:11 ` Paweł Staszewski
2017-08-15 9:19 ` Paweł Staszewski
2017-08-15 10:05 ` Jesper Dangaard Brouer
@ 2017-09-21 21:26 ` Paweł Staszewski
2017-09-21 21:34 ` Eric Dumazet
2017-09-21 21:41 ` Florian Fainelli
2 siblings, 2 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-09-21 21:26 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
> index
> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
> 100644
> --- a/net/8021q/vlan_netlink.c
> +++ b/net/8021q/vlan_netlink.c
> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
> struct net_device *dev,
> vlan->vlan_proto = proto;
> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
> vlan->real_dev = real_dev;
> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
> vlan->flags = VLAN_FLAG_REORDER_HDR;
> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
> vlan->vlan_id);
Any plans for this patch to go normal into the kernel ?
So far im using it for about 3 weeks on all my linux based routers - and
no problems.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 21:34 ` Eric Dumazet
@ 2017-09-21 21:34 ` Paweł Staszewski
0 siblings, 0 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-09-21 21:34 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
W dniu 2017-09-21 o 23:34, Eric Dumazet pisze:
> On Thu, 2017-09-21 at 23:26 +0200, Paweł Staszewski wrote:
>> W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
>>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>>> index
>>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>>> 100644
>>> --- a/net/8021q/vlan_netlink.c
>>> +++ b/net/8021q/vlan_netlink.c
>>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>>> struct net_device *dev,
>>> vlan->vlan_proto = proto;
>>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>>> vlan->real_dev = real_dev;
>>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>>> vlan->vlan_id);
>> Any plans for this patch to go normal into the kernel ?
>>
>> So far im using it for about 3 weeks on all my linux based routers - and
>> no problems.
> Yes, I was about to submit it, as I mentioned it few hours ago to you ;)
>
>
>
>
Yes i saw Your point 2) in previous emails :)
But there was no patch in previous reply for this so was thinking that
maybee too many things to do and You forgot about it :)
Thanks
Paweł
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 21:26 ` Paweł Staszewski
@ 2017-09-21 21:34 ` Eric Dumazet
2017-09-21 21:34 ` Paweł Staszewski
2017-09-21 21:41 ` Florian Fainelli
1 sibling, 1 reply; 37+ messages in thread
From: Eric Dumazet @ 2017-09-21 21:34 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On Thu, 2017-09-21 at 23:26 +0200, Paweł Staszewski wrote:
>
> W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
> > diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
> > index
> > 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
> > 100644
> > --- a/net/8021q/vlan_netlink.c
> > +++ b/net/8021q/vlan_netlink.c
> > @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
> > struct net_device *dev,
> > vlan->vlan_proto = proto;
> > vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
> > vlan->real_dev = real_dev;
> > + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
> > vlan->flags = VLAN_FLAG_REORDER_HDR;
> > err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
> > vlan->vlan_id);
>
> Any plans for this patch to go normal into the kernel ?
>
> So far im using it for about 3 weeks on all my linux based routers - and
> no problems.
Yes, I was about to submit it, as I mentioned it few hours ago to you ;)
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 21:26 ` Paweł Staszewski
2017-09-21 21:34 ` Eric Dumazet
@ 2017-09-21 21:41 ` Florian Fainelli
2017-09-21 21:43 ` Paweł Staszewski
` (2 more replies)
1 sibling, 3 replies; 37+ messages in thread
From: Florian Fainelli @ 2017-09-21 21:41 UTC (permalink / raw)
To: Paweł Staszewski, Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On 09/21/2017 02:26 PM, Paweł Staszewski wrote:
>
>
> W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>> index
>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>> 100644
>> --- a/net/8021q/vlan_netlink.c
>> +++ b/net/8021q/vlan_netlink.c
>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>> struct net_device *dev,
>> vlan->vlan_proto = proto;
>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>> vlan->real_dev = real_dev;
>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>> vlan->vlan_id);
>
> Any plans for this patch to go normal into the kernel ?
Would not this apply to pretty much any stacked device setup though? It
seems like any network device that just queues up its packet on another
physical device for actual transmission may need that (e.g: DSA, bond,
team, more.?)
--
Florian
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 21:41 ` Florian Fainelli
@ 2017-09-21 21:43 ` Paweł Staszewski
2017-09-21 21:54 ` Eric Dumazet
2017-10-18 21:49 ` Paweł Staszewski
2 siblings, 0 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-09-21 21:43 UTC (permalink / raw)
To: Florian Fainelli, Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
W dniu 2017-09-21 o 23:41, Florian Fainelli pisze:
> On 09/21/2017 02:26 PM, Paweł Staszewski wrote:
>>
>> W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
>>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>>> index
>>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>>> 100644
>>> --- a/net/8021q/vlan_netlink.c
>>> +++ b/net/8021q/vlan_netlink.c
>>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>>> struct net_device *dev,
>>> vlan->vlan_proto = proto;
>>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>>> vlan->real_dev = real_dev;
>>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>>> vlan->vlan_id);
>> Any plans for this patch to go normal into the kernel ?
> Would not this apply to pretty much any stacked device setup though? It
> seems like any network device that just queues up its packet on another
> physical device for actual transmission may need that (e.g: DSA, bond,
> team, more.?)
Some devices libe bond have it.
Just maybee when there was first patch vlans were not taken into account.
Did not checked all :)
But I know Eric will do :)
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 21:41 ` Florian Fainelli
2017-09-21 21:43 ` Paweł Staszewski
@ 2017-09-21 21:54 ` Eric Dumazet
2017-09-21 22:07 ` Florian Fainelli
2017-10-18 21:49 ` Paweł Staszewski
2 siblings, 1 reply; 37+ messages in thread
From: Eric Dumazet @ 2017-09-21 21:54 UTC (permalink / raw)
To: Florian Fainelli
Cc: Paweł Staszewski, Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On Thu, 2017-09-21 at 14:41 -0700, Florian Fainelli wrote:
> Would not this apply to pretty much any stacked device setup though? It
> seems like any network device that just queues up its packet on another
> physical device for actual transmission may need that (e.g: DSA, bond,
> team, more.?)
We support bonding and team already.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 21:54 ` Eric Dumazet
@ 2017-09-21 22:07 ` Florian Fainelli
2017-09-22 0:37 ` Eric Dumazet
0 siblings, 1 reply; 37+ messages in thread
From: Florian Fainelli @ 2017-09-21 22:07 UTC (permalink / raw)
To: Eric Dumazet
Cc: Paweł Staszewski, Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On 09/21/2017 02:54 PM, Eric Dumazet wrote:
> On Thu, 2017-09-21 at 14:41 -0700, Florian Fainelli wrote:
>
>> Would not this apply to pretty much any stacked device setup though? It
>> seems like any network device that just queues up its packet on another
>> physical device for actual transmission may need that (e.g: DSA, bond,
>> team, more.?)
>
> We support bonding and team already.
Right, so that seems to mostly leave us with DSA at least. What about
other devices that also have IFF_NO_QUEUE set?
--
Florian
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 22:07 ` Florian Fainelli
@ 2017-09-22 0:37 ` Eric Dumazet
0 siblings, 0 replies; 37+ messages in thread
From: Eric Dumazet @ 2017-09-22 0:37 UTC (permalink / raw)
To: Florian Fainelli
Cc: Paweł Staszewski, Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On Thu, 2017-09-21 at 15:07 -0700, Florian Fainelli wrote:
> On 09/21/2017 02:54 PM, Eric Dumazet wrote:
> > On Thu, 2017-09-21 at 14:41 -0700, Florian Fainelli wrote:
> >
> >> Would not this apply to pretty much any stacked device setup though? It
> >> seems like any network device that just queues up its packet on another
> >> physical device for actual transmission may need that (e.g: DSA, bond,
> >> team, more.?)
> >
> > We support bonding and team already.
>
> Right, so that seems to mostly leave us with DSA at least. What about
> other devices that also have IFF_NO_QUEUE set?
It wont work.
loopback has IFF_NO_QUEUE, but you need to keep dst on skbs...
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-09-21 21:41 ` Florian Fainelli
2017-09-21 21:43 ` Paweł Staszewski
2017-09-21 21:54 ` Eric Dumazet
@ 2017-10-18 21:49 ` Paweł Staszewski
2017-10-18 21:54 ` Eric Dumazet
2 siblings, 1 reply; 37+ messages in thread
From: Paweł Staszewski @ 2017-10-18 21:49 UTC (permalink / raw)
To: Florian Fainelli, Eric Dumazet
Cc: Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
W dniu 2017-09-21 o 23:41, Florian Fainelli pisze:
> On 09/21/2017 02:26 PM, Paweł Staszewski wrote:
>>
>> W dniu 2017-08-15 o 11:11, Paweł Staszewski pisze:
>>> diff --git a/net/8021q/vlan_netlink.c b/net/8021q/vlan_netlink.c
>>> index
>>> 5e831de3103e2f7092c7fa15534def403bc62fb4..9472de846d5c0960996261cb2843032847fa4bf7
>>> 100644
>>> --- a/net/8021q/vlan_netlink.c
>>> +++ b/net/8021q/vlan_netlink.c
>>> @@ -143,6 +143,7 @@ static int vlan_newlink(struct net *src_net,
>>> struct net_device *dev,
>>> vlan->vlan_proto = proto;
>>> vlan->vlan_id = nla_get_u16(data[IFLA_VLAN_ID]);
>>> vlan->real_dev = real_dev;
>>> + dev->priv_flags |= (real_dev->priv_flags & IFF_XMIT_DST_RELEASE);
>>> vlan->flags = VLAN_FLAG_REORDER_HDR;
>>> err = vlan_check_real_dev(real_dev, vlan->vlan_proto,
>>> vlan->vlan_id);
>> Any plans for this patch to go normal into the kernel ?
> Would not this apply to pretty much any stacked device setup though? It
> seems like any network device that just queues up its packet on another
> physical device for actual transmission may need that (e.g: DSA, bond,
> team, more.?)
How far it is from applying this to the kernel ?
So far im using this on all my servers from about 3 months now without
problems
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-10-18 21:49 ` Paweł Staszewski
@ 2017-10-18 21:54 ` Eric Dumazet
2017-10-18 22:45 ` Paweł Staszewski
0 siblings, 1 reply; 37+ messages in thread
From: Eric Dumazet @ 2017-10-18 21:54 UTC (permalink / raw)
To: Paweł Staszewski
Cc: Florian Fainelli, Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
On Wed, 2017-10-18 at 23:49 +0200, Paweł Staszewski wrote:
> How far it is from applying this to the kernel ?
>
> So far im using this on all my servers from about 3 months now without
> problems
It is a hack, and does not support properly bonding/team.
( If the real_dev->privflags IFF_XMIT_DST_RELEASE bit changes,
we want to update all the vlans at the same time )
We need something more sophisticated, and I had no time to spend on
this topic recently.
^ permalink raw reply [flat|nested] 37+ messages in thread
* Re: Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on
2017-10-18 21:54 ` Eric Dumazet
@ 2017-10-18 22:45 ` Paweł Staszewski
0 siblings, 0 replies; 37+ messages in thread
From: Paweł Staszewski @ 2017-10-18 22:45 UTC (permalink / raw)
To: Eric Dumazet
Cc: Florian Fainelli, Paolo Abeni, Jesper Dangaard Brouer,
Linux Kernel Network Developers, Alexander Duyck
W dniu 2017-10-18 o 23:54, Eric Dumazet pisze:
> On Wed, 2017-10-18 at 23:49 +0200, Paweł Staszewski wrote:
>
>> How far it is from applying this to the kernel ?
>>
>> So far im using this on all my servers from about 3 months now without
>> problems
> It is a hack, and does not support properly bonding/team.
>
> ( If the real_dev->privflags IFF_XMIT_DST_RELEASE bit changes,
> we want to update all the vlans at the same time )
>
> We need something more sophisticated, and I had no time to spend on
> this topic recently.
>
>
>
>
>
ok
^ permalink raw reply [flat|nested] 37+ messages in thread
end of thread, other threads:[~2017-10-18 22:45 UTC | newest]
Thread overview: 37+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-11 17:51 Kernel 4.13.0-rc4-next-20170811 - IP Routing / Forwarding performance vs Core/RSS number / HT on Paweł Staszewski
2017-08-12 12:23 ` Jesper Dangaard Brouer
2017-08-12 17:27 ` Paweł Staszewski
2017-08-13 16:58 ` Paweł Staszewski
2017-08-14 16:19 ` Jesper Dangaard Brouer
2017-08-14 16:33 ` Eric Dumazet
2017-08-14 16:57 ` Paolo Abeni
2017-08-15 0:45 ` Paweł Staszewski
2017-08-15 1:07 ` Eric Dumazet
2017-08-15 1:17 ` Eric Dumazet
2017-08-15 9:11 ` Paweł Staszewski
2017-08-15 9:19 ` Paweł Staszewski
2017-08-15 10:05 ` Jesper Dangaard Brouer
2017-09-21 21:26 ` Paweł Staszewski
2017-09-21 21:34 ` Eric Dumazet
2017-09-21 21:34 ` Paweł Staszewski
2017-09-21 21:41 ` Florian Fainelli
2017-09-21 21:43 ` Paweł Staszewski
2017-09-21 21:54 ` Eric Dumazet
2017-09-21 22:07 ` Florian Fainelli
2017-09-22 0:37 ` Eric Dumazet
2017-10-18 21:49 ` Paweł Staszewski
2017-10-18 21:54 ` Eric Dumazet
2017-10-18 22:45 ` Paweł Staszewski
2017-09-09 9:03 ` Paweł Staszewski
2017-09-11 16:57 ` Paweł Staszewski
2017-09-11 22:11 ` Paweł Staszewski
2017-08-15 9:35 ` Jesper Dangaard Brouer
2017-08-15 0:38 ` Paweł Staszewski
2017-08-15 9:23 ` Jesper Dangaard Brouer
2017-08-15 9:30 ` Paweł Staszewski
2017-08-15 9:57 ` Jesper Dangaard Brouer
2017-08-15 10:02 ` Paweł Staszewski
2017-08-15 10:05 ` Paweł Staszewski
2017-08-15 10:28 ` Jesper Dangaard Brouer
2017-08-14 0:07 ` Alexander Duyck
2017-08-14 15:07 ` Paweł Staszewski
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.